I've never loaded the 32 ACP but I have put together some very mild 308 loads. I used a 32 cal cast SWC for about 1000 FPS. Instead of worrying about the bullet squeezing down a couple thousandths, I did have to be cautious of the neck diameter of the finished round. Once I found a brand of brass that didn't require neck turning, the loads worked fine. My goal was a load that roughly duplicated the 22 LR, and with a bullet that was twice as heavy I think it had a bit more punch. I have no way of measuring it, but it seemed quieter than a high speed 22 LR.
So if I was to load 32 for that adapter, I think I would use a cast SWC bullet. The lead would greatly reduce the chance of sticking a bullet in the barrel, and because it would be in the adapter there would not be any feeding issues. Good luck, Griz