This isn't really an answer for which is better, the 32 or 380. More of a justification of either.
When you are at home, and someone breaks in, you are in "Your Element". Yes, there is adrenaline and your heart is racing, but you are in familiar surroundings. As such, your mind can concentrate on other things. Such as the noise and recoil of the 9mm, 45acp, 38spl, 357mag, etc...
When I am out in public, outside of my little town; walking down the street; parking garages; etc... I am not in "My Surroundings". When forced into a situation, my mind has to do a lot of thinking. With the adrenaline and unfamiliarity, my mind will be concentrating on options, escape routes, fear, etc... Much more DEFENSIVE than in my house where I can "Feel" a little more OFFENSIVE. Because of that, I like the 32acp. Whenever I go shooting, before I leave, I ALWAYS shoot 1 clip of 45acp; 1 cylinder of 38spl; 1 cylinder of 357mag; 1 clip of 32acp; and 3 shots of 12 gauge. And I ALWAYS do this WITHOUT hearing protection. WITHOUT safety glasses. I make it as "REAL WORLD" as possible. I've also done this is a HOT-HOUSE where it's indoors. For me, the 32acp can be shot without any flinching or blinking. Just like when you shoot a .22; you can do so without even ANTICIPATING the noise or recoil. I can shoot the 32acp without even subconsciously thinking about it. In a real world situation out in public that ISN'T my surroundings, I know that I can dump a clip of 32acp into a 6"-10" area without thinking.
I feel comfortable with my 45acp also, but there is that anticipation of the shot in the back of the mind. But with my Walther (And hopefully the new FEG I picked up-haven't shot it yet); I feel more comfortable with it in unfamiliar territory. In my home town/neighborhood, my Sig 45acp works fine. I know the surroundings. i can concentrate on the gun. In unfamiliar areas, I'd rather concentrate on the surrounding and not think about the gun. And I'd rather have 7 definite hits with a 32acp, than only a couple with my 45acp.