It's about range...
For most of us. For those who would limit shots to bowhunting ranges with a .357 Magnum, which isn't a bad call, would you feel more confident in your shooting ability at longer ranges with any iron-sighted handgun, regardless of caliber?
I sure wouldn't. If I can't put the bullet where it needs to go, at the range I'm shooting at, more gun won't help. Now, if I were capable of shooting out to 100 yards at game with a handgun, a bigger gun is needed. But I can't do it, regardless of the round I'm shooting, so it's not an issue.
At the ranges I'd be comfortable shooting any handgun at game, I doubt there's much difference between any of the sufficient calibers. And a properly loaded .357 is sufficient, at any range I'd shoot a handgun at an animal.
The critical question is "how good of a shot are you?" not "How big is your gun?" Find your maximum range. Can you put the first bullet in the kill zone, every time? If not, that's too far for you to shoot at game. I'm pretty sure that your accuracy will limit your shots much more than your gun will.