I agree to a point.
Once you DO draw, there should be a ton of threat involved...you're less likely to have to shoot. I suspect there's a real difference in the odds you'll have to shoot between pulling, say, a small .32 or .380 and something the size of an N-Frame S&W.
Your demeanor makes possibly an even bigger difference. If you look scared even when hauling out a real hogleg, that's a problem (for you). On the other hand, looking either absolutely stone-cold determined to kill is I think the single scariest, followed (closely!) by looking seriously ****** off or outright crazy.
A clean draw and a competent stance matter too.
Finally (and most controversially!), if you DO have to fire into the first of an attacking group, doing "gruesome damage" may cause the rest to re-think their activity. Very hot 357Magnums (125gr up over 1,500fps fr'instance) have been linked to phrases like "a spray of lung tissue" or similar (ewww). The major muzzle blast of a full-house 357 is going to create psychological effects too.
Again: this all takes place once there's a lethal threat. At that point big power early can make the difference, esp. against a mob.