Rationalization, denial, fantasy, emotion, pride it will all get you killed if the right situation comes along. The unfortunate thing is that we do not get to pick when, where and what happens. The fact that you live a "low threat lifestyle" and live in the nice part of town is great, but criminals have cars too.
A double action revolver in the hands of a trained dedicated shooter is very effective so long as it packs enough punch. You guy's that carry a single action six shooter with 5 shots for self defense; I hope your fantasy does not get you killed.
I am a big guy too, 6' 275lbs, I hike and train and work out. I am still scared of little guys with knives ( been stabbed 3 times).
My rifle and pistol are tools, I am the weapon.