I hit a guy once while carrying. He was dragging his wife around by the hair and she asked me to help. He was drunk and I really didn't want to get involved in a marital thing, but also didn't want to hear her Kitty Genovese story on the news later either so went ahead and popped the guy a couple times. The gun didn't even come into my mind as it was not needed.
It's good to anaylize scenarios beforhand but methinks some of you are over-anaylizing this and not giving credit to your own intuition. I suppose I could be wrong on this but trust your instincts no matter the size of the guy.
It was Heinlein that said
"Never frighten a little man, He'll kill you." Size don't mean a whole lot nowadays. Martial artists abound. You just never know until its too late.
Listen to your spidey sense and act appropriately.