I've seen some physical confrontations for "line jumping" at the grain elevator but those involved guys who knew what they were doing and did it anyway.
At the very least you get credit for being aware that an individual was approaching in a belligerent manner and realizing your error. Regadless of the appearance of the other guy, his aggressive approach and demeanor put him in the crosshairs so to speak. He knew his size and appearance would intimidate and his manner of dealing with your error would indicate he had used this ploy previously.
If he had opened the door, my first thought would be to produce the weapon and say"Sorry I made a mistake but you just made a bigger one. Now get out of my face so I can move out of the way". That is NOT the way to handle this type of situation. I'm 60ish and medium build so a physical assault by this individual would very possibly be fatal but I wouldn't want to have to justify brandishing my pistol until he actually made contact.
My son repeats the line" don't mess with an old guy, he won't fight he'll just kill you".