While this is always a possible but are we to draw our firearms every time there is an physical altercation?
We draw our firearms to PREVENT
a physical altercation.
If you carry a handgun, you simply CANNOT allow yourself to get in to a physical altercation.
There is too great of a chance that your opponent might discover and get hold of your concealed weapon....and then you're done for.
Besides, the guy might have a knife or other weapon on him that you will not be able to see till it's too late.