It's different for different people. I'm not all that big, but at 6', 220#, I have to figure that if they are screwing with me, they are serious. If I was 5'6" and 140 pounds, they might just be pushing me because I look like an easy target.
It doesn't matter if they are openly armed or not. A hand strike to the right region can render unconscious, or worse. 12' rule and all...
The first shooting involving a TX CHL holder was a guy shooting another, unarmed guy who was hitting him. It was ruled a clean shoot and provides some precedence.
*Opinion worth what you paid.