One of the most argumentative and obnoxious clowns I have ever met had a cwp, and he was an aggressive, nasty, confrontational a-hole.
One day, he let slip in conversation that he is "a lot less mouthy and obnoxious when I'm packing."
I congratulated him on finally admitting that he is an obnoxious loud mouthed jerk, and suggested that maybe he should go packing every day and make the world a quieter, more peaceful place.
I don't think that people like him should be hauling around firearms, being a confrontational and aggressive type. There is nothing in this world that requires more self control than having a lethal weapon in your hands when a bad situation is escalating. Drawing that weapon should be the very last step, and in plenty of situations, there are many opportunities to de-escalate before that point is reached.
I really hope that most of the CCW holders are aware of this, and that they believe in holding deadly force in reserve until you almost have no choice but pull the trigger.
You did the right thing by not showing your weapon in the face of simple physical intimidation.