Don't get me wrong. I am generally a VERY open fellow and have no harbored ill will toward anyone on what I would call a 'racial or religious bias'. But I HAVE lived on the 'wrong side of the tracks' where a certain color of clothing will get you beat down or shot...regardless of whether or not you are playing the game.
It is a real combination of action and perception that sends up a flag. That does not mean the tidy, middle aged 'buffy and tad' flavored golf goof is NOT gonna try to take my car by force...but you cannot ignore the reality of crime and the general appearance of those who commit the crimes.
Observe. Overreact. Destroy. Apologize.