One of the past ladies in my life was assaulted as a child by someone she knew - a vagrant her minister father had taken in. Charity is fine, but even "2%" is a risk.
As far as how I respond to panhandlers, I will normally start with direct eye contact and a polite but firm, "No thanks."
If the panhandler gets more assertive, but not implicitly aggressive, I will point out that last I looked, military recruiters were hiring; or, "yeah, the economy sucks, I just got back from Afghanistan."
That usually gives them a hint, if the first "no thanks" did not suffice.
So far, I have not even had to use the Sicilian glare I inherited from my mother's side.