Damn! What part of California do you live in? I work in one of the crappiest areas of L.A. I see junkies and winos begging for money, but they are just an annoyance. If you keep going to the same place time and time again and keep telling them no, they'll eventually leave you alone.
Why are you so scared? As others have told you, if you look like a victim, you'll be a victim. I'm not the biggest guy, but I'm big enough. I look them in the eye and tell them NO! before they even finish the sentence. If I see them accosting older folks or women, I tell them to get lost and I take the heat for these poor people. If I do it often enough, they go elsewhere or clear out when I'm around.
My dad uses a routine that I love. They ask Do you have any any spare change? He reaches into his pocket and jingles the coins, and says yep. the he just smiles at them.
Seriously, what station do you go to. I need a new crew of bums to -CENSORED--CENSORED--CENSORED--CENSORED- off.