As the quote goes ...
A coward dies a thousand deaths; the valiant taste of death but once.
There are always going to be BG's out there and random acts of violence, and if you let their presence leave you cowering in your thrice locked home or afraid to venture out without your hand on the butt of a .44 magun then they've won.
Sure, you should keep your eyes on those around you and your SO, but it's not like every stop to pick up gas is a tactical situation requiring a plan and converging fields of fire.
Start carrying that piece as soon as you legally can, and try not to freak out everytime someone approaches. With or without a gun there's nothing serious you can do until someone makes their intentions known.
A brain is your most important weapon. Every kid in a hood isn't a thug, and not every white guy in a suit is a good guy.
Go wherever you want, carry whenever you can, and keep your eyes open.