I think keeping the gun in the safe does you little good should the day come that you need it... and judging by the activities in your neighborhood that is highly possibly if not likely.
I was 7 when I got my first gun and learned how to shoot. Before I ever EVER got to fire a shot my grandfather drilled the four rules into my head (as well as a fifth).
RULE I: ALL GUNS ARE ALWAYS LOADED
RULE II: NEVER LET THE MUZZLE COVER ANYTHING YOU ARE NOT WILLING TO DESTROY
RULE III: KEEP YOUR FINGER OFF THE TRIGGER UNTIL YOUR SIGHTS ARE ON THE TARGET
RULE IV: BE SURE OF YOUR TARGET
This elaborates on the 4:
http://www.thefiringline.com/Misc/safetyrules.html
Make sure your kids know these rules, I don't think 4 is too young to learn the rules and understand them. Also make sure that they never NEVER touch any gun without your permission and if they do they will be in LOTS of trouble. Just be very direct and clear about what you expect and what can happen with guns if they're not careful. My younger brother and I used to get into WORLDS of trouble but we always understood that we were never to touch my grandfather's guns without him knowing.
If you have to retrieve the gun from a safe when you're in a pinch it might prove to be a critical handicap. This is your decision as the head of your house, if you think getting to your gun in time is not a problem then the extra safety is probably worth it, you must be pretty darn fast though. I would definitely suggest drilling your kids on gun safety though regardless of where you decide to keep your guns. Just my $.02
As for this situation I would definitely have had the gun at least within arm's reach and I would probably have stayed inside my house, but yelling at them is probably alright, just be prepared to back it up if push comes to shove. Again just my $.02