I own guns, shoot about once a month or so, and do not carry.
It's mostly due to circumstances; I work about 45 hours a week, no guns allowed, am a full time college student, no guns allowed, and I have a 1 year old who uses me as a jungle gym, which probably woulndt be a problem given a good holster, but not worth taking the chance.
Also, quite frankly, I don't have the at least $700-800 bucks to lay out for a gun, license and carry system. (I have only one pistol: a 10 inch barrel Ruger Mk II target model.)
Truthfully, I doubt I will ever carry 24/7. I just don't know that I could make the time commitment. Also, I frequently go to so many places which prohibit carry it would end up as a car gun anyway...