but if you do have a single safety that's reasonably firm, then it should never come off when secured in your holster.
I suppose a lot has to do with your lifestyle here. Mine is pretty active, and clothing and other things, still get into places you wouldn't think they would.
All my Colts had the factory (left side) safeties, and I used holsters with a sweat guard that covered the safety and part of the grip. I still had the safeties on all my guns I carried, off at the end of the day on a pretty regular basis.
This is just the nature of the beast with this type gun, and you just learn to deal with it.