On my guns with safeties, if they are loaded and a round chambered, then yes, I use the safety (e.g. 1911 cocked-n-locked on the bedside table). And when I practice with them, I practice using the safety as well.
However, most of mine do not have a safety, being either DA revolvers or DA/SA pistols with dockers (in which case they are loaded, round chambered and decocked).
My feeling is, that if you do not want to learn to use a mechanical manual safety, then pick firearms without them in the first place. This is the very reason many people actually prefer a Glock or a revolver as their carry/SD handgun (I do not own a Glock, but that's because I do not own any striker fired pistols at all).
What if, as in your stated case, a safety inadvertently got swiped ON and you grabbed the gun in a true life-or-death crisis expecting the safety to already be OFF? By the time, in your adrenalin hammered mind, you realize the gun ain't going bang, it may be too late to rectify.