You should need to handle your firearm when you holster it in the morning and when you put it away at night. In between, you shouldn't be messing with it unless you're about to USE it.
Ahh, if only I lived in a state where this was really possible. I worry most about discharging my firearm while un holstering in my vehicle before I enter a CPZ.
I agree on not using that holster if you need to un-holster to go to the restroom though.
Avoid public restrooms. When available, a tree with smooth bark is preferred.