You say you don't carry with one in the chamber, but what if your life is dependent on having a gun ready like in a deflect / fire situation? Seconds count.
As far as firing at someone with a ccw, no never have and hope to never have to do so. I have unfortunetly fired a large surplus of 5.56 at other human beings overseas and I'll tell you one thing, my weapon was loaded. It just seems like a saftey that could cost you your life. Use your own judgement, but why hinder yourself?