I don't worry about it too much.
However, I did have someone pounding on my door at 3am at an apartment my wife and I lived in during college. It was dark and they had their finger over the peephole.
I answered the door with my Ruger P89 behind my back. It was just some drunk girl trying to be funny but at the wrong address. Never showed her I was armed but sent her on her way.