I've been using earbuds/phones a lot in the past 5 years, my hearing is fine. Hell, I listen to some loud, angry, and often distorted death metal and the like. I agree it's not likely good, but I'm fine.
That aside, I never really have a problem with it. I'm alone in the house, so I just keep the volume low. If I am elsewhere, or have visitors I usually don't have such a problem. My neighbors two sons are druggies, and known thieves. If I worried about all of it, every moment, every day, I would go insane. Not saying I'm right in the head but you get the idea.
Headphones can indeed hinder your senses, so can staring at a computer screen. So can sitting in a car, so can frustration, so can a child, so can alcohol, and worst of all, you could be cleaning your gun when someone breaks in!