I have mild permanent hearing damage from a single incident of my father's safety de-cock lever malfunctioning. (by the way, I have seen S&W, Beretta, and Ruger safety decock levers 'fire' on decock. please treat that feature with great caution). I was forward of the muzzle (not in the line of fire, just forward of the gun), and it took about 20 minutes to hear normal conversation speech, several days to hear semi-normally, and I still have some permanent tinnitus in both ears. It was just a 9mm hydroshock (it was harder on the ceiling than on me).
It's not fun. hearing damage is a form of injury, and potentially a minor disability. I don't think that it's something you want to marginalize or downplay. Not if you've experienced it. I mean, I'd rather someone stole my bigscreen tv, than have permanent hearing loss from trying to stop them, ya know? Hearing isn't life/death, but it's not insignificant.
Short answer to the question though, is that outdoors, you likely won't suffer hearing loss from handgun fire. Not small amounts of it anyway. A rifle outdoors could damage hearing more significantly though.
Indoors, things get LOUD. The perceived intensity of the sound is at least double or triple outdoors fire. The pitch of the sound makes a difference too, for example, the boom-thud sound of a shotgun seems less piercing to the ears than a high powered rifle.
Subsonic ammunition really is amazingly quiet. I don't recommend 9mm subsonic for SD... in my view, it doesn't work very well. But even a big .45acp, at standard subsonic velocity, will be noticeably quieter than a supersonic 9mm.
Also, ported guns and revolvers are louder.
Leaves me thinking that a .45 with naturally subsonic 230gr JHP ammo is probably the ideal way to go for home defense. in terms of hearing loss anyway. Or perhaps get a gun with a screw-in "suppressor" as they are called. While they are not silencers (and thus infinitely more legal), they certainly do take the sharp edge off of gunfire noise. Or both... a .45 with a canister-style flash suppressor.
For reference, a .357 magnum out of a ported revolver, is 'very loud', compared to every other handgun I own. Which makes me hesitate to grab it for indoors home defense.
also, i could be crazy, but I think Golden Sabers run just a tad quieter than most other SD ammo.