I'm no expert on which caliber is the worst but from my own experiences, I'd have to say any caliber. I'm 60 and have shot for 50 years. When we were young, we didn't wear any hearing protection. I also worked around machinery and didn't give it a whole lot of thought. As a result, I've lost about 80% in my left ear - part of that by shooting muskets in team competitions with earplugs that weren't the best. I also shot full size Civil War artillery but was at least smart enough to wear ear plugs and ear muffs while doing that. I recently got back in to shooting 22s - pistol and rifle. I wear ear muff protection all the time now as even the smaller 22 LR seems to bother me as much as 38 spl, 9mm, etc.
In a SD situation though, in an enclosed room in the middle of the night, anything is going to be loud . . and has the potential for ear damage . . . but I doubt that any of us, if faced with a SD situation such as that, are going to bother to take the time to put on hearing protection. We'll be lucky if we're able to even think clearly to react as needed.
If a pair of '51 Navies were good enough for Billy Hickok, then a single Navy on my right hip is good enough for me . . . besides . . . I'm probably only half as good as he was anyways. Hiram's Rangers Badge #63