All calibers can and will cause hearing damage without the use of a suppressor, although many shoot .22 out of rifles without ear protection which I am guilty of every now and then.
I will share an experience of mine. I had the indoor range to myself and slipped off my hearing protection. I reloaded my 3" S&W model 60 with hot 125gr .357 magnums and pulled a shot off. My immediate reaction was to cover my ears the blast was so great. For a good 6 months after my left ear rang non stop at an unbearing level which was driving me crazy. Its been about 2 years since the incident and my left ear still slightly rings and has high frequency hearing loss. The condition is called Tinnitus, it is a non stop ringing usually resulting from hearing loss/damage. Since then I have no interest in the .357 as a self defense round where I may have to fire it again with unprotected ears. The .357 is considerably louder than most handgun rounds out there, even louder than the .44 magnum believe it or not. I was naive then thinking I needed a big boomer as a carry gun but have since then switched to .380, .38 special, and 9mm for my carry guns.