I shoot without hearing protection when doing self defense drills with my 357 magnum, 22LR, 40sw, and 12 gauge every once and a while. Chances are, bad guys wont wait for me to put in my ear plugs before I have to defend myself. I personally think one should be comfortable with the loudness. That is a personal preference only.
”An unarmed man can only flee from evil, and evil is not overcome by fleeing from it.”
~Col. Jeff Cooper