I'm not a doctor, nor do I play one on TV, but I'm pretty sure it takes more than one shot to permanently damage your hearing. A friend of mine was in the Army before ear plugs came into use for training situations, and he can hear me speaking to him when we're in mcDonald's having coffee. However, I can't recall whether he uses hearing aids. I think he does. However, he went through officer basic without ear plugs and he still is able to enjoy recorded music at home. I'd say ask your doctor and go from there.
I would hope your right, but firing a decent sized caliber inside the small confines of a vehicle doesn't sound like a good thing.
There was a guy who lost most of his hearing in one ear from firing a single .308Win round.