My 10 year old grandson shoots a .30-06 and handles the recoil just fine. If you shoot shotguns, the .30-06 recoil is roughly equivalent to 12 gauge shotgun recoil. Get a rifle that fits you, a scope with decent eye relief and tuck the rifle in before you shoot it and you will be fine. As far as hearing protection, muffs or plugs are fine, muffs over plugs to me are gilding the lilly unless you're firing artillery or something big with a nasty muzzle brake.
As far as first rifle after .22 and maybe for elk, for sure a decent bolt action 30-06 will serve you very well.