Don't feel bad. 30-06 is too much recoil for me, too. I breifly owned a custom Springfield in 30-06 but half box of ammo convinced me to sell it. I hunt with either my .308 or 30-30 carbine. Or sometimes my .243 rifle. My level of shooting accurasy is directly related to recoil.
Fire up the grill! Deer hunting IS NOT catch and release.