Yes, I have 'knocked' my rifle 'out of zero' a FEW times over almost 30 years of big game hunting. I've done it to scoped rifles and 'iron sighted' rifles. Hell, when I was a kid, I even fell on a rem 870 barrel hard enough and 'bent' it so that it didn't shoot point of aim!
I've seen rifles knocked off zero with the slightest bump, and others get the $hit banged out of them, and remained zeroed. the best insurance is a well bedded stock, good, secure mounts[low as possible], and a well made scope. And check you zero often.
Our deer season in South Carolina lasts from August 15 till Jan. 1. I hunt usually at least once a week, and check my rifle's zero AT LEAST every couple of weeks. ALMOST NEVER any deviation, but I sure have more confidence in my gear when the big buck walks out
Spend at least as much money on your scope and mounts as you do on your rifle, and you should be fine. Check your zero often, and you will gain confidence in your rifle. Besides, a little practice is good for you!
Will Fennell/CAMILLUS CUTLERY