It sounds to me like your wood stock needs a good pillar bedding job.
The most accurate rifle I have is a Rem 700 BDL that has been blue-printed & a 22" med heavy SS Hart installed with a standard size .270 chamber. I kept the original Remington stock except that it was pillar bedded, glass bedded, floated, plastic forend & grip cap replaced with ebony & a limbsaver butt pad replaced the steel cap. The hardest part was getting that so called "bowling pin" finish off the stock so I could put on the TruOil! LOL
I torque the action screws down 60lbs. I can take the action in & out of the stock & the zero never changes. With it's favorite load & me having a good day, it makes nice single hole "bumblebugs". I can load & mix several different brands of 130gr bullets & it will still shoot < MOA! As a matter of fact, most 100gr, 130gr, & 150gr loads all shoot to the same practical zero at 100 yds! This gun originally came from the factory 42+ yrs ago so I figure, this wood stock is as stable as it will ever get!
My Texas sendero rifle.