The first thing to do is free float the barrel and bed the action.
The bedding in my 6.5 Sweded M700 was just awful, the barrel touched the sides of the stock in the barrel channel and there was a bedding point in the forend. Clearing that out brought round groups.
But, I have a M96 Swede that I have done all the above and nothing keeps the barrel from walking as it heats up.
If it still walks after bedding and free floating, its the barrel.
If I'm not shooting, I'm reloading.