I think that BartB, in post #5, had the best suggestion. First free float the barrel and see how that works. If it doesn't help, I'd suggest a different stock all together - a synthetic one with the bedding channel or pillar bedding already in place (Hogue makes one, and others do also). If neither of those fixes the problem, I'd go with a new barrel.
I had a friend with a 'walking group' bolt gun in 243 some years ago, and his problem sounds just like the OP's problem. Nothing seemed to fix it, so he finally sold the rifle. The first shot from a cold barrel was Ok, but starting with the second shot and any following shot, the impact point went up and to the right. Had to be the barrel, and that honestly sounds like the OP's problem. Sometimes you just get a bad barrel, though I don't think that it's a common problem.