Properly bedded, and with action screws torqued to the same values, I would see little reason for a significant shift in poi.
You said "always have"- so you've removed the action from the stock before, replaced it, and have not had this issue?
I said "properly bedded" because a poorly done bedding job can actually introduce-rather than relieve- stress in the receiver and cause poi shifts.
"Always have" = on any rifle even with scope.
"Properly bedded" — I did a bang up job on this one. I put pillars in front and rear, but only bedded the chamber and the recoil lug. We'll see...