Well.. the thing that intrigues me is that with iron sights... being directly attached to the completely floating barrel, so not the action, they should "follow" any changes that happen during the re-torquing.
I think they probably do. I suspect the contact of the stock with the barrel is what gets changed and that relationship is the root of the POI changes.
Just my opinion though.
The only gun I have ever taken out of the stock is my M1A and it took a few shots to get it back to normal. It has been a long time since I have done that and I don't plan on doing it any time soon.