Okay, I am to understand that the scope will stay on the slug barrel, then you will change out your slug barrel for a shot barrel. Whenever you reinstall the slug barrel, you are questioning if you will have to resight it?? ... Right?
In a given season, I do just that and no, have not see any accuracy problems. However, at the beginning of every season, I put in some range time to confirm it's accuracy. Now, if you were talking about a reciever mount, then I'd have greater concerns. Hope this helps ....
Be Safe !!!