I agree. The one thing I tried was after each barrel change I counted the clicks on the scope adjustment between the .22 and the .17's. It gets you pretty close but still needs finer adjustment to be dead zero.
I did have a Mauser 660 that had 3 interchangeable barrels but each scope was attached to the barrels. All calibers were uncannily accurate after each barrel change.