16" is a lot more than is common. I have no good explanation for the amount of difference, other than it may be that your rifle and load combinations are just a "perfect storm" of factors combining.
500fps difference in velocity and 40% differnce in bullet weight means that odds are they are going to strike in quite different places.
Drop may be similar, but drop is due to gravity, a constant, varied results coming from velocity over the time of flight. And remember drop is referenced to point of aim and point of impact being the same at a given distance. At another distance, the difference between impact and original point of aim is the "drop".
Vibration (harmonics) is, I think the culprit here. The fact that your groups are small means that it is consistant, not that it isn't the cause of the radical impact shift. It could very well be that as that big heavy bullet rumbles down the bore your rifle is actually pointed in a much different place when it leaves the muzzle than when using the lighter, faster bullet.
Any difference (change) in the way the rifle is held can make a difference as well. Shooting offhand with a tight sling can mean a different point of impact than shooting from a bipod, which could also be different than shooting from a bench. Each rifle & load combination is an individual in this regard, some being very different with each set of conditions, some not so much.
16" difference in the point of impact at 100 yards is way more than normal, but if consistant, can be lived with, although I wouldn't be all that happy with it.
Do you have enough adjustment in your scope to sight in (bullet hits point of aim at specific distance) for both loads? If so, memorize the clicks needed to change and go back and forth as you change ammo.
If not, decide on which load best suits your shooting, and stick with it. While most rifles are pretty flexible about changing loads, some are not, and it seems yours might be one of those rare ones which isn't.
All else being equal (and it almost never is) bigger bullets tend to work better.