Take any sporter weight barreled rifle and shoot 4 different weight bullets for groups. All on the same target. Use the same poa for all loads. Each bullet weight will have its own poi, in different locations, probably similar group sizes.
As a bullet travels down the barrel the muzzle moves in a pattern that is consistent. The speed of the bullet determines where in that movement the bullet exits the barrel. Therefore heavier, slower bullets leave the barrel at a different axis from the line of sight. This is part of what makes ammunition that is more consistent in velocity more accurate. Not the only thing but it certainly helps.
If you doubt me, go to the range and prove me wrong.