A 500 grain bullet at 2570 fps!?
You sure that 16" consistent lower impact is not due to a consistent flinch?
That's what I would be doing
Barrel harmonics are weird, with the amount of horsepower the action is absorbing, I would check to ensure that the barrel is not making anysort of contact anywhere, even if it is supposed to be "free floating", and I would bed the action in a 1/4" thick layer of stainless steel filled Devcon epoxy.