Given the range capability of the 300Wby, this much change could be a real problem with a followup shot. I've seen similar variances with a cold,clean bore vs second shot dirty bore but you're saying this happens again with a clean, fouled bore. I'd look at the bedding and action screws first.
Has anyone else shot the rifle? In my experience, a harder recoiling rifle like this is very prone to vertical stringing dependant on shoulder contact and how it's held. I shot a 300 Win mag target rifle for a period of time and can assure you that it takes very little difference in recoil attenuation or shoulder contact to throw the shots several inches.