Never rule out shooter error. But...
It's quite possible that the Varget was showing some temperature sensitivity.
The best load my .270 Win has ever seen was something like 52 gr RamShot Hunter, under a 140 gr Partition. It provided the proverbial "one-hole groups".
But... it would only do that around 40-50 degrees F. Once the temperature came up to even the mid-60s, the groups opened.
And... don't buy manufacturer's claims of powders not being temperature sensitive. I've seen the same sensitivity with some of Hodgdon's "Extreme" powders.
Yet... one of the most temperature insensitive powders I've encountered is RL-19. ....something that most reloaders, ballisticians, and gun writers constantly flaunt as very temperature sensitive, due to its "primitive" double-base chemistry and extruded shape.
Just like charge weights and accuracy... you never know how a powder will react at any given temperature, until you try it. (Getting data for RL-19's temperature tolerance took me nearly 2 years, because I had to wait on the weather. (I tested a few others, as well.))
"Such is the strange way that man works -- first he virtually destroys a species and then does everything in his power to restore it."