If that Berger round performed as you say on a light-boned deer, I have to say DON'T DO IT on an elk!!!! While I'm not a fan of .264 Mags because of the barrel length needed to actually achieve the published velocities, that's not the point. I'm sure that yours will do fine IF the right bullet is used. As many others have said, it's hard to go wrong with a Nosler partition.
That being said, I've killed elk with a round as light as the .25-06. Would I unconditionally recommend it to everyone? No. The animals that I've dropped with the .25-06 (love those things) were undisturbed, broadside shots where I was absolutely sure of the bullet placement, and those Noslers were deadly. But, I've also passed up a big bull that was quartering away, because even with Noslers I couldn't be sure of penetrating to the vitals through so much tissue and tough hide. My .300WM w/180's would have piled him up for a follow up, I'm sure.
In my long experience, there just "ain't no free lunch" when it comes to light bullets and big, heavy-boned animals.
Happiness is a steaming gut pile.