I think that there is a fairly significant difference between 308 & 30/06 on your shoulder (& muzzle blast), and there is absolutely nothing lacking with a 308 on Elk. It's largely preference between the two cartridges, but I prefer the 308.
My son took an Elk with one shot at about 80 yards with a 308/180. He only went over one ridge and into the next ravine before going down for the count. It was a perfect shoulder shot. If the slightly reduced recoil of the 308 would let you shoot even a smidge better, then it would be the better choice.
If you handload then it's 6 of one half dozen of the other. Bullet selection and placement is everything, and more important than velocity or energy, even on Elk.