I don't see the downside to shooting something more powerful unless the recoil bothers you. You are the only one who knows that.
The chances of you losing an elk because the 308 was too small is pretty slim.
Although I think a 300 magnum might be better on a long quartering shot or whatever I still use a 280 and just don't take a long quartering away shot.
I got tired of heavy recoil a while back.