I'm with you here!
I used to hunt with a 30'06. I shot 150gr at first. They were fine. When we really got into Antelope hunting and loading our own, I found out that BC gets noticeably higher around 180gr.
So, even at a lower starting velocity, the 180's will catch up.
Play with JBM Ballistics
and you will see.