I recently (ok, a couple of years) went thru some of your deliberations. First I checked what the group I went deer hunting with used. Answer: almost exclusively .30-06!
I ended up acquiring a used pump .30-06 (left hand) and spending just as much on a Leupold Vari-X III 3.5-10x40mm, since I can only afford one scope, one time. I used iron sighter mounts, (for close shots I can see thru the iron sights. question my ZERO, well use the iron sights. Problem solved.)
The result? in 3 years of hunting Wisconsin Whitetails I have taken 5 shots and brought home 4 deer. I did take it to Texas wild russian boar hunting. (didn't see much) I am planning to try a Wyoming elk hunt next fall.. with, yup you guessed it the same rifle.
I had one other factor influence me. One of the guys I shoot with reloads .30-06. I am able to reload with his equipment, and get many more shots for the dollar. I practiced
almost 100 rounds thru this rifle before the first time I went hunting with it.
I recommend you practice practice practice. A good tip is to try to hit in a hunt situation, i.e. if you aren't going to have a handy bench/or some other rest in the woods then don't use one at the range, (after zeroing with scope/ammo/etc) try offhand, 50-100 yards, try prone if you can. (ask range officer first...) Try a lifesize target once, so you know about what sight picture you are looking for. (but don't get too hung up on this, deer move, targets don't)
If not for the commonality of .30-06 ammo, high percentage of use, and the ability to use it out west for larger game,
I would have looked long and hard at .270 or .308. Best part is if you actually do go once and decide you need to go back with more gun you have FIRST hand experience to tell you what you need. There are people who say you NEED a .338 magnum just for Wisconsin whitetails. Sheesh, that is overkill. ok, a .30-06 is overkill on a deer, but at least some of the meat survives the shot.
The theory I subscribe to is being able to hit with a .308 at a limited range beats not being able to hit with an expensive BIG gun, especially if you have limited funds.
Sorry this is a long post, but FWIW my $.02
p.s. to date I have not used the iron sights in the woods, but I had the option!