I only have a .30-06, and I feel if you can only have one then it is the one to have. 110, 135, and 150 grain are all sufficient for various deer.
I am looking at a .243 for my next hunting rifle. Lever Action, open sights, 150 yards or less, quick action rifle for shooting those targets that like to move.
If you find your back is up against a wall, maybe you have been backing up for too long.