I have never owned a 35 Rem,but by reputation .ordinarily,I would support it as a very good choice for a deer hunting round.
There were some other factors that seemed important to you that stand out to me.
You want occasional capability for 300 yd shots.
You want the ammo to be readily available.
You want the option to download/subload with handgun bullets.
I can fit the .308 to all of your parameters pretty darn well.From 32 acp,30 Luger,30 carbine,a raft of cast bullet options,and rifle bullets from 110 gr on up...Do what makes you happy,its an option.