You are OK with the 165 gr bullets. I will have to try your Varget load. IMR 4064 seems to work the best in my rifle.
For over 10 yrs now all I have ever used is 150g Nosler BT in my 308 with excellent results. All but a few dropped in their tracks and the ones that did run only went about 25 yds.