seconding other who have said:
a) first, shoot whichever is more accurate, since shot placement matters most. This will often be influenced by the twist rate/barrel length of your particular rifle.
b) However, assuming that 165gr bullets can be shot accurately for your rifle; I then HIGHLY recommend these mid-weight bullets over the 180gr for almost all NorthAmerican game. A few quick reasons. Modern "premium" bullet design is most-often built around controlled expansion -> which is best aided by higher velocity -> with higher velocities possible in 165gr than 180gr. Simple algebra.
That said, my handloads are typically non-premium Sierra Bullets (MK, GK & Pro-Hunter), though I still load 165-168gr almost exclusively for .308 diameter. If I need more than 165gr to knock it down, I recommend adding the weight width-wise rather than length-wise (meaning: moving up a caliber, like to .338 diameter).