I'd shoot the load that proves to be most accurate. Any bullet weight from 150-180 will be fine at 300 yards or less.
If you are only hunting deer sized game and at only 300 yards, I'd start with 150's. If they shoot well that is what I'd use. They are more than adequate for that purpose and will recoil less. Not a huge difference, but the heavier 165's and 180's tend to be more accurate in my guns.
The difference in trajectory is highly over rated. While the 150's are faster, the heavier bullets are more aerodynamic and perform better at long range. With a 200 yard zero (and my handloads) 165's are only 1" lower at 400 yards than 150's, 180's are another 1" lower, but the heavier bullets hit with far more energy at extended ranges, are less effected by wind and are generally more accurate at extended ranges.
So where's the happy medium? I sure don't think it's 180gr!
180's are certainly more bullet weight than you need, but if it proves to be the most accurate load in your rifle that is what I'd use. Actually for really long range work bullets in the 180-210 gr range are what you'd want. While lighter bullets start out faster, they also slow down faster. Once you reach a certain range, around 600 yards, the heavier 180 gr bullet will be moving faster and be more accurate than the 150 at that range. 300 yards is not long range and deer are not hard to kill, so any bullet accurate in your gun will work.