I'd try what FM suggested and use 165 grain bullets. I used to be a die hard 180 grain shooter in my 06 rifles. However with todays premium bullets I see no need to go heavier than 165 for hunting purposes in an 06. The velocity gain from going to a lighter bullet is nice to have at normal hunting ranges. Another big plus is the less recoil you get from a 165 grain bullet which makes it a lot more comfortable to shoot and easier IMO to find accuracy with especially at the bench.
Thread about upgrading the XL7 rifle.