I was going to suggest the dollar bill or even the business card test on the stock, too. A card should slip between the barrel and the stock without interference. If there's an intentional vibration node contact point (I think you have a Bell & Carlson Carbelite stock that won't likely have this) in the front, inadequate upward pressure could be an issue, too (the barrel hopping off the points).
Everything recoil related that causes trouble in a heavier rifle will be more likely in a light one. I would check the scope mounts. I would either swap in the scope from another rifle or swap out yours to another rifle to be sure it hasn't shaken loose inside. I would also check that the action screw torque spec is being met, as any slipping in the bedding would be unhelpful.
I would also do as you yourself suggested in the beginning, and that is to check different bullet weights. Every once in awhile you hear of someone with a rifle whose barrel just is not sympathetic with the barrel times typical of a particular bullet weight. I would try both 150's and 180's for this reason, if you can. For a systematic method of finding a best load, take a look at
Dan Newberry's site.
Finally, if your bedding and scope are good and you still can't tune a load, you may want to look at some kind of barrel tuner. They will add a few ounces. Also, there is one device that wouldn't add significant weight, and that is an upward barrel tension device whose position can be changed with a set screw to tune the barrel to a load.
That's here.