How is the trigger? More than likely you have some bedding issues with the rifle as well. I'd glass bed it and depending on the stock maybe add pillars. Get it off a bi-pod and get it resting on sand bags, with a rear bag under the butt of the stock.
Try some Sierra bullets, I find most rifles shoot them very well. I'd try some different powders as well most 6.5X55's like H4350, RL22 and H4831 seem a little slow for the 6.5 Swede.