Without seeing you shoot, I can't say for sure. However, I suspect improper follow through and/or inconsistent/improper form/grip/stance.
I would first start with follow through. The barrel of a long gun is much longer than a handgun, and improper/inconsistent follow through has more potential to affect accuracy on a rifle than a handgun.
Here is a video that may be of help to you. Please keep us posted with your progress.
http://www.youtube.com/watch?v=BGs_GuPiDbI