2 things jump to mind: 1) you are flinching in some form or 2) you are accurate but the gun recoils differently in your grip than in a rest.
Flinching has been covered by other people, but if you are not sure, try shooting off of the bench. Be VERY conscious of your trigger pull - slow and deliberate. See if you are still off by the same amount.
Another issue could be that you are 100% consistent, but the gun just shoots to a different POI in your hand v in a rest. You said that you sighted in the gun "in a rest." I'm assuming that this means some sort of ransom rest arrangement, or something where you aren't holding the gun 100% on your own? If so, the way that the gun interacts with the rest can be different enough from how it recoils in your hand to throw off the POI. For this reason, I just shoot for consistency/groups and move my sights accordingly. yes, this is harder with fixed sights, but set your windage and then adjust your elevation as necessary.
To see for sure if this is the case, you need to shoot the gun from you hands with your arms in a rest. A sandbag rest works best. Hold the gun as you normally would, then rest your wrists on the bag. Put your wrists such that they are well supported, but the gun isn't 100% supported by your hand, not the rest itself.
Head shots are hard, but nut shots are twice as hard.