That is exactly the way my beam scale behaves if there is dust in the v-groove where the knife edge rests.
So, I ALWAYS clean the v-groove AND the knife edge with a Q-tip before I zero and use the check weights.
That has always resulted in the scale holdng zero and giving proper check weight readings for me.
If that doesn't work for you, then I would suspect that the knife edge has been damaged.
When you clean it, be careful that you don't damage the knife edge. You are just trying to remove dust, not scrub it or polish it.