I'm a lefty as well, and I'll get a low-right shot from jerking. Be sure that you're not attempting to stage the trigger - don't hesitate once you begin the trigger stroke, bring it back with constant pressure on the trigger face.
And try dry firing. Don't accept any deflection of the front sight. Work on it until there's no disruption of the sight alignment when you release the shot.