Sounds like the handgun grip may be too small or too large. If you have adjustable backstraps try a different one. If not then practise your form keep your draw and presentation almost mechanical with the same movements each time. Also get the same grip on the stock every time with the web of the hand high up on the beaver tail.
With my Gen 4 glock I have a better shooting grip and I'm a little more accurate with the large backstrap, but I found myself shooting left during stress fire and or speed shooting during competitions. I went back to the medium and the problem went away. I had a similar, but not as bad, problem with no backstrap and having too much finger on the trigger.