Your statement defies the laws of physics. If your sights are on target when the gun fires, whether by luck or design, the bullets hit in the same place.
It's called jirking the trigger. Happens very very often. And if the pressure is not strait back you tend to milk the rounds to one side also.
The handgun weighs maybe 2 lbs. The trigger can be anywhere from 3 to 10 or more pounds. Do the math.