As others already mentioned, for me it was peeking at the target in between shots. It's like golfing and pulling your head up as you swing to see where it goes, it never turns out good. I practiced and still do a whole lot with a .22 to instill good shooting habits and work on fundamentals.
As for people saying don't drink coffee before you shoot, I often drink coffee before and during some of my shooting sessions and see no ill effects compared to when I don't drink any. I can also down coffee and fall asleep so caffeine doesn't really have a big effect on me.