What gets me a lot, both in pistol and rifle offhand shooting is miss anxiety induced flinching or "the yips". When this happens, I can flinch with a .22. It's not the recoil and noise, it's yanking the trigger while the sights are on the X while I am getting tireder and tireder holding the aim.
It starts with "chicken finger" where you can't get yourself to increase pressure on the trigger because when you do, it seems to turn on some sort of magnet that repels the bullseye, then you start getting tired and you want to get the shot over with and BANG!!!, another bullet hole in the white part of the target.