That sounds like anticipation of the shot, or over adjustment for recoil. One quick way to test is to use your weaker hand, or to turn the firearm upside down (not joking). If it is corrected with these "distractions," then your sights are fine.
Thank you Aunt Cindy 