[carguychris]

Barrel movement DOES affect the final point of impact. This is why handguns always have the barrel pointed downwards at rest relative to the level of the sights. (This is masked by the slide on many automatics; it's more obvious on revolvers, particularly snubbies.) Since the barrel centerline of a handgun is almost always well above the centerline of the grip, the barrel pivots up under recoil, so the sight plane is typically set to point the barrel downwards to compensate for this. This also explains why, at typical handgun ranges, a heavier bullet hits higher on the target than a lighter bullet loaded to a similar power level. Heavier bullet = more recoil and (typically) longer time spent within the barrel = more barrel movement before the bullet exits.

Another factor is torque (twisting force). The bullet engaging the rifling creates a torque on the gun in the opposite direction of the rifling; remember Newton's Third Law of Motion. This is most obvious if you shoot a powerful handgun using a relatively limp one-handed hold; you'll notice that the gun doesn't kick straight up, it moves off to one side.

In addition to the uncertainty in barrel lockup mentioned by Skadoosh (good post!), remember that guns and bullets are manufactured by humans and therefore aren't ever dimensionally flawless. Barrels are never perfectly straight, muzzles are never cut exactly perpendicular to the bore axis, rifling is never cut to a perfectly uniform depth, bullets are never perfectly concentric, and so on. Consequently, bullets don't leave the bore going perfectly straight- they wobble, resulting in a slightly helical path as they leave the bore. The bullet's spin subsequently straightens this out for the most part, but the variables in what happens before the bullet stabilizes have an effect in accuracy. A more precisely manufactured (and more expensive!) firearm can minimize these effects but never eliminate them completely.