A tiny difference in the position of the muzzle at the instant (I use the word "instant" advisedly) the bullet exits can make a huge difference in where the bullet hits the target. We are talking microseconds here.
And a properly and precisely bedded action, stress free, proper and consistent torque on action screws with pillars... is the 800 pound gorilla here that no one's mentioned yet.
I think if the term "accuracy" with respect to our hobby were replaced with "consistency" it might create a light bulb moment for some shooters. As you noted, getting the bullet to exit the muzzle while it is at the exact same point in space at every shot is what we're seeking.
Found this, and it made my brain hurt....