they do pivot, they also tend to pivot under recoil so the gun and you are running away from the sight.
it is just physics, remember the law "a body in motion tends to stay in motion and a body at rest tends to stay at rest"
the mass of the sight doesn't want to get moving so as the gun recoils the sight is actually moving towards the front of the gun.
that is unless you buy a cheap/junk one that locks up solid.
I have never been wacked by any of the sights even shooting my 45-110's (which is what Quigley shot)
and yes they are that accurate, maybe not offhand that is a stretch (but I am sure there is someone who can shoot that good). fundamentally there are lots of bolt actions that aspire to the degree of accuracy possible
they are not NOT hunting sights though