Did I find the culprit?
The bearings (not sure of the correct term. Agate?) are not located symmetrically. While the one nearest the user is sitting flush in its recess, the opposite bearing is not.
Unfortunately I dont have a phillips both small and narrow enough to reach the set screw from the access hole on the back side. Should these be torqued down or allowed to float a bit, as one user mentioned? Mine are quite tight.