If you load using the loading lever, you put a lot of stress on the frame during the loading process. On a Colt all of that stress is focused as tension on the arbor/frame union.
Loosening of the arbor is not the only damage to the frame to be considered. In addition you have the cylinder impressions on the recoil shield.
My reading of history convinces me that most bad government results from too much government. Thomas Jefferson