I do not think it's about age to a reasonable extent. Some of it is the guns and the cartridges that they are chambered for (modern metallurgy has made it possible to chamber too much cartridge in too small of a gun for it to be comfortable to shoot.) The SP101 is a prime example.
Bigger guns are more technique than strength to control (to a certain extent, of course). I have seen big guys with poor technique fail to control magnums and dislike them, and they are certainly big enough to do so. I have seen small women shoot magnums very well and not complain, because of good technique. I am not a big guy and love shooting big bores, but I have been doing so for a long time and have good technique.