There is 44mags you carry and 44mags you shoot. There is a huge trade off in recoil and accuracy.
As a huge .44 fan (and handgun hunter), I couldn't disagree more. My deer-loads are what some would consider moderate (10.9gr Unique pushing a 240gr JSP or XTP), and I can shoot those in my 5-shot tracker no problem.
With handloading, I can get even milder trail loads (when I'm thinking "coyotes", not "bears"). In bear country my normal hunting loads do just fine.
I enjoy shooting my big hunting handguns, and my 4" 629, and my Tracker. I think you can (and should) shoot them all. If you need to load down a bit due to recoil sensitivity, you should. No sense carrying a gun you can't practice with, in my opinion.