depending on the gun and load...
Weight makes a bigger difference in the felt recoil than the difference in velocity.
10" .44 Mag Contender kicks harder than a 6" S&W, its lighter.
3.5" S&W .44 mag kicks harder than 6" S&W .44 mag, its lighter, even though velocity is less. Also the shorter barrel gun has a much greater "torque" in the hands.
With smaller lighter calibers, the difference may be even more pronounced.
All else being equal (and it almost never is) bigger bullets tend to work better.