Fifties are far easier and cheaper to get ammo for. If you going to be buying bullets instead of casting your own get a fifty.
The .45s are easier to shoot, and the .40s are easier still and are considered more accurate by a lot of knowledgeable shooters. But a .40 would mean casting your own balls or bullets, or you might struggle to find them as they are more rare.
Some states have a minimum size for deer and big game hunting usually around .45 for good reason.
Recoil is an issue. I have .54s and they let you know it every time. Kills on one end and maims on the other!