Heavier bullets will recoil more than light bullets in my experience. And of course the more powder you have with each bullet weight, the more recoil. I say with each bullet weight because an 80gr bulletloaded with 47gr powder will still recoil less than a 100gr bullet loaded with 44gr even though it has less powder.
But since you want to take a deer, you cant just load up some 55gr'ers and call it good. The recoil would be mild, but it would be far from ideal for a deer. From what I have heard, 80gr is about as low as you want to go.