Ok here's a caveat: I've NEVER shot a deer with a .357mag handgun. I have however taken deer with a lever action .357mag rifle.
At the ranges you're talking about and with the load you're planning to use you should be ok. That'll be plenty enough to take deer. As someone stated though I'd lose the scope for 25yards or less. At that range just practice a lot and get good with irons. A scope will just get in the way.
Heck even with my .44mag revolver I don't use a scope. I use irons out to 75 yards and if its farther than that then I don't take the shot unless I've got a rifle to shoot - not the gun's fault, just my bad aim.
This is who we are, what we do.