I've taken a number of deer, mostly northern deer in Michigan, with my Dan Wesson 15-2 .357. Most around 50 yards, but a few near 100 yds. Difference is that I know I can consistently keep my shots within 6" at 100 yds, or I wouldn't take the shots.
As far as getting a .44, I think maybe you should, given your stated lack of confidence in the Vaquero. I recommend you look at a longer barrel rather than a short one if you are primarily interested in handgun hunting. Two reasons, longer sighting radius and typically better accuracy for the average shooter. Plus, it won't look so odd when you decide to add that pistol scope.
In any case, practice, practice, practice...'till you absolutely know what shots to take and which to let pass you by. Welcome to the exciting world of handgun hunting.