You're worrying a lot about nothing, IMO. Your bullet will hit the ground and even if it "bounces" it will be destabilized and make a crash landing in short order. In all likelihood, it'll stay in the ground more often than not.
Plus, there's not a great deal of lethality difference between, say, a 223, 243 and 7mm-08, as examples.
Many areas of the country have had rifle hunting for decades and accident rates are no higher, and in fact often lower, than shotgun areas.
NY has slowly been transitioning from all shotguns in the southern zone to now nearly all rifle, with just a few "hold out" counties. The accident rates have not increased.
If you practice safe shooting, be sure of your target and what is beyond, you have nothing to worry about.
Pick a good deer cartridge and don't worry about it. Safety is in your hands, literally, not in the cartridge you select.
(I recommend 243, 25-06 or 7mm-08)