Muzzle brakes are fine for range shooting, but most of us don't wear muffs for hunting deer or other large game where hearing the animal approaching is important, and only one or two shots may be fired in maybe 20 outings. That shot shouldn't be louder than necessary.
That said, there are some brakes with sleeves that vent forward only.
To minimize velocity of escaping gases, they tend to be larger than direct vented ones.
If I can't be with the rifle I love, I love the rifle I'm with!!