While I understand the frustration, I have to side with those who say the annoying people also had a right to be there.
If I find people annoying in a public venue, I go somewhere else.
There are typically campsites, trailheads, etc that prohibit powered vehicles, or dogs, etc. These may require a hike to get to, but if peace and solitude are that important, the hike may be worthwhile.
Other such sites may be accessible by canoe, or horse.
For those who don't want to deal with the great unwashed, and don't want to hike or paddle in, there are two other good options:
1) Buy or lease your own hunting land; or
2) Make friends with somebody who will let you hunt their private land.