I'm guessing that the people who don't want to lose the boar as game don't own their own ranch or farm; they really need to think of the rights of other people, and the continued harm they cause them by keeping these non-indigenous pests around (not that we have a choice, cos the pigs are probably here to stay). Oh, joy.
It isn't an issue about the rights of others. Just because you like something does not mean that you are ignoring the rights of others. However, where one person's rights stop is where another person's rights start and vice versa. Those that want feral hogs obliterated would be just as guilty of your stipulated "ignoring the rights of others" as well.
Those that don't want to lose boar as game (hunting definition, not legal definition) aren't fighting legal battles to keep them protected. In many cases, the boar provide hunting and food when no other larger animal hunting is legal (no season).
Personally, I would like to see everyone's right to hunt feral hogs in the US ruined by a lack of hogs. However, like others who enjoy hunting them and reaping their benefits, I would miss being able to hunt them if they were gone, though that isn't stopping me from shooting every last one I can when I can and how I can.
Sadly, I am secure in the notion that nobody's hog hunting is apt to be ruined anytime soon by an actual lack of hogs.
Farmers and ranchers have the right to fence out hogs and to shoot every last one of them as well (in most states in accord with depredation laws). This would go for many types of animals that damage their crops or threaten livestock. Yes, it might be expensive, but if the issue is one of rights, then they have the right to implement such measures.