Consider if our other rights were to be restricted in similar ways...
They are, though each right is restricted in ways unique to the societal issues unique to the exercise of that right.
First Amendment rights, for instance, are subject to all sorts of restrictions...narrowly tailored, for the most part, but restrictions nonetheless.
The constitution is a constitution, not a legal code. The details get worked out over the years, in the political and judicial arenas.