...if it moves shot it, be it legal with papers or just not protected,...
Yes, that's correct. If it's legal game, during the proper season and you have the correct permits, then it is perfectly fine to shoot it as long as you abide by the other imposed hunting regulations that apply. Those regulations include specific rules about the type that may be taken, number that may be taken, and other related aspects. The money spent on permits and collected on sporting goods taxes is used to insure that the populations are properly managed and that animals are not hunted to extinction or even to an unhealthy level.
So, for example, in some areas you might be able to shoot only a single whitetail buck with a particular antler point count, while in other areas, with much higher populations, you may be able to legally harvest more animals, even possibly including does. There are specific restrictions about how the animal must be processed and tagged, etc.
Here is an 80 page document from my state that summarizes the hunting and fishing regulations and restrictions. About 44 pages deal with hunting rules.
As far as shooting unprotected species, I believe you would be surprised to find out how few species are actually unprotected. For example, when it comes to birds, in my state, there are only 3 species that are unprotected. European starlings (an imported species), English sparrows (another imported species) and feral pigeons, a ubiquitous pest species.
There are 11 unprotected animal species in my state, not including feral animals and "exotics" which are non-native species. The unprotected animals are pest species and/or animals which are extremely common, or that are, for one reason or another, not in any danger of extinction from hunting.
...if hunters keep taking the large animals out it will take years for them to recover in these areas so you see smaller and smaller animals being taken until there is not much left to hunt and the hunters have to move to new hunting grounds.
I don't know what to tell you other than what I've already said. Hunting is carefully regulated by aggressively enforced game laws which prevent the kind of gradual, localized extinction you're concerned about.
The system isn't perfect, but it is working quite well.