I think you probably can get some immunity from peppers.
Cayenne may be 30-50,000, but it isn't a very hot pepper.
I regularly use olive oil with a Ghost Pepper
in it. Even in a mixture it is many times hotter than relatively mild cayenne(800,000- 1,500,000 SHU for the ghost). I usually use chop sticks to handle the peppers when preparing the mix as using my hands then going to the restroom, rubbing my eyes, etc is unpleasant.
The dose may also be much larger with food. Popping a habanero in your mouth likely delivers a larger dose than a spray that is landing mostly on your skin or going into the wind.