Some of them have even decided to take up residence inside our walls.
I believe that is due to an overpopulation of squirrels in the area. All the good tree houses are taken up by other squirrels, so the displaced population decides to move into your house!
The only effective solution: thin them out!
I don't bother the albino squirrels.