The angle and direction of the light has a lot to do with it.
Try it at different times of the day and cloudy days to see if that's the cause.
There's polarizing and sun-glass type lenses for the the scopes available that can help, too.
Some scopes come with them.
If you're having trouble with the reflex type, try a tube type.
Lock the doors, they're coming in the windows.