The angle and direction of the light has a lot to do with it.
Try it at different times of the day and cloudy days to see if that's the cause.
There's polarizing and sun-glass type lenses for the the scopes available that can help, too.
Some scopes come with them.
If you're having trouble with the reflex type, try a tube type.
Walt Kelly, alias Pogo, sez:
“Don't take life so serious, son, it ain't nohow permanent.”