Clean brass has got to prolong the life of your dies, and certainly makes the brass easier to inspect for defects. Why not do it? It certainly isn't very labor-intensive.
I also pick up brass at the range I go to. I deprime and size before I tumble, I seriously doubt my carbide handgun dies will wear any with any dirt they are supjected to, after all the next hardest substance to the carbide is diamonds.
Even if per chance it would wear the dies, I'm now 69, by the time the dies get worn, I'm worn out.