I think you should move on, cause unless you know that the safety isn't going to continue wearing on the frame, any re-finishing will be a short lived fix. The reality is: this is a fairly common level of "wear" for a lot of pistols, of many makes and models. Thumb safeties quite often end up rubbing a bit.
If you worry too much about leaving room in the fitment of the safety, instead of worrying about getting a nice and tight fit, you'd end up with more sloppy safeties and a few less rub marks.
I don't think it's something to worry about, or spend money fixing. It's just not worth it.
But hey, check out my perfect finish!
Just kidding. Just like almost every other pistol in the world, I have a couple spots where the finish is worn. CZ's in particular may not even have the finish completed on the inside of the pistol. Mine is that way.