Many years ago(late 1990s), I spoke to firearms industry coating expert Walter Birdsong(
www.Black-T.com ).
Birdsong told me about a class of sworn LE officers he was working with. They were talking about sidearms & holsters and he asked the cops who cleaned their gear or holsters. Few of the officers raised hands.
Proper care & inspections can help maintain a firearm's plate job or coating.
Dust, grit, gunk, etc can collect or store inside a holster. Many armed professionals and gun license holders advise using unlined models.
I haven't had major problems with nylon or synthetic fabric designs. I have seen some kydex/polymer holsters scuff or mar a handgun but not a lot.