Speaking from experience...
My duty holsters were Safariland with the Elk suede linings. These did a fair job of reducing or slowing the wear from the tighter fitting holsters in the 70's. Today's holsters are even tighter as a rule.
We did a comparison test. A buddy used silicone spray on the inside of his holster while I did not. At the end of a year, his gun showed much more wear than mine. The reasons, we surmised were twofold. First, his holster's lining collected more dust due to the silicone spray. While the silcone obviously helped things slide, the collected fine airborne dust in any metro area collects and acts as an abrasive.
Secondly, every month I would use a brass "suede" toothbrush and a nylon brush to clean the lining of my holster. I also used a suede cleaner without any "conditioner" to help remove the dirt around the top and open muzzle end.
The bad news is that my gun still showed some wear at the end of the year. We did take satisfaction from the 3rd member of our test group who used an unlined Bianchi holster for most of the year. His poor Colt Lawman took a beating with a lot of wear showing on the gun.
Talk to some holster makers or read their online FAQ's about keeping the lining clean. Find out what products they recommend and how to apply them properly. Keep the holster as clean as you can to minimize wear and if the gun doesn't fit snugly, get it restitched or a new holster.
BillCA in CA (Unfortunately)