If I do not want wear on the finish of one of my guns I do not put it in a holster, and am careful about how it is cased. I do have a few that I inherited that are safe queens for a number of reasons I will not go into at this time. The same is true of a few rifles.
My carry guns have carry wear on them. Some of my rifles have dings, and scratches from riding in the truck, or going through the brush. It is like trying to keep a car in mint condition. If you want to keep it that way then it would not be a good idea to use it as a daily driver.
No matter how many times you do it and nothing happens it only takes something going wrong one time to kill you.