I've worn a shoulder holster off and on for over 50 years. Never had a problem. I do find it slower to draw but that's not a major factor. I also use belt holsters.
The only holsters that find problematic are pocket holsters. They all advertise that the gun comes out of the holster while the holster stays in the pocket. I have yet find one that stays in the pocket. I have tried about seven.
Cogito, ergo armatum sum