I've never had any issues with my SERPA. Granted it's for a 1911, so I have the thumb safety too, but I honestly can't see how it would be any easier to have an AD with it than any other holster.
For me, when I grab the gun, and position my finger out along the line of the frame - where I'd put it if there was no holster - it hits the release. When I draw my finger is left - where it started - on the frame above the trigger guard.
Clearly, people have shot themselves using the holster, but I'd guess that has more to do with trying to draw quickly out of an unfamiliar holster than it does with some inherent design flaw.
As far as it getting ripped off easily, has anyone actually tried it or is that just another internet rumor? Maybe mine is different, but the holster attaches to the belt loops with 4 fairly beefy screws. Tightened properly, with a drop of locktite, I don't see them coming out very easily.