Ignoring all the back and forth, I'd say if you shoot a single action well you'll do just fine with it as your primary.
I'm no "pistolero" but I can thumb back a hammer just about as fast as I can line my gun back up, so I don't see aimed fire happening all that much slower.
Reloads are going to be quite a lot slower, but needing to reload before you can escape - at least to cover - is pretty unlikely.
Honestly the biggest downside I see is the size of the typical SA revolver, but I also know that a lot of guys manage to carry big guns just fine.
Open a history book and you'll see countless examples of people sucessfully defending themselves - even against multiple opponents - with single action six guns.
si vis pacem para bellum