I can get off a shot quicker with my single actions, and accurately, than many folks can with double action revolvers. All depends on what you're used to.
"Many" folks might be slower but most
folks, everything else being equal in terms of training and what they're "used to", will get off a shot (first and subsequent ones) faster with a double-action revolver than they ever would with a single-action revolver. However, in terms of what the op has in mind (and most other envisioned shooting scenarios for that matter) a person competent with a typical sa revolver would not be functioning at a significant disadvantadge when compared to the speed of operation offered by a typical da revolver.