If you are down to fast-draw, you've already screwed up. It will be hard to dig out from the hole the momentary loss of situational awareness dropped you into.
Not meant as a slam, either. We all have those momentary losses of SA, condition orange be damned.
But in any of those cases where you have any inkling that something is going down, draw speed should not be a factor; the weapon should be in a concealed-ready state, unless you need to fire right then and there, or need to attempt to immediately deter the BG prior to firing.
Now, if I'm worried about something like bears, then the fast-draw becomes more of a valid issue, because quite frankly in most of the accounts I've read of bear attacks, the victim did not see it coming. This goes back to my own tendency to only OC when out in the great outdoors; even then, I typically CC, because I'm not normally in places where bears are a major concern. (Although we did seem to have had one in the barn a couple years ago....)