You did good. Good awareness. No one got shot.
In that area/situation, I would have probably already drawn by the time I was asking their business. Not pointed or aimed, just held it down by my side while clicking the safety off.
I've done some measuring. An average human, running full out, can cover about 30 feet in 2 seconds. That's two mini-van lengths (for reference), and 2 seconds is the amount of time we're supposed to be able to draw and get a shot off. Don't let'em get too close.
+1 on checking 6.