As he advanced yelling I'd have already been drawing from the ankle, because if things did get violent I wouldn't have time to. You did well, though, to de-escalate the situation and get out of there without risking your daughter.
Remember, no one was harmed. You achieved directive #1 of any self-defense scenario.
That said, this is why I pocket carry. If things get tense, I can already have a grip on my LCP without letting anyone know I am ready.
Well, I have big hands. I can have like a 3-finger grip to get it out of the pocket.
Beretta PX4; 92FS; 92S; 8040G | Sig Sauer SP2022; P6 | Colt Python 4"; 1968 Detective Special 2"| Smith & Wesson M642-1 | Ruger LCP; 10/22 | Daniel Defense M4V1 | Remington 870 | Alpine M1 Carbine | Steyr M1895 |1943 Izhevsk Mosin M91/30 | Walther PPS