As he advanced yelling I'd have already been drawing from the ankle, because if things did get violent I wouldn't have time to. You did well, though, to de-escalate the situation and get out of there without risking your daughter.
Remember, no one was harmed. You achieved directive #1 of any self-defense scenario.
That said, this is why I pocket carry. If things get tense, I can already have a grip on my LCP without letting anyone know I am ready.
Well, I have big hands. I can have like a 3-finger grip to get it out of the pocket.
Beretta PX4 .40 S&W - Beretta 92FS 9x19 Para - Rock Island Armory GI 1911 .45 ACP - Colt 1968 Detective Special .38 Spc. - Ruger LCP .380 ACP - Daniel Defense M4V1 Carbine 5.56x45 - Ruger 10/22 .22LR - Remington 870 Express Tactical 12GA - 1943 Izhevsk Mosin-Nagant M91/30 7.62x54R