As he advanced yelling I'd have already been drawing from the ankle, because if things did get violent I wouldn't have time to. You did well, though, to de-escalate the situation and get out of there without risking your daughter.
Remember, no one was harmed. You achieved directive #1 of any self-defense scenario.
That said, this is why I pocket carry. If things get tense, I can already have a grip on my LCP without letting anyone know I am ready.
Well, I have big hands. I can have like a 3-finger grip to get it out of the pocket.
Beretta PX4 .40 - Colt 1968 Detective Special .38 Spc. - Ruger LCP .380 - Daniel Defense M4V1 Carbine 5.56 - Ruger 10/22 .22LR - Remington 870 Express Tactical 12GA