As he advanced yelling I'd have already been drawing from the ankle, because if things did get violent I wouldn't have time to. You did well, though, to de-escalate the situation and get out of there without risking your daughter.
Remember, no one was harmed. You achieved directive #1 of any self-defense scenario.
That said, this is why I pocket carry. If things get tense, I can already have a grip on my LCP without letting anyone know I am ready.
Well, I have big hands. I can have like a 3-finger grip to get it out of the pocket.
14 Pistols, 5 Rifles, 1 Shotgun, no time to shoot them