Leaving does sound like the best option but you dont want to turn your back on him neccessarily. I was in a similar situation but had my pregnant wife with me so I couldn't just run off to leave her with two thugs!
I could feel him getting ready to suckerpunch me, so backstepped off the curb and took a drawing grip on my concealed pistol without saying a word. It worked. That time.
They could probably have you charged with menacing threats, even if you didn't say anything and just gripped it. We were lucky, God walked with us that day. No easy answers for this one. Moment by moment decision process.