Thanks for the input so far.
One of the first thoughts that went through my head when I realized he was indeed following me was to head towards a police station. Unfortunately, the closest one was about 10 minutes away, maybe more with traffic. The gas station I was headed to is one I frequent often, I knew that the area would be well lit and I also knew it was protected by security cameras. At the time, I figured it was better than letting the guy follow us home
It wasn't until afterwards that I really started to think he didn't have one. At the time, I was so surprised by his "live or die" remark that I reacted to it without really thinking. Like I said, since we didn't actually see a weapon, neither of us drew. Part of me thinks that he registered both of us preparing to draw and that helped cool him off.
As Geetarman said, it was a learning experience for me. Thanks again for the input so far.