Personally, I think you would have been justified to shoot. You were the passenger. How could you safely retreat? The best way to retreat is to speed away in the car, and you were in no position to do that. So you jump out of the car and lose your best means of retreat? Doesn't make sense.
And there had to be some reason you felt he was a threat. Even if he hadn't pulled a weapon, did he allude to one somehow? And having your sidearm out possibly prevented him from pulling his. I'd like to point put that in such close quarters, he could have had a weapon if he had taken yours, even if he didn't bring one himself.
The law isn't black and white. There is a certain point at which there is no right or wrong answer. At this point you would have to look to case law, and at convincing a jury to see it your way. If I were on a jury I wouldn't convict you. However, this starts to turn into a game of "Who's got the best lawyer, the most money and lots of luck?"