Please tell me that you saw him with his back to the door from someplace other than that door, window, side door etc.....................
Never go directly to the door. LOOK FIRST.
And you did good by not answering. Nothing says you have to open the door when someone knocks.
Here's a drill for ya:
Turn your phone over so you can't see who is calling next time it rings. When it stops ringing check to see if you are still alive. Same concept applies to someone knocking on the door.
If someone chooses to enter when you did not answer the door then they have taken things to whole nother level. That's what the Glock is for.
.44 Special: For those who get it, no explanation is necessary. For those who don't, no explanation is possible.