You did good. You recognized something unusual and did not answer the door to be "polite" to a stranger. You got your gun and called the nearest trained responder.
However, no need to go back upstairs to confront him. Your grandpa or other responder could do that.
We just installed a wireless IP camera over our front porch so we can see who is there before getting close to the door. We will not answer the door if it is a stranger. Family and friends have the courtesy to call us before stopping by. The only stranger I expect is UPS or Fedex and they knock and go.