2123, my philosophy is also similar. But that me. Others may be better advised to just stay inside such as a woman alone or someone not well prepared. Fortunately I have cameras that can see all areas around the house at .001 lux.
If someone was trying to actually break in, I would wait and let them come on in after calling Sherriff to come pick them up. This is a area where neighbors are 1/4 mile apart and police are at least 30 min away. Anyone knocking or banging on door anytime after 10-11 pm would be just as odd as 4 am
We each have to act in such a way that we are comfortable and prepared to do so.
If you live in a semi-remote or remote location, your options will differ from those living in a gated community where houses are close to one another.
Also, as you mentioned, timing is an important consideration. If someone knocks on my door after 10pm or so, I will approach the door in a different manner, than I would at 11am or so.
Bottom line is to know what is or what isn't normal or routine in your community and to pay attention to the little things that don't quite measure up. Being aware is a very good start to being safe.