If he didn't enter the open door, and didn't try to force the locked doors, then he wasn't trying to break into anything. You probably saw a guy who was confused as to where he was and once he realized that move on. No harm no foul.
There is a fine line between being aware of what is happening and being paranoid, I think the lien was crossed in this case.
the main thing though is no one was harmed.