It is very unlikely that he attacks. He has a lot to lose if he does. Also if he does decide to attack you then you should defend yourself to the legal extent. I myself train with MMA guys when I do Kali. I suggest you learn to play nice with the neighbor instead of fearing him. I mean keeping everything cordial is all well and good. It is always good to know your neghbor. Keep it that way. No need to fear a friend right?
Faith Reason Fraternus