He knows where you live now. This has several implications.
1.) Will he be back? Probably not. But if he gets particularly desperate, he may remember that you own a shotgun (I know you don't, but HE doesn't), and he may want to get his hands on that to obtain money from other people or to sell for drugs.
2.) If you didn't carry 24/7 (when legal) before, you certainly should now, and I would actually invest the 300-400 in an 870 and place it someplace secure and/or inconspicuous. You may need a more potent home defense weapon than a handgun.
You did the right thing. You avoided violence. You're probably clear of that guy forever. But you absolutely must keep alert now.
Beretta PX4; 92FS; 92S; 8040G | Sig Sauer SP2022; P6 | Colt Python 4"; 1968 Detective Special 2"| Smith & Wesson M642 No-Lock | Ruger LCP; 10/22 | Daniel Defense M4V1 | Remington 870 | Alpine M1 Carbine | Steyr M1895 |1943 Izhevsk Mosin M91/30 | Walther PPS