He knows where you live now. This has several implications.
1.) Will he be back? Probably not. But if he gets particularly desperate, he may remember that you own a shotgun (I know you don't, but HE doesn't), and he may want to get his hands on that to obtain money from other people or to sell for drugs.
2.) If you didn't carry 24/7 (when legal) before, you certainly should now, and I would actually invest the 300-400 in an 870 and place it someplace secure and/or inconspicuous. You may need a more potent home defense weapon than a handgun.
You did the right thing. You avoided violence. You're probably clear of that guy forever. But you absolutely must keep alert now.
Beretta PX4 Type F Fullsize .40 S&W | Beretta 92FS 9x19 Para | Beretta 8040 Cougar Type G .40 S&W | Rock Island Armory GI 1911 .45 ACP | Sig Sauer SP2022 Stainless 9x19 Para | Sig Sauer P6 9x19 Para | Colt Python 4" .357 Magnum | Colt 1968 Det. Spec. .38 Spc. | Ruger LCP .380 ACP | Daniel Defense M4V1 5.56x45 | Alpine M1 Carbine .30 | Steyr M1895 8x56 | Ruger 10/22 .22LR |1943 Izhevsk Mosin-Nagant M91/30 7.62x54R | Remington 870 12GA