He knows where you live now. This has several implications.
1.) Will he be back? Probably not. But if he gets particularly desperate, he may remember that you own a shotgun (I know you don't, but HE doesn't), and he may want to get his hands on that to obtain money from other people or to sell for drugs.
2.) If you didn't carry 24/7 (when legal) before, you certainly should now, and I would actually invest the 300-400 in an 870 and place it someplace secure and/or inconspicuous. You may need a more potent home defense weapon than a handgun.
You did the right thing. You avoided violence. You're probably clear of that guy forever. But you absolutely must keep alert now.
Beretta PX4 .40 S&W - Beretta 92FS 9x19 Para - Rock Island Armory GI 1911 .45 ACP - Colt 1968 Detective Special .38 Spc. - Ruger LCP .380 ACP - Daniel Defense M4V1 Carbine 5.56x45 - Alpine Industries M1 Carbine .30 - Steyr-Mannlicher M1895 8x56 - Ruger 10/22 .22LR - Remington 870 Express Tactical 12GA - 1943 Izhevsk Mosin-Nagant M91/30 7.62x54R