He knows where you live now. This has several implications.
1.) Will he be back? Probably not. But if he gets particularly desperate, he may remember that you own a shotgun (I know you don't, but HE doesn't), and he may want to get his hands on that to obtain money from other people or to sell for drugs.
2.) If you didn't carry 24/7 (when legal) before, you certainly should now, and I would actually invest the 300-400 in an 870 and place it someplace secure and/or inconspicuous. You may need a more potent home defense weapon than a handgun.
You did the right thing. You avoided violence. You're probably clear of that guy forever. But you absolutely must keep alert now.
14 Pistols, 5 Rifles, 1 Shotgun, no time to shoot them