In most states, an officer has to OBSERVE the crime being committed before he can make an arrest.
Thats why you can't just claim you got beat up and expect the police to go arrest your attacker. It doesn't work that way.
Ask any domestic violence victim about that.
At worst, the person would have to swear out a statement and try to get a warrant and appear to testify before a magistrate or judge. As there is no actual evidence it is just his word against yours. Most judges won't give that the time of day. Only a realy shady prosecutor who is out to make Charles Schumer brownie points would even bother persuing it, so far as that goes.
The fact that you HAVE a permit shows that you passed all the societal rules that your opponent probably would not be able to.
Remember, most sleazebags are too ball-less to even make a report to the police because they are afraid of getting tripped up into whatever they are doing.
If somebody claimed that you harrassed them and were packing, the only thing an officer could do is see if you are packing. If the other guy could not say what you were packing any cop with street savvy would smell bull$#!+ all over the other guy;s claims and get suspicous of him.
The only way you would be in trouble would be if you were packing without a permit.