Someone with enough concern (and suspicion) might, if unsatisfied with a pleasant smile and a polite non-answer might lead to a conversation with the police investigating a suspected "man with a gun" call.
If the officer has probable cause I think the officer has a duty to check it out. Will one citizens suspicion constitute a probable cause ? I think the Leos on this board need to answer that.
If I am carrying legally why would I worry about it ?