Hi gabby! Don't worry about it! I bought a vest when I was staying in NY city just because I thought they were cool and all the photojournalist seemed to wear them. I never had a cop stop me to check (all guns are illegal in NYC) and lots of people wear them who don't have a gun. In fact most people with those kind of vests don't pack a gun. People who have a concealed pistol just think everyone else has one too
In fact my mom wore one for the same reason on a few trips, so she wouldn't have to carry a purse and she's never considered carrying a pistol.