I don't think most people would be so bold as to ask you what was in your pocket unless you're trying to conceal a S&W 500 or something.
As for "gun free zones", it depends on what it is. If it's a government building, school, etc. I leave it in the car. I usually am not going to anyplace like that where I'm not driving so it's not a problem. If it's the mall or something like that I carry even though they are posted. It is not a criminal violation to do so where I live. If they knew I was carrying then they could ask me to leave, but concealed is concealed.