17 years as a full time LEO here. While I carry both on and off-duty I do not wear my vest off-duty and I don't envision ever doing so. The handgun is for unanticipated problems. Off-duty, If I feel that an area or location was such a risk that I needed to wear a vest, I don't go there.
You specifically mention taking your children to the movie with you. Where's their vest?
In the end, wear it if you like. I'm just giving my thoughts on how I chose to be.
All that is neccessary for evil to triumph is for good men to do nothing.