My opinion is to do what you feel is necessary. If I saw an off duty officer wearing a vest, I would not think they were paranoid. I am glad to hear you feel that law abiding citizens with firearms are friends of LE. It is good to hear that you carry while off duty. Some agencies have policies that do not authorize their officers to carry off duty - especially if they will be outside of their jurisdiction at any point.
I am not sure if this is still the case, but stab resistant vests use to be a hot item in the UK. Parents would buy them for their kids, as street gangs and school gands in the UK are a big problem in some areas. Since LE is usually unarmed in the UK, they do little to nothing with these vicious gangs who armed themselves with knives and/or firearms. So much for the safety of an unarmed society.