I wouldn't sweat it. Keep on trucking with what works for you. If you're overseas where guns are far less common than tourists I suspect you'll not have a problem. Even if civilans occassionally think you might be carrying - it should only serve to make the BGs more conservative.
If officials (foreign or domestic) question you and you're clean - so what? Do your best "Aunt Bea" routine and go on about your business.
I'm not female - and I'm swiftly moving past the "middle" part of middle age - and I often wear an old fishing vest for exactly the reasons you do. In this hot East Texas climate I often wear a light tee shirt and jeans. Since I need glasses to read, use tobacco, often have a dog with me that sometimes needs a leash, carry a checkbook that I don't want crushed and so on - I don't appologise for the vest.
In my case it often helps conceal a firearm. If it didn't conceal a firearm, I'd still use it because it works.
My general advice is "Don't let someone else's problem become your problem" if you can help it.
Have fun in your travels.