I thought long and hard as well and then tried some different holsters, of course I used the search button here and over at hs2000talk.com first
Anyway, what I ended up doing (I have two XD's, a 9 sub and a 40) is buying a owb leather holster labeled Old World and it fit's both guns. This puts the firearm in the same position no matter where/when I am carrying.
I carry the 40 around the house and the 9 when I go out (along with a .380 keltec in the pocket).
The way I dress is that I have a t-shirt (or a wife beater depending on how hot it is) as the first layer to keep the firearm from rubbing a sweaty body and then a loose shirt on top of that, in which I usually just button the middle button or two.
I think you are probably just going to have to get over the "it's hot here and two shirts don't work" mentality and let your body adjust to it.
I tried the IWB's and never found one that I liked, the OWB has been the most comfortable...