Like others the belt added to your holster should not be needed and when a pistol is as larger as the xds it does not work real well in the pocket. Mine a little smaller and will never be pocket CC'd.
I to am a heavier guy and found a couple years back what I think is a great iwb holster thans very comfortable to wear on the front half of my body. I wear a comp-tac minotaur neutral set just high enought so I can set and drive with out problems at about a 2:15 position. Over my front pocket.
I have since trimmed the bottom leather off the holster. And that pocket holster is history too but allways worked fine.
The pistol is a kahr cw9. The width and height or the same with your xds being only 4/10ths longer. I can cover with a loose t-shirt or polo even tuck in a botton down or T. I also pocket a cm9 and in real life I will say walking and sitting with a cw9 sized pistol in a pocket shows. Prints just just to much. Still photos don't do justise to accual movement.
I wear dockers, columbia ,full cut jeans and bibs. Out of view yes but not really hide from knowing eyes. CM sized is way better and way easy to retreve when needed. I have pocket CC'd for 20+ years as a pimary way to carry and have found bad and good pocket holsters but never had issues with a handgun getting out of position. Most of those years as a carpenter.
Take a look at Recluse holster for pocket. They have a different design that simply works and pass's is a wallet. http://www.recluseholster.com/Produc...4B2-1349841735