I don't have any personal experience with the XD or XDm, but I've owned several M&Ps - still do - and find the ergonomics to be superior on them.
But, on the topic of switching between 40 S&W and 357-SIG, I tried that on my Sig P226R. It was easy - the Sig 357-SIG barrel is a drop-in. Nothing else is needed. Except for Mr. Murphy. See Murphy is a close relative of mine, or so he seems to be. After the second time I found myself with the wrong ammo in the magazine for the barrel I had in the gun, I decided to sell the 357 barrel and stick with one caliber in one gun. OK, I make exceptions for my 22LR conversions. If I make a mistake with one of those, I need to take up basket weaving.