A little late to the party here but I'll chime in. The number one issue for me is the fact that you have to depress the grip safety to rack the slide. This is a no-go for me on a defensive weapon for a miriad of reasons which should be pretty obvious. Also, as others have mentioned its just another part that could break and if it does you are left with a gun you cannot rack the slide on.
If Springfield would change the function of the grip safety on the XD/XDm line like they did on the XDs where you don't have to depress the grip safety to rack the slide I think that you would see more training professionals and LEO's jump on board with the XD. The current design is prohibitive to serious work in my book.