Don't do it
I have a poly Baby Eagle (Jericho) in 9mm and while the trigger on mine might be awful, I LOVE that little gun. At some point I will try to have the trigger reworked by a good smith, and at some point I will also buy a steel version.
XD's are fine guns but they are a dime-a-dozen. You don't see Jericho's/Baby Eagles's too often. I wouldn't part with mine!