Congrats on the XSE...
You probably know all this already, but...
Series 80 firing pin safety parts are seldom the issue with a heavy trigger pull... and as they came with the gun, I'd not be removing them unless the gun is to be used purely for target shooting.
You might try detail stripping the gun, cleaning and applying proper lubrication to all fire control parts. With some guns, this will make for a very noticeable improvement. If you lived in the Puget Sound area, I'd be happy to show you how to do this, if you don't.
If you are mechanically inclined, you can, relatively easily install a "drop-in" fire control kit, such as the one sold by Cylinder and Slide for the series 80. If you're
really mechanically inclined... you can read up on (there's tons of DIY guides with photos on the internet) and do your own trigger job. If you can't pound sand into a rat hole... then get to know a good 1911 gunsmith.
Same deal with the safety... although in reality, there is no such animal as a drop-in safety, unless you just get really lucky. It's not really difficult to fit one, but you do need to understand the relationship of the safety and sear.
Thanks to John Holbrook/THR forums for the excellent cutaway photos. The secondary sear angle is nicely visible here as well.
Cheers,
C