Get the whole shabang hot with a heat gun. Then apply the corner of a piece of dry ice to the sight. Tap it out before it all comes back to the same temperature. It might work with regular ice, but dry ice ice is better and more fun to play with anyway.
I haven't tried this with sights, but it has worked with other interference fit parts.