If you end up removing metal from anywhere it should be from the bottom of the sight, not the frame.
I do not suspect the grips. Are you shooting from a rest?
"A Liberal is someone who doesn't care what you do, as long as it's mandatory". - Charles Krauthammer