Leave the screw be. It stops the wedge from falling out once its loosed, and stops the wedge from being hammered in too far.
I use a fairly hard rubber mallet to get out the wedge if it's too tight. Put it to half cock, then give it a good whack with the mallet directly from the side opposite of the screw. I usually just hold the pistol in my other hand, under the frame, not by the handle. A solid whack seems to do it most of the time. Actually seems to work much better than a bunch of tapping or pounding with a brass drift.