I actually try to use my press every chance I get.
If you must use a drift and hammer also keep in mind that if your holding fixture (most likely a vise) is padded it will absorb much of the blow making the sight harder to drift.
You might find using a smooth jaw vise and card stock to keep from marring your gun will work best for a holding fixture.
Best Regards
Bob Hunter
www.huntercustoms.com