The gunsmith gave you good advice. The mother of pearl is very brittle and it really isn't worth the risk to a beautiful set like that to risk cracking them. I would put some more functional set of grips on to shoot it. It's a perfect place for a pair of Pachmayr's. The pearls are great for "show" but not as good for "go".
Be careful in removing the old pearls and don't put any pressure on them. You are going to be surprised at just how heavy they are.