This is what I meant.
Clamps hold the base in one spot. and the wooden hold downs hold the grips in one spot.
Then it is just a matter of drilling and counterboring each hole one at a time.
Change only the bit and the workpiece stays put.
I used a real small bit for a pilot hole. Easier to locate the smaller hole, then move up to the right bit for the shank hole. Made a counterbore bit from an old broken drill bit.