Experiment with various loads, FIRST
Before you punch and file on it, try some different loadings.
If you are shooting round-ball and the patch is too thin, it will affect your accuracy. Loading HOT will make that ball fly erratically. Loading LIGHT will affect accuracy. Different density of lead projectiles will act differently during the trip down the barrell.
Slugs shoot different than round ball and various models of slugs shoot differently as well.
Different powders will shoot slightly different.
Experiment with some loadings and look for stable consistency down range. Then move the sights if you need to. You might find that way back in the day, someone probably had that gun sighted in and had it hitting the mark with a particular load. You might stumble onto the load.
Keep some written notes on black powder guns - what worked well. Keep the notes stored in the case with the gun. It really comes in handly later.