As long as the ball is seated tightly against the powder and shaves a thin ring of lead at the edge of the chamber, you should have no troubles. For years, nobody used a wad when loading a cap and ball revolver. We smeared grease in the space in front of the ball. We thought we were preventing chain-fires, but what we were actually doing was keeping the fouling soft. I know it's messy, but after 43 years, I'm used to it.
I also use black powder only, not the newer powders.
I think of my gun as a tool for converting worthless human garbage into valuable fertilizer.