Slug your bore using a dead soft fishing weight or slightly OVERSIZED bullet. Measure it and use bullets that are .001 or .002 OVER this measurement.
If you are ordering your bullets pre lubed, see if they have lube specific for BP. Otherwise, you may want to order them without lube and do it yourself using proven homemade lubes.
Back off if you are using hot loads. With BP in a handgun I don't believe this is the problem.