Kevin, you probably have already done this but I used to have some similar problems. Not as bad as yours I don't think so maybe this isn't it at all. Anyway, I was using wd-40 down my barrel after I cleaned it to stop rust. I would spray enough in there to run a stream out the nipple. When I would use the rifle, I would run several dry patches down the bore and snap a few caps on the nipple before I loaded up. If I shot it pretty soon It mostly worked but it seemed the longer it was loaded, the more likely it was to misfire or hang fire. Once I got it to shoot a couple of times, the problem would be over with for that session. This was in my .32 squirrel rifle which would get fired several times, generally, every time it was out. If it had been used for deer, you know, load it up and wait..and wait.. and then do it some more, I imagine I would have had a bigger problem with it.
What I decided was that I was having the wd-40 stay down on the face of the breechplug or somewhere and not being removed by the method I was using before I loaded. I don't think it takes much of that stuff to kill the powder. I stopped using any petroleum based lubricant and my problem stopped. After I clean now, I heat the barrel and lubricate it with the same stuff I use for patch lube. A couple of days later I run a patch or two to check for rust and re-apply if there is any. I keep checking every other day or so until the patches stay white. I grease the nipple threads with it, too. If there's some pitting in your chamber or gouges on the face of the breechplug there may be enough oil staying in there to cause your problem.