I'm a member of the N-SSA and we shoot Civil War Competitive shooting. In an average year we run about 800 round a year through our muskets and it is a yearly thing to pull the breach and polish it. Polishing it cuts down on the buildup you can get on a breach plug making a difference in performance. They make a breech scraper and you should have one of those too, you'd be surprised all the stuff that can build up on that plug. Getting a clean patch dose not mean there is not a build up on the plug. Take a mini Mag light and see if you can see a shinny plug at the bottom of the barrel, chances are you will not. There is nothing scary about pulling the breech when you have the proper wrench. The barrel maker would not go to all that expense to put it there if it wasn't a good thing to have.
For more info visit
www.n-ssa.org
Tom