I think you'd have a hard time damaging the barrel with anything that's meant to go into it. You might see some little surface scratches, but I imagine they'll disappear after you put a few rounds through it.
The steel of the barrel is far harder than the brush - assuming you're using something designed for guns. before you push it out try loading it with oil and see if it comes out front ways.
Either way you're going to have a hard time butting more stress on the barrel than a bullet.
si vis pacem para bellum