Could just be copper build-up. Most gun cleaners claim to remove copper, even when they don't do it very well. Ballistol could loosen the edges of the layers in a serious build up, so that it feels rougher on a patch. Read
this article as a starting point.