Does Ruger use anything like the Glock copper-colored lubricant stuff out of the factory? That may be what you're seeing, which would look like powder as you describe, in which case there's no harm.
Can you get a pic of the area in question? Might be hard to get to, but then a good macro shot may help you better see what's in there. In any case, sounds like you take care of it so I wouldn't worry unless it comes back.
Also, do you remember what you shot last? I know when I shot, e.g. Remington goldenwhatevers, they sometimes left copper/gold-colored residue.
Shoot smart. Shoot S-Mart.