Regarding rifled slugs from rifled barrels, I've never heard that it is unsafe. Brenneke claims that their rifled slugs can be shot in smooth or rifled barrels, but that shooting them through rifled barrels actually improves accuracy a little. I've never heard or seen that rifled barrels hurt accuracy of rifled slugs.
The biggest issue may be that swaged lead slugs traveling down a rifled barrel with no plastic sleeve (like sabot's have) could foul up the barrel quickly. Maybe it that got really bad, accuracy would start to suffer.