Just leave it fouled. Slightly fouled would seem best if your rifle's accuracy suffers with fouling buildup. I don't think I have a rifle that isn't slightly fouled at this time.
I just cleaned my Dad's rifle and the bore hasn't been cleaned in 5 to 8 years, or maybe 10 years. The bore wasn't rough at all, and it shoots great. Don't get overly concerned about having a dirty bore.