I wrestled with that problem for a few years, before doing something about it.
I just couldn't use the .220 Swift more than about once a year (if that), unless I was shooting paper. I tried to find ways to make it more useful, but I couldn't. The biggest problem was that it was illegal for most game, where I hunt.
In the end, I rebarreled the rifle to my own 6mm wildcat (consider it a .243 Win), with a 25" Shilen 1:10" twist barrel. I can hunt big game. I can pop varmints. I can shoot paper. It's a great compromise. I do miss the laser beam performance of .220 Swift ...but not much. The 6mm is pretty close.