It's the shell holder.
Lee's shell holder is not actually designed for .220 Swift. It's designed for 6.5 Jap, and just happens to usually work for .220 Swift (until they let their tolerances drift...).
The problem is almost always the portion of the shell holder that's supposed to engage the extractor groove of the case. It's too narrow (around 0.415"), so the smaller rebated extractor groove of some .220 Swift brass won't fit (~0.420-0.425").
Sometimes, a sloppy .30-06 shell holder will work, but not always.
If you happen to have one, a .41 Mag shell holder sometimes works. (But you may need to manually aid alignment while the case enters a die.)
When I was still reloading .220 Swift, I had to keep 4 ".220 Swift" shell holders on hand; just to be able to handle all of the varied case head dimensions. ...and I only had three lots of brass: boxed Norma (same lot), W-W (one lot), and newer Winchester (one lot). It's not bad brass... it's just that almost no one makes a shell holder specifically for .220 Swift.
It wasn't until nearly the end of my .220 Swift experience, that I bought a Hornady shell holder. It never gave me trouble. ...which isn't surprising, considering it was designed specifically for .220 Swift/.225 Win.
Just to be sure...
Check the rim thickness of your brass, as well. I found the rims on Norma .220 Swift brass to be quite variable in all dimensions. They're usually within tolerance, but not very consistent.
"Such is the strange way that man works -- first he virtually destroys a species and then does everything in his power to restore it."