If you actually got a trace of lug bearing on the light contact lug,I think you have no problem. Get your micrometer out,wipe a little magic marker across it,let it dry,and decide how much clearance "only a light contact" means.
I don't think you are looking at more than .0002.
So now imagine you lap with 500 grit on the tight lug till you get lug contact.Surprise!!!The rocks in 500 grit are about .002.320 grit,about .003.Guess how much clearance you have when you clean it out?
Now,there is some diametral clearance around the bolt.Lets say .003 for fun.
When you touch the bolt handle,the bolt is deflected to the limits of this clearance.If both lugs were bearing equally,the force of your hand of the bolt would lift one lug off contact very slightly as the bolt deflects .0015
I'd say,shoot it!!