Do you have a floating handguard installed on it? Was the guy that put barrel on a real AR mechanic or someone with tools that knew how it came off and went on?
What kind of sights? If scope the mount could be loose. Also the internal adjustments on the scope may be worn out and shift constantly.
What kind of bullets are you using to handload with?
You are not giving us much to work with here.
Did Bill Wylde build it or did you just get a barrel with a Wylde chamber? I know Bill and it doesn't sound like one of his.