Since I am not shooting benchrest rifles, the silencers I use don't noticably reduce accuracy unless something is wrong.
It sounds as if the silencer you were using caused baffle strikes. I have a 22lr can that caused baffle strikes and changed the point of impact by two feet at 25 yards. It normally had no effect on accuracy and was repaired by repairing the seating surface of the barrel adaptor. Alignment is critical to say the least.
My ar-15's accuracy does not degrade, but point of impact changes a bit. My 510 whisper has a large 2"x18" can that changes point of impact several inches at 100 yards, but all I have to do is adjust the scope to put it back on the bullseye.