You're fortunate that the beating on the front sight the wrong way didn't seem to dent or distort the barrel, otherwise you wouldn't have grouped 3" at that distance.
Any distortion of the bore near the muzzle is prone to causing the bore to be altered at the most critical part of the barrel. Even cleaning from the bore end without a guide can wear the crown as can dropping the rifle on the muzzle.
My B-I-L had a Marlin 336 in 35 Rem that had the front ramp screw drilled a bit too deep. After repeated firing, the pressure dimpled the bore upward into the screw hole and bullets were coming out tumbling and landing sideways on targets at 50 yards. It couldn't even hit paper often at 100 yards.