Loctite makes a product called Quick Metal that's specifically for installing bearings where the hole has worn sloppy. That or one of the high strength Loctite's might be just the ticket.
Gunsite Orange Hat Family Member
CMP Certified GSM Master Instructor
NRA Certified Rifle Instructor
NRA Benefactor Member