It's not the sights, you're not used to shooting a Glock. The grip, especially the angle of the grip, is much different from other pistols. The trigger also takes a lot of getting used to and breaking in.
The slippery texture of the grip doesn't help any, I have a Hogue HandAll grip on my G23. It adds bulk but fits my hands well. I'll probably get a Talon Grip for my G27 in the future.