Time for some outside help - get a coach or go to a class. Untold benefits await if you get someone who knows how to shoot to observe and critique your form... maybe that guy who was punching out 1" holes could be a buddy? Find one...
Another possibility in the meantime - try a different stance/hold. Sounds like you're using iscoceles (both arms straight out and squared to the target). Very strong and rigid stance, but all one directional and some people are jittery with it. So switch - try bending your weakside elbow while keeping your stringside straight and shift your weakside toward the target fourty five degrees or whatever's comfortable. Feet follow too, weakside a little forward of weak. Pull toward yourself with the bent arm's hand while pressing forward with the strong hand, providing isometric (opposing) tension. What you'll have is a sort of Weaver stance. (Threre's more to it than that, but a knowledgeable coach/instructor can help) This may help steady you if you have involuntary wiggles.
And it may not. The joys of studying form!
