I squeeze the trigger and my shots are not all pulling one direction. I shoot a fair amount of pistol also. No problem with 2" group at 25 off hand, when I shoot my bolt 22 at 50 I can put group average about 5/8".
You are ready for 100 yards now. Shoot both long gun and pistol at 100 yards and you will learn to concentrate on that front sight/trigger like you never have before.
When you go back to 25 and 50 yards you will find the concentration needed for intermediate range shooting has made you better at the close ranges. As soon as you are shooting most of your shots inside the black bull at 100 move to 200 yards. As soon as you get over the mental hurdle of it being so far away and find that the bullet drop isn't near as much as you thought it would be your groups will tighten up again.
The big thing is between the ears, convincing yourself that you can and then you will do. In the meantime pay attention to the light, it will affect you more at longer distances, learn to judge wind and know how far to adjust your sights to accommodate it. Then you can get past these intermediate distances and start working on long range. Don't hurry it, nothing happens in a day. Just relax, think it out have fun and it will come quicker than you think.