I thought you said the table was purpose built for shooting,but the chair was low? What happens when you shoot a 10 round group or 20. Shooting off a bipod and towels in a chair that's too low with just plain hunting ammo doesn't add up to the best chances for accuracy. Your bipod most likely makes the table seem too high or chair too low. my guess is that the equipment you have is capable of doing the job and you are somehow moving the gun when pulling the trigger.
Your theory about the rounded bullets being affected by wind at 100 yds causing a shot to go 4" high is unlikely. You would have to have a hell of a wind to affect a bullet 4" at 100 yds - especially upwards.
The real answer is that it could be a number of things, but attention to details, eliminating the most likely suspects and lots of practice should reveal the correct answer. From my point of view, you are doing pretty good with that set up.... Everyone has "fliers"