I was disappointed with the stock as well. Its suprising they dont offer the FVT with the accu-stock. The added weight and rigidity make a big difference.
Correct me if I'm wrong but the sling shouldn't change your POA. The barral comes floated out of the box so neither ataching point for your sling should cause any issues. I have a mark 2 that I frequently hunt/target shoot with and have never had this problem.
The floated barral may be part of why it feels 'flimsy' as the soft synthetic does flex a bit, but I've never seen it do so to the point that it contacts the barral.
Do you use a sling at the indoor range or did you have some other intention for the rifle?