hey birch, my walker does the exact same thing. 6" high and about 3-4" right.
If I aim COM on a human silhouette, I group at the low left shoulder...

I have seen some ingenious ideas on how to "add" to the front sight with minimal cosmetic distraction. The smith that worked on mine said to add a new front sight would mean a really ugly dove tail cut to accommodate the new sight. but she draws a crowd and I like introducing people to the world of BP.
BTW- My 2nd Gen. dragoon isn't much more accurate than the walker..