At face value, I do not see anything wrong with your shot-string. Perhaps you could try some 2F and see if that makes a difference. I know a lot of guys that shoot 3F in their 54's with good performance. Are you shooting BP or Pyrodex? Older Pyrodex and other subs, lose power, with time. Black powder stays good longer. If that is what your rifle wants to shoot then that's might just be it. Need more info but try some fresh powder.
Be Safe !!!