The reasons are many and varied. They're why black powder is interesting. Every load is different, sometimes by just minute things and sometimes by significant things. Sometimes due to esoteric scientific facts (barrel dynamics), sometimes due to simple facts (chamber/bore dimensions) and sometimes due to things we can't really account for (powder variations, powder compression, voids in projectiles).
I'm always amused by the interest in chronograph test results; people act like they really mean something, to three significant figures, when in fact they're looking at numbers from uncalibrated instrumentation measuring data from a test with uncontrolled, changing variables.
FTIW, my Uberti Walker is most accurate, as determined by 5 shot group size, when shooting .457 rb's over 50 gr fffg Goex with a lubed felt overpowder wad. Change any of those variables and group sizes increase (I've not tried Swiss, however). I have no idea how fast the balls were moving when they left the barrel.
From my data, the presence of the wad improved group sizes by a small amount. I presume this is due to moving the ball closer to the chamber mouth, but I don't know why that's 'better'.