I have not yet observed any negative effects on the wedge or arbor. You'll note that I've taken into consideration the bullet mass versus the charge amount. The 141 grain roundball with 60 grains of BP is the exact max charge that the revolver in cap-n-ball mode is designed to handle. That's the rumor anyway. The 150 grain bullet is as close an approximation that one can get with a conical style bullet. The decreasing Bp charges for the increasing bullet weight are meant to minimize the risk of wedge/arbor damage.
Not sure what the results would be with Swiss but as you indicate there most likely would be an increase in performance.
I've always wanted to get a Walker, prefer carrying a big bore pistol rather than a heavy rifle when camping in the hill country, but kept putting it off, because from diagrams I've seen, the arbor is not replaceable in case it get damaged, and I don't want to risk ruining a pricey gun due to that little part.