Well, you don't have to fire maximum loads out of a Walker. It is fun to shoot with 50 grains of BP. The only reports I recall of damage to the wedge or arbor were on Walkers made by Armi San Marco (ASM). This occured with repeated firings with 60 grains of BP and a round ball. I suspect not all ASM Walkers have this problems but it's probably limited to some that got out of the factory with improperly heated treated steel. Just a guess.
I plan to use the Walker as a hunting gun when camped in the highlands. I like wearing my regular .45 LC Remington on my hip and then the Walker in a shoulder rig, and use all the packs and bags for just gear/fuel/tools instead of a rifle and heavier ammunition. 50 grains sound like a good idea since some shooters claimed that 60grs didn't givve them much noticeable increase in velocity compared to 50 grains.
One question I have: Is your Walker's arbor screwed onto the frame as well as pinned? If damage occurs, is it replaceable? If I do get one, should I bring a spare arbor since I usually live out in the high country for months at a time.