No, you are not worrying too much.
Using too hard an alloy can cause all sorts of nasty things. The hard ball can resist the pressure of the rammer to the point of breaking or bending the loading arm.
If you cast a harder alloy and have casting flaws, the ball may not swage properly into the chamber and leave an air gap along the side of the ball. this results in a delightful experience called a chain fire. Two or more chambers going at the same time. Don't ask me how I know and don't ask why it took four occurances before I figured it out.
Use the softest lead you can find.