As Gdawgs says, powder can have a big effect.
However, what is your handgun to be used for?
There are hunting guns which may also be used for fun plinking and there are those which are best suited for and best used for personal protection.
If your firearm is for the last use, personal protection/social issues, 25' (or less) and not 25yds is what you should be concerned about.
I have some of both, but for the personal protection/social issue firearms, that practice is up close and personal, my concern being able to rapidly place my shots in such a fashion as to cancel the possible threat ASAP.
But, not being there and seeing for myself just what was taking place, I'd still say it is most likely the bullet and not the 231 causing the problem.
Try the loads at living room distances and see what they do.
Do they function the firearm with 100% relyability?
If so, Maybe the best answer is buying a RUGER Mark II or III and a back seat full of .22 ammo and learn how to shoot with a firearm likely much better suited to the 25yd targets or plinking.
Crusty Deary Ol'Coot