I have had great results in .41 Mag loads using AA#9. Yes the recoil is near to what the factory loads are, though there is much less flash, and muzzle flip to my point of perception.
Oh and if you are going to use it for hunting. Instead of grouping. Try hanging up paper plates for targets. Shoot each one once. If all of the shots are in the non ribbed area it will do the job on any deer you put the sights behind the shoulder of. In the opening you said the first two were close then the group opened up. That tells me it is from one of three things. 1. Flinching (it happens to many of us with large bore mags after a few shots. The good news is when hunting you only will get one shot anyhow.) 2. Not crimped enough ( as has been mentioned.) 3. Barrel heat (wait at least 3 minutes between shots if the barrel feels warm. Though I have never had this problem with the single actions I use for hunting hogs.
No matter how many times you do it and nothing happens it only takes something going wrong one time to kill you.