"I ... spend lots of time ensuring my powder charge is accurate to a single granule...
Am I wrong in this? "
maybe not wrong, but you are wasting your time on this step.
And be sure you don't let the bullet get "smooshed out of true."!!
... still waiting for that stimulus to kick in ...