"I ... spend lots of time ensuring my powder charge is accurate to a single granule...
Am I wrong in this? "
maybe not wrong, but you are wasting your time on this step.
And be sure you don't let the bullet get "smooshed out of true."!!