Frankly, I came to realize that the pistol was of no doggoned use at all. You can't shoot a driver for attacking you. You can't cap a dog on the run. there just aren't any real situations that a pistol can help you out, except for really, seriously crazy stuff that have million to one odds. So, I leave the shoulder holster and .357 at home now if biking.
I'm a long-time road rider, and see no practical use in carrying while riding for the same reasons you point out. If I were mountain biking in a remote area, I'd consider it. On the road, though? Nope, though I do carry pepper spray. Otherwise, I ignore dogs (they're mostly interested in the chase) and idiots behind the wheel (anything more is to just invite a confrontation).
BTW, in 30+ years of riding, I've noted a correlation between being very concerned about crashing, dogs & idiots on the road, and actually having those incidents. Riders seem to attract what they fear.