At close quarters it is easier to wrest control of a longer gun, and/or to get inside the weapon. (I work out a lot with cops and MPs, and weapon retentions and disarms are kind of a hobby/specialty.)
It is easier to draw the shorter barreled weapon if the threat is close.
As has already been noted, a 6" is harder to carry, particularly if seated.
For range use, or hunting (typically carrying crossdraw or on the chest) a 6" is probably a better choice for pure ballistic energy, longer sight radius, etc.
For defense, I would go 3" or 4", in that order.