Bulky? Nah. Don't use a 7.5" barrel, lol. The smallish size is one of the appealing aspects of a SA for defensive use. Slow? Situational awareness is the key here. If one is not aware of their environment and the potential threats present, then their draw is going to be just as slow with any gun.
The only possible drawback to a SA for defense in my mind would be multiple attackers. This can be tactically considered and dealt with. I marvel at the size/horsepower available in such a small action with SA's. And they are usually easily twice as accurate as any semi. Another consideration..