A stray pellet or two is unlikely to be lethal, and may not even cause serious damage, but the full shot charge (and wad) certainly can.
When its my life and the life of my family on the line, I'm not going to put my eggs in the "certianly can" basket.
There is more than enough evidence out there that shows the ineffectiveness of birdshot for stopping the attacker. Add to this some tweaker who hasn't come down yet and you have serious problems.
With low recoil loads there is no excuse for using buck in your shotgun.