The disk/raid driver was getting stuck, and the kernel wasn't rebooting from the subsequent soft lockup like it was supposed to.
After last weekend's fiasco we had a new linux kernel ready, so we're on 2.6.28 now.
I've also converted 2 of the 3 filesystems that were xfs over to ext3 (the database/web/root directories have always been ext3, but a few non-critical directories were xfs to speed things up). There's a recent xfs bug report I'm worried might have been the cause of these lockups.
Next time just give it a good whack right at the epicenter of the fulcrum, always works for me.