The leap second issue was only indirectly responsible; there were at least a couple of leap second bugs in the linux kernel: the first would have caused a crash on Saturday, but didn't affect this server. The second caused excessive CPU utilization starting at midnight July 1st.
While rebooting to try to clear up that leap second CPU bug, the server ended up hanging while trying to start mysql, before the webserver or ssh was started. I'm still not sure why it didn't timeout eventually and continue with the startup, but it didn't. I haven't had a chance today to poke around and look.
Ideally the problem would have been fixed quickly, but the colo situation is less than ideal.
“Compleχity is a symptom of confusion, not a cause.” —Jeff Hawkins
“The egg hatched...” “...the egg hatched... and a hundred baby spiders came out...”
Last edited by tyme; July 2, 2012 at 10:22 PM.