The leap second issue was only indirectly responsible; there were at least a couple of leap second bugs in the linux kernel: the first would have caused a crash on Saturday, but didn't affect this server. The second caused excessive CPU utilization starting at midnight July 1st.
While rebooting to try to clear up that leap second CPU bug, the server ended up hanging while trying to start mysql, before the webserver or ssh was started. I'm still not sure why it didn't timeout eventually and continue with the startup, but it didn't. I haven't had a chance today to poke around and look.
Ideally the problem would have been fixed quickly, but the colo situation is less than ideal.
“The egg hatched...” “...the egg hatched... and a hundred baby spiders came out...”
“Who are you?” “A friend. I'm here to prevent you from making a mistake.” “You have no idea what I'm doing here, friend.” “In specific terms, no, but I swore an oath to protect the world...”
“It's a goal you won't understand until later. Your job is to make sure he doesn't achieve the goal.”
Last edited by tyme; July 2, 2012 at 10:22 PM.