THR Outage right now

Derek Zeanah · April 7, 2007, 01:33 PM

Sorry folks, but THR is unavailable.

From what I can tell it's a network-related issue. My alarms went off this morning as we saw a 6 minute outage, then things came up fine, then they went down again, and are still down 86 minutes later.

It's not the THR box, it's all of them I have at the datacenter. It looks like it's a datacenter issue, as my colo provider's web page is down, no-one's available via AIM, and their voicemail box is full.

Just wanted to let folks know what's happening. I'm aware of the situation, but am sitting here waiting for a problem that's in someone else's hands...

Sorry.

Redneckrepairs · April 7, 2007, 01:44 PM

Thanks for the info , Likely to be some dammed fool with a backhoe and a lazy swamper involved lol .

sm · April 7, 2007, 02:40 PM

Derek,

I hope everything works out soon, I know you have a life besides tending to Networks.
As always, I appreciate your time and hard work, as I do others responsible for keeping THR , TFL and Sister sites up and running.

Steve

Larry Ashcraft · April 7, 2007, 03:08 PM

Thanks, Derek.

Derek Zeanah · April 7, 2007, 03:30 PM

OK, we're back up.

Bogie · April 7, 2007, 03:40 PM

Not quite...

Derek Zeanah · April 7, 2007, 03:41 PM

Crap. Looks like we might be going back down again. Wish I knew what was going on.

Bogie · April 7, 2007, 03:47 PM

Take a step back, and look around for Mr. Chainsaw. That always makes _me_ feel better...

(warning: If Mr. Chainsaw is actually close at hand, this can have Bad Consequences. It is recommended that one modify this error protocol to include a device that is not readily available.)

Derek Zeanah · April 7, 2007, 03:52 PM

Don't start. You're not getting 6 SMS messages every time the status changes.

stellarpod · April 7, 2007, 04:00 PM

Whew! I thought it was due to the tactless posting I made in Legal and Political...

stellarpod

Derek Zeanah · April 7, 2007, 04:07 PM

Don't know that we're up permanently, but I was able to get through to the firewall. Here are two graphs of bandwidth data: one from the last 2 months to give you a feel for averages, and one from the last 2 hours. Note the increments -- normal inbound traffic might be .3 megabits/sec. For at least a short while there, we were reporting 36? I thought I was on a 10 megabit port.

This might end up being an interesting explanation...

Gewehr98 · April 7, 2007, 04:25 PM

Isn't that what a DDoS attack looks like?

Redneckrepairs · April 7, 2007, 04:32 PM

damm lol will withold other comment for the rest of the story .

Derek Zeanah · April 7, 2007, 04:32 PM

Quote:

Isn't that what a DDoS attack looks like?

Yeah, but it doesn't seem consistent. We just went down again, but I grabbed this from the firewall while we were up:

Why just the 2 spikes, and why isn't it constant? Either my colo is doing a great filtering job, or it's generalized around the IPs that the datacenter owns. I'm still feeling a bit blind here though, to be honest. Colo still seems slammed.

Derek Zeanah · April 7, 2007, 04:56 PM

Thinking through the "is it a DDOS against THR" question a bit further, it still doesn't make much sense. Basically, I'm on a 10 mb/s pipe for all of my sites -- if someone's sending more than 10 megabits per second, there's a good chance packets are gonna get dropped. Well, maybe 20 megabits -- my firewall is plugged into a 10 Mbit port which should allow us to duplex.

So, it wouldn't be that hard to knock my sites offline. What we're seeing appears to be a more general failure -- my provider's got big pipes from 7-8 providers coming into his cages, and it would take a lot more to knock him offline. It's still possible (maybe the routers can't keep up with the demand if this is way more than normal traffic would predict), but if he was getting slammed that hard with an attack on THR or Oleg's site or whatever you'd think he would just blackhole the affected IP addresses so the data was dropped before it hit the local network.

For that not to work would mean a huge attack.

No-one hates us that much.

Gewehr98 · April 7, 2007, 05:02 PM

But that's an awfully big amount of inbound hits, regardless.

Sindawe · April 7, 2007, 05:04 PM

Quote:

No-one hates us that much

Don't forget the "mad lemur" that has been such fun for for Oleg on APS.

Maybe somebody did a router upgrade somewhere along the line that your not aware of.

sm · April 7, 2007, 05:12 PM

Quote:

No-one hates us that much.

I don't know about that, Missouri Legislature and wise cracks about Baking Soda being "restricted" might have hurt THR, I know I tried real hard ...

Steve,

Who knows where the big rubber hammer is in his IT classroom...works great on Routers and Switches...
Gives a whole new meaning to "switchport security".

Gewehr98 · April 7, 2007, 05:36 PM

"The Mad Lemur".

**tyme** · April 7, 2007, 06:40 PM

Looks like an attack to me, or horrible routing screw-up or IP conflict. I'm intermittently managing to connect, but it never serves a webpage.

Code:

64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=4 ttl=51 time=53.9 ms
64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=14 ttl=51 time=60.5 ms
64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=16 ttl=50 time=74.2 ms
64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=25 ttl=50 time=67.4 ms
64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=32 ttl=50 time=103 ms

On second thought, looking at those TTLs, maybe there's a route flapping somewhere. It's still switching between 51 and 50 as of 0008 UTC

Kestrel · April 7, 2007, 07:43 PM

Being in the IT industry myself, this is the kind of stuff that gives me gray hair...

... and a tight stomach...

Derek, I would be willing to bet you have a drawer stocked with Maalox, Advil, Tylenol and Alka-Seltzer. (I wonder how I know...)

Good luck with running it down.

Kaylee · April 7, 2007, 07:56 PM

Derek, you're my hero.

Thanks.

J.J. · April 7, 2007, 08:17 PM

I have been trying to get on APS or THR all Morning and someone missed these threads when I came to TFL.

I am confused is it an attack or not? I ask because on the THR Outage thread Derek seems to think it is an attack.

I eagerly await the forums coming online.

Derek Zeanah · April 7, 2007, 08:37 PM

Quote:

I am confused is it an attack or not?

We really don't have enough information to be able to tell.

Bogie · April 7, 2007, 08:54 PM

Guys, there are nice folks out there who just flat out HATE guns. I could see one of 'em doing an attack, and then bragging to his little buddies about putting the evil gun nuts out of business.

Then again, I'm a little paranoid.

April 7, 2007, 01:33 PM	#1
Derek Zeanah Senior Member Join Date: March 30, 2000 Location: South GA Posts: 267	THR Outage right now Sorry folks, but THR is unavailable. From what I can tell it's a network-related issue. My alarms went off this morning as we saw a 6 minute outage, then things came up fine, then they went down again, and are still down 86 minutes later. It's not the THR box, it's all of them I have at the datacenter. It looks like it's a datacenter issue, as my colo provider's web page is down, no-one's available via AIM, and their voicemail box is full. Just wanted to let folks know what's happening. I'm aware of the situation, but am sitting here waiting for a problem that's in someone else's hands... Sorry. __________________ -- Derek "An elective despotism was not the government we fought for." --Thomas Jefferson

April 7, 2007, 02:40 PM	#3
sm Senior Member Join Date: February 5, 2002 Posts: 1,819	Derek, I hope everything works out soon, I know you have a life besides tending to Networks. As always, I appreciate your time and hard work, as I do others responsible for keeping THR , TFL and Sister sites up and running. Steve __________________ Use Enough Gun TFL Alumni

April 7, 2007, 03:08 PM	#4
Larry Ashcraft Senior Member Join Date: February 22, 2001 Location: Pueblo, CO, Home of Heroes Posts: 251	Thanks, Derek. __________________ Larry Ashcraft, formerly TrophyShop

April 7, 2007, 03:30 PM	#5
Derek Zeanah Senior Member Join Date: March 30, 2000 Location: South GA Posts: 267	OK, we're back up. __________________ -- Derek "An elective despotism was not the government we fought for." --Thomas Jefferson

April 7, 2007, 03:40 PM	#6
Bogie Senior Member Join Date: June 5, 2000 Location: Job hunting on the road... Posts: 3,827	Not quite... __________________ Job hunting, but helping a friend out at www.vikingmachineusa.com - and learning the finer aspects of becoming a precision machinist. And making the world's greatest bottle openers!

April 7, 2007, 01:44 PM	#2
Redneckrepairs Senior Member Join Date: March 9, 2006 Posts: 666	Thanks for the info , Likely to be some dammed fool with a backhoe and a lazy swamper involved lol .

April 7, 2007, 03:41 PM	#7
Derek Zeanah Senior Member Join Date: March 30, 2000 Location: South GA Posts: 267	Crap. Looks like we might be going back down again. Wish I knew what was going on. __________________ -- Derek "An elective despotism was not the government we fought for." --Thomas Jefferson

April 7, 2007, 03:47 PM	#8
Bogie Senior Member Join Date: June 5, 2000 Location: Job hunting on the road... Posts: 3,827	Take a step back, and look around for Mr. Chainsaw. That always makes _me_ feel better... (warning: If Mr. Chainsaw is actually close at hand, this can have Bad Consequences. It is recommended that one modify this error protocol to include a device that is not readily available.) __________________ Job hunting, but helping a friend out at www.vikingmachineusa.com - and learning the finer aspects of becoming a precision machinist. And making the world's greatest bottle openers!

April 7, 2007, 03:52 PM	#9
Derek Zeanah Senior Member Join Date: March 30, 2000 Location: South GA Posts: 267	Don't start. You're not getting 6 SMS messages every time the status changes. __________________ -- Derek "An elective despotism was not the government we fought for." --Thomas Jefferson

April 7, 2007, 04:00 PM	#10
stellarpod Senior Member Join Date: August 18, 2001 Location: OKC,OK Posts: 263	Whew! I thought it was due to the tactless posting I made in Legal and Political... stellarpod __________________ Things are a lot more like they used to be than they are now

April 7, 2007, 04:25 PM	#12
Gewehr98 Senior Member Join Date: June 30, 2000 Location: Token Creek, WI Posts: 4,067	Just out of curiosity... Isn't that what a DDoS attack looks like? __________________ "Bother", said Pooh, as he chambered another round... Neural Misfires

April 7, 2007, 04:32 PM	#13
Redneckrepairs Senior Member Join Date: March 9, 2006 Posts: 666	damm lol will withold other comment for the rest of the story .

April 7, 2007, 04:56 PM	#15
Derek Zeanah Senior Member Join Date: March 30, 2000 Location: South GA Posts: 267	Thinking through the "is it a DDOS against THR" question a bit further, it still doesn't make much sense. Basically, I'm on a 10 mb/s pipe for all of my sites -- if someone's sending more than 10 megabits per second, there's a good chance packets are gonna get dropped. Well, maybe 20 megabits -- my firewall is plugged into a 10 Mbit port which should allow us to duplex. So, it wouldn't be that hard to knock my sites offline. What we're seeing appears to be a more general failure -- my provider's got big pipes from 7-8 providers coming into his cages, and it would take a lot more to knock him offline. It's still possible (maybe the routers can't keep up with the demand if this is way more than normal traffic would predict), but if he was getting slammed that hard with an attack on THR or Oleg's site or whatever you'd think he would just blackhole the affected IP addresses so the data was dropped before it hit the local network. For that not to work would mean a huge attack. No-one hates us that much. __________________ -- Derek "An elective despotism was not the government we fought for." --Thomas Jefferson Last edited by Derek Zeanah; April 7, 2007 at 05:25 PM. Reason: wasn't clear the first time

April 7, 2007, 05:02 PM	#16
Gewehr98 Senior Member Join Date: June 30, 2000 Location: Token Creek, WI Posts: 4,067	That's why I asked. But that's an awfully big amount of inbound hits, regardless. __________________ "Bother", said Pooh, as he chambered another round... Neural Misfires

April 7, 2007, 05:36 PM	#19
Gewehr98 Senior Member Join Date: June 30, 2000 Location: Token Creek, WI Posts: 4,067	That's the first thing that came to my mind, too. "The Mad Lemur". __________________ "Bother", said Pooh, as he chambered another round... Neural Misfires

April 7, 2007, 06:40 PM	#20
tyme Staff Join Date: October 13, 2001 Posts: 3,355	Looks like an attack to me, or horrible routing screw-up or IP conflict. I'm intermittently managing to connect, but it never serves a webpage. Code: 64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=4 ttl=51 time=53.9 ms 64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=14 ttl=51 time=60.5 ms 64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=16 ttl=50 time=74.2 ms 64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=25 ttl=50 time=67.4 ms 64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=32 ttl=50 time=103 ms On second thought, looking at those TTLs, maybe there's a route flapping somewhere. It's still switching between 51 and 50 as of 0008 UTC __________________ “The egg hatched...” “...the egg hatched... and a hundred baby spiders came out...” (blade runner) “Who are you?” “A friend. I'm here to prevent you from making a mistake.” “You have no idea what I'm doing here, friend.” “In specific terms, no, but I swore an oath to protect the world...” (continuum) “It's a goal you won't understand until later. Your job is to make sure he doesn't achieve the goal.” (bsg)

April 7, 2007, 07:43 PM	#21
Kestrel Senior Member Join Date: October 28, 2001 Posts: 363	Being in the IT industry myself, this is the kind of stuff that gives me gray hair... ... and a tight stomach... Derek, I would be willing to bet you have a drawer stocked with Maalox, Advil, Tylenol and Alka-Seltzer. (I wonder how I know...) Good luck with running it down.

April 7, 2007, 07:56 PM	#22
Kaylee Senior Member Join Date: June 14, 2000 Location: The Last Homely House Posts: 1,677	Derek, you're my hero. Thanks.

April 7, 2007, 08:17 PM	#23
J.J. Junior Member Join Date: May 19, 2004 Location: Central Texas Posts: 3	I have been trying to get on APS or THR all Morning and someone missed these threads when I came to TFL. I am confused is it an attack or not? I ask because on the THR Outage thread Derek seems to think it is an attack. I eagerly await the forums coming online.

April 7, 2007, 08:54 PM	#25
Bogie Senior Member Join Date: June 5, 2000 Location: Job hunting on the road... Posts: 3,827	Guys, there are nice folks out there who just flat out HATE guns. I could see one of 'em doing an attack, and then bragging to his little buddies about putting the evil gun nuts out of business. Then again, I'm a little paranoid. __________________ Job hunting, but helping a friend out at www.vikingmachineusa.com - and learning the finer aspects of becoming a precision machinist. And making the world's greatest bottle openers!