The Firing Line Forums

Go Back   The Firing Line Forums > Forum Support > Site Questions and Tech Support (NO FIREARMS QUESTIONS)

Closed Thread
 
Thread Tools
Old April 7, 2007, 01:33 PM   #1
Derek Zeanah
Senior Member
 
Join Date: March 30, 2000
Location: South GA
Posts: 267
THR Outage right now

Sorry folks, but THR is unavailable.

From what I can tell it's a network-related issue. My alarms went off this morning as we saw a 6 minute outage, then things came up fine, then they went down again, and are still down 86 minutes later.

It's not the THR box, it's all of them I have at the datacenter. It looks like it's a datacenter issue, as my colo provider's web page is down, no-one's available via AIM, and their voicemail box is full.

Just wanted to let folks know what's happening. I'm aware of the situation, but am sitting here waiting for a problem that's in someone else's hands...

Sorry.
__________________
-- Derek

"An elective despotism was not the government we fought for."
--Thomas Jefferson
Derek Zeanah is offline  
Old April 7, 2007, 01:44 PM   #2
Redneckrepairs
Senior Member
 
Join Date: March 9, 2006
Posts: 658
Thanks for the info , Likely to be some dammed fool with a backhoe and a lazy swamper involved lol .
Redneckrepairs is offline  
Old April 7, 2007, 02:40 PM   #3
sm
Senior Member
 
Join Date: February 5, 2002
Posts: 1,810
Derek,

I hope everything works out soon, I know you have a life besides tending to Networks.
As always, I appreciate your time and hard work, as I do others responsible for keeping THR , TFL and Sister sites up and running.

Steve
__________________
Use Enough Gun
TFL Alumni
sm is offline  
Old April 7, 2007, 03:08 PM   #4
Larry Ashcraft
Senior Member
 
Join Date: February 22, 2001
Location: Pueblo, CO, Home of Heroes
Posts: 250
Thanks, Derek.
__________________
Larry Ashcraft, formerly TrophyShop
Larry Ashcraft is offline  
Old April 7, 2007, 03:30 PM   #5
Derek Zeanah
Senior Member
 
Join Date: March 30, 2000
Location: South GA
Posts: 267
OK, we're back up.
__________________
-- Derek

"An elective despotism was not the government we fought for."
--Thomas Jefferson
Derek Zeanah is offline  
Old April 7, 2007, 03:40 PM   #6
Bogie
Senior Member
 
Join Date: June 5, 2000
Location: Job hunting on the road...
Posts: 3,827
Not quite...
__________________
Job hunting, but helping a friend out at www.vikingmachineusa.com - and learning the finer aspects of becoming a precision machinist.

And making the world's greatest bottle openers!
Bogie is offline  
Old April 7, 2007, 03:41 PM   #7
Derek Zeanah
Senior Member
 
Join Date: March 30, 2000
Location: South GA
Posts: 267
Crap. Looks like we might be going back down again. Wish I knew what was going on.
__________________
-- Derek

"An elective despotism was not the government we fought for."
--Thomas Jefferson
Derek Zeanah is offline  
Old April 7, 2007, 03:47 PM   #8
Bogie
Senior Member
 
Join Date: June 5, 2000
Location: Job hunting on the road...
Posts: 3,827
Take a step back, and look around for Mr. Chainsaw. That always makes _me_ feel better...

(warning: If Mr. Chainsaw is actually close at hand, this can have Bad Consequences. It is recommended that one modify this error protocol to include a device that is not readily available.)
__________________
Job hunting, but helping a friend out at www.vikingmachineusa.com - and learning the finer aspects of becoming a precision machinist.

And making the world's greatest bottle openers!
Bogie is offline  
Old April 7, 2007, 03:52 PM   #9
Derek Zeanah
Senior Member
 
Join Date: March 30, 2000
Location: South GA
Posts: 267
Don't start. You're not getting 6 SMS messages every time the status changes.
__________________
-- Derek

"An elective despotism was not the government we fought for."
--Thomas Jefferson
Derek Zeanah is offline  
Old April 7, 2007, 04:00 PM   #10
stellarpod
Senior Member
 
Join Date: August 18, 2001
Location: OKC,OK
Posts: 263
Whew! I thought it was due to the tactless posting I made in Legal and Political...



stellarpod
__________________
Things are a lot more like they used to be than they are now
stellarpod is offline  
Old April 7, 2007, 04:07 PM   #11
Derek Zeanah
Senior Member
 
Join Date: March 30, 2000
Location: South GA
Posts: 267
Data

Don't know that we're up permanently, but I was able to get through to the firewall. Here are two graphs of bandwidth data: one from the last 2 months to give you a feel for averages, and one from the last 2 hours. Note the increments -- normal inbound traffic might be .3 megabits/sec. For at least a short while there, we were reporting 36? I thought I was on a 10 megabit port.

This might end up being an interesting explanation...



Attached Images
File Type: gif 2-month.gif (18.1 KB, 584 views)
File Type: gif 2-hour.gif (13.0 KB, 588 views)
__________________
-- Derek

"An elective despotism was not the government we fought for."
--Thomas Jefferson
Derek Zeanah is offline  
Old April 7, 2007, 04:25 PM   #12
Gewehr98
Senior Member
 
Join Date: June 30, 2000
Location: Token Creek, WI
Posts: 4,068
Just out of curiosity...

Isn't that what a DDoS attack looks like?
__________________
"Bother", said Pooh, as he chambered another round...

Neural Misfires
Gewehr98 is offline  
Old April 7, 2007, 04:32 PM   #13
Redneckrepairs
Senior Member
 
Join Date: March 9, 2006
Posts: 658
damm lol will withold other comment for the rest of the story .
Redneckrepairs is offline  
Old April 7, 2007, 04:32 PM   #14
Derek Zeanah
Senior Member
 
Join Date: March 30, 2000
Location: South GA
Posts: 267
Quote:
Isn't that what a DDoS attack looks like?
Yeah, but it doesn't seem consistent. We just went down again, but I grabbed this from the firewall while we were up:



Why just the 2 spikes, and why isn't it constant? Either my colo is doing a great filtering job, or it's generalized around the IPs that the datacenter owns. I'm still feeling a bit blind here though, to be honest. Colo still seems slammed.
Attached Images
File Type: gif spike.gif (17.5 KB, 583 views)
__________________
-- Derek

"An elective despotism was not the government we fought for."
--Thomas Jefferson
Derek Zeanah is offline  
Old April 7, 2007, 04:56 PM   #15
Derek Zeanah
Senior Member
 
Join Date: March 30, 2000
Location: South GA
Posts: 267
Thinking through the "is it a DDOS against THR" question a bit further, it still doesn't make much sense. Basically, I'm on a 10 mb/s pipe for all of my sites -- if someone's sending more than 10 megabits per second, there's a good chance packets are gonna get dropped. Well, maybe 20 megabits -- my firewall is plugged into a 10 Mbit port which should allow us to duplex.

So, it wouldn't be that hard to knock my sites offline. What we're seeing appears to be a more general failure -- my provider's got big pipes from 7-8 providers coming into his cages, and it would take a lot more to knock him offline. It's still possible (maybe the routers can't keep up with the demand if this is way more than normal traffic would predict), but if he was getting slammed that hard with an attack on THR or Oleg's site or whatever you'd think he would just blackhole the affected IP addresses so the data was dropped before it hit the local network.

For that not to work would mean a huge attack.

No-one hates us that much.
__________________
-- Derek

"An elective despotism was not the government we fought for."
--Thomas Jefferson

Last edited by Derek Zeanah; April 7, 2007 at 05:25 PM. Reason: wasn't clear the first time
Derek Zeanah is offline  
Old April 7, 2007, 05:02 PM   #16
Gewehr98
Senior Member
 
Join Date: June 30, 2000
Location: Token Creek, WI
Posts: 4,068
That's why I asked.

But that's an awfully big amount of inbound hits, regardless.
__________________
"Bother", said Pooh, as he chambered another round...

Neural Misfires
Gewehr98 is offline  
Old April 7, 2007, 05:04 PM   #17
Sindawe
Member
 
Join Date: November 5, 2002
Location: Just outside of the PRoB
Posts: 56
Quote:
No-one hates us that much
Don't forget the "mad lemur" that has been such fun for for Oleg on APS.

Maybe somebody did a router upgrade somewhere along the line that your not aware of.
__________________
When you dream, there are no rules.
People can fly, anything can happen...
Sindawe is offline  
Old April 7, 2007, 05:12 PM   #18
sm
Senior Member
 
Join Date: February 5, 2002
Posts: 1,810
Quote:
No-one hates us that much.
I don't know about that, Missouri Legislature and wise cracks about Baking Soda being "restricted" might have hurt THR, I know I tried real hard ...

Steve,

Who knows where the big rubber hammer is in his IT classroom...works great on Routers and Switches...
Gives a whole new meaning to "switchport security".
__________________
Use Enough Gun
TFL Alumni
sm is offline  
Old April 7, 2007, 05:36 PM   #19
Gewehr98
Senior Member
 
Join Date: June 30, 2000
Location: Token Creek, WI
Posts: 4,068
That's the first thing that came to my mind, too.

"The Mad Lemur".
__________________
"Bother", said Pooh, as he chambered another round...

Neural Misfires
Gewehr98 is offline  
Old April 7, 2007, 06:40 PM   #20
tyme
Staff
 
Join Date: October 13, 2001
Posts: 3,185
Looks like an attack to me, or horrible routing screw-up or IP conflict. I'm intermittently managing to connect, but it never serves a webpage.

Code:
64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=4 ttl=51 time=53.9 ms
64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=14 ttl=51 time=60.5 ms
64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=16 ttl=50 time=74.2 ms
64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=25 ttl=50 time=67.4 ms
64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=32 ttl=50 time=103 ms
On second thought, looking at those TTLs, maybe there's a route flapping somewhere. It's still switching between 51 and 50 as of 0008 UTC
__________________
“The egg hatched...” “...the egg hatched... and a hundred baby spiders came out...” (blade runner)
“Who are you?” “A friend. I'm here to prevent you from making a mistake.” “You have no idea what I'm doing here, friend.” “In specific terms, no, but I swore an oath to protect the world...” (continuum)
“It's a goal you won't understand until later. Your job is to make sure he doesn't achieve the goal.” (bsg)
tyme is offline  
Old April 7, 2007, 07:43 PM   #21
Kestrel
Senior Member
 
Join Date: October 28, 2001
Posts: 363
Being in the IT industry myself, this is the kind of stuff that gives me gray hair...

... and a tight stomach...

Derek, I would be willing to bet you have a drawer stocked with Maalox, Advil, Tylenol and Alka-Seltzer. (I wonder how I know...)

Good luck with running it down.
Kestrel is offline  
Old April 7, 2007, 07:56 PM   #22
Kaylee
Senior Member
 
Join Date: June 14, 2000
Location: The Last Homely House
Posts: 1,670
Derek, you're my hero.

Thanks.
Kaylee is offline  
Old April 7, 2007, 08:17 PM   #23
J.J.
Junior Member
 
Join Date: May 19, 2004
Location: Central Texas
Posts: 3
I have been trying to get on APS or THR all Morning and someone missed these threads when I came to TFL.


I am confused is it an attack or not? I ask because on the THR Outage thread Derek seems to think it is an attack.

I eagerly await the forums coming online.
J.J. is offline  
Old April 7, 2007, 08:37 PM   #24
Derek Zeanah
Senior Member
 
Join Date: March 30, 2000
Location: South GA
Posts: 267
Quote:
I am confused is it an attack or not?
We really don't have enough information to be able to tell.
__________________
-- Derek

"An elective despotism was not the government we fought for."
--Thomas Jefferson
Derek Zeanah is offline  
Old April 7, 2007, 08:54 PM   #25
Bogie
Senior Member
 
Join Date: June 5, 2000
Location: Job hunting on the road...
Posts: 3,827
Guys, there are nice folks out there who just flat out HATE guns. I could see one of 'em doing an attack, and then bragging to his little buddies about putting the evil gun nuts out of business.

Then again, I'm a little paranoid.
__________________
Job hunting, but helping a friend out at www.vikingmachineusa.com - and learning the finer aspects of becoming a precision machinist.

And making the world's greatest bottle openers!
Bogie is offline  
Closed Thread

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -5. The time now is 05:38 PM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.
This site and contents, including all posts, Copyright © 1998-2014 S.W.A.T. Magazine
Copyright Complaints: Please direct DMCA Takedown Notices to the registered agent: thefiringline.com
Contact Us
Page generated in 0.12747 seconds with 10 queries