April 7, 2007, 01:33 PM | #1 |
Senior Member
Join Date: March 30, 2000
Location: South GA
Posts: 267
|
THR Outage right now
Sorry folks, but THR is unavailable.
From what I can tell it's a network-related issue. My alarms went off this morning as we saw a 6 minute outage, then things came up fine, then they went down again, and are still down 86 minutes later. It's not the THR box, it's all of them I have at the datacenter. It looks like it's a datacenter issue, as my colo provider's web page is down, no-one's available via AIM, and their voicemail box is full. Just wanted to let folks know what's happening. I'm aware of the situation, but am sitting here waiting for a problem that's in someone else's hands... Sorry.
__________________
-- Derek "An elective despotism was not the government we fought for." --Thomas Jefferson |
April 7, 2007, 01:44 PM | #2 |
Senior Member
Join Date: March 9, 2006
Posts: 666
|
Thanks for the info , Likely to be some dammed fool with a backhoe and a lazy swamper involved lol .
|
April 7, 2007, 02:40 PM | #3 |
Senior Member
Join Date: February 5, 2002
Posts: 1,819
|
Derek,
I hope everything works out soon, I know you have a life besides tending to Networks. As always, I appreciate your time and hard work, as I do others responsible for keeping THR , TFL and Sister sites up and running. Steve
__________________
Use Enough Gun TFL Alumni |
April 7, 2007, 03:08 PM | #4 |
Senior Member
Join Date: February 22, 2001
Location: Pueblo, CO, Home of Heroes
Posts: 251
|
Thanks, Derek.
__________________
Larry Ashcraft, formerly TrophyShop |
April 7, 2007, 03:30 PM | #5 |
Senior Member
Join Date: March 30, 2000
Location: South GA
Posts: 267
|
OK, we're back up.
__________________
-- Derek "An elective despotism was not the government we fought for." --Thomas Jefferson |
April 7, 2007, 03:40 PM | #6 |
Senior Member
Join Date: June 5, 2000
Location: Job hunting on the road...
Posts: 3,827
|
Not quite...
__________________
Job hunting, but helping a friend out at www.vikingmachineusa.com - and learning the finer aspects of becoming a precision machinist. And making the world's greatest bottle openers! |
April 7, 2007, 03:41 PM | #7 |
Senior Member
Join Date: March 30, 2000
Location: South GA
Posts: 267
|
Crap. Looks like we might be going back down again. Wish I knew what was going on.
__________________
-- Derek "An elective despotism was not the government we fought for." --Thomas Jefferson |
April 7, 2007, 03:47 PM | #8 |
Senior Member
Join Date: June 5, 2000
Location: Job hunting on the road...
Posts: 3,827
|
Take a step back, and look around for Mr. Chainsaw. That always makes _me_ feel better...
(warning: If Mr. Chainsaw is actually close at hand, this can have Bad Consequences. It is recommended that one modify this error protocol to include a device that is not readily available.)
__________________
Job hunting, but helping a friend out at www.vikingmachineusa.com - and learning the finer aspects of becoming a precision machinist. And making the world's greatest bottle openers! |
April 7, 2007, 03:52 PM | #9 |
Senior Member
Join Date: March 30, 2000
Location: South GA
Posts: 267
|
Don't start. You're not getting 6 SMS messages every time the status changes.
__________________
-- Derek "An elective despotism was not the government we fought for." --Thomas Jefferson |
April 7, 2007, 04:00 PM | #10 |
Senior Member
Join Date: August 18, 2001
Location: OKC,OK
Posts: 263
|
Whew! I thought it was due to the tactless posting I made in Legal and Political...
stellarpod
__________________
Things are a lot more like they used to be than they are now |
April 7, 2007, 04:07 PM | #11 |
Senior Member
Join Date: March 30, 2000
Location: South GA
Posts: 267
|
Data
Don't know that we're up permanently, but I was able to get through to the firewall. Here are two graphs of bandwidth data: one from the last 2 months to give you a feel for averages, and one from the last 2 hours. Note the increments -- normal inbound traffic might be .3 megabits/sec. For at least a short while there, we were reporting 36? I thought I was on a 10 megabit port.
This might end up being an interesting explanation...
__________________
-- Derek "An elective despotism was not the government we fought for." --Thomas Jefferson |
April 7, 2007, 04:25 PM | #12 |
Senior Member
Join Date: June 30, 2000
Location: Token Creek, WI
Posts: 4,067
|
Just out of curiosity...
Isn't that what a DDoS attack looks like?
|
April 7, 2007, 04:32 PM | #13 |
Senior Member
Join Date: March 9, 2006
Posts: 666
|
damm lol will withold other comment for the rest of the story .
|
April 7, 2007, 04:32 PM | #14 | |
Senior Member
Join Date: March 30, 2000
Location: South GA
Posts: 267
|
Quote:
Why just the 2 spikes, and why isn't it constant? Either my colo is doing a great filtering job, or it's generalized around the IPs that the datacenter owns. I'm still feeling a bit blind here though, to be honest. Colo still seems slammed.
__________________
-- Derek "An elective despotism was not the government we fought for." --Thomas Jefferson |
|
April 7, 2007, 04:56 PM | #15 |
Senior Member
Join Date: March 30, 2000
Location: South GA
Posts: 267
|
Thinking through the "is it a DDOS against THR" question a bit further, it still doesn't make much sense. Basically, I'm on a 10 mb/s pipe for all of my sites -- if someone's sending more than 10 megabits per second, there's a good chance packets are gonna get dropped. Well, maybe 20 megabits -- my firewall is plugged into a 10 Mbit port which should allow us to duplex.
So, it wouldn't be that hard to knock my sites offline. What we're seeing appears to be a more general failure -- my provider's got big pipes from 7-8 providers coming into his cages, and it would take a lot more to knock him offline. It's still possible (maybe the routers can't keep up with the demand if this is way more than normal traffic would predict), but if he was getting slammed that hard with an attack on THR or Oleg's site or whatever you'd think he would just blackhole the affected IP addresses so the data was dropped before it hit the local network. For that not to work would mean a huge attack. No-one hates us that much.
__________________
-- Derek "An elective despotism was not the government we fought for." --Thomas Jefferson Last edited by Derek Zeanah; April 7, 2007 at 05:25 PM. Reason: wasn't clear the first time |
April 7, 2007, 05:02 PM | #16 |
Senior Member
Join Date: June 30, 2000
Location: Token Creek, WI
Posts: 4,067
|
That's why I asked.
But that's an awfully big amount of inbound hits, regardless.
|
April 7, 2007, 05:04 PM | #17 | |
Member
Join Date: November 5, 2002
Location: Just outside of the PRoB
Posts: 56
|
Quote:
Maybe somebody did a router upgrade somewhere along the line that your not aware of.
__________________
When you dream, there are no rules. People can fly, anything can happen... |
|
April 7, 2007, 05:12 PM | #18 | |
Senior Member
Join Date: February 5, 2002
Posts: 1,819
|
Quote:
Steve, Who knows where the big rubber hammer is in his IT classroom...works great on Routers and Switches... Gives a whole new meaning to "switchport security".
__________________
Use Enough Gun TFL Alumni |
|
April 7, 2007, 05:36 PM | #19 |
Senior Member
Join Date: June 30, 2000
Location: Token Creek, WI
Posts: 4,067
|
That's the first thing that came to my mind, too.
"The Mad Lemur".
|
April 7, 2007, 06:40 PM | #20 |
Staff
Join Date: October 13, 2001
Posts: 3,355
|
Looks like an attack to me, or horrible routing screw-up or IP conflict. I'm intermittently managing to connect, but it never serves a webpage.
Code:
64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=4 ttl=51 time=53.9 ms 64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=14 ttl=51 time=60.5 ms 64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=16 ttl=50 time=74.2 ms 64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=25 ttl=50 time=67.4 ms 64 bytes from wellbuiltnetworks.net (209.51.144.70): icmp_seq=32 ttl=50 time=103 ms
__________________
“The egg hatched...” “...the egg hatched... and a hundred baby spiders came out...” (blade runner) “Who are you?” “A friend. I'm here to prevent you from making a mistake.” “You have no idea what I'm doing here, friend.” “In specific terms, no, but I swore an oath to protect the world...” (continuum) “It's a goal you won't understand until later. Your job is to make sure he doesn't achieve the goal.” (bsg) |
April 7, 2007, 07:43 PM | #21 |
Senior Member
Join Date: October 28, 2001
Posts: 363
|
Being in the IT industry myself, this is the kind of stuff that gives me gray hair...
... and a tight stomach... Derek, I would be willing to bet you have a drawer stocked with Maalox, Advil, Tylenol and Alka-Seltzer. (I wonder how I know...) Good luck with running it down. |
April 7, 2007, 07:56 PM | #22 |
Senior Member
Join Date: June 14, 2000
Location: The Last Homely House
Posts: 1,677
|
Derek, you're my hero.
Thanks. |
April 7, 2007, 08:17 PM | #23 |
Junior Member
Join Date: May 19, 2004
Location: Central Texas
Posts: 3
|
I have been trying to get on APS or THR all Morning and someone missed these threads when I came to TFL.
I am confused is it an attack or not? I ask because on the THR Outage thread Derek seems to think it is an attack. I eagerly await the forums coming online. |
April 7, 2007, 08:37 PM | #24 | |
Senior Member
Join Date: March 30, 2000
Location: South GA
Posts: 267
|
Quote:
__________________
-- Derek "An elective despotism was not the government we fought for." --Thomas Jefferson |
|
April 7, 2007, 08:54 PM | #25 |
Senior Member
Join Date: June 5, 2000
Location: Job hunting on the road...
Posts: 3,827
|
Guys, there are nice folks out there who just flat out HATE guns. I could see one of 'em doing an attack, and then bragging to his little buddies about putting the evil gun nuts out of business.
Then again, I'm a little paranoid.
__________________
Job hunting, but helping a friend out at www.vikingmachineusa.com - and learning the finer aspects of becoming a precision machinist. And making the world's greatest bottle openers! |
|
|