Hipointtalk.net
http://hipointtalk.net/

Today's outage.
http://hipointtalk.net/viewtopic.php?f=1&t=2338
Page 1 of 2

Author:  995 [ Sun Feb 01, 2009 3:22 pm ]
Post subject:  Today's outage.

Alright, guys, I'm sure you noticed the "ACCOUNT SUSPENDED" message today.

Let me explain what went on...

Our provider, had apparently been observing that "your account regularly causes a load on our servers due to MySql processes". (This has to do with an upcoming feature that I had really wanted to add to this site, that we were working on)

Now, instead of e-mailing me, letting me know, so I could fix it, they chose to wait, until someone got upset at it, and just shut us down.

I've just completed a full site backup, while we look into alternate solutions

Bear with us folks, this might get a little bumpy.

Author:  mcdonl [ Sun Feb 01, 2009 3:28 pm ]
Post subject:  Re: Today's outage.

We are with you. Don't be afraid to ask for a few bucks either.

We will give.

Leroy

Author:  Pudds995 [ Sun Feb 01, 2009 3:30 pm ]
Post subject:  Re: Today's outage.

+1 to mcdonl :yes2:

Author:  995 [ Sun Feb 01, 2009 3:39 pm ]
Post subject:  Re: Today's outage.

Turned out not to be the new feature we were testing btw, but, this site itself malfunctioning, and putting down massive queries.

So, for the time being, I've put a band-aid on a gunshot wound, and wrote a program to "kill" any MySQL queries that have run for more than 60 seconds. This shouldn't affect anyone, as the ones I saw running long (this takes up a LOT of CPU time), were running for 10-15 mins... I don't think anyone would wait that long for a page to load :P

And, to be fair, it got to the point where we were nearly disabling their server, which, really wasn't fair to the other users.

Author:  waltham41 [ Sun Feb 01, 2009 3:53 pm ]
Post subject:  Re: Today's outage.

at least we are back up in just a couple of hours. Thats the important part.

Author:  Sunset [ Sun Feb 01, 2009 3:56 pm ]
Post subject:  Re: Today's outage.

Rats, take a nap and you miss all the fun! Glad everything is cooking again. +1 on the $'s also.

God Bless,

Ed

Author:  995 [ Sun Feb 01, 2009 4:00 pm ]
Post subject:  Re: Today's outage.

To those that are techies, I'm looking into using some sort of distributed SQUID solution right now, maybe with a cold backup site as well :P

Author:  Wildman [ Sun Feb 01, 2009 4:34 pm ]
Post subject:  Re: Today's outage.

Well I'll be dipped down deep daily in dog dung, I missed all the "fun". Sounds like the right guys are on the job though. It's like the old saying says you don't know what you have until it is gone.

Author:  Bushman98 [ Sun Feb 01, 2009 4:46 pm ]
Post subject:  Re: Today's outage.

995 wrote:
To those that are techies, I'm looking into using some sort of distributed SQUID solution right now, maybe with a cold backup site as well :P


I'm looking for someone to translate that into English. :wink:

Author:  woodyubet [ Sun Feb 01, 2009 4:51 pm ]
Post subject:  Re: Today's outage.

Do whatever you need to do. We got your back.

Author:  Wildman [ Sun Feb 01, 2009 5:00 pm ]
Post subject:  Re: Today's outage.

I'm able to send some $$ if need be. Please holler!

Author:  erc [ Sun Feb 01, 2009 5:20 pm ]
Post subject:  Re: Today's outage.

LOL, We'll be fine folks, no different than losing power with no backup.
We are working on a solution for the future, trust me.

Having said that, all who read this need to view the emergency announcement, as I'm about to update it, here:
http://hipointtalk.com/viewtopic.php?f=1&t=786

This was a minor glitch that could have been much worse. Thankfully, a couple Admins. happened to be here at the time it happened, and were able to contact 995 about it.
Some settings have been changed so that mail to staff will go to the gmail account, in an effort to relieve the flood that happened today to 995. This change will allow us to better deal with any problem, versus answering mail about a WTF? situation.
Please write this down, or otherwise, save it. In the future, please use it. We will respond as time allows while trying to solve the issue.

Author:  pills [ Sun Feb 01, 2009 6:21 pm ]
Post subject:  Re: Today's outage.

I notice a lot of bot activity on the site.

Any correlation?

Author:  995 [ Sun Feb 01, 2009 6:32 pm ]
Post subject:  Re: Today's outage.

pills wrote:
I notice a lot of bot activity on the site.

Any correlation?


Nah, bug in bb3 was causing LOOONG (1500+s) cpu-intensive queries.

A server of those specs should be able to easily host us a few times over.

Author:  redhawk56 [ Sun Feb 01, 2009 7:40 pm ]
Post subject:  Re: Today's outage.

Glad we are back.

Page 1 of 2 All times are UTC - 5 hours
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/