Unexpected outage

Status
This old topic is closed. If you want to reopen this topic, contact a moderator using the "Report Post" button.
Switches things on and off again
Joined 2000
Paid Member
The hosting company's monitoring service detected that port 80 was not responding on the server. I'm currently trying to find out if that was the case or a false alarm. Things appeared to be going swimmingly up until a point and then everything dropped to zero. It could well have been a teething problem in the new setup. I'm going to be investigating that and monitoring things closely as time goes on.

When that happened, one of their techs logged in to save the day, but didn't follow their standard operating procedures which included instructions on how to properly restart the web server (this was documented 2 years ago when we last had an issue). A sequence of other non SOP events unfolded from there, resulting in the site being down for 4 hours. The host has taken full responsibility for their failure to following SOP and promises this won't happen again in the future. I'll be taking proactive steps to "future proof" things so even if SOP is not followed in the future, it will be hard to bork it up again.

Sorry for the downtime.

Other than this very consequential "glitch", the server is coasting along at record low load averages and serving up pages faster than ever. Hopefully it stays that way, I'll be baby sitting things in the meantime...
 
I live on a farm and we use electric fencing to keep the cattle on the correct side. About 10kV but very low joules.
Works really well, one individual tests the envelope and gets a zap and the rest of the herd learns from the squeal and respects all fencing for the next few weeks.

I have thought about introducing similar "learning" tools at work but the health and safety people tell me there are a number of issues and will not let me. Wonder if the issues apply to contractors too?
 
I thoght you did that on purpose Jason to try and correct the slowness some ppl are gertting,i didnt know it has unplanned!

Another VBB site im on is also down .. I tried to goto it just now and I got a database error message .. I emailed the admin and let him know!


Im glad ya got things corrected :) -- This is a good site!
 
Switches things on and off again
Joined 2000
Paid Member
Thanks guys...

Today, also while I was sleeping, either the web server or database server went down. This time, the techs at the hosting company detected the problem, logged in, and restarted everything perfectly - fantastic.

So now at least we know when things do need restarting and I am asleep / away, they will be restarted correctly.

As for why the server went down in the first place - I'm not sure. Certainly up until that point, things were going very well. I'm pleased to announce that other than these periods of undetermined total downtime, the server load is at record lows and pages being served faster than ever which is great news.
 
Just another Moderator
Joined 2003
Paid Member
Hi Jason, Is it just the site that is going down or the whole server? I've been in situations in the past where major changes were made, and the problem turned out to be completely unrelated and mere co-incidence, makes troubleshooting hard as you are always looking at the recent changes for the cause!!

Tony.
 
Switches things on and off again
Joined 2000
Paid Member
I've now upgraded mod_pagespeed - we were using quite an old version before and are now on the current version. It might optimize some things differently out of the box so if you experience any strange behavior (eg: images not showing up, javascript not working, etc) please let me know.
 
Switches things on and off again
Joined 2000
Paid Member
I believe I just found the culprit - a badly written query - both nights the system crashed at exactly the same time which is when this query is being automatically run (actually it just locked up the database, causing a massive backlog of requests). I'm 99% sure that was the problem both yesterday and today.
 
Official Court Jester
Joined 2003
Paid Member
well - it's not exactly outage but more like other glitch , repetitive in last 48h

see attachment , just checked - same from FF , GC and IE

I believe not dependable of my ISP , because I experienced the same in last few days , using two completely independent ISPs

disclaimer - I'm not complaining ...... just letting you know ....... ;)

edit - everything is OK , after posting it
 

Attachments

  • screen-DiyA.jpg
    screen-DiyA.jpg
    79.5 KB · Views: 79
Status
This old topic is closed. If you want to reopen this topic, contact a moderator using the "Report Post" button.