diyAudio

diyAudio (http://www.diyaudio.com/forums/)
-   Forum Problems (http://www.diyaudio.com/forums/forum-problems/)
-   -   Downtime today (http://www.diyaudio.com/forums/forum-problems/191762-downtime-today.html)

Jason 30th June 2011 06:08 AM

Downtime today
 
My sincere apologies for the 9 hours of downtime today. When I "improved" the backups a few months ago I switched the write location to the main partition instead of the backup partition and today during the "PST/EST dawn backups" the main drive filled up which linux really doesn't like happening, and it killed the webserver. I was just going to sleep at that time and didn't get notification until I woke up, hence the 8 hour delay in my response.

For the record, our server is monitored and managed by our host, and they did detect the problem immediately, logged in and attempted to resolve it, but weren't familiar enough with the configuration to be able to resolve the problem. We're working with them to ensure that in the future they have enough information on our (rather hardy/secure/locked down) configuration to be able to fix this kind of problem without additional assistance from myself in the future.

As this is the kind of model we're shooting for now (if stuff breaks it gets fixed immediately by the server management company), I guess this was a good real world test, and in this case it failed and again my sincere apologies for the interruption to your service. We'll do what it takes to ensure this can't happen again in the future.

Jason

geraldfryjr 30th June 2011 06:55 AM

Thanks Jason for letting us know what happened which is more than what other services would do!

Ironicaly I was just on and then went off to do a virus scan and then came right back and then nothing ,so ,I thought that my scanner had messed up something as it has happend to me before.

But by using one of my backup computers I figured that it must have been something with your server,Shooo, WoW I'm glad that you are back online !:)

jer

gpapag 30th June 2011 06:58 AM

I was a bit worried :scratch: as to what happened.
I am illiterate on such stuff (what you wrote sound Greek :D to me ), but I don’t think it is a bad idea to deliberately program for some regular downtimes in the future, to give us a chance to do some diyaudio real work off-line. :rolleyes:

Jason, I am glad that You are back and catering for this wonderful site. :)
I hope You are doing well.

Regards
George

PS. Against the “common wisdom” here, a few hours sleep each day is a normal activity. :eek:

Jason 30th June 2011 11:08 AM

I just sorted out a few more problems with the revised backup routines which just caused an extended maintenance period, things should be back to normal now.

ptempel 30th June 2011 12:45 PM

+1 on geraldfryjr's comment. Good to be informed but its ok to have a glitch once in a while. Unlike audio and electronics, I have some experience with being a Unix admin (only over 12 years). Let me know if I can help with anything. Getting the operations team on board will require some documentation. Maybe they have a wiki site for that? That's what I used for the last three jobs to document proceedures and changes. If for nothing else than to remind me of "how it was done." ;)

503Timber 30th June 2011 12:52 PM

Glad we're back......wish I had a dollar for every time I tried to log on yesterday......I could get a few projects done!

c2cthomas 30th June 2011 12:57 PM

The down time proved to me that I'm waaaaaaay to addicted to diyAudio!!! Man - I wuz Jones'n for my diyAudio!!!! Maybe I should get a life......:rolleyes:

Thanks Jason for getting things put right and letting us know what is going on.:D

geraldfryjr 30th June 2011 02:41 PM

I second that c2cthomas I am on here more helping others than getting my own projects done!
But thats okay it is what I enjoy doing.
Thanks DIYAudio for everything that you stand for!


jer

Jason 1st July 2011 04:04 AM

Thanks ptempel for the offer.

Our host now has all the information and our special requirements in case of an emergency, backups are now back on their own partition (rolleyes), I'm getting texts and phone calls if disk space ever gets unplentiful, or the site becomes unresponsive, and our senior admins on here have a direct line and verbal passwords to our host in case of extended drama. It's been a long time since we last had any downtime, hopefully it will be another year or two until the next problem :)

ThorstenL 1st July 2011 08:58 AM

Hi Jason,

Quote:

Originally Posted by Jason (Post 2623703)
My sincere apologies for the 9 hours of downtime today.

Don't worry about it. It's not like DIYA is "business critical".

It's just a "downtime" thing for those who post here.

Occasional breaks are probably a good thing.

You should probably schedule 24 Hour outages every two weeks or so... ;)

Appreciate the hard work to keep everything running and getting things back up to scratch when Murphy strikes...

Ciao T

PS, I remember once nearly 2 Weeks outage on a business critical Mini system, it was the Payroll... :(


All times are GMT. The time now is 10:08 AM.


vBulletin Optimisation provided by vB Optimise (Pro) - vBulletin Mods & Addons Copyright © 2014 DragonByte Technologies Ltd.
Copyright ©1999-2014 diyAudio


Content Relevant URLs by vBSEO 3.3.2