Go Back   Home > Forums > >
Home Forums Rules Articles diyAudio Store Blogs Gallery Wiki Register Donations FAQ Calendar Search Today's Posts Mark Forums Read

Forum Problems If there is a forum related problem please leave a message here so an Admin will see it.

2017-07-01: Server Downtime / Database Restore / Lost Posts
2017-07-01: Server Downtime / Database Restore / Lost Posts
Please consider donating to help us continue to serve you.

Ads on/off / Custom Title / More PMs / More album space / Advanced printing & mass image saving
Reply
 
Thread Tools Search this Thread
Old 1st July 2017, 12:03 PM   #1
Jason is offline Jason  Australia
:)
diyAudio Administrator
 
Jason's Avatar
 
Join Date: Oct 2000
Location: Melbourne
2017-07-01: Server Downtime / Database Restore / Lost Posts
Default 2017-07-01: Server Downtime / Database Restore / Lost Posts

Tonight our database suffered an irreparable data corruption and was unable to be repaired.

Several different methods were attempted to salvage the data, so that nobody had the hard work they put into posting lost. Unfortunately none of these methods were able to bring the database back into a stable state.

We then reviewed our options, which included two different forms of database backups, one from ~4 hours prior to the corruption, and one from ~10 hours. The more current backup was chosen (a text based SQL backup) as it was newer, and being as that it would ensure the database was restored "from the ground up" with no possible corrupted vestiges lurking in the background that might cause another issue down the line, we (slowly) restored everything from this SQL backup.

The good news is the backup appears to have worked perfectly, the bad news is we have lost ~4 hours of posts. We have since also upgraded our database server software to the latest patch level, which hopefully will eliminate whatever was the cause of the occasional database hangs we've experienced for the past few months, as well as tonight's data corruption. It's also possible that the binary database files themselves contained some issues created a long time ago, which then came to a head tonight, in which case restoring from SQL will let us start with a "clean slate". Hopefully this approach will lead us to improved stability in the future.

I will use this chance to perform a review and do a few other upgrades that have been lagging for various reasons.

This post will be updated as more news arrives and as I watch to make sure things "settle in" ok.

It was great to see the first of several layers of backups worked successfully in this emergency situation. However I do extend apologies to those affected and who lost their posts and will do everything I can to ensure this doesn't happen again.

Last edited by Jason; 1st July 2017 at 12:26 PM.
  Reply With Quote
Old 1st July 2017, 01:39 PM   #2
Duck-Twacy is offline Duck-Twacy
diyAudio Member
 
Join Date: Oct 2002
Thanks for the hard work done
__________________
Is that all there is?
  Reply With Quote
Old 1st July 2017, 02:01 PM   #3
wintermute is offline wintermute  Australia
just another
diyAudio Moderator
 
wintermute's Avatar
 
Join Date: Aug 2003
Location: Sydney
2017-07-01: Server Downtime / Database Restore / Lost Posts
Thanks Jason, four hours is not too bad really (24 hours is not uncommon in situations like this). I suspect it was not a lot of posts. I've sent you a couple of screen shots which show the front page from just after you got it back online the first time. There were only 32 posts in the three hours prior to that time.

Tony.
__________________
Any intelligence I may appear to have is purely artificial!
Some of my photos
  Reply With Quote
Old 1st July 2017, 02:02 PM   #4
gpapag is offline gpapag  Greece
diyAudio Moderator
 
gpapag's Avatar
 
Join Date: Nov 2002
Location: Athens-Greece
2017-07-01: Server Downtime / Database Restore / Lost Posts
Thank you Jason!

George
__________________
["Second Law is a bitch." - SY]
["I insist on respecting the means of the average person working in their garage/basement." -Scott Wurcer]
  Reply With Quote
Old 1st July 2017, 02:03 PM   #5
jan.didden is offline jan.didden  Europe
diyAudio Member
 
jan.didden's Avatar
 
Join Date: May 2002
Location: The great city of Turnhout, BE
Except that I lost my best post ever! Really!

Just pulling your leg. Well done Jason, you must have shed some sweat.

Jan
__________________
Cable: a potential source of trouble interconnecting two other potential sources of trouble - Erik Margan
Linear Audio pubs and articles . The SilentSwitcher now at diyaudio store SilentSwitcher. Keeping in touch with SY.
  Reply With Quote
Old 1st July 2017, 02:26 PM   #6
zman01 is offline zman01  Bangladesh
diyAudio Member
 
Join Date: Feb 2011
Location: Dhaka
2017-07-01: Server Downtime / Database Restore / Lost Posts
Jason,

Thanks for the update, and appreciate the hard work to get things up and running again!
  Reply With Quote
Old 1st July 2017, 02:37 PM   #7
wintermute is offline wintermute  Australia
just another
diyAudio Moderator
 
wintermute's Avatar
 
Join Date: Aug 2003
Location: Sydney
2017-07-01: Server Downtime / Database Restore / Lost Posts
Below is an almost complete list of what was lost minus a few posts that managed to be made in between Jason posting the original outage message and it going down again (which I didn't screen shot). I've checked and the Led drivers post in the second attachment is still there, but the one above it is not.

Tony.
Attached Images
File Type: png lost_posts1.png (174.3 KB, 198 views)
File Type: png lost_posts2.png (53.8 KB, 196 views)
__________________
Any intelligence I may appear to have is purely artificial!
Some of my photos
  Reply With Quote
Old 1st July 2017, 03:21 PM   #8
jackinnj is offline jackinnj  United States
diyAudio Member
 
jackinnj's Avatar
 
Join Date: Apr 2002
Location: Llanddewi Brefi, NJ
2017-07-01: Server Downtime / Database Restore / Lost Posts
Tnx for your hard work, jack
  Reply With Quote
Old 2nd July 2017, 03:02 AM   #9
Jason is offline Jason  Australia
:)
diyAudio Administrator
 
Jason's Avatar
 
Join Date: Oct 2000
Location: Melbourne
2017-07-01: Server Downtime / Database Restore / Lost Posts
Thanks for the kind words and glad as Tony has pointed out it happened in the wee hours of Friday night / Saturday morning for the US/EU, so posting was at a relative lull.
  Reply With Quote

Reply


2017-07-01: Server Downtime / Database Restore / Lost PostsHide this!Advertise here!
Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
2017-05-06: Server Downtime Report Jason Forum Problems 1 7th May 2017 07:23 AM
2017-03-28: Server Downtime Report Jason Forum Problems 0 27th March 2017 09:58 PM
2017-02-20: Server Downtime Report Jason Forum Problems 2 19th February 2017 11:49 PM
2017-02-02: Server Downtime Report Jason Forum Problems 0 1st February 2017 01:08 PM
2016-11-14: Server Downtime Report Jason Forum Problems 3 15th November 2016 10:57 AM


New To Site? Need Help?

All times are GMT. The time now is 09:50 AM.


Search Engine Optimisation provided by DragonByte SEO (Pro) - vBulletin Mods & Addons Copyright © 2018 DragonByte Technologies Ltd.
Resources saved on this page: MySQL 15.00%
vBulletin Optimisation provided by vB Optimise (Pro) - vBulletin Mods & Addons Copyright © 2018 DragonByte Technologies Ltd.
Copyright ©1999-2018 diyAudio
Wiki