OVH founder says UPS fixed up day before blaze is early suspect as source of data centre destruction
"Cloud company offers slightly confused roadmap to restoration"
I have watched halon deploy from camera views we had up on screens though it was not a real fire. I forget why they tripped. Just interesting to see the cloud and be glad I was not there. I figure I would have been able to get out before I ran out of oxygen.
I have also seen a datacenter lose power and the transfer switch failed to switch to the generators that had managed to start. No fun to watch the two UPS devices slowly run out of battery for no good reason.
This sort of thing happens even when you really try to protect the servers.
I have also seen a datacenter lose power and the transfer switch failed to switch to the generators that had managed to start. No fun to watch the two UPS devices slowly run out of battery for no good reason.
This sort of thing happens even when you really try to protect the servers.
I can also say I have swapped enough tapes in and out of a tape library to consider it a form of exercise. It was usually 80 tapes in and out each week at some times.
Really, get the tapes offsite? And confirm restores work.
I wish I knew the required steps ahead of time for a Windows domain controller. Starting the restore job is one thing. Starting the restore and having it work is another.
Someone whacked an OU labeled "Employees" during a change of some sort. Did not bother to check if that contained all of the employee accounts. Was the third strike for him.
Really, get the tapes offsite? And confirm restores work.
I wish I knew the required steps ahead of time for a Windows domain controller. Starting the restore job is one thing. Starting the restore and having it work is another.
Someone whacked an OU labeled "Employees" during a change of some sort. Did not bother to check if that contained all of the employee accounts. Was the third strike for him.
Wow, this is the worst nightmare in a datacenter. Normally UPSes go to bypass (or switch themselves off) when they fail and then fire extinguishing systems solve this. Something went very wrong over there and possibly more than 1 issue occurred. Is SBG1 unattended at night? As it is a relatively young building built in 2011, I am very curious to know what went wrong as this kind of catastrophes are normally eliminated by design as much as possible.
BTW the way of reporting the issue is non standard in the business.
Last edited:
Well, I was lucky with my webmaster, he insisted at the time to run daily backups on his own server at his office. I thought he was being overly concerned. But he got it right.
The datacenter also had backups for me but these were in the same building ...
In the end, I was up and running on Wednesday 6PM.
Jan
The datacenter also had backups for me but these were in the same building ...
In the end, I was up and running on Wednesday 6PM.
Jan
That is mostly backup/software and it is a good decision by the webmaster. At the root of it all lays the infrastructure where things went wrong. OVH states 2 UPSes caught fire and at least 1 of them was "tuned by the manufacturer" the morning before the fire.
OVH founder says UPS fixed up day before blaze is early suspect as source of data centre destruction • The Register
Sorry to say but the statements are somewhat fishy:
If the cause of the blaze was a UPS, it could be somewhat exculpatory for OVH because while data centres are equipped with fire suppression systems the kind of conflagration caused by a runaway battery fire can be very hard to quell. And of course if the device was not properly maintained, OVH's culpability could diminish further. Klaba did, however, say that SBG2 was commissioned in 2011, was built to the standards of the day, and that OVH now constructs more resilient facilities.
OVH founder says UPS fixed up day before blaze is early suspect as source of data centre destruction • The Register
Sorry to say but the statements are somewhat fishy:
If the cause of the blaze was a UPS, it could be somewhat exculpatory for OVH because while data centres are equipped with fire suppression systems the kind of conflagration caused by a runaway battery fire can be very hard to quell. And of course if the device was not properly maintained, OVH's culpability could diminish further. Klaba did, however, say that SBG2 was commissioned in 2011, was built to the standards of the day, and that OVH now constructs more resilient facilities.
Last edited:
The OVH SBG2 was not a typical datacenter. It was built in 2016 according to the OVH "tower" blueprint, also deployed on other OVH facilities. The OVH founder was proud of this concept back then, and described it extensively. It is a lightweight structure with wooden floors and minimal internal partitioning, designed to host OVH lower tier services in a cost optimized fashion. It was touted to enable higher density than the company "datacenter in a container" blueprint, also deployed at the Strasbourg site (SBG1) and also damaged by the fire. The building was equipped with old servers that were previously retired from the OVH enterprise service tier, paired with cheap consumer grade hardware. Basically it was a temporary building filled with expendable hardware. I would not be surprised if the fire suppression systems and crew incident response training would have been geared mainly toward the prevention of personal injuries. No one was hurt despite the extent of the fire, so this worked out. Services at SBG2 had a "best effort" service level agreement.
OVH datacenters are staffed 24/7 and have internal video surveillance, it does make sense even in extreme cost-cutting scenarios, but I guess that people were instructed to just turn on the fire suppression system they had, call the fire brigade and rush outside. According to reports on web hosting forums, power to server rooms was left turned on while internal temperature was rising, so this is consistent with rapid evacuation.
Attachments
Hi, I asked this because sometimes even 24/7 staff is too expensive to be able to compete with larger companies. Video surveillance is therefor standard since ages and the least one should have to run an (unmanned) datacenter and to have a TIER or other similar certification. I do note differences in disaster scenarios with regards to leaving equipment switched on. Sometimes there isn't a working/proven/tested EOP.
All in all something nobody wants to experience and also pities the competitor.
All in all something nobody wants to experience and also pities the competitor.
Last edited:
Things are not going well. And this is not new news.
OVH writes off another data centre – SBG1 – as small second fire detected, doused • The Register Forums
“We don’t plan to restart SBG1. Ever.”
....news of a second fire on the site.
....a post .... from November 2017 that discussed a power outage .... “the worst incident that can happen to us.”
Meanwhile fire takes out Japanese chip plant:
Fire takes out Japanese chip plant, owner Renesas warns of more silicon shortages • The Register
OVH writes off another data centre – SBG1 – as small second fire detected, doused • The Register Forums
“We don’t plan to restart SBG1. Ever.”
....news of a second fire on the site.
....a post .... from November 2017 that discussed a power outage .... “the worst incident that can happen to us.”
Meanwhile fire takes out Japanese chip plant:
Fire takes out Japanese chip plant, owner Renesas warns of more silicon shortages • The Register
My sites have been moved to a Contabo server center in Germany. All seems well again.
Contabo Locations 🌎 Global Availability & Growing Data Center Footprint
Jan
Contabo Locations 🌎 Global Availability & Growing Data Center Footprint
Jan
Last edited:
Like a phoenix rising from the smouldering ruins of its data centre, OVH sets sights on IPO
"One other financial aspect mentioned in today's document is the commitment OVH received from insurers to pay $58m to "cover the consequences of the fire" at its data centre in Strasbourg, France, in March.
"The blaze took out SBG2 data centre, OVH was forced to write off SBG1, and the blaze hit service in other bit barns on site. The suspected cause was a UPS refresh a day before the fire broke out, but a full report on the investigation is not expected to be made public for some time.
"Some clients lost data due to the incident and OVH has put in place measures to prevent a similar mishap from taking place, including a lab to model the impact of fires on data centres."
Jan> My sites have been moved
Just a year later...
'The OVHcloud datacenter ....destroyed in a fire last year had no automatic fire extinguisher system nor an electrical cutoff mechanism'
'arriving firefighters were met with "electric arcs of more than one meter around the exterior door of the energy room"'
'firefighter group leader stating that the temperature in the ground floor room measured 400 degrees Celsius'
https://www.theregister.com/2022/03/22/ovhcloud_fire_datacenter_report/
Just a year later...
'The OVHcloud datacenter ....destroyed in a fire last year had no automatic fire extinguisher system nor an electrical cutoff mechanism'
'arriving firefighters were met with "electric arcs of more than one meter around the exterior door of the energy room"'
'firefighter group leader stating that the temperature in the ground floor room measured 400 degrees Celsius'
https://www.theregister.com/2022/03/22/ovhcloud_fire_datacenter_report/
