Member
Joined 2009
Paid Member
According to Reuters in 'over 50%' of the inspected planes but other sources mention 70%.
Boeing finds debris in fuel tanks of many undelivered 737 MAX jets - Reuters
And yet another ooopsie:
https://www.washingtonpost.com/tran...spections-fixes-engine-panels-boeing-737-max/
https://www.washingtonpost.com/tran...spections-fixes-engine-panels-boeing-737-max/
Hmmmmm.
"The program manager in charge of Boeing’s Starliner crew capsule program said Friday that additional checks
would have uncovered problems with the spaceship’s software that plagued the craft’s first unpiloted orbital test
flight in December, but he pushed back against suggestions that Boeing engineers took shortcuts during ground testing."
Boeing says thorough testing would have caught Starliner software problems – Spaceflight Now
"The program manager in charge of Boeing’s Starliner crew capsule program said Friday that additional checks
would have uncovered problems with the spaceship’s software that plagued the craft’s first unpiloted orbital test
flight in December, but he pushed back against suggestions that Boeing engineers took shortcuts during ground testing."
Boeing says thorough testing would have caught Starliner software problems – Spaceflight Now
As I said for a long time now the problem is systemic within Boeing, the 737Max and Starliner are just symptoms.
And to think Toyota got publicly roasted for sloppy software on one of their autos . . . And here we have airplanes that fall out the sky and spacecraft where the same manufacturer admits ‘more thorough testing’ would have uncovered problems.
I’m thinking about the hours and hours the guys in my team spent doing FMEA analysis on power semiconductors for ABS and EMU systems. Projects were delayed and I had to fly off to HQ to explain . . .
Houston, we have a problem . . .
I’m thinking about the hours and hours the guys in my team spent doing FMEA analysis on power semiconductors for ABS and EMU systems. Projects were delayed and I had to fly off to HQ to explain . . .
Houston, we have a problem . . .
If you look at my linky it is not just software, it is also the physical lightning protection of the engines for which the FAA has now issued an airworthy directive.
Honestly, having some experiences with large-scale software systems testing, more thorough testing ALWAYS uncovers problems or faults.
We used to have a standing mantra: 'a manager who demands error-free software should be quietly replaced by someone who knows what he's talking about'.
The art of it, if you will, is to be able to determine when it is 'good enough' (and I mean that in a positive way) to be unleashed on the public. I agree that Boeing has a bad track record here.
Jan
I have a long background in pacemaker design, where software errors are not tolerated. verification of software (I sometime spell it softwhere, softwear) take 5-6 times longer than hardware verification, that is why we are used to say: hardware you can change, software not.
^+1
And a pacemaker is a very neat and tidy problem compared to the full avionics package of an airplane. I'm with Jan -- which is why critical paths *should* (and we can safely say the weren't) be checked with the thoroughness you apply to your work, basreflex.
And a pacemaker is a very neat and tidy problem compared to the full avionics package of an airplane. I'm with Jan -- which is why critical paths *should* (and we can safely say the weren't) be checked with the thoroughness you apply to your work, basreflex.
today my daily work is more related to design of neuro-type implants,trying to prevent errors by young and enthousiastic engineers. a must-view for every biomedical engineer is the netflix production "the bleeding edge" . it all goes back to workethics and attitude, similar to the max737 story.
We'd get along great, I'm sure. 🙂 I do somewhere between biomedical engineering and instrumentation, so I'd probably have a lot to learn from you about new and creative ways we screw things up. (I mean that sincerely)
Do you have some report or a source for that? The only concrete problem I read about with the Toyota accelerator was a NASA report that the two potentiometers in the accelerator pedal failed due to tin whiskers that grew from lead-free solder. I'm sure I've posted that link on DIYaudio, maybe earlier in this thread.
I presume a similarly detailed analysis of the software wouldn't be made public, as it would include source code which Toyota would claim is all under copyright, trade secret and such.
Yet, if you test deep enough, go through even more unlikely combinations of conditions, you WILL find errors. There is no way you can know whether all errors are found.
If you are convinced there are no more errors, you're on dangerous terrain. The only thing you can hope for, and work for, is that the non-found errors are not catastrophic.
Jan
No, I am quite correct. Google Toyota software problem in US.
All manner of software design experts were brought in to pass expert opinion on how they screwed up. Circa 2011/12 IIRC.
The tin whiskers thing is wel understood. Every semi manufacturer was battling with this 15 years ago.
I didn't say or mean to imply you were incorrect, I just asked for a cite.
This is the document I recall:
https://nepp.nasa.gov/whisker/reference/tech_papers/2011-NASA-GSFC-whisker-failure-app-sensor.pdf
I recall the late 1990s, there was a lot in EE Times about the then-new RHOS standards and the need for lead-free solder. Even then there was concern about tin whiskers in lead-free solder and its use in satellites. There was talk of allowing leaded solder in safety critical areas, but it looks like not everyone got the message.
I would never discount the role of potential faults in software, and I'd agree that it's a likely source of failure. The problem is that hardware can also fail.
interestingly the guy that I knew that wrote bulletproof software, also was a spare time pilot. He mentioned back then when he had a choice between boeing and airbus (fly by wire) , he would always choose boeing as there was always a mechanical connection betweeen the pilots and the rudders. so boeing apparently was not used to the rigorous standards of fly by wire.
So I'm reading this article:
https://www.edn.com/toyotas-killer-firmware-bad-design-and-its-consequences/
and it says this:
2009–11 Toyota vehicle recalls - Wikipedia
Regardless, it looks like there may have been different causes of the different unintended acceleration events of Toyota cars.
The software analysis discussed in the EDN article certainly aims at bad software as a cause, and mentions many possible failures, but there's nothing that says any particular error caused a specific event. This would require a trace of the processor, a recording of every CPU register for every instruction executed This can be done in development and testing, but not production. Some of these errors can be discovered more easily, such as regularly checking the stack pointer to find stack overflow, but even this adds overhead to the CPU for a safety check function that, as I could hear someone say, "would almost never get used."
This is what's been so unsatisfying about the Toyota events. There's no exact "this part of the code did this, which caused full throttle" smoking gun. In many if not all of these cases, one can't know whether the causal event was software or hardware, much less which part of the software. Part of this is using a small CPU that's "not worth" making bigger to add real-time tests and logging functions and such. Another part was that Toyota didn't follow good standards and procedures enough to make "good code" (good enough to where there would be only one or a very few causes of failure), so it has MANY possible failure modes (!). No one knows what really happened.
Compare to this, where they did better to write good software (Ada was supposed to be a "better language" for this sort of thing, created specifically by the military to write reliable code for jet fighters, and admittedly it's better than C, but it still dates from way back when, circa 1980). But still, they failed, and an analysis points to exactly how and why the code failed:
ARIANE 5 Failure - Full Report
https://www.edn.com/toyotas-killer-firmware-bad-design-and-its-consequences/
and it says this:
This appears to refer to earlier Toyota unintended acceleration cases. There was a recall on Toyota accelerator pedals/mechanisms, but I never heard (through the usual news media) what exactly that was supposed to fix. Wikipedia only discusses (other than the software problems) mechanical problems, such as plastic bits in the mechanism causing the accelerator to get stuck partway down:
2009–11 Toyota vehicle recalls - Wikipedia
Regardless, it looks like there may have been different causes of the different unintended acceleration events of Toyota cars.
The software analysis discussed in the EDN article certainly aims at bad software as a cause, and mentions many possible failures, but there's nothing that says any particular error caused a specific event. This would require a trace of the processor, a recording of every CPU register for every instruction executed This can be done in development and testing, but not production. Some of these errors can be discovered more easily, such as regularly checking the stack pointer to find stack overflow, but even this adds overhead to the CPU for a safety check function that, as I could hear someone say, "would almost never get used."
This is what's been so unsatisfying about the Toyota events. There's no exact "this part of the code did this, which caused full throttle" smoking gun. In many if not all of these cases, one can't know whether the causal event was software or hardware, much less which part of the software. Part of this is using a small CPU that's "not worth" making bigger to add real-time tests and logging functions and such. Another part was that Toyota didn't follow good standards and procedures enough to make "good code" (good enough to where there would be only one or a very few causes of failure), so it has MANY possible failure modes (!). No one knows what really happened.
Compare to this, where they did better to write good software (Ada was supposed to be a "better language" for this sort of thing, created specifically by the military to write reliable code for jet fighters, and admittedly it's better than C, but it still dates from way back when, circa 1980). But still, they failed, and an analysis points to exactly how and why the code failed:
ARIANE 5 Failure - Full Report
- Status
- Not open for further replies.