[conspire] 737 MAX story keeps getting more fractally bad

[Rick Moen]

As often occurs with a complex story, the 737 MAX story twists and
morphs as more details emerge.

To rewind, in 2011, Boeing hurriedly cancelled[1] the next-generation 
Y1 Project and instead quickly up-revved the aging 737 design to
compete with a critical challenge in the form of Airbus's fuel-thrifty 
A320neo.  The first stage (2012), wind-tunnel tests, immediately showed
a severe aerodynamics problem:  Owing to the product line's trend of
ever-larger engines mounted high and forward, if put into a specific
extreme manoeuvre -- a banked spiral called a 'wind-up turn' -- at high
speed, the MAX pitched upwards and stalled. 

This was a difficult problem (albeit one of narrow scope) that Boeing
tried, at first, to fix by tweaking the MAX's aerodynamic shape in
several ways to make its handling self-correcting, but such tweaks
couldn't produce enough effect during wind-up turns.  So, engineers 
working under chief test pilot Ray Craig turned to a software solution:
A routine called Manoeuvring Characteristics Augmentation System (MCAS)
triggered only if two distinct sensors indicated extreme high-speed
handling -- both a high angle of attack and high G-force.  If that
situation were detected, MCAS would automatically swivel up the leading
edge of the MAX's horizontal tail by, at most if required to produce the
needed trim effect, adding (at most) 0.6 degrees of downwards pitch to
the fuselage, over 10 seconds.  This solution appeared to work perfectly
and was included in Boeing's certification documents approved by the
FAA.  

In the process of implementing MCAS and as described in its System
Safety Assessment filed with FAA, Boeing made MCAS able to apply 0.27
degrees of trim per second, making the assumption that pilots would
respond in under three seconds to any inappropriate MCAS triggering
during normal flight, i.e., that MCAS would induce the maximum permitted
0.61 degrees of trim in 0.61/0.27 = 2.26 seconds.  

FAA's certification standard for almost any commercial aircraft
components dictates that if it malfunctioning cannot cause 'hazardous
failure' scenarios involving injury or fatalities, but only 'major
failure' scenarios where injury isn't expected, then it may be reliable
all but one time in 100,000.  If 'hazardous failure' (one in 10 million)
might result, then additional hardware redundancy would be required to
reduce risk (such as adding a check against a G-force sensor to the
angle-of-attack check) -- but there's an exception to that requirement:
If the hazardous events in question are outside normal flight conditions
and unlikely to be encountered, then the extra redundancy is waived.

In the case of MCAS's initially contemplated use only in (some) wind-up
turns, Boeing rated MCAS somehow triggering during normal flight as only
a 'major risk' -- largely because of their assumption that pilots' quick
reaction would serve as a backup system (something rather difficult for
them to do given that they were not even informed of MCAS's existence).
They rated the only 'hazardous failure' scenario, the one where the
plane went through an extremely rare wind-up turn and simultaneously
MCAS failed to trigger when needed, as almost astronomically unlikely:
about once every 223 trillion hours of flight.

Based on these figures, Boeing implicitly had ruled the G-force sensor
check (for redundancy) unnecessary, but initially left it in the design.
But then a further and catastrophic change was introduced, one _not_
submitted to FAA for a revised Safety System Assessment:

During initial flight testing in 2016, test pilots found more
aerodynamics problems:  The MAX tended to pitch dangerously nose-up at 
lower speeds, too.  Chief test pilot Ray Craig had by now left Boeing
and been replaced by Mark Forkner.  Engineers decided to greatly expand
the scope of MCAS's operation to also cover low-speed flight, but never
revisited their 2012 conviction that a single sensor was sufficient, and
dropped the G-Force sensor -- which was convenient to the new use-case, 
as there are no excessive G-forces at low speed,  This now meant that
the angle-of-attack (AoA) sensor became an unexamined single point of
failure (SPoF) -- unexamined because of the now-outdated assumption that
only 'major failure' malfunctions could result.

At the lower speeds in question (where a tendency to pitch-up is far
more dangerous), control surfaces must be deflected more to produce the
same effect, so engineers allowed MCAS a great deal more force,
permitting it to apply up to 2.5 degrees of trim (over 10 seconds) each
time it activated, which it could now do multiple times, ultimately
reaching about four times the certified 0.6 degrees of deflection,   No
additional risk analysis was done, and documentation to the FAA wasn't
revised.  

Further, Boeing's assumption of pilot correction in less than three
seconds wasn't examined or revisited, either, until after the two 737
MAX crashes' tape recordings showed the crews to have been distracted
and confused by multiple alerts from the moment of takeoff -- made worse
by the crews having not even been informed of MCAS's existence, a
_removal of description of MCAS_ from the pilot manual.  This was 
literally requested of FAA in a March 30, 2016 e-mail from Boeing's MAX
program chief technical pilot and former FAA employee, Mark Forkner.
Based on Forkner's guidance (who did not see fit to mention that MCAS
had been overhauled since certification, but in fairness he may have
been not informed of that change), FAA OK'ed the removal and approved
certifying pilots for the MAX based on an hour of training through an
iPad about the differences between the MAX and the previous generation,
737ng, with no simulator training and no mention of MCAS anywhere.  FAA
justified the removal on grounds that MCAS is merely code running in
background and not part of the flight controls, and the system is
'relatively benign and rarely used'.

Both FAA and Boeing's basic defence in all this is 'Hey, we followed all
the rules.  The rules permitted us to do stupid, dangerous things for
bottom-line purposes and so 346 people died.  Why are you suggesting
it's our fault?'


In other news, back in 2017 when Boeing discovered after rollout that
the extra-cost safety option 'AoA Disagree' was no-functional (where
present) on existing MAX jets unless a second extra-cost safety measure
('AoA Indicator') was also present, it turns out that Boeing had no plan
to release a software update to fix the AoA Disagree issue until 2020
(in addition to telling nobody including the FAA about this bug until
after the Lion Air crash).  According to a statement in May Boeing had 
unilaterally decided in 2017 that the issue did not 'adversely impact
the safety or operation' of the plane.  Oops.  The statement added:
'Senior company leadership was not involved in the review.'

Trump-toady and ex-lobbyist Acting FAA Administrator Daniel Elwell also
endorsed Boeing's perspective that the light wasn't a safety-critical
issue.  Double-oops.

In May, Elwell chided members of the House Transportation Committee for
making so much of a simple AoA Disagree boo-boo:  'Don’t make something
that isn't a critical safety item a critical safety item, because
there's enough critical safety items for us to focus on', he helpfully
advised Congress.


Also in May, Boeing rolled out its proposed omnibus response to the two
MAX disasters:  a software patch that makes MCAS listen to both AoA
sensors, a revision to the flight manual, and about an hour of
self-directed pilot training on an iPad.  Among the observers
unimpressed by this response:  US Airways Captain Chesley 'Sully'
Sullenberger (ret.), hero with First Officer Jeffrey Skiles of the 2009
Miracle on the Hudson no-engine landing, who told the House
Transportation Committee that realistic training on a simulator should
be absolutely required.  After initial signs that Elwell's FAA intended
to rubber-stamp Boeing's proposed fix, non-USA aviation regulators
suggested that -- once again -- they would be inclined to disregard
FAA's view as worthless, which then caused additional consultation and
political wrangling that is still ongoing.


Another revelation:  For quite a few years, Boeing has been outsourcing
and offshoring software coding for avionics and flight-test equipment to
temporary workers earning as little as $9/hour from two large coding
companies in India, HCL Technologies Ltd. (formerly Hindustan Computers)
and Cyient Ltd.  (Boeing recently claimed that neither of these firms
was involved in the coding specifically of the MCAS software.)

At an all-company meeting in 2015 where Boeing announced it was laying
off many of its senior software engineers, managers explained to the
departing veteran coders that they were no longer needed because
Boeing's products were 'mature'.  (And how's that going, guys?)


Recent coverage (read the IEEE Spectrum piece, if no other):

https://spectrum.ieee.org/aerospace/aviation/how-the-boeing-737-max-disaster-looks-to-a-software-developer
https://www.salon.com/2019/04/27/boeing-might-represent-the-greatest-indictment-of-21st-century-capitalism_partner/
https://www.nytimes.com/2019/06/01/business/boeing-737-max-crash.html
https://www.nytimes.com/2019/04/11/business/boeing-faa-mcas.html
https://www.seattletimes.com/seattle-news/times-watchdog/the-inside-story-of-mcas-how-boeings-737-max-system-gained-power-and-lost-safeguards/
https://www.seattletimes.com/business/boeing-aerospace/how-much-was-pilot-error-a-factor-in-the-boeing-737-max-crashes/
https://www.seattletimes.com/business/boeing-aerospace/famed-pilot-sully-sullenberger-tells-lawmakers-that-simulator-training-needed-for-boeing-737-max-pilots/
https://www.seattletimes.com/business/boeing-aerospace/failed-certification-faa-missed-safety-issues-in-the-737-max-system-implicated-in-the-lion-air-crash/
https://www.seattletimes.com/business/boeing-aerospace/boeing-didnt-plan-to-fix-737-max-warning-light-until-2020/
https://www.thedailybeast.com/how-boeing-bean-counters-courted-the-737-max-disaster
https://www.cnn.com/2019/05/30/politics/737-max-computer-simulator/index.html
https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers
https://www.bbc.com/news/business-48528383
https://www.washingtonpost.com/local/trafficandcommuting/changes-to-flawed-boeing-737-max-were-kept-from-pilots-defazio-says/2019/06/19/553522f0-92bc-11e9-aadb-74e6b2b46f6a_story.html
https://www.cnn.com/2019/05/21/politics/boeing-737-max-lawsuit-1990s-crashes/index.html
https://www.asiatimes.com/2019/06/opinion/why-boeing-may-never-recover-from-737-debacle/



[1] This decision was taken by CEO James McNerney, Boeing's first chief
executive to have no background whatsoever in aviation, having worked 
before at Procter & Gamble, McKinsey, General Electric, and 3M.
McNerney changed company culture, even more than his successor
Muilenberg, away from engineering and towards tight-fisted focus on
financial results.