hn-classics/_stories/2009/12016234.md

---
created_at: '2016-07-01T14:23:37.000Z'
title: Automated to Death (2009)
url: http://spectrum.ieee.org/computing/software/automated-to-death
author: mcspecter
points: 100
story_text: 
comment_text: 
num_comments: 45
story_id: 
story_title: 
story_url: 
parent_id: 
created_at_i: 1467383017
_tags:
- story
- author_mcspecter
- story_12016234
objectID: '12016234'
year: 2009

---
The passengers and crew of Malaysia Airlines Flight 124 were just
settling into their five-hour flight from Perth to Kuala Lumpur that
late on the afternoon of 1 August 2005. Approximately 18 minutes into
the flight, as the Boeing 777-200 series aircraft was climbing through
36 000 feet altitude on autopilot, the aircraft—suddenly and without
warning—pitched to 18 degrees, nose up, and started to climb rapidly. As
the plane passed 39 000 feet, the stall and overspeed warning indicators
came on simultaneously—something that’s supposed to be impossible, and a
situation the crew is not trained to handle.

At 41 000 feet, the command pilot disconnected the autopilot and lowered
the airplane’s nose. The auto throttle then commanded an increase in
thrust, and the craft plunged 4000 feet. The pilot countered by manually
moving the throttles back to the idle position. The nose pitched up
again, and the aircraft climbed 2000 feet before the pilot regained
control.

The flight crew notified air-traffic control that they could not
maintain altitude and requested to return to Perth. The crew and the 177
shaken but uninjured passengers safely returned to the ground.

The [Australian Transport Safety Bureau
investigation](http://www.atsb.gov.au/publications/investigation_reports/2005/aair/aair200503722.aspx)
discovered that the air data inertial reference unit (ADIRU)—which
provides air data and inertial reference data to several systems on the
Boeing 777, including the primary flight control and autopilot flight
director systems—had two faulty accelerometers. One had gone bad in
2001. The other failed as Flight 124 passed 36 571 feet.

The fault-tolerant ADIRU was designed to operate with a failed
accelerometer (it has six). The redundant design of the ADIRU also meant
that it wasn’t mandatory to replace the unit when an accelerometer
failed.

Photo: John Boyd

Watch a forensic simulation of key moments of Malaysian Airlines Flight
124

However, when the second accelerometer failed, a latent software anomaly
allowed inputs from the first faulty accelerometer to be used, resulting
in the erroneous feed of acceleration information into the flight
control systems. The anomaly, which lay hidden for a decade, wasn’t
found in testing because the ADIRU’s designers had never considered that
such an event might occur.

The Flight 124 crew had fallen prey to what psychologist Lisanne
Bainbridge in the early 1980s identified as the ironies and paradoxes of
automation. The irony, she said, is that the more advanced the automated
system, the more crucial the contribution of the human operator becomes
to the successful operation of the system. Bainbridge also discusses the
paradoxes of automation, the main one being that the more reliable the
automation, the less the human operator may be able to contribute to
that success. Consequently, operators are increasingly left out of the
loop, at least until something unexpected happens. Then the operators
need to get involved quickly and flawlessly, says Raja Parasuraman,
professor of psychology at George Mason University in Fairfax, Va., who
has been studying the issue of increasingly reliable automation and how
that affects human performance, and therefore overall system
performance.

“There will always be a set of circumstances that was not expected, that
the automation either was not designed to handle or other things that
just cannot be predicted,” explains Parasuraman. So as system
reliability approaches—but doesn’t quite reach—100 percent, “the more
difficult it is to detect the error and recover from it,” he says.

And when the human operator can’t detect the system’s error, the
consequences can be tragic.

![DC crash](/image/1464827)

Photo: Win McNamee/Getty Images

In June of this year, a Washington Metropolitan Area Transit Authority
(Metro) red line subway train operated by Jeanice McMillan rear-ended a
stationary subway train outside Fort Totten station in northeast
Washington, killing McMillan and eight others and injuring 80. The cause
is still under investigation by the U.S. National Transportation Safety
Board (NTSB), but it appears that a safety-signal system design anomaly
was at fault, in which a spurious signal generated by a track circuit
module transmitter mimicked a valid signal and bypassed the rails via an
unintended signal path. The spurious signal was sensed by the module
receiver, which resulted in the train not being detected when it stopped
in the track circuit where the accident occurred. So the safety system
thought the track was clear when it was not. When she saw the other
train in her path, a surprised McMillan hit the emergency brake in an
attempt to slow her train, which may have been traveling nearly 95
kilometers per hour (59 miles per hour), but it was too late.

To put this accident in perspective, however, it was only the second
fatal crash involving Washington, D.C.’s Metro in its 33 years of
operation. In 2008, customers took 215 million trips on the system. Not
counting train-vehicle accidents, a total of 27 people were killed and
324 people were injured in [train accidents in the United States
in 2008](http://safetydata.fra.dot.gov/OfficeofSafety/publicsite/Query/statsSas.aspx).
This compares with statistics from 1910, when W.L. Park, general
superintendent of the Union Pacific Railroad, asserted that “one human
being is killed every hour, and one injured every 10 minutes.”

Not only has automation improved train safety, but travel by plane,
ship, and automobile is safer too. [According to
Boeing](http://www.boeing.com/commercial/safety/pf/pf_howsafe.html), in
2000 the world’s commercial jet airlines carried approximately 1.09
billion people on 18 million flights and suffered only 20 fatal
accidents. The NTSB estimates that traffic deaths in the United States
may drop by 30 percent after electronic stability control becomes
mandatory in 2012 for automobiles.

Charles Perrow, professor emeritus of sociology at Yale University and
author of the landmark book Normal Accidents: Living With High-Risk
Technologies (Princeton University Press, 1999), contends that
“productivity, successful launches, successful targeting, and so on,
increase sharply with automation,” with the result being that “system
failures become more rare.”

One can see this in aviation. As automation has increased aircraft
safety, the rarity of crashes has made it harder to find common causes
for them, the NTSB says.

However, the underlying reason for this rarity, namely the ubiquity of
increasingly reliable automation, is also becoming a concern for system
designers and safety regulators alike, especially as systems become ever
more complex. While designers are trying to automate as much as they
can, complex interactions of hardware systems and their software end up
causing surprising emergencies that the designers never considered—as on
Flight 124—and which humans are often ill-equipped to deal with.

“The really hard things to automate or synthesize, we leave to the
operator to do,” says Ericka Rovira, an assistant professor of
engineering psychology at the U.S. Military Academy at West Point. That
means people have to be alert and ready to act at the most crucial
moments, even though the monotony of monitoring supposedly reliable
systems can leave them figuratively or physically asleep at the wheel.

![Royal Majesty cruise ship](/image/1464842)

Photo: Steven Senne/AP Photo

That was the case in June 1995, when the 568-foot-long [cruise ship
Royal Majesty ran
aground](http://www.ntsb.gov/publictn/1997/MAR9701.htm) onto the sandy
Rose and Crown Shoal about 10 miles east of Nantucket Island, off the
coast of Massachusetts. Fifty-two minutes after leaving St. George’s,
Bermuda, on its way to Boston, the Royal Majesty’s GPS antenna cable
became detached from the GPS antenna. This placed the GPS in
dead-reckoning mode, which does not take into consideration wind or sea
changes. The degraded GPS continued to feed the ship’s autopilot. No one
noticed the change in GPS status, even though the GPS position was
supposed to be checked hourly against the Loran-C radio navigation
system, which is accurate by roughly one-half to 1 nautical mile at sea,
and a quarter-mile as a ship approaches shore. The Royal Majesty
proceeded on autopilot for the next 34 hours until it hit the Rose and
Crown Shoal.

Why hadn’t the watch officers noticed something was wrong? One major
reason, the NTSB said, was that the ship’s watch officers had become
overreliant on the automated features of the integrated bridge system.

Watch officers, who in less-automated times actively monitored the
current environment and used this information to control the ship, are
now relegated to passively monitoring the status and performance of the
ship’s automated systems, the NTSB said. The previous flawless
performance of the equipment also likely encouraged this overreliance.
Checking the accuracy of the GPS system and autopilot perhaps seemed
like a waste of time to the watch officers, like checking a watch
against the Coordinated Universal Time clock every hour.

In many ways, operators are being asked to be omniscient systems
administrators who are able to jump into the middle of a situation that
a complex automated system can’t or wasn’t designed to handle, quickly
diagnose the problem, and then find a satisfactory and safe solution.
And if they don’t, the operators, not the system’s designers, often get
the blame.

Adding another system to help detect the error and recover from it isn’t
a straightforward solution either. In Flight 124, the fault-tolerant,
redundant system design helped to mask the problem. In fact, such
redundancy often merely serves to act as yet another layer that
abstracts the human operator from the system’s operational control.

“In other words, the initial control loop is done by one system, and
then you have a computer that is backing up that system, and another is
backing up that one,” according to Parasuraman. “Finally, you have to
display some information to the operator, but the operator is now so far
from the system and the complexity is so great that their developing a
\[mental\] model of how to deal with something going wrong becomes very,
very difficult.”

Economics also figures into the equation. The ADIRU in Flight 124’s
Boeing 777 was designed to be fault-tolerant and redundant not only to
increase safety but also to reduce operating costs by deferring
maintenance.

“The assumption is that automation is not only going to make \[what you
are doing\] safer but that it will make it more efficient,” says Martyn
Thomas, a Fellow of the UK’s Royal Academy of Engineering. “This creates
a rather nasty feedback loop, which means that when adverse events
become relatively rare, it is taken as an opportunity to deskill the
people you’re employing or to reduce their number in order to reduce a
cost.”

This erosion of skills in pilots was a major concern raised in the last
decade as glass cockpits in aircraft became common \[see IEEE Spectrum’s
article [The Glass
Cockpit,](http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=406460&isnumber=9128)
September 1995\].

Peter Ladkin, a professor of computer networks and distributed systems
at Bielefeld University, in Germany, is heavily involved in aircraft
accident investigations and is a pilot himself. “Line pilots are trained
and told—and their procedures also say—to use the automation all the
time. Many older pilots are really worried that when they get into
difficulties, they aren’t going to \[know how to\] get out of them,” he
says.

![Turkish Airlines photo](/image/1464857)

Photo: United Photos/Reuters

The crash in February of Turkish Airlines Flight 1951 just short of
Amsterdam’s Schiphol International Airport, which killed 9 people and
injured 86 others, raised this concern anew. As the aircraft passed
through 1950 feet, [the left radio altimeter failed and indicated an
altitude of –8
feet](http://www.onderzoeksraad.nl/en/index.php/onderzoeken/Neergestort-tijdens-nadering/),
which it passed on to the autopilot, which in turn reduced engine power
because it assumed the aircraft was in the final stages of approach. The
pilots did not initially react to the warnings that something was wrong
until it was too late to recover the aircraft.

“When we start removing active learning for the operator, the operator
begins to overtrust the automation,” Rovira says. “They’re not going
back and gathering those data pieces that they need” to make an
effective decision.

Another issue associated with overtrusting the automation is that it can
encourage “practical drift,” a term coined by Scott Snook in his book
Friendly Fire: The Accidental Shootdown of U.S. Black Hawks over
Northern Iraq (Princeton University Press, 2002). The phrase refers to
the slow, steady uncoupling of practice from written procedures.

![NW Flight 188 map](/image/1464873)

Illustration: Boswell/MCT/Landov

We see how that happened in the Royal Majesty incident, where the watch
officers failed to follow established procedures. This was also the case
in the October incident involving Northwest Airlines Flight 188 on its
way to Minneapolis-St. Paul International Airport, which overshot the
airport by 150 miles. The pilots claimed they were working on their
laptops and lost track of the time and their location. The aircraft was
on autopilot, which in normal circumstances leaves the pilots with
little left to do other than monitor the controls.

 Again, when you are only a system’s monitor, especially for an
automated system that rarely, if ever, fails, it is hard not to get
fatigued or bored and start taking shortcuts.

The situation isn’t hopeless, however. For some time now, researchers
have been working to address the ironies and paradoxes of automation.
One new approach has been to address the issues from the human point of
view instead of the point of view of the system.

“We draw a system’s boundary in the wrong place,” Thomas states. “There
is an assumption that the system boundary that the engineer should be
interested in \[sits\] at the boundary of the sensors and actuators of
the box that is being designed by the engineers. The humans who are
interrelating with these systems are outside it. Whether they are
operators, pilots, controllers, or clinicians, they are not part of the
system.

“That is just wrong,” Thomas adds. “The system’s designer, engineer, and
overall architect all need to accept responsibility for the ways those
people are going to act.”

Victor Riley, associate technical fellow in Boeing Flight Deck, Crew
Operations, argues that there needs to be a two-way dialogue between the
operator and the automated system.

“The operator-to-the-system part of the dialogue is more important than
the system-to-the-operator part,” Riley says. “People see what they
expect to see, and what they expect to see is based on what they thought
they told the system to do.”

Studies by Parasuraman, Rovira, and others have found that operators of
highly reliable automated systems will often perform worse than if they
were operating a lower-reliability system, which seems paradoxical.

Parasuraman explains that “if you deliberately engineer anomalies into
the automation, people rely less on it and will perform a little bit
better in monitoring the system. For example, if the system is 90
percent reliable, operators will be better at picking up the 10 percent
of the errors than if the system is 99 percent reliable.”

Rovira also says that operators need to be able to see how well the
automation is working in a given context.

“The goal for us as designers is to provide an interface that allows a
drill-down if the operator needs to query the system, in the event they
have a different perspective of the decision than the automation has
given them,” Rovira says. “Or if not a drill-down, there should be some
visibility or transparency right up front about what the underlying
constraints or variables are that make this decision not totally
reliable.”

Maybe one way to remind ourselves of the potential effects of the
ironies and paradoxes of automation is to simply pull the plug.

“If we don’t want people to depend on automated systems, we need to turn
them off sometimes,” Thomas observes. “People, after all, are the backup
systems, and they aren’t being exercised.”

## About the Author

Robert N. Charette, an IEEE Spectrum contributing editor, is a
self-described “risk ecologist” who investigates the impact of the
changing concept of risk on technology and societal development.
Charette also writes Spectrum ’s blog [The Risk
Factor](/blog/computing/it/riskfactor).

Advertisement