Betting
Against Nature: Human Fallibility and Dangerous Technologies
Lloyd
J. Dumas,
Professor
of Political Economy, University of Texas at Dallas.
Prepared
for the Annual Meeting of the Friends Committee on National
Legislation: Wyndham Hotel, Washington DC, November 9, 2001
The
devastating terrorist attack that struck the United States on
September 11, 2001 shattered New York's massive World Trade
Center, a piece of the Pentagon, thousands of innocent lives
-- and the illusion that sophisticated technology and powerful
weapons could keep us safe.
Thousands
of ordinary people going about their day-to-day lives became
the victims of an enemy who cared nothing about our fleets of
warships, bombers and missiles -- an enemy who turned the fruits
of our own technological brilliance against us. Over 33 years,
more than 14,000 international terrorist attacks by sub-national
groups around the world had taken a total of 9,000-10,000 lives.
In one terrible day, more than 5000 new victims were added to
the list.
It
has been 135 years -- more than 5 generations -- since this
kind of deliberate slaughter has been seen on the mainland of
the United States. It became natural for us to assume that this
was something that always happened somewhere else. Now we know
that we too are vulnerable. But I don't think we have yet understood
just how vulnerable we have become.
The
development of more and more powerful technologies has dramatically
increased our ability to affect the physical world around us.
But the very technologies that gave us this extraordinary power
have also made us increasingly vulnerable to both human error
and malevolence. The vulnerability that derives from the clash
between our growing technological power and our unchanging fallibility
lies at the heart of my book, Lethal
Arrogance (NY: St. Martin's Press/Palgrave, 1999).
This
problem is most serious for "dangerous technologies" -- those
capable of killing or injuring very large numbers of people
by design (as in the case of nuclear, biological and chemical
weapons of mass destruction) or by accident (as in the case
of nuclear power or highly toxic industrial chemicals). Here
human error and malevolence threaten not only our tranquility,
but perhaps our very survival.
To
begin to understand this threat -- and more importantly, what
must be done to overcome it -- we will briefly explore the pervasiveness
of human error, and then look at the character of malevolence
in one of its more virulent forms, terrorism.
Human
Error
According
to a 1998 study by the U.S. General Accounting Office, human
error was a contributing factor in almost 75% of the most serious
class of U.S. military aircraft accidents in 1994 and 1995.
A 1998 study by the Union of Concerned Scientists of ten nuclear
power plants (representing a cross section of American civilian
nuclear industry) concluded that nearly 80% of reported problems
resulted from worker mistakes or the use of poorly designed
procedures. In November 1999, the Institute of Medicine of the
U.S. National Academy of Sciences issued a report finding that
medical errors cause more deaths each year in the United States
than breast cancer or AIDS.
As
we talk about human error in dangerous technology systems, keep
two key points in mind. The first is that failures do not have
to be continuous in order to be dangerous. A drug or alcohol
impaired nuclear weapons guard is not a problem most of the
time, because most of the time nothing happens. But if that
guard is not alert and ready to act the moment terrorist commandos
try to break into the storage area, there could be a major disaster.
Because there is no way to know when those critical moments
will occur, every failure of reliability must be taken seriously.
The
second point is that the difference between a trivial error
and a catastrophic error lies not in the error itself, but in
the surrounding situation. Many of the most trivial kinds of
mistakes that all of us make on a daily basis would be disastrous
if made in a very different context. For example, making a telephone
call begins by entering a sequence of numbers on a keypad that
is fed into computers that switch the call. If we make a mistake
entering the number, we get the wrong person. The error is trivial.
But on a clear night in December 1995, the pilots of American
Airlines Flight 965 made essentially the same mistake as they
were flying toward Cali, Colombia. They entered the wrong sequence
of numbers into a computer, the plane's navigational computer.
The plane steered into the side of mountain, and 160 people
died.
Boredom.
For all the potential risk involved, much of the day-to-day
work of many of those who deal with dangerous technologies is
really quite boring. Guarding nuclear weapons storage areas,
going through checklists in missile silos, monitoring control
panels at nuclear power plants is not all that stimulating.
Boring work dulls the mind, leading to a lack of vigilance.
Laboratory studies have shown that, after a few weeks, people
exposed to extremely monotonous living and working environments
sometimes experience serious mood swings, diminished judgment,
and even hallucinations.
The
things people sometimes feel driven to do to cope with grinding
boredom can also cause serious reliability problems. They may
try to distract themselves by focusing their attention on more
interesting or amusing thoughts, which means they are not paying
close attention to the task at hand. They may play games. For
example, in the late 1970s, Tooele Army Depot in Utah contained
enough GB and VX nerve gas to kill the population of the earth
100 times over. According to newspaper reports, the guards at
Toole sometimes distracted themselves from the boring routine
by drag racing their vehicles. They played marathon card games.
Arsonists burned down an old building inside the Army Depot
while guards on the night shift played poker. There were also
reports of frequent sleeping on the job.
Sometimes
people try to make the boredom more palatable by drinking or
taking drugs. An American sailor who served as helmsman on the
nuclear aircraft carrier Independence during the late 1970s/early
1980s claimed that he used LSD almost every day on duty. He
said it was the only way to get through eight hours of extremely
boring work.
Stress.
Working with dangerous technologies can also be very stressful.
Sustained high levels of stress can lead to serious physical
problems, such as a compromised immune system, and serious emotional
problems, such as severe depression and even post-traumatic
stress disorder (PTSD). PTSD includes difficulty concentrating,
extreme suspicion of others, recurrent nightmares and emotional
detachment, all of which tend to reduce reliability. At least
500,000 of the 3.5 million American soldiers who served in Vietnam
have been diagnosed as suffering from PTSD, as many as 30% of
them may never lead a normal life without medication and/or
therapy.
Drug
and Alcohol Abuse. Boredom and stress can lead
to drug and alcohol abuse. Data released by the Pentagon for
the years 1975-1990 show that almost 20,000 American military
personnel were permanently removed from nuclear duty over that
period as a result of drug abuse. Alcohol abuse added about
another 7000 to the total.
The
Fallibility of Groups. One common strategy for
assuring that an unreliable individual cannot cause a disaster
in dangerous technology systems is to require that a group act
together to, say, launch a nuclear missile attack. But sometimes
groups can be less reliable than individuals.
In
bureaucracies, the flow of information from subordinates to
superiors is often distorted. One classic example is the "good
news" syndrome: subordinates edit problems out of the information
they send to higher management in order to pass along a more
pleasant picture. The result of all this good news is that top-level
decision makers have a very distorted picture of what is really
going on. And this problem tends to get worse, not better, when
there is more at stake, as in organizations dealing with dangerous
technologies.
"Groupthink"
occurs when the quality of decisions made by a group deteriorates
as a result of the pressure to maintain consensus among its
members. Increasingly isolating themselves from other points
of view, group members can develop an illusion of invulnerability
that sets the stage for very risky decision making. For example,
during the Korean War, groupthink was involved in the U.S. decision
to invade North Korea, after the North Koreans had been successfully
driven out of the South. That drew the Chinese into the war,
overwhelming American forces and driving deep into South Korea.
Years of fighting followed to regain the ground lost, at a cost
of millions of lives.
Group
psychosis is a situation in which a crazy but charismatic leader
is able to draw the otherwise sane members of a group into his/her
own delusional worldview by isolating them and controlling the
conditions in which they live. Twentieth century examples include
the Reverend Jim Jones and his followers at Jonestown, Guyana
in the 1970s and David Koresh and the Branch Davidian at Waco,
Texas in the early 1990s.
Suppose
a particularly charismatic military commander, who seemed fully
functional and apparently sound, had in fact developed a deep
psychic crack. With great control over the lives of his/her
troops, troops already primed for obedience by the very nature
of military life, such a commander might be able to draw them
into his/her delusional world. The crew of a nuclear missile
submarine is isolated for months at a time. The captain has
nearly complete control of the conditions in which they live
and work. And every nuclear missile submarine carries enough
firepower on board to devastate any nation on earth.
In
short, relying on groups does not fix the human reliability
problem.
Terrorism
Calling
violent groups "terrorists" when we don't like their objectives
and "freedom fighters" when we do is a political game. It doesn't
help us understand what terrorism is or figure out what to do
about it. We need a working definition that is more than just
propaganda or opinion.
Terrorism
is the threat or use of violence for the purpose of spreading
fear and alarm. The idea is to frighten the public in order
to affect their behavior or put pressure on their government
to change its behavior to help accomplish the terrorists' goal.
It is a method independent of the ultimate goal of those who
use it. Whether a group is trying to establish a homeland for
a long disenfranchised people or trigger a religious war, if
it uses terrorist means, it is a terrorist group.
Governments
themselves can be involved in terrorism, not only by supporting
sub-national terrorist groups, but also more directly. There
is no doubt that the Nazi Gestapo and the Stalinist era Soviet
Secret Police were terrorist organizations. Nor is there any
doubt that the whole nuclear strategy of mutually assured destruction
fits the definition of terrorism. It was often officially called
a "balance of terror". A balance of terror is still terror.
The
sub-national form of terrorism is a perverse form of theater.
Because shock and fear are the currency of terrorism, terrorists
like to stage acts of destruction in the most spectacular and
public way possible -- to capture media attention and to scare
the daylights out of the public. That the brutal attacks of
September 11 certainly succeeded in doing.
It
is important to remember that terrorism is not a new phenomenon
inside the U.S. Long before 1995, when Timothy McVeigh brought
down the Murrah Federal Building in Oklahoma City with a truck
bomb, the Ku Klux Klan was committing sickening acts of violence
in order to terrorize blacks emancipated by the Civil War. New
York's World Trade Center itself was the target of a deadly
international terrorist attack in 1993. What was different about
the events of September 11, 2001 was chiefly the scale of the
carnage. But that is a very important difference.
Sub-national
terrorists have crossed a critical line. As bad as they were,
the attacks of September 11 could have been worse.
The
Terrorism of Mass Destruction. There are two basic ways
in which the terrorism of mass destruction might yet escalate.
One is that terrorists might somehow get their hands on horrendously
destructive nuclear, chemical or biological weapons. There has
especially been a lot of talk in the last weeks about bioterrorism.
It is not nearly as easy to carry out a massive bioterrorist
attack as many people suppose. It's not just a matter of breeding
virulent germs in a petri dish and getting your hands on a crop
duster. There are serious technical and tactical difficulties.
Still,
bio-weapons have clearly been on the minds of some terrorists
long before September 11. In 1995, four members of the extremist
right wing Minnesota Patriots Council were convicted in federal
court of conspiracy to murder federal agents with ricin, a biological
toxin. They had manufactured enough to kill 1400 people. That
same year, a member of the American white supremacist group
Aryan Nations pled guilty to buying three vials of frozen bubonic
plague bacteria -- by mail.
Mass
destruction by toxic chemicals may be less difficult. Many readily
available chemicals can be used to produce poison gas. But as
the 1995 nerve gas attack by the Japanese Doomsday cult Aum
Shinrikyo illustrates, there are formidable problems here as
well. With millions of dollars and some serious scientific talent
on their side, they successfully manufactured the nerve gas
sarin. But they only managed to kill about a dozen people in
the attack, even though they released the deadly gas in crammed,
rush hour subways.
There
is, however, evidence that this attack was only a dress rehearsal.
When police raided Aum Shinrikyo buildings around Japan after
the attack, they reportedly found a cult newsletter proclaiming
that 90% of the people living in major cities like Tokyo would
die the following year from earthquakes, epidemics -- or poison
gas. They specifically mentioned sarin.
It
is also not a simple matter to build a nuclear weapon. But all
the information necessary to design at least a crude, inefficient
nuclear explosive -- still many times as powerful as the bomb
used in Oklahoma City -- has been available in the public literature
for decades. More than 20 years ago, two undergraduate students
-- one at Princeton, one at MIT -- independently designed workable
nuclear weapons using only publicly available sources. In 1996,
Time magazine reported that 17 scientists at Los Alamos nuclear
weapons labs had been given the assignment of designing AND
building terrorist-type nuclear weapons using "technology found
on the shelves of Radio Shack and the type of nuclear fuel sold
on the black market". They successfully assembled more than
a dozen "homemade" nuclear bombs.
Terrorists
might also be able to steal -- or buy -- a ready-made weapon.
In 1997, on American television, Russian General Alexander Lebed
claimed that Russia had lost track of some 100 "suitcase"
nuclear bombs.
The
other, more familiar route by which terrorists could escalate
the level of destruction would be to follow the same basic strategy
as on September 11 -- using our own technology and equipment
against us. They could stage a conventional terrorist bombing
of an industrial plant that uses or produces highly toxic chemicals,
a nuclear power plant, or a toxic chemical or nuclear waste
storage area. According to the Nuclear Regulatory Commission,
nearly all of the almost 700 bomb-related threats against nuclear
facilities in the U.S. from 1976-1994 were hoaxes. Even so,
such facilities were clearly on the minds of at least some potential
terrorists.
The
fourth jetliner that crashed in Pennsylvania during the barrage
of hijackings on September 11, 2001 was circling back toward
and went down only about 120 miles (about 15 minutes flying
time) from the Three Mile Island nuclear power plant. Maybe
Camp David was not its intended target.
Ending
Terrorism
On
September 11, we learned the hard way that massive military
force does not -- in fact cannot -- prevent devastating terrorist
attack. Like it or not, it is also true that massive military
force is unlikely to be effective in fighting broad-based international
terrorism, after-the-fact. Among the reasons for this are: a)
well-organized terrorist networks are dispersed and multi-headed;
b) the likelihood of killing many more innocent civilians than
terrorists is too high; and c) killing large numbers of innocent
civilians causes their families and friends same kind of anger
and pain we are now feeling, breeds hatred and creates more
terrorists in the future.
In
the short run, the terrorism of mass destruction is most effectively
fought pretty much the same way that we can most effectively
fight any other form of terrorism. It is much more like high
quality police work than military tactics -- a combination of
improved intelligence, greater international cooperation and
a far better understanding of the character of terrorist groups.
Not
all terrorist groups are equally likely to use the techniques
of mass destruction. Lethal Arrogance includes a taxonomy of
terrorists that I believe is helpful in sorting out these differences.
The
essence of the matter is this: groups with well-defined, rational
and limited political goals -- goals such as political independence
for their people -- are likely to limit the amount of violence
they commit. If they overdo it, they will undercut any chance
they have of winning enough public support to achieve their
objectives. Groups with vague ideological goals, driven by motives
that are not rational -- such as ancient traditional hatreds
and violence-prone doomsday religion -- are much more dangerous.
For them, revenge for past injury or the desire to hasten Armageddon,
make violence on a massive scale not only thinkable but even
attractive. They bear close scrutiny.
In
the long run, the best way to end terrorism is to drain the
pool of marginalized and humiliated people from which demagogues
like Osama Bin Laden recruit people who are so frustrated that
they are willing to die to strike a blow against those they
hold responsible for their pain. The only way to permanently
defeat terrorist demagogues is to give those frustrated people
less violent ways to really get their grievances heard, to give
them a sense of dignity and some hope for the future.
That
cannot be done with military strikes or better police work.
It can only be done by helping them develop politically and
economically, by taking their rights just as seriously as we
take the rights of those whose worldview aligns more closely
with our own. No one who feels respected and taken seriously
by the world flies airliners into buildings.
Preventing
Human-Induced Disaster
There
are a number of ways in which our own fallibility can lead to
disaster. For example, Lethal Arrogance catalogues 89 nuclear
weapons-related accidents from 1950-1994 -- an average of one
accident every six months for 45 years! There has also been
at least one major explosion of nuclear waste that heavily contaminated
close to 400 square miles of the former USSR, rendering it uninhabitable.
Chernobyl
and Three Mile Island are only the most public warnings we have
been given in the realm of nuclear power. There have been many
more. And we still do not have a safe and cost-effective way
to store nuclear waste over the long term, yet we generate more
of it day-by-day.
Without
doubt, the most terrible way that human error and technical
failure could cause catastrophe is through accidental nuclear
war. This is too complex and important an issue to do justice
to in the brief time available today. Suffice it to say that
there have been many false warnings of attack that could have
played a key role in unleashing nuclear forces by mistake. For
example, in 1995, Russian warning radars detected the launch
of a rocket from the Norwegian Sea that appeared to be a U.S.
submarine-launched Trident missile targeted at Moscow. The warning
was relayed all the way up to President Yeltsin, who had only
a few minutes to decide whether to launch a nuclear attack in
response. Fortunately, the Russian military determined that
the missile they were tracking was headed far out to sea. The
rocket was American, but it was a scientific probe designed
to study the Northern Lights. The Russian government had been
told of the launch, but apparently through human error, word
never reached key military commanders.
Today,
a decade after the Cold War, we still keep much of the U.S.
nuclear force on high alert, increasing the probability of accidental
war.
The
idea that we can always control
the technologies we create, no matter how powerful or complex
they may be, is a lethal bit of arrogance. Accidents, mistakes,
misunderstandings, oversights are a normal
part of human life, not some bizarre aberration.
The
amazing things that technology can accomplish has given us the
hope that it can solve most of our problems and lead us to a
better future. But the reality is technology cannot solve any
problem whose roots lie in any
other realm of human experience.
Technology
can solve the technical problem of getting astronauts to Mars,
but it cannot solve the socioeconomic and political problem
of getting food to those who are chronically hungry. It can
solve the technical problem of providing people with huge amounts
of information quickly, but it cannot solve the educational
problem of understanding what that information means and how
to use it wisely. Technology can give us the technical means
to threaten and destroy each other, but it cannot solve the
social, psychological, economic and political problem of security.
When
we understand this critical distinction, we will finally realize
that we do not need nuclear weapons, nuclear power or most of
the other dangerous technologies that expose us to human-induced
disaster.
There
are effective and much less dangerous
ways to achieve security than through the threat or use of weapons
of mass destruction. There ARE much safer energy sources than
nuclear power. There are ways
to protect crops from insects and disease, to produce plastics
and other desirable materials without relying on dangerous technologies.
There are better ways to do all these things, ways that allow
us to make very big mistakes without causing disaster.
There
is no such thing as life without risk. We would be foolish indeed
to try to achieve it. But we would be even more foolish to
forget who we are, and so, to endanger all that
we have accomplished. We must find a way to rid the earth of
nuclear weapons, replace nuclear power with alternative, environmentally
benign energy technologies, and phase out highly toxic chemicals.
We have shown that we are brilliant enough to create extraordinary
technologies. Now we must show we are wise enough to choose
among them well.
Posted
with permission
Reviewed: 8/19/05
|
