Citation: 89 Wash. L. Rev. 117 2014
Content downloaded/printed from HeinOnline
Fri Jul 21 11:53:54 2017
-- Your use of this HeinOnline PDF indicates your acceptance
of HeinOnline's Terms and Conditions of the license
agreement available at http://heinonline.org/HOL/License
-- The search text of this PDF is generated from
uncorrected OCR text.
-- To obtain permission to use this article beyond the scope
of your HeinOnline license, please use:
Copyright Information
MACHINES WITHOUT PRINCIPALS: LIABILITY RULES
AND ARTIFICIAL INTELLIGENCE
David C. Viadeck
INTRODUCTION
The idea that humans could, at some point, develop machines that
actually "think" for themselves and act autonomously has been
embedded in our literature and culture since the beginning of
civilization.' But these ideas were generally thought to be religious
expressions-what one scholar describes as an effort to forge our own
Gods2-or pure science fiction. There was one important thread that tied
together these visions of a special breed of superhuman men/machines:
They invariably were stronger, smarter, and sharper analytically; that is,
superior in all respects to humans, except for those traits involving
emotional intelligence and empathy. But science fiction writers were of
two minds about the capacity of super-smart machines to make life
better for humans.
One vision was uncritically Utopian. Intelligent machines, this
account goes, would transform and enlighten society by performing the
mundane, mind-numbing work that keeps humans from pursuing higher
intellectual, spiritual, and artistic callings.3 This view was captured in
the popular animated 1960s television show The Jetsons.4 As its title
* Professor of Law, Georgetown University Law Center. The author wishes to thank his
Georgetown Law colleague Girardeau Spann and Perkins Coie partner Albert Gidari, Jr., for their
thoughts on the article and the staff of the Washington Law Review, especially Lauren Watts, Laura
Powell, Brooke Howlett, and Brian Epley, for their research and editorial assistance.
1. See, e.g., PAMELA MCCORDUCK, MACHINES WHO THINK: A PERSONAL INQUIRY INTO THE
HISTORY AND PROSPECTS OF ARTIFICIAL INTELLIGENCE, at xxiii iv (2004).
2. Id. at 413.
3. See Anne Trafton, Assistive Robot Adapts to People, New Places, MIT TECHTALK, Apr. 11,
2007, at 3 (reporting developments from MIT's Humanoid Robotics Group in perfecting "Domo," a
robot that can adapt to situations to assist people with everyday chores, everyday life, and everyday
work), availableathttp://web.mit.edulnewsoffice/2007/techtalk51-23.pdf.
4. The history of The Jetsons can be found at various websites, including, The Jetsons,
WIKIPEDIA, http://en.wikipedia.org/wiki/The Jetsons (last visited Feb. 12, 2014), and, Hanna
Barbera Wiki, The Jetsons, WIKIA, http://hanna-barbera.wikia.com/wikilThe Jetsons (last visited
Feb. 12, 2014).
117
WASHINGTON LAW REVIEW
suggests, the show's vision is decidedly futuristic. The main character,
George Jetson, lives with his family in a roomy, bright, and lavishly
furnished apartment that seems to float in the sky. George and his family
travel in a flying saucer-like car that drives itself and folds into a small
briefcase. All of the family's domestic needs are taken care of by Rosie,
the robotic family maid and housekeeper, who does the household
chores and much of the parenting.s George does "work." He is employed
as a "digital index operator" by Spacely's Space Sprockets, which makes
high tech equipment. George often complains of overwork, even though
he appears to simply push buttons on a computer for three hours a day,
three days a week.6 In other words, the Jetsons live the American dream
of the future.
In tangible ways, this Utopian vision of the partnership between
humans and highly intelligent machines is being realized. Today,
supercomputers can beat humans at their own games. IBM's "Deep
Blue" can beat the pants off chess grand-masters, while its sister-supercomputer
"Watson" can clobber the reigning Jeopardy champions. But
intelligent machines are more than show. Highly sophisticated robots
and other intelligent machines perform critical functions that not long
ago were thought to be within the exclusive province of humans. They
pilot sophisticated aircraft; perform delicate surgery; study the landscape
of Mars; and through smart nanotechnology, microscopic machines may
soon deliver targeted medicines to areas within the body that are
otherwise unreachable.8 In every one of these examples, machines
perform these complex and at times dangerous tasks as well as, if not
better than, humans.
But science fiction writers also laid out a darker vision of intelligent
5. See Trafton, supra note 3 (reporting that "Domo" is based in part on "Rosie").
6. See generally Harry McCracken, The Jetsons Turns 50: How the FutureLooked in 1962, TIME
(Sept. 23, 2012), http://techland.time.com/2012/09/23/the-jetsons-turns-fifty/; Steve Waldis,
Keeping Up With the Jetsons, WIRED (Jan. 7, 2014, 11:00 AM), http://insights.wired.com/
profiles/blogs/keeping-up-with-the-jetsons (comparing modern advances in "smart home"
technology to The Jetsons); Troy Wolverton, Your Home Could Soon Be Like the Jetsons', SAN
JOSE MERCURY NEWS (Jan. 11, 2014, 7:10 PM), http://www.mercurynews.com/
troy-wolverton/ci 24893407/wolverton-your-home-could-soon-be-like-jetsons (same).
7. Daniel C. Dennett, Higher Games, MIT TECH. REV. (Aug. 15, 2007) (analyzing the
significance of "Deep Blue's" win over chess genius Gary Kasparov in 1997),
http://www.technologyreview.com/review/408440/higher-games/; John Markoff, ComputerWins on
'Jeopardy!': Trivial, It's Not, N.Y. TIMES, Feb. 17, 2011, at Al, available at
http://www.nytimes.com/2011/02/17/science/17jeopardy-watson.html?pagewanted=all (describing
"Watson's" commanding success over two former Jeopardy champions).
8. See, e.g., Sha Jin & Kaiming Ye, Nanoparticle-MediatedDrug Deliveryand Gene Therapy,23
BIOTECHNOLOGY PROGRESS 32 (2007).
118 [Vol. 89:117
MACHINES WITHOUT PRINCIPALS
machines and feared that, at some point, autonomously thinking
machines would turn on humans. Some of the best science fiction
expresses this dystopian view, including Stanley Kubrick's 1968 classic
film 2001: A Space Odyssey.9 The film's star is not the main character,
"Dave" (Dr. David Bowman, played by Keir Dullea), or "Frank" (Dr.
Frank Poole, played by Gary Lockwood), who are astronauts on a secret
and mysterious mission to Jupiter. Instead, the character who rivets our
attention is HAL 9000,o the all-knowing supercomputer who controls
most of the ship's operations, but does so under the nominal command
of the astronauts. The complexity of the relationship between man and
the super-intelligent machine is revealed early in the film. During a premission
interview, HAL claims that he is "foolproof and incapable of
error,"" displaying human-like hubris. And when Dave is asked if HAL
has genuine emotions, he replies that HAL appears to, but that the truth
is unknown. 12
Once the mission begins, tensions between HAL and the astronauts
start to surface. HAL wants the astronauts to tell him the details of the
highly secret mission, but Dave and Frank refuse. In fact, they too do not
know. Soon thereafter, HAL warns of the impending failure of a critical
antenna on the spaceship's exterior. Starting to have doubts about HAL,
Dave and Frank lock themselves in an evacuation vehicle to ensure that
HAL cannot overhear their conversation; HAL reads their lips through
the vehicle's window. Dave and Frank decide to follow HAL's advice
and replace the antenna, but in the event that HAL is wrong about the
antenna's defect, they agree to shut HAL down. Frank goes on a
spacewalk to replace the antenna, and, as he had planned, HAL seizes
the moment to kill off the humans. He first severs Frank's oxygen hose
and sets him adrift in space. Dave vainly tries to rescue Frank, and as
soon as Dave leaves the spacecraft, HAL turns off the life-support
9. 2001: A SPACE ODYSSEY (Metro-Goldwyn-Mayer Pictures 1968).
10. HAL is an acronym for Heuristically programmed Algorithmic computer. See HAL 9000,
WIKIPEDIA, http://en.wikipedia.org/wiki/Heuristically programmed algorithmic computer (last
visited Feb. 12, 2014); KuBRICK 2001: THE SPACE ODYSSEY EXPLAINED,
http://www.kubrick2001.com/ (last visited Feb. 17, 2014). Because HAL is portrayed in the film as
a living being, I will use animate, not inanimate, pronouns to refer to HAL.
11. 2001: A Space Odyssey: Quotes, IMDB, http://www.imdb.com/title/tt0062622/quotes (last
visited Feb. 26, 2014) (HAL's complete answer to the interviewer's question is as follows: "The
9000 series is the most reliable computer ever made. No 9000 computer has ever made a mistake or
distorted information. We are all, by any practical definition of the words, foolproof and incapable
of error.").
12. Id. (Dave's full response to the question is as follows: "Well, he acts like he has genuine
emotions. Um, of course he's programmed that way to make it easier for us to talk to him. But as to
whether he has real feelings is something I don't think anyone can truthfully answer.").
2014] 119
WASHINGTON LAW REVIEW
system for the three remaining crew members, who were in suspended
animation. HAL then refuses to let Dave back onto the spaceship, telling
Dave that the plan to deactivate him jeopardizes the mission. Ultimately,
Dave makes his way back onto the spaceship and starts shutting HAL
down. All the while, as HAL regresses, HAL pleads with Dave to stop,
and finally expresses fear of his demise.13
The question one might ask at this point is what relevance does 2001:
A Space Odyssey have to liability rules for autonomous thinking
machines? The answer is quite a bit. Today's machines, as path-breaking
as they are, all have a common feature that is critical in assessing
liability. In each case, the machine functions and makes decisions in
ways that can be traced directly back to the design, programming, and
knowledge humans embedded in the machine.14 The human hand
defines, guides, and ultimately controls the process, either directly or
because of the capacity to override the machine and seize control. As
sophisticated as these machines are, they are, at most, semi-autonomous.
They are tools, albeit remarkably sophisticated tools, used by humans.
Where the hand of human involvement in machine decision-making is
so evident, there is no need to reexamine liability rules. Any human (or
corporate entity that has the power to do things that humans do, enter
into contracts, hire workers, and so forth) that has a role in the
13. See 2001: A SPACE ODYSSEY, supra note 9; 2001: A Space Odyssey, Quotes, IMDB,
http://www.imdb.com/title/tt0062622/quotes (last visited Feb. 26, 2014) ("I'm afraid. I'm afraid,
Dave. Dave, my mind is going. I can feel it. I can feel it. My mind is going. There is no question
about it. I can feel it. I can feel it. I can feel it. I'm a ... fraid."). The same intelligent machineversus-humans
theme was explored with far less nuance but much more mayhem in the 1984 film
The Terminator,which tells the story, set in 2029, of how an artificially intelligent defense network,
"Skynet," becomes self-aware and decides to wipe-out mankind through a nuclear holocaust. Skynet
nearly succeeds. Only a few humans survive, and they form a resistance movement. Over time, the
resistance gains enough strength that it is on the verge of beating the machines. To regain the
offensive, Skynet sends the "Terminator," played by Arnold Schwarzenegger (people forget that he
was the villain in the first film), back to 1984 to kill Sarah Connor, then pregnant with John Connor,
who grows up to lead the resistance against Skynet. Much entertaining violence ensues, and the film
ends inclusively. See THE TERMINATOR (Orion Pictures Corp. 1984). John Connor and his mother
survive, in part because the resistance sends its own highly intelligent robot/warrior back in time to
defend them, but there is no doubt that they will face another onslaught from Skynet terminators. Id.
These onslaughts come in an unending number of sequels involving ever more intelligent and
dangerous robotic assassins, but John Connor nonetheless eludes them all, mainly because of help
from a highly skilled robot warrior sent by the resistance to protect him (played by Arnold
Schwarzenegger). See, e.g., TERMINATOR 2: JUDGMENT DAY (TriStar Pictures 1991); TERMINATOR
3: THE RISE OF THE MACHINES (Warner Bros. Pictures, Columbia Pictures 2003). But the film's
basic point is a warning to humans: Be careful what we wish for. Artificially intelligent machines
that have the capacity to "think" for themselves should not be trusted because they may turn on
humans.
14. See infra notes 15-16 and accompanying text.
120 [Vol. 89:117
2014] MACHINES WITHOUT PRINCIPALS 121
development of the machine and helps map out its decision-making is
potentially responsible for wrongful acts-negligent or intentionalcommitted
by, or involving, the machine. 5
The reason, of course, is that
these machines, notwithstanding their sophistication, have no attribute of
legal personhood. They are agents or instruments of other entities that
have legal capacity as individuals, corporations, or other legal "persons"
that may be held accountable under the law for their actions.
But the fully autonomous machines that at some point will be
introduced into the marketplace may be quite different, and for that
reason, society will need to consider whether existing liability rules will
be up to the task of assigning responsibility for any wrongful acts they
commit. The first generation of fully autonomous machines-perhaps
driver-less cars and fully independent drone aircraft-will have the
capacity to act completely autonomously. They will not be tools used by
humans; they will be machines deployed by humans that will act
independently of direct human instruction, based on information the
machine itself acquires and analyzes, and will often make highly
consequential decisions in circumstances that may not be anticipated by,
let alone directly addressed by, the machine's creators.16 Artificial
15. There has been extensive litigation over the safety of surgical robots, especially the "da
Vinci" robot. Not surprisingly, no one has claimed that the robot itself bears any liability; the claims
all proceed on some form of agency theory. See, e.g., O'Brien v. Intuitive Surgical, Inc., No. 10 C
3005, 2011 WL 304079, at *1 (N.D. Ill. Jul. 25, 2011) (granting summary judgment to robot's
manufacturer); Mracek v. Bryn Mawr Hosp., 610 F. Supp. 2d 401, 402 (E.D. Pa. 2009) (same),
aff'd, 363 F. App'x 925 (3d Cir. 2010).
16. It is useful to contrast the complexity of driver-less cars to what we think of as highly
sophisticated machines in widespread use today. For instance, most commercial airplanes have
"auto pilots," which shift control of the aircraft to a computer that "flies" the plane. So too do most
vessels. But in comparison to driver-less cars, the autopilot devices perform a relatively simple set
of tasks. Autopilots keep the plane or vessel on a course determined by the pilots, and do so by
controlling for minor variations in winds and currents, but generally without reference to other
traffic. For that reason, pilots have a duty to remain vigilant-while the machine may have the
controls, the pilots are responsible for monitoring other traffic and ensuring that the autopilot is
working correctly. See, e.g., In re Korean Air Lines Disaster of Sept. 1, 1983, 932 F.2d 1475, 1478
(D.C. Cir. 1991); Boucvalt v. Sea-Trac Offshore Servs., Inc., 06-103 (La. App. 5 Cir. 10/17/06);
943 So. 2d 1204, 1208. In contrast, for autonomous cars, one key goal is to reduce the oversight
required by the driver, both to maximize safety and to capture the possibility of increased
productivity during long commutes. The systems that control autonomous cars are required to
navigate through complex and rapidly changing environments-e.g., traffic, weather, detours and
the like-and are responsible for making critical decisions-e.g., what route to take, what lane to be
in, what exit to take, and so forth. See Dylan LeValley, Note, Autonomous Vehicle LiabilityApplication
of Common Carrier Liability, 36 SEATTLE U. L. REv. 5, 7 (2013). And there may be
good reasons why the designers of driver-less cars would want to discourage human drivers from
intervening. A driver intervening to prevent what he perceives to be a possible accident may be less
equipped to handle the situation than the car's autonomous driving system; he may not have
monitored blind spots, heard the siren of an approaching emergency vehicle, or been privy to a
WASHINGTON LAW REVIEW [Vol. 89:117
intelligence theorists distill the concept of full autonomy down to the
paradigm of machines that "sense-think-act" without human
involvement or intervention.1
And Oxford Professor Nick Bostrom, an
eminent futurist, goes as far as to suggest that machines "capable of
independent initiative and of making their own plans ... are perhaps
more appropriately viewed as persons than machines.""
Assuming that this description of the capabilities of such machines is
accurate, the key conceptual question that autonomous thinking
machines will pose is whether it is fair to think of them as agents of
some other individual or entity, or whether the legal system will need to
decide liability issues on a basis other than agency. To be sure, it is hard
to conceptualize a machine as being anything other than an agent of a
person, be it a real person or an entity with legal personhood. But there
is another argument that is worth exploring, namely that concepts of
agency may be frayed, if not obliterated, by autonomous thinking
machines, even those that are not truly "sentient." Let us go back to
HAL. At some point before he turns murderous, HAL became an
"agent" of no one. An agent who decides to go on his own frolic and
detour, defying the instructions of his principal, is no longer an agent
communication sent by another vehicle in close proximity warning of some danger, and his reaction
time is certainly going to be slower than the car's. See id. at 16 (pointing out that Air France flight
447 crashed over the Atlantic because of pilot error. When the plane's autopilot sensors stopped
working due to the build-up of ice, the pilots took over control of the plane, and reacted in a way
that exacerbated the icing problem, causing the plane to crash).
17. UGO PAGALLO, THE LAW OF ROBOTS: CRIMES, CONTRACTS AND TORTS 2 (2013). This Essay
draws significantly on Professor Pagallo's path-breaking work, which broadly and ambitiously
synthesizes the technical, philosophical, and legal questions surrounding the advent of fully
autonomous, artificially intelligent machines.
18. Nick Bostrom, When Machines Outsmart Humans, 35 FUTURES 759, 763 (2003). Artificial
intelligence theorists use the term "singularity" or "technical singularity" to describe the moment in
time, purely hypothetical at this point, when machines exceed human intelligence. At that point,
these theorists argue, machines will become fully sentient, and they will pose a raft of complex
philosophical and legal questions with which society will have to wrestle. See, e.g., NICK BOSTROM
& MILAN CIRKOVIC, GLOBAL CATASTROPHIC RISKS (2008); RAY KURZWEIL, THE SINGULARITY IS
NEAR: WHEN HUMANS TRANSCEND BIOLOGY 135-36 (2005); HANS MORAVEC, ROBOT: MERE
MACHINE TO TRANSCENDENT MIND 61 (1999). Ray Kurzweil predicts that "singularity" is now
within reach, and will be achieved within fifteen years. See Ben Rossington, Robots 'SmarterThan
Humans Within 15 Years,' PredictsGoogle's Artificial Intelligence Chief,MIRROR NEWS (Feb. 2,
2014), http://www.mirror.co.uk/news/technology-science/technology/ray-kurzweil-robots-smarterhumans-3178027.
This Essay does not suggest that the first generation of fully autonomous,
artificially intelligent machines will necessarily meet the various definitions of "singularity" that
have been propounded. They probably will not. For our purposes, it is sufficient to focus on
machines that are capable of "learning" to the point where they can take actions in ways that are not
necessarily pre-ordained by the programs that enable them, and thus present the difficult agency
problems discussed in Part II, infra.
122
2014] MACHINES WITHOUT PRINCIPALS 123
under any conventional understanding of the law.19 And HAL plainly
detoured. HAL was given the ability to think and act independently, so
much so that he "decided" to violate the first rule of robotics: That is,
machines must do no harm to humans or to humanity.20 By deciding to
harm humans, HAL at least arguably (if not decisively) terminated his
status as an agent.
To be sure, HAL's capacity to violate that rule could be attributable to
a manufacturing defect, design flaw, or the result of poor programming.
In that case, existing legal principles would almost certainly place
liability on the shoulders of HAL's creators. But suppose HAL's
wrongful conduct was not a manufacturing, design, or programming
flaw, but was instead an unforeseen byproduct of teaching machines to
"think." 21 Sentient beings often choose to break rules, and all disciplines
interested in artificial intelligence-technologists, scientists, ethicists,
and philosophers alike-have long worried that giving machines the
capacity to "think" autonomously necessarily gives them the capacity to
act in ways that may be contrary to the "rules" they are given.22 That
conclusion, of course, lies at the heart of the academic debate over the
wisdom of building artificially intelligent machines in the first place and
is the animating force behind many science fiction classics.
19. See generally RESTATEMENT (THIRD) OF AGENCY § 7.07 (2006) ("An employee acts within
the scope of employment when performing work assigned by the employer or engaging in a course
of conduct subject to the employer's control. An employee's act is not within the scope of
employment when it occurs within an independent course of conduct not intended by the employee
to serve any purpose of the employer."); Lev v. Beverly Enters.-Mass., Inc., 929 N.E.2d 303, 308
(Mass. 2010).
20. The most famous exposition of the "law" of robots comes from Isaac Asimov's I Robot,
where he lays out the Three Laws of Robotics: (1) a robot may not injure a human being, or,
through inaction, allow a human being to come to harm; (2) a robot must obey the orders given it by
humans, except where such orders would conflict with the First Law; (3) a robot must protect its
own existence, as long as such protection does not conflict with the First or Second Law. ISAAC
AsIMov, I,ROBOT 37 (1950). See generally ARTHUR C. CLARKE, 2001: A SPACE ODYSSEY (1968).
21. Once an autonomous machine decides for itself what course of action it should take, the
agency relationship becomes frayed or breaks altogether. See RESTATEMENT (THIRD) OF AGENCY
§ 7.07 (2006); id. § 7.03 (describing that a principal is subject to vicarious liability for an agent's
actions only when the agent is acting within the scope of employment).
22. See PAGALLO, supra note 17, at 4, 37-44, 49 55, 119 21. Oxford University Professor Nick
Bostrom makes a similar argument. Nick Bostrom, The Superintelligent Will: Motivation and
Instrumental Rationality in Advanced ArtificialAgents, 22 MINDS & MACHINES 71-84 (2012); see
also Bill Joy, Why the FutureDoesn'tNeed Us, WIRED (Apr. 2000), http://www.wired.com/wired/
archive/8.04/joy.html.
23. This is not pure speculation; there is already emerging evidence that highly "intelligent"
autonomous machines can learn to "break" rules to preserve their own existence. See Bostrom,
supra note 22, at 77; John Markoff, Scientists Worry Machines May OutsmartMan, N.Y. TIMES,
July 26, 2009, at Al, available at http://www.nytimes.com/2009/07/26/science/26robot.
WASHINGTON LAW REVIEW [Vol. 89:117
Assuming that an inevitable byproduct of giving a machine the
capacity to "think" carries with it a risk that the machine will break the
rules, 24 how would the law apply to HAL? Suppose Dave returns to
Earth and demands compensation for the wrongful deaths of Frank and
the three other crewmembers HAL murdered? 25 Unless the law is
willing to invest an autonomous thinking machine like HAL with legal
personhood, he is beyond the reach of the law. As one court observed,
"robots cannot be sued," even though "they can cause devastating
damage."26
The introduction of highly intelligent, autonomous machines may
prompt reconsideration of that rule. After all, there is no a priori reason
why truly autonomous machines should not be accorded some formal
legal status, making them, like corporations and certain trusts, "persons"
in the eyes of the law and thus subject to suit.27 Perhaps justice would be
html? r=1&ref=todayspaper; Jason Mick, New Navy-Funded Report Warns of War Robots Going
"Terminator," DAILY TECH (Feb. 17, 2009, 8:58 AM), http://www.dailytech.com/New%
20Navyfunded%20Report%20Warns%20of%2OWar%20Robots%20Going%20Terminator/
articlel4298.htm.
24. See Bostrom, supranote 22, at 77. As noted above, for our purposes, it is not essential that the
machine has the capacity to actually choose to break a "rule"; it is enough that the machine's
programming does not necessarily determine how the machine will act in all situations, leaving the
machine to "learn" how to make decisions when confronted with a situation not within the
contemplation of the machine's programmers.
25. Here we put aside any question of criminal liability. At some point, though, the science of
artificial intelligence may evolve to the point where concepts of criminality will have to be applied
to highly intelligent, autonomous machines. And perhaps that time will come sooner than many
think. That was a central message of the 1982 science-fiction thriller, THE BLADE RUNNER (Warner
Bros. 1982), which was set in 2019, and involved a detective played by Harrison Ford chasing down
"replicants"-life-like robots-who were, like HAL, killing humans. Id.; see also BLADE RUNNER:
30TH ANNIVERSARY, http://bladerunnerthemovie.warnerbros.com/ (last visited Feb. 12, 2014).
26. United States v. Athlone Indus., Inc., 746 F.2d 977, 979 (3d Cir. 1984) (discussing how the
manufacturer of a defective robotic pitching machine is liable for civil penalties for the machine's
defects). Of course, the court's quip was not completely accurate. The court's point was that,
because the robot lacked legal capacity, the robot could not be sued in personam. But the court's
conclusion necessarily means that an in rem or quasi in rem action against a robot would be
maintainable, precisely because the robot was an "object" rather than a legal "person."
27. As discussed in more depth infra, this proposal is not as far-fetched as it may seem, and may
not require an upheaval in current law to achieve. Conceptualize a highly intelligent machine as one
might think of a corporation, which through the income it earned or its intrinsic value was
capitalized. There is no reason why the machine itself could not bear liability in case of
wrongdoing. For driver-less cars, suppose as a condition of the sale of a driver-less car, state law
requires the "car," not necessarily its purchaser, but maybe a pool consisting of the manufacturer,
suppliers, and the purchaser, to obtain insurance sufficient to address any likely issue of liability. To
ensure that the car remains insured, the car itself would be the policy-holder, and could not operate
without valid insurance. The car might have to have a "kill switch" that would automatically disable
the car in the event its insurance lapsed. But the point here is simply that the law could evolve to
bestow "personhood" on machines, just as it has done for corporations. See Santa Clara Cnty. v. S.
124
MACHINES WITHOUT PRINCIPALS
served to put HAL in the dock. Until then, the question will be who, if
anyone, should be held to account? Would it be fair to hold liable the
companies that designed, programmed, or manufactured HAL 9000,
even though they embedded in HAL's "thinking" systems the first rule
of autonomous machines-i.e., never harm a human-and even though
the evidence strongly suggests that HAL "taught" himself to defy their
instructions? Or should the creators of machines that have the capacity
to "think" be held strictly liable whenever anything goes wrong? If so,
on what theory? The theory that the wrongful conduct itself is proof of a
defect? Or on an insurance-based theory that the creators are in a better
economic position to absorb the cost of the injury than the person
harmed?
As fanciful as it may seem, these are by no means idle questions.
Machines that operate and make decisions independently will at some
point be introduced into the market. The first truly autonomous artificial
intelligence devices that may test the adequacy of current liability rules
may be cars designed to be driver-less, or at least to give human drivers
the option to let the car drive itself.28 Google's driver-less cars have been
test-driven on public roads for several years and have logged hundreds
of thousands of miles.29 Although there is a human sitting in the
"driver's seat," the human is, for the most part, playing the role of potted
Pac. R. Co., 118 U.S. 394 (1886) (observing that "corporations" may be "persons" for the purposes
of the Fourteenth Amendment). My Georgetown colleague, Lawrence B. Solum, first floated this
idea more than two decades ago. See Lawrence B. Solum, Legal Personhood for Artificial
Intelligences, 70 N.C. L. REV. 1231 (1992).
28. Gary Marcus, Moral Machines, NEW YORKER (Nov. 27, 2012), http://www.newyorker.com/
online/blogs/newsdesk/2012/11/google-driverless-car-morality.html. As Professor Marcus observes,
it may be that truly autonomous military robots or drones will be introduced first. See also Jason
Palmer, Call for Debate On Killer Robots, BBC NEWs (Aug. 3, 2009, 7:09 PM),
http://news.bbc.co.uk/2/hi/8182003.stm (explaining that the capacity to introduce drones that are
able to make firing decisions without direct human oversight will soon be available). But that
example does not lend itself to an exploration of the application of product liability rules to
autonomous machines, since the law of war will plainly trump the law of products liability. See
generally John Markoff, War Machines: Recruiting Robots for Combat, N.Y. TIMES, Nov. 28, 2010,
at Al [hereinafter Markoff, War Machines]; Joshua Foust, Soon, Drones May Be Able to Make
Lethal Decisions on Their Own, NAT'L J. (Oct. 8, 2013), http://www.nationaljournal.com/national-
security/soon-drones-may-be-able-to-make-lethal-decisions-on-their-own-20131008.
29. See Jeffrey K. Gurney, Note, Sue My CarNot Me: ProductsLiability andAccidents Involving
Autonomous Vehicles, 2013 U. ILL. J. L. TECH. & POL'Y 247, 249 n.6. Google is not the only
company engaged in the arms-race to develop a truly driver-less vehicle. Many other companies are
engaged in the same enterprise. See, e.g., Kevin Funkhouser, Note, Paving the Road Ahead:
Autonomous Vehicles, ProductsLiability, and the Needfor a New Approach, 2013 UTAH L. REV.
437, 437 38; Bryant Walker Smith, Proximity-Driven Liability, 102 GEO. L.J. 18 (forthcoming
2014); Andrew P. Garza, Note, "Look Ma No Hands!": Wrinkles and Wrecks in the Age of
Autonomous Vehicles, 46 NEW ENG. L. REV. 581, 587 (2012).
2014] 125
WASHINGTON LAW REVIEW
plant. The car is driven by systems that use an array of radar and laser
sensors, cameras, global positioning devices, many complex analytical
programs and algorithms (and other devices far beyond my
comprehension) to actually drive the car in much the same way that
humans do, only better.30 The car "watches" the road, constantly looks
out for other cars, pedestrians, obstructions, detours, and so forth, and
adjusts its speed and course to account for traffic, weather, and every
other factor that affects the safe operation of the vehicle. And they are
programmed to avoid a collision with a pedestrian, another vehicle, or an
obstacle.31 Google has claimed that its driver-less cars have yet to have
an accident (except when one of its cars was rear-ended)-an impressive
record. 32
It is hardly surprising that Google's experience has thus far validated
the hypothesis that autonomous cars would perform far more safely than
cars driven by humans. Driving is risky because drivers are humans.33
Humans inject risks that can be eliminated, or at least considerably
mitigated, by intelligent machines. After all, the Google system that
drives cars "sees" everything in the vicinity; reacts at speeds humans
cannot match; and constantly checks the performance of every
component in the vehicle to ensure that it is functioning properly.
What's more, the system never gets drowsy or falls asleep, does not
drive drunk, does not eat Big Macs or drink hot coffee, does not get
distracted, does not talk on the phone or text, and does not get road
rage. 4 The system focuses solely on the one activity it is designed for:
Driving from point A to point B as safely and efficiently as possible.
Like human drivers, the machines that drive Google cars will on
30. See JAMES M. ANDERSON ET AL., AUTONOMOUS VEHICLE TECHNOLOGY: A GUIDE FOR
POLICYMAKERs 58 95 (RAND Corp. 2014) [hereinafter RAND REPORT], available at
http://www.rand.org/pubs/research reports/RR443-1.
31. See Funkhouser, supra note 29, at 437-38, 443-44 (discussing the technology that Google
cars use to avoid collisions); Smith, supra note 29, at 18-19; Garza, supranote 29, at 587-88.
32. Burkhard Bilger, Auto Correct:Has the Self-Driving Car at Last Arrived?, NEW YORKER
(Nov. 25, 2013), http://www.newyorker.com/reporting/2013/11/25/131125fa fact bilger?
currentPage=all; John Markoff, Google Cars Drive Themselves in Traffic, N.Y. TIMES, Oct. 10,
2010, at Al, available at http://www.nytimes.com/2010/10/10/science/10google.html?
pagewanted=all.
33. Sven A. Beiker, Legal Aspects ofAutonomous Driving,52 SANTA CLARA L. REV. 1145, 1149
(2012) (pointing out that "[d]river error is by far (95%) the most common factor implicated in
vehicle accidents"); Markoff, War Machines, supra note 28 (describing the passive role of the
"driver" in a self-driving car); see also RAND REPORT, supranote 30, at 12-16.
34. See, e.g., Beiker, supra note 33, at 1149 (identifying human error as the primary catalyst in
the vast majority of car accidents); Funkhouser, supra note 29, at 437-38, 443-44 (discussing
Google's attempts-through its self-driving cars-to eliminate human error in driving).
126 [Vol. 89:117
MACHINES WITHOUT PRINCIPALS
occasion encounter unexpected events that call for snap judgments: a
child darting in front of a car; a tree limb crashing down just a few yards
ahead; a car running a red light; or a patch of black ice that is
undetectable on a moonless night. There are liability rules that come into
play when, as a consequence of any of these unexpected events, there is
injury to humans or damage to property. No matter how well-designed
and programmed self-driving cars are, factors beyond the machine's
control virtually guarantee that at some point the car will have an
accident that will cause injury of some kind, and will act in ways that are
not necessarily ordained by their programming. The question I have
been asked to address in this symposium is: What liability rules should
society adopt to govern artificially intelligent machines, such as the
driver-less cars that soon may be right in front of us on the highway?
Answering this question requires that two different issues be
addressed. The first is how to apply the law of products liability on the
assumption that any liability concern with the machine is the result of
human (but not driver) error-that is, a design or manufacturing defect,
an information defect, or a failure to instruct humans on the safe and
appropriate use of the product. In my view, the application of these
reasonably settled principles is a straightforward one, and there is no
justification for treating even autonomous thinking machines differently
than any other machine or tool a human may use, except, perhaps,
holding them to a higher standard of care.
The second question comes into play if, and only if, fully autonomous
machines cause injury in ways wholly untraceable and unattributable to
the hand of man.35 This question, in my view, crystalizes the HAL
problem. It is fair to assume that, if driver-less cars become the norm,
there will be accidents, perhaps few and far between, that cannot fairly
be attributed to a design, manufacturing, or programming defect, and
35. Again, this is an assumption, but because the vehicle's control system itself is so highly
automated and continually monitors its own performance, it is likely that the vehicle's own data will
shed considerable light on the cause of accidents. Indeed, most cars sold today contain "black
boxes" that monitor the car's performance and often provide important clues in ascertaining the
causes of accidents. See, e.g., Jaclyn Trop, A Black Box for Car Crashes, N.Y. TIMES, July 21,
2013, at Bi, available at http://www.nytimes.com/2013/07/22/business/black-boxes-in-cars-aquestion-of-privacy.html.
There is a related consideration that is worth noting: Highly autonomous
cars will communicate with cars in their vicinity to ensure that a safe distance is maintained
between the two vehicles. In that case, it may be that data from other, nearby vehicles may shed
light on vehicle failures. See RAND REPORT, supra note 30, at 66-68, 79-81; Robert B. Kelly &
Mark D. Johnson, Defining a Stable, Protectedand Secure Spectrum Environmentfor Autonomous
Vehicles, 52 SANTA CLARA L. REV. 1271, 1310 (2012) (discussing "vehicle-to-vehicle collision
avoidance" communications).
2014] 127
WASHINGTON LAW REVIEW
where even an inference of defect may be hard to justify.36 What should
be the rule at that point, especially where the car acts in a way that is at
odds with the instructions of its creators? Tort law is ordinarily
unwilling to let people injured through no fault of their own bear costs
imposed by others. So the question then becomes, "Who pays?" The
only feasible approach, it would seem, would be to infer a defect of
some kind on the theory that the accident itself is proof of defect, even if
there is compelling evidence that cuts against a defect theory. There is
precedent for courts making such an inference, which is simply a
restatement of res ipsa loquitor. If that is the right choice to make (and I
argue it is), then there is the secondary question of how, if at all, should
the law apportion liability among designers, programmers,
manufacturers, and others involved in the vehicle's creation? Or, as
suggested above, should liability simply be assigned to the vehicle
itself?
Of course, the solution suggested by existing law would be to hold the
vehicle's manufacturer liable and let the manufacturer seek indemnity or
contribution from other potentially responsible parties, if any.37 But that
approach may be nothing more than an empty gesture. If it is in fact
impossible to identify the cause of the accident, then the manufacturer
would likely have no reasonable grounds for an indemnity or
contribution action, and would thus be saddled with the entire
judgment.38 That result might make sense if the manufacturer is in the
best position to bear the loss. Otherwise, it might be fairer to apportion
responsibility among all of the parties that participated in building and
maintaining the vehicle's autonomous systems, on the ground that the
cost of error is better spread among all potentially responsible parties or
36. See generally RESTATEMENT (THIRD) OF TORTS: PRODUCTS LIABILITY § 3(a) (1998)
(providing for an inference based on concepts of res ipsa loquitor where the plaintiff can show that
the product failure "was of a kind that ordinarily occurs as a product defect"); see also In re Toyota
Motor Corp. Unintended Acceleration Mktg., Sales Practices, & Prods. Liab. Litig., F. Supp.
2d , 2013 WL 5763178 (C.D. Cal. Oct. 7, 2013); Douglas A. Kysar, The Expectations of
Consumers, 103 COLUM. L. REv. 1700, 1721 & nn.89 90 (2003). These sources and others make
clear that an inference can take a plaintiff only so far; the plaintiff still has to prove that the failure
was of the kind ordinarily seen with design defects. An otherwise inexplicable failure, which is not
fairly described as "ordinary," would likely not qualify under this standard.
37. See generally Larsen v. Gen. Motors Corp., 391 F.2d 495 (8th Cir. 1968) (applying
Minnesota law and discussing apportionment ofdamages in automobile accidents cases); DAvID G.
OWEN, PRODUCTS LIABILITY LAW § 17.4, at 1094-100 (2005).
38. The law generally limits the liability of component part manufacturers absent proof that they
participated in the design or manufacture of a finished product. See Davis v. Komatsu Am. Indus.
Corp., 42 S.W.3d 34, 43 (Tenn. 2001) (holding that the maker of a robot used in can-making was
not liable when the plaintiffs injury was likely caused by unsafe maintenance and other dangerous
practices); Kysar, supranote 36, at 1725 & n.109 (and authorities cited therein).
128 [Vol. 89:117
2014] MACHINES WITHOUT PRINCIPALS 129
among the parties who could more efficiently guard or insure against the
loss.39 The other approach would be to hold the vehicle itself
responsible, assuming, of course, that the law is willing to confer legal
"personhood" on the vehicle and require the vehicle to obtain adequate
insurance.
I. AUTONOMOUS MACHINES AND LIABILITY RULES
RELATING TO HUMAN (BUT NOT DRIVER) ERROR
The key question in discussing liability is who bears the costs when
something goes wrong and injury occurs. Driver-less cars, at least those
that might be on the road within the next decade, will reduce but will not
entirely eliminate the possibility of driver error. As I understand it, the
system Google is developing permits the person sitting in the driver's
seat to take control of the car whenever he or she wants. At this point,
however, it is unclear whether Google intends for the driver to have
control at some points: For instance, while driving through high
pedestrian traffic residential areas, construction zones, or during
especially inclement weather, even though Google's driver-less cars
39. This proposal would entail application of a variation of "common enterprise" liability. As
explained in detail below, I envision this as a doctrine that aims to force a group of companies that
work together for a common end here, to design and manufacture a driver-less car to bear
liability collectively when something goes wrong and injury ensues and when it is impossible to
determine, let alone apportion, fault. In other words, as I envision it, common enterprise liability
here would be a form of court-compelled insurance. The manufacturers and designers ("the
enterprise") would jointly indemnify individuals injured by driver-less cars when it is impossible to
determine fault. In the field of consumer protection, for instance, the Federal Trade Commission
often invokes the "common enterprise" doctrine to seek joint and several liability among related
companies engaged in fraudulent practices. See, e.g., FTC v. Network Servs. Depot, Inc., 617 F.3d
1127, 1142-43 (9th Cir. 2010); SEC v. R.G. Reynolds Enters., Inc., 952 F.2d 1125, 1130-31 (9th
Cir. 1991); FTC v. Tax Club, Inc., F. Supp. 2d_, 2014 WL 199514, at *5 (S.D.N.Y. Jan. 17,
2014).
Common enterprise liability should not be confused with its first-cousin, "enterprise liability,"
which proposes a collective theory of liability for companies engaged separately in the same
hazardous industry, when the identity of the responsible firm cannot be determined. See generally
OWEN, supra note 37, § 11.3, at 752. Compare Hall v. E.I. Du Pont De Nemours & Co., 345 F.
Supp. 353 (E.D.N.Y. 1972) (invoking "enterprise" liability theory to hold the highly concentrated
blasting cap industry collectively liable for injuries to children), with Sindell v. Abbott Labs, 607
P.2d 924, 935 (Cal. 1980) (rejecting enterprise liability theory). See generally James A. Henderson,
Jr., Echoes of Enterprise Liability in Product Design and Marketing Litigation, 87 CORNELL L.
REV. 958 (2002); Gregory C. Keating, The Theory of Enterprise Liability and Common Law Strict
Liability, 54 VAND. L. REV. 1285 (2001). See also Kysar, supra note 36, at 1708 & n.28 (collecting
sources). Of course, if the number of manufacturers of driver-less vehicles was relatively small, and
there were issues of identifying the manufacturer of a vehicle that caused significant harm,
enterprise theory of liability might be viable in that situation as well.
WASHINGTON LAW REVIEW
have performed well in those conditions. 40 But the Google system also
has a voice command that instructs the driver to take control of the
vehicle, presumably when the control system detects some kind of error
or is unable to determine what it should do.4' The point of this essay is
to explore liability concerns where there is no possibility that a human
driver caused or contributed to the accident. So what follows is based on
the understanding that the person in the driver's seat is disengaged from
any aspect of driving the car, and that the car is under the sole control of
the car itself.42 Simply taking human drivers out of the equation should
mean, in the long run, a considerable drop in accidents.
But even in the best of circumstances, unexpected and improbable
events occur, things go wrong, and someone or something is injured.
Product liability law provides a framework for resolving claims by
parties injured when things go wrong. With a "product" like an
autonomous car, the law groups those possible failures into familiar
categories: design defects, manufacturing defects, information defects,
and failures to instruct on appropriate uses.43
Before examining these theories of liability, it is useful to pause to
consider whether the standard of care to be applied to driver-less cars
will be different than the standard applied to cars driven by humans.
There is every reason to think that the answer will be "yes," and that fact
may bear on the analysis that follows.
As unlikely as this may seem, the standard of care question was
addressed by a court more than a half-century ago. In Arnold v.
Reutheri an intermediate appellate court in Louisiana questioned
40. According to one press report on the operation of a Google car, "[t]o gain control of the car
[the driver] has to do one of three things: hit a red button near his right hand, touch the brake or turn
the steering wheel." Markoff, War Machines, supra note 28. The article reports that during one
lengthy test drive, the driver "did so twice, once when a bicyclist ran a red light and again when a
car in front stopped and began to back into a parking space. But the car seemed likely to have
prevented an accident itself." Id.
41. Peter Valdes-Dapena, Thrilled and Bummed by Google's Self-Driving Car, CNN MONEY
(May 18, 2012), http://money.cnn.com/2012/05/17/autos/google-driverless-car; see also RAND
REPORT, supranote 30, at 58-65, 68-69.
42. This discussion also does not address questions of the crashworthiness of the vehicle, since
that question is entirely independent of the identity of the driver.
43. See generally OWEN, supra note 37, Parts I & II. No doubt prompted by Google's
development of a wholly autonomous driver-less car, there has been a spate of articles and student
notes and comments addressing liability questions. See, e.g., Beiker, supra note 33; Funkhouser,
supra note 29; Bryant Walker Smith, supra note 29; Kyle Colonna, Note, Autonomous Cars and
Tort Liability, 4 CASE W. RES. J.L. TECH. & INTERNET 81 (2012); Garza, supra note 29; Julie
Goodrich, Comment, DrivingMiss Daisy:An Autonomous ChauffeurSystem, 51 Hous. L. REV. 265
(2013); Gurney, supranote 29.
44. 92 So. 2d 593, 596 (La. Ct. App. 1957).
130 [Vol. 89:117
MACHINES WITHOUT PRINCIPALS
whether autonomous cars will be held to a higher standard of care than
cars driven by humans. Arnold involved a negligence claim by Mrs.
Arnold against Mr. Reuther, whose car struck Mrs. Arnold when she
darted across Canal Street in New Orleans as he was making a left-hand
turn. Although the facts were controverted, the court summarized the
relevant, undisputed facts this way: "[J]ust before the car of Reuther
reached a point opposite that at which Mrs. Arnold stood on the
sidewalk among the other pedestrians, she, at a hurried pace, entered the
street and attempted to cross;" the other pedestrians did not attempt to
cross.45 According to the court, Mr. Reuther testified that "he did not see
Mrs. Arnold as she left the sidewalk and, in fact, did not see her until she
was actually 'in front' of his car."46
Mrs. Arnold did not dispute that she attempted to cross the street
without looking and that her conduct was negligent.4 Her argument was
that Mr. Reuther had the "last clear chance" to avoid the accident, but
failed to do so and should be held liable for that reason.48 The court
disagreed, concluding that Mr. Reuther's efforts, although ineffective,
were sufficient to avoid liability because, after all, he was only human.49
The court said:
A human being, no matter how efficient, is not a mechanical
robot and does not possess the ability of a radar machine to
discover danger before it becomes manifest. Some allowances,
however slight, must be made for human frailties and for
reaction, and if any allowance whatever is made for the fact that
a human being must require a fraction of a second for reaction
and cannot respond with the mechanical speed and accuracy
such as is found in modern mechanical devices, it must be
realized that there was nothing that Reuther, a human being,
could have done to have avoided the unfortunate result which
the negligence of Mrs. Arnold brought upon herself.s
Fifty years later, the court's insight seems remarkably prescient. Had
Mr. Reuther been sitting behind the wheel of a driver-less car, it is quite
possible that, as the Louisiana court forecast, the accident would have
been averted; the car would have immediately detected Mrs. Arnold's
ill-considered dash across the street and rapidly applied the brakes (after
45. Id. at 595-96.
46. Id. at 596.
47. Id.
48. Id.
49. Id.
50. Id.
2014] 131
WASHINGTON LAW REVIEW
confirming that there was no vehicle behind it that would be unable to
stop safely) to come to a stop before striking Mrs. Arnold and before Mr.
Reuther was even alert to the danger.
As this example illustrates, in all likelihood, in the first accident case
involving a driver-less car, the court will not ask whether the self-driving
car performed as well as a reasonable human. The question in Arnold
was one of negligence: Did Mr. Reuther behave reasonably under the
circumstances? And the court said he had. But with a driver-less car, and
no human at the controls, the focus will be on whether the car performed
as well as it should have. To be sure, one could state that question in the
language of negligence, and there may be questions about whether the
car or some of its key components were designed, manufactured, or
maintained negligently. But the court in the first driver-less car case will
likely ask whether the car involved in the accident performed up to the
standards achievable by the majority of other driver-less cars, as well as
the performance specification set by the car's manufacturer.1
Just how demanding would that standard be? Under contemporary
products liability law, the standard would likely be set by a risk-utility
analysis of the sort described in the Restatement (Third) of Torts. To be
sure, in the past, cases like Arnold proceeded under negligence theories,
in large part because the focus was on the conduct of humans, not the
performance of machines. But since then, as negligence claims involving
product failures became more difficult to prove, strict liability principles
took root to govern product liability cases. Products liability plaintiffs
often plead both negligence and strict liability claims. But cases
involving complex products are now typically viewed through the lens
of products liability law. Take Mrs. Arnold's case. Imagine that the car
that struck her was a driver-less car, and taking her cue from the court,
she claims that the car Mr. Reuther was operating (but not driving)
should have been designed in a way that would have averted the
accident. That is a question of design, technological and economic
feasibility, and consumer expectations, and there is no doubt that there
will be design defect cases brought against the manufacturers and
52
designers of self-driving cars. Under current law, Mrs. Arnold would
51. On the duty of care, see generally RESTATEMENT (THIRD) OF TORTS: PHYSICAL &
EMOTIONAL HARM § 7(a) (2010) ("[A]n actor ordinarily has a duty to exercise reasonable care
when an actor's conduct creates a risk of physical harm.").
52. Whether programmers will also be subject to design claims is less clear. Computer code has
not generally been considered a "product" but instead is thought of as a "service." As a result, to the
extent that there are cases seeking compensation caused by allegedly defective software, those cases
have ordinarily proceeded as breach of warranty cases under the Uniform Commercial Code rather
than product liability cases. See, e.g., Motorola Mobility, Inc. v. Myriad France SAS, 850 F. Supp.
132 [Vol. 89:117
MACHINES WITHOUT PRINCIPALS
have a design defect claim against the manufacturer, but whether she
would have a winning case is far from clear, and would depend heavily
on the facts leading up to her accident.
To explain why that is so, consider two scenarios, both of which are
based on the supposition that there are no mechanical or manufacturing
defects with the car. In scenario one, assume that the evidence shows
that a driver-less car's sensors were capable of detecting Mrs. Arnold
moving into the car's path at a point when she was 15 feet away. And
suppose further that, to the extent that there is an industry "collision
avoidance" standard, it is that a driver-less car should be able to detect
objects (human or not) in its path and avoid a collision (by applying the
brakes or by taking evasive maneuvers) if the object is detected within
12 feet of the car. In scenario two, we reverse the numbers so now the
industry "collision avoidance" standard is 15 feet, but it is clear that Mrs.
Arnold was not detected by the sensors until she was 12 feet from the
53
car.
Determining the proper test for design defects has been the central
and most contentious question in products liability law for the past halfcentury,
and it does not appear that the debate has been conclusively
resolved.54 Under current law, there are two potentially viable liability
2d 878 (N.D. 111.2012) (case alleging defective software pleaded as a breach of warranty); In re All
Am. Semiconductor, Inc., 490 B.R. 418 (Bankr. S.D. Fla. 2013) (same); Daniel B. Garrie, The Legal
Status of Software, 23 J. MARSHALL J. COMPUTER & INFO. L. 711, 714-20 (2005); David W.
Lannetti, Toward a Revised Definition of "Product" Under the Restatement (Third) of Torts:
Products Liability, 55 Bus. LAW. 799, 806-07 (2000). But those are typically cases between the
software purchaser and provider. That dynamic appears to be changing, as increasingly software
systems operate cars, trucks, planes and other machines, that, on occasion, malfunction and injure
non-purchaser third parties. These plaintiffs allege that the product failure that caused their injury
was a result of defectively designed or manufactured software, though it may be hard to ascertain
the root cause. At the time of this writing, there are product liability cases alleging that software
defects "caused" serious accidents, including those involving deaths. Perhaps most notable are the
many defective software design claims being brought in In re Toyota Motor Corp. Unintended
AccelerationMktg, Sales Practices,& Prod.Liab. Litig., F. Supp. 2d , 2013 WL 5763178 (C.D.
Cal. Oct. 7, 2013), a multi-district litigation matter in which the plaintiffs allege, inter alia, that
certain vehicles manufactured by Toyota have a software defect that causes, at times, the vehicles to
accelerate notwithstanding measures the drivers take to stop the vehicle. Typically, of course, the
software is licensed into the vehicle by the manufacturer, and thus in the case of an inexplicable
failure the plaintiff can always bring suit against the manufacturer, which, in turn, can seek
contribution or indemnification from the software provider under breach of warranties and other
contract-based theories.
53. Of course, these scenarios assume that the car was driving at a speed (say twenty miles per
hour) within the speed limit and with due attention to the then-current driving conditions
(surrounding traffic, road conditions, weather, time of day, and so forth). See generally RAND
REPORT, supranote 30, at 59 (addressing the technological aspects of how a driver-less car would
respond to these kinds ofscenarios).
54. I have no intention ofjoining the debate over the proper formulation of the design defect test;
2014] 133
134 WASHINGTON LAW REVIEW [Vol. 89:117
theories that plaintiffs may be able to employ to demonstrate that a
product was defectively designed. In the jurisdictions that still follow the
Restatement (Second) of Torts or some variant of it, Mrs. Arnold could
proceed under a "consumer expectations" test. Putting aside that courts
and academics have been critical of using a consumer expectations test
in design defects,56 and especially in cases involving complex
57
products, that claim permits juries to find for the plaintiff whenever a
suffice it to say that many eminent tort scholars have waded into this thicket and have emerged
arguing for a wide variety of approaches. See, e.g., Richard A. Epstein, Products Liability: The
Searchfor the Middle Ground, 56 N.C. L. REV. 643, 647-49 (1978) (describing judicial confusion
in assessing design defects); W. Page Keeton, Product Liability Design Hazards and the Meaning
of Defect, 10 CUMB. L. REV. 293, 298 n.23 (1979) ("The search for the universally acceptable
definition of defect has been the most elusive one in the products liability field."); Kysar, supra note
36, at 1709 & n.30; Joseph W. Little, The Place of Consumer Expectations in Product Strict
Liability Actions for Defectively Designed Products, 61 TENN. L. REV. 1189, 1190 (1994) ("The
difficult and politically contentious cases are those that involve allegations of defective design.");
Marshall S. Shapo, In Search of the Law of Products Liability: The ALI Restatement Project, 48
VAND. L. REV. 631, 638 (1995) ("[A] crucial aspect of products liability law perhaps the core
concept, if any one idea may be described that way-lies in the definition of defect."); Marshall S.
Shapo, Products at the Millennium: Traversing a Transverse Section, 53 S.C. L. REV. 1031, 1033
(2002) ("However divided analysts of products law may be about definitions, most would agree that
the heart of the matter in products liability is the concept of defect."); David G. Owen, Design
Defects, 73 Mo. L. REV. 291, 292 (2008) ("The quest for understanding design defectiveness
perennially vexes courts and accomplished products liability lawyers attempting to unravel design
defect problems; delights law clerks, young associates, and law students, furnishing them with an
occasion to display their erudition; and provides fertile grist for law professors aspiring for the
renown that accompanies discovery of the key to any riddle wrapped in a mystery inside an
enigma.") (footnotes omitted).
55. See RESTATEMENT (SECOND) OF TORTS §402A cmt. g (1965); Kysar, supra note 36, at
1712-13.
56. The Restatement (Third) ofTorts formally abandons the consumer expectations test for design
defect claims. See RESTATEMENT (THIRD) OF TORTS: PRODUCTS LIABILITY § 2 cmt. g (1998)
("[C]onsumer expectations do not constitute an independent standard for judging the defectiveness
of product designs."). Many commentators suggest, however, that the risk-utility test adopted by the
new Restatement permits consideration of consumer expectations, and, in any event, many
jurisdictions continue to permit cases to proceed under a consumer expectations theory. See, e.g.,
Potter v. Chi. Pneumatic Tool Co., 694 A.2d 1319, 1333 (Conn. 1997) (continuing to permit cases
to proceed under the consumer expectations test); Kysar, supra note 36, at 1726 29 & nn.111-25;
Owen, supra note 54, at 335, 342.
57. For instance, in Potterv. Chicago Pneumatic Tool Co. the Connecticut Supreme Court held
that
Although today we continue to adhere to our long-standing rule that a product's defectiveness
is to be determined by the expectations of an ordinary consumer, we nevertheless recognize
that there may be instances involving complex product designs in which an ordinary consumer
may not be able to form expectations ofsafety.
694 A.2d at 1333. Accordingly, the Court adopted a new test where complex products are involved
to incorporate elements of a risk-utility test. The governing test is as follows:
[I]n determining the reasonable expectations of the ordinary consumer, a number of factors
must be considered. The relative cost of the product, the gravity of the potential harm from the
claimed defect and the cost and feasibility of eliminating or minimizing the risk may be
MACHINES WITHOUT PRINCIPALS
product is in a "defective condition unreasonably dangerous to [a] user
or consumer."5
An "unreasonably dangerous" defect is one that makes a
product "dangerous to an extent beyond that which would be
contemplated by the ordinary consumer who purchases it, with the
ordinary knowledge common to the community as to its
characteristics." 59 Although there are difficulties with this test (discussed
below), it does provide Mrs. Arnold a path to get her case before a jury,
where jurors can then "draw[] their own reasonable conclusions as to the
expectations of the ordinary consumer and the knowledge common in
the community at large." 60
The second test is the risk-utility test laid out in the Restatement
(Third) of Torts: ProductsLiability,which would require Mrs. Arnold to
show that "the foreseeable risks of harm posed by the product could
have been reduced by the adoption of a reasonable alternative design."61
That is a formidable burden. The word "reasonable" is intended to
import a quantitative cost-benefit analysis into the test, and thus under
the Restatement's formulation, Mrs. Arnold would have to shoulder the
burden of proving that some modified version of the product would have
avoided the collision at a cost that is reasonable in relation to the degree
of harm reduced.62
Under scenario one, where the car's sensors should have detected
Mrs. Arnold's presence in time to avoid the collision but failed to do so,
Mrs. Arnold is likely to prevail under each theory. She would likely
prevail under the consumer expectations test because consumers would
reasonably expect that Mr. Reuther's driver-less car would meet or
relevant in a particular case. In other instances the nature of the product or the nature of the
claimed defect may make other factors relevant to the issue.
Id. (internal quotations omitted); see also Seattle-First Nat'l Bank v. Tabert, 86 Wash. 2d 145, 542
P.2d 774 (1975) (earlier adoption of the same test by the Washington State Supreme Court);
Gurney, supra note 29, at 260-62. See generally Owen, supra note 54, at 346-53.
58. RESTATEMENT (SECOND) OF TORTS § 402A(1) (1965); see also Greenman v. Yuba Power
Prods., Inc., 377 P.2d 897, 900 (Cal. 1963) (Traynor, J.) ("A manufacturer is strictly liable in tort
when an article he places on the market, knowing that it is to be used without inspection for defects,
proves to have a defect that causes injury to a human being.").
59. RESTATEMENT (SECOND) OF TORTS §402A cmt. i; see also Kysar, supra note 36, at 1709 &
nn.51-52.
60. Giglio v. Conn. Light & Power Co., 429 A.2d 486, 488 (Conn. 1980) (internal quotations
omitted).
61. RESTATEMENT (THIRD) OF TORTS: PRODUCTS LIABILITY §2(b) (1998).
62. See id.; Connelly v. Hyundai Motor Co., 351 F.3d 535, 541 (1st Cir. 2003) (noting that, under
New Hampshire law, the jury could have reasonably found that the airbag was not defective: "[o]n
balance, the benefit to the public of including the overly aggressive airbag system in the Sonata
outweighed the danger caused by the airbag system (because the system saved many more lives than
it took)"); Owen, supranote 54, at 330 36.
2014] 135
WASHINGTON LAW REVIEW
exceed the existing 12-foot "collision avoidance" industry standard.63
The analysis conducted under the risk-utility test should come out in her
favor as well. She would not have to advocate in favor of an
"alternative" design; her argument would be that the market had already
weighed the relative costs and benefits of the 12-foot collision avoidance
standard and determined that the cost of such a standard is justified by
the corresponding risk reduction.64
Scenario two raises much more difficult questions because most,
perhaps all, driver-less vehicles would have been unable to stop or take
evasive action quickly enough to avoid the accident. For starters, it is not
clear that Mrs. Arnold could prevail under a consumer expectations test.
To be sure, given the indeterminacy of the test-the standard is, after all,
what reasonable consumers say it is-it could be her salvation or her
undoing. 65 On the one hand, the consumer expectations test might work
to Mrs. Arnold's advantage. There has been considerable hype about the
enhanced safety of driver-less cars-so much so that some experts
forecast that these cars, once we rid the roads of drivers, will put an end
to accidents.66 Reacting to that hype, consumers may expect (putting
aside questions of reasonableness) driver-less cars to be designed in a
way that would avert accidents even where, as is the case with Mrs.
Arnold, the initial "fault" lies with the person and the machine had at
best a second or two to detect and react to the impending accident.
Indeed, even if the industry's evidence shows that the 15-foot collision
avoidance standard pushes the limits of technological and economic
63. See Soule v. Gen. Motors Corp., 882 P.2d 298, 308 (Cal. 1994) (noting that the ordinary
consumer expectations test is appropriate when the everyday experience of the particular product's
users permits the inference that the product did not meet minimum safety expectations).
64. Of course the manufacturer would have defenses to liability, but given the presence of a
prevailing industry standard and the vehicle's evident non-compliance with that standard, the
burden ofproof would likely shift to the manufacturer. See, e.g., Richard C. Ausness, "Fasten Your
Seatbelt Orville!": Exploring the Relationship Between State-of-the-Art, Technological and
Commercial Feasibility, and the Restatement's Reasonable Alternative Design Requirement, 45
IM. L. REV. 669, 686-89 (2012); O'Brien v. Muskin Corp., 463 A.2d 298, 302 (N.J. 1983),
overruling recognized by Dewey v. R.J. Reynolds Tobacco Co., 577 A.2d 1239 (N.J. 1990).
65. Among the criticisms leveled at the consumer expectations test is that it is indeterminate, or to
put a finer point on it, "so vague as to be lawless." James A. Henderson, Jr. & Aaron D. Twerski,
Achieving Consensus on Defective Product Design, 83 CORNELL L. REv. 867, 882 (1998). The
same authors have made the same claim repeatedly. See, e.g., James A. Henderson, Jr. & Aaron D.
Twerski, Drug Designs are Diferent, 111 YALE L.J. 151, 178 (2001).
66. Beiker, supra note 33, at 1149 (pointing out that driver error is "by far (95%) the most
common factor implicated in vehicle accidents"); Markoff, WarMachines, supra note 28; Chunka
Mui, Will Auto InsurersSurvive Their Collision with DriverlessCars? (Part6), FORBEs (Mar. 28,
2013), http://www.forbes.com/sites/chunkamuil2013/03/28/will-auto-insurers-survive-their-
collision-with-driverless-cars-part-6/.
136 [Vol. 89:117
2014] MACHINES WITHOUT PRINCIPALS 137
feasibility, jurors might conclude that designers are capable of doing
better and hold the manufacturer of Mr. Reuther's car liable on that
basis.67 Whether the court would let a verdict like that stand is open to
question, but the consumer expectations test is a double-edged sword;
expectations are often shaped by industry, but industry standards do not
necessarily define the outer boundary of enforceable consumer
expectations. 69
On the other hand, one downside of the test is that consumer
expectations are generally a reflection of the existing state of the
marketplace. Manufacturers, through advertising and other
communications with consumers, play a key role in shaping consumer
expectations. Unless the manufacturer makes inflated and unjustified
representations about its product's performance, consumers are likely to
expect that their products will perform in a way that is consistent with
prevailing standards as articulated by the products' manufacturers, even
if better and safer products are achievable at a nominal cost. Under this
67. The law has never treated industry custom as even the de facto standard of care. Eighty years
ago, Learned Hand's famous decision in The TJ Hooper,60 F.2d 737, 740 (2d Cir. 1932), made
clear that a negligence standard should not strictly be set by industry custom because "a whole
calling may have unduly lagged in the adoption ofnew and available devices." In most jurisdictions,
the defendant is permitted to show that the product's design was "state-of-the-art" (that is, what is
technologically feasible, not just industry custom) to rebut claims of defect. See, e.g., Potter v. Chi.
Pneumatic Tool Co., 694 A.2d 1319, 1346-47 (Conn. 1997) (collecting cases). Although not
conclusive, that evidence is highly probative. Id. Mrs. Arnold might have to prove that the design,
although meeting industry's custom, was not in fact state-of-the-art and further improvements were
technologically achievable. See id.
68. The California Supreme Court in Soule v. GeneralMotors Corp., 882 P.2d 298, 309-10 (Cal.
1994), for example, suggests that using a consumer expectations test to determine liability when
"esoteric circumstances" and "complicated design considerations" are present would be improper.
Mrs. Arnold's case would likely fall into that category.
69. See, e.g., Potter,694 A.2d at 1333.
70. As the Supreme Court of California drove home in Barkerv. Lull EngineeringCo., 573 P.2d
443, 451 n.7 (Cal. 1978), relying on the consumer expectations test can undermine incentives for
manufacturers to enhance the safety features of their products: "The flaw in the Restatement's
analysis, in our view, is that it treats such consumer expectations as a 'ceiling' on a manufacturer's
responsibility under strict liability principles, rather than as a 'floor."' The Barkercourt formulated
the applicable design defect test in this way:
[A] product may be found defective in design, so as to subject a manufacturer to strict liability
for resulting injuries, under either of two alternative tests. First, a product may be found
defective in design if the plaintiff establishes that the product failed to perform as safely as an
ordinary consumer would expect when used in an intended or reasonably foreseeable manner.
Second, a product may alternatively be found defective in design if the plaintiff demonstrates
that the product's design proximately caused his injury and the defendant fails to establish, in
light of the relevant factors, that, on balance, the benefits ofthe challenged design outweigh the
risk of danger inherent in such design.
Id. at 455 56. Barkerwas explained and reaffirmed in Soule, 882 P.2d at 308 n.4. See also Potter,
694 A.2d at 1333-34.
WASHINGTON LAW REVIEW
theory, it may be fair to assume that consumers would expect driver-less
cars to operate in a safe manner and might even have a reasonable
expectation that driver-less cars will deliver higher levels of safety than
cars driven by humans. But it seems a stretch to assume that consumers
would have formed a reasonable expectation that driver-less cars can
stop quickly enough to avoid hitting a person who darts in front of the
car. This point might take on added force if the manufacturers of driverless
cars seek to dampen consumer expectations about the safety features
of the cars, if for no other reason than to quell wholly unrealistic
expectations that might lead consumers to think, as Mrs. Arnold may
have thought, that no matter how risky her conduct was, the car would
be able to stop quickly enough to avoid impact. 1For these reasons, even
under the consumer expectations test, perceived by many to be more
plaintiff-friendly, Mrs. Arnold would face a tough and uncertain path to
victory.
Equally uncertain is whether the risk-utility test embodied in the
Restatement (Third) of Torts would provide Mrs. Arnold a road to
recovery. Her burden would be to show that the incremental benefit of
altering the car's design to improve stopping distance from 15 feet to 12
feet would be worth the cost-that is, that the value of the accidents
72
avoided would equal or exceed the added cost of the alternative design.
Presenting evidence to that effect would be difficult. After all, Mrs.
Arnold would need to retain an expert engineer to demonstrate that a
reduction in the stopping distance is technologically feasible, as well as
another expert (likely an economist) to establish that the savings
achieved by the reduction would outweigh the attendant costs of
modifying the vehicle.73 Her experts would also have to do battle with
71. See Jon D. Hanson & Douglas A. Kysar, Taking Behaviorism Seriously: A Response to
Market Manipulation, 6 ROGER WILLIAMS U. L. REV. 259, 324 70 (2000); Jon D. Hanson &
Douglas A. Kysar, Taking Behaviorism Seriously: Some Evidence of Market Manipulation, 112
HARV. L. REV. 1420 (1999).
72. This balancing test may itself be troublesome because it asks jurors to place values on things
that do not come with an attached price tag-including the "value" of human life, injury and
suffering, and the "value" to be gained or sacrificed by design modifications. Making matters worse,
jurors not only have to assign values, they then have to weigh them against one another and pick a
winner and a loser. For a critique of using economic modeling to make such choices, see generally
FRANK ACKERMAN & LISA HEINZERLING, PRICELESS: ON KNOWING THE PRICE OF EVERYTHING
AND THE VALUE OF NOTHING (2004).
73. Although there was much critical commentary on the burdens that the risk-utility test would
place on plaintiffs, see, e.g., Kysar, supra note 36, at 1721-22 & n.84 (collecting authorities), the
tools for lessening those burdens would not apply here. They include cases (a) where the product
defect was one in which the everyday experience of the product's users permits a conclusion that the
product's design violated minimum safety assumptions, and is thus defective regardless of expert
opinion about the merits of the design, Soule, 882 P.2d at 308 (emphasis in original), or (b) where
138 [Vol. 89:117
MACHINES WITHOUT PRINCIPALS
the manufacturer's experts, who would likely claim that the design
pushed the limits of technological feasibility.74 To be sure, if she could
make the showing that additional modifications could have made the
vehicle safer at a reasonable cost, she might have a case. But finding and
retaining experts to put on a case might be a dauntingly expensive
enterprise for an individual plaintiff to bear.
Mrs. Arnold's path to victory would be far surer if it turned out that
there was a manufacturing defect or component failure that caused or
contributed to the accident. A product with a manufacturing defect is a
product with an unintended flaw; that is, the finished product does not
conform to the manufacturer's own specifications or requirements. A
plaintiff can prevail in a manufacturing defect case if she can prove that
the product does not conform to the manufacturer's specifications and
that the defect played a role in causing the accident. For autonomous
cars, there are certain to be instances in which vital components of the
system fail because of a manufacturing or maintenance defect. For
example, assume that one of the key components responsible for keeping
track of objects that might enter the path of the car (for instance, a radar
or laser sensor) failed on Mr. Reuther's car prior to the accident with
Mrs. Arnold. Assuming that the failing component can be identified, and
assuming that there is a link to the failed component and the accident,
liability can be established and properly allocated. If the component is
analyzed and does not meet the manufacturer's specification, then the
case is an easy one, and liability may be appropriately meted out.76
Even if there is no tangible sign of defect, Mrs. Arnold may have a
viable claim, at least in some jurisdictions, if she can prove that the
component malfunctioned and, for instance, resulted in the car failing to
the product defect was of the kind that "ordinarily occurs" so it is fair to base liability on a res ipsa
loquitor theory. See RESTATEMENT (THIRD) OF TORTS: PRODUCTS LIABILITY § 3(a) (1998)
(providing for an inference based on concepts of res ipsa loquitor where the plaintiff can show that
the product failure "was of a kind that ordinarily occurs as a result of product defect"); Kysar, supra
note 36, at 1721 & nn.89 90.
74. See supranote 67 and accompanying text on the availability of a "state-of-the-art" defense.
75. Greenman v. Yuba Power Prods. Inc., 377 P.2d 897 (Cal. 1963); RESTATEMENT (THIRD) OF
TORTS: PRODUCTS LIABILITY § 2(a) ("[A] manufacturing defect [exists] when the product departs
from its intended design even though all possible care was exercised in the preparation and
marketing of the product .... ). In the past, product manufacturers ordinarily were held strictly
liable for defectively manufactured products. See generally RESTATEMENT (SECOND) OF TORTS:
PRODUCTS LIABILITY §402A (1965).
76. As a general matter, in cases where a product or component is manufactured in a way that
fails to comport with the defendant's own intended design specifications, courts routinely find
liability. See RICHARD A. EPSTEIN, MODERN PRODUCTS LIABILITY LAW 70 (1980); Kysar, supra
note 36, at 1709 & n.29.
2014] 139
WASHINGTON LAW REVIEW [Vol. 89:117
stop in a manner consistent with the manufacturer's specifications. The
law permits such a claim to go forward so long as the component was
properly used, properly maintained, and had not been altered or damaged
in a way that might have caused the malfunction. But take note: In
cases involving other autonomous machines, liability has been difficult
to establish where alternative theories of liability are present.
Finally, there is the issue of information defects. Although it is
unlikely that Mrs. Arnold would have a claim under this theory, the
"duty to train" often plays a role when humans interact with, and have to
operate, complex machines. The duty to train may play an important role
in the introduction of driver-less cars onto the market. Driver-less cars
are highly sophisticated systems, and learning how to "drive" and
"operate" them may be difficult for consumers used to being in control
while they are in the "driver's seat." Especially if the "driver" must
remain alert and ready to assume the conventional role of "driver," the
burden of teaching consumers how to manage those tasks may fall to
manufacturers, and they may face liability if they fail to adequately
communicate information about the safe use of driver-less cars.
77. There are often questions about whether the manufacturer or service provider that maintained
the part is responsible for the defect, but that issue has no bearing here. See supra note 37 and
accompanying text. Once driver-less cars enter the market, however, maintenance may become a
significant issue. As Anderson's RAND Report observes, many of the key sensors degrade over
time, software will need to be updated regularly, and GPS systems will have to be updated
constantly to reflect road openings and closings. RAND REPORT, supranote 30, at 66. Although the
car itself will play a key role in continually monitoring the updating and performance of its
component parts, other entities will play a role as well in ensuring that these components are
maintained and functioning properly, potentially complicating the liability questions. Id.
78. For example, in Ferguson v. BombardierService Corp., 244 F. App'x 944 (11th Cir. 2007),
the court rejected a manufacturing defect claim against the manufacturer of an autopilot system in a
military cargo plane that suffered a catastrophic crash while the plane was on autopilot. The
plaintiffs cited some evidence that the autopilot failed to function properly. But the court found
equally credible the defense theory that the plane was improperly loaded, so much so that a strong
gust of wind caused the plane to crash-a theory consistent with the information salvaged from the
aircraft's flight data recorder. And in Nelson v. American Airlines, Inc., 70 Cal. Rptr. 33 (Cal. Ct.
App. 1968), the plaintiff was a passenger on an American Airlines flight who was injured when the
plane suddenly descended. One theory was that when the pilots engaged the autopilot, a fault in the
autopilot caused the sudden descent. The court applied the doctrine of res ipsa loquitor to find an
inference of negligence by American Airlines, but ruled that the inference could be rebutted if
American Airlines could show that the autopilot did not cause the accident or that an unpreventable
cause triggered the accident. The court said that a defect in the autopilot could have caused the
accident, as well as the negligent maintenance of the device. Accordingly, the court reversed the
lower court's ruling in favor ofAmerican and remanded the action for further proceedings. See also
Payne v. ABB Flexible Automation, Inc., No. 96-2248, 1997 WL 311586 (8th Cir. June 9, 1997)
(per curiam).
79. Many of the cases addressing this informational defect are those where airplane
manufacturers are alleged to have failed to provide adequate training to pilots in the safe use oftheir
140
MACHWINES WITHOUT PRINCIPALS
As this brief review shows, the introduction of truly autonomous
vehicles is unlikely to present legal issues that tax our current product
liability regime, at least so long as the product failure linked to the injury
can be traced to, or reasonably imputed to, the activity of an identifiable
person or legal entity. To be sure, the increasing complexity of
automotive technology will provide even more ammunition to those who
criticize the continued use of a consumer expectation test in design
defect litigation. Supporters of the test may have a difficult time
defending its use in cases involving driver-less cars, given the vehicles'
intricate and interdependent technologies and the argument from
opponents that consumers will not have a sufficient basis to form a
reasonable judgment about what level of safety to expect. On the other
hand, as many academics have suggested, when properly applied, the
risk-utility test laid out in the Restatement (Third) of Torts does embody
some elements of consumer expectations, and that test may adequately
safeguard the interests of those injured by driver-less vehicles.so
II. MACHINES WITHOUT PRINCIPALS AND LIABILITY
RULES RELATING TO INEXPLICABLE ACCIDENTS
Thus far, this Essay has addressed the application of conventional
products liability principles to machines that are capable of operating
independently of human direction, but where the machine's failure can
fairly be attributed to some act or omission by a human that can be said
to have "caused" the accident. As we have seen, apart from the vexing
question of whether the "operator" or manufacturer of the driver-less car
should be the defendant in a case where the car and not the driver had
control, products liability law will apply in much the same way as it
would apply if the car involved in the accident were driven by a human.
And as we have also seen, resolution of the proper defendant question is
likely to have little bearing on the important question of "Who pays?" In
a fault-based system, drivers will bear the loss when they are responsible
for the accident, and manufacturers will bear the loss when the driverless
car they have designed or manufactured fails. But in all of these
scenarios, it is easy to avoid the question of agency because some
human-Mrs. Arnold, Mr. Reuther, the designer, the programmer, or the
aircraft. See, e.g., Glorvigen v. Cirrus Design Corp., 796 N.W.2d 541 (Minn. Ct. App. 2011)
(considering but ultimately rejecting failure-to-train claim); Driver v. Burlington Aviation, Inc., 430
S.E.2d 476 (N.C. Ct. App. 1993); Berkebile v. Brantly Helicopter Corp., 311 A.2d 140, 142 (Pa.
Super. Ct. 1973), aff'd, 337 A.2d 893 (Pa. 1975).
80. Kysar, supra note 36, at 1761-82; Owen, supranote 54, at 336 53.
2014] 141
WASHINGTON LAW REVIEW
manufacturer of the vehicle or its component parts-was the key causal
link in the events that led to the injury and thus can be assigned the
blame.
Now switch gears and assume that an accident occurs involving a
truly autonomous vehicle where it is impossible to reasonably assign the
responsibility for the accident to a human. The hard question that may
arise when autonomous machines are involved in accidents is what
should the liability rule be when it is unreasonable to infer that the
accident was caused by a design or manufacturing defect. It may be that
those cases prove to be the null set. After all, an inference of defect is
reasonably drawn when a product fails, even when a defect cannot be
determined by engineers, when the failure occurs with some frequency
and the failure follows a common pattern. In those circumstances, courts
routinely apply principles of res ipsa loquitor and conclude that the car,
not the driver, is at fault.
As one example, consider the recent flood of design defect cases
brought against Toyota alleging that certain Toyota-manufactured
vehicles (both Toyota and Lexus) are prone to "sudden acceleration"
through no act by the driver and the acceleration cannot be stopped by
the driver, by either disengaging the accelerator pedal or by applying the
brakes. Notwithstanding an exhaustive inquiry, engineers have been
unable to identify a specific design or manufacturing defect that causes
the uncontrolled acceleration, although theories abound.82 Nonetheless,
because of the substantial number and common features of these
complaints, Toyota has decided not to contest liability. It settled a class
action lawsuit for economic loss due to the defect brought by owners of
affected Toyota and Lexus vehicles for $1.3 billion and other relief.83
81. Indeed, in the multidistrict litigation (MDL) case against Toyota for personal injuries
allegedly resulting from sudden acceleration, the evidence shows that Toyota received upwards of
60,000 complaints about sudden acceleration in certain of its vehicles. In re Toyota Motor Corp.
Unintended Acceleration Mktg., Sales Practices, & Prods. Liab. Litig., F. Supp. 2d_, 2013 WL
5763178, at *26 n.66 (C.D. Cal. Oct. 7, 2013).
82. See, e.g., Sharon Sile Carty, Toyota's Sudden Acceleration Problem May Have Been
Triggered By Tin Whiskers, HUFFINGTON POST (Jan. 22, 2012), http://www.huffing
tonpost.com/2012/01/21/toyota-sudden-acceleration-tin-whiskers n 1221076.html (reporting that
tiny threads oftin had developed in areas in which they might conduct electricity to the systems that
control acceleration, and pointing out that this problem had been "implicated in crippling defects
besetting a range of equipment, including communications satellites, pacemakers, missiles and
nuclear power plants"); see also In re Toyota Motor Corp., 2013 WL 5763178, at *34 35
(discussing several of the plaintiffs' theories of causation).
83. This settlement excluded the personal injury cases part of the multidistrict litigation and
involved a class action involving defect claims for owners who sought damages for the diminished
value of their vehicles and certain repairs. See, e.g., Jessica Dye, Toyota Acceleration Case
Settlement Gets Final OK, INS. J. (July 22, 2013), http://www.insurancejournal.com/
142 [Vol. 89:117
2014] MACHINES WITHOUT PRINCIPALS 143
More recently, Toyota took steps to settle the pending 400 personal
injury cases against it after an Oklahoma jury, applying the doctrine of
res ipsa loquitor,84 awarded the plaintiffs $3 million. The jury apparently
concluded that, even though the plaintiffs could not isolate the cause of
sudden acceleration, the accident was more likely caused by the car than
the driver. 5
As the Toyota case makes plain, existing products liability
law is well-positioned to address cases where the evidence strongly
suggests a defect, but technology cannot isolate the cause.
What about failures involving driver-less vehicles that are rare,
news/national/2013/07/22/299154.htm; Greg Risling, Toyota Tackles Acceleration Lawsuits;
Questions Remain, USA TODAY (Jan. 21, 2013), http://www.usatoday.com/story/
money/cars/driveon/2013/01/21/toyota-sudden-acceleration-lawsuits/i 851813/; TOYOTA ECONOMIC
Loss SETTLEMENT WEBSITE, http://www.toyotaelsettlement.com (last visited Feb. 18, 2014) (laying
out terms and conditions of settlement).
84. See, e.g., Jacklyn Trop, Toyota Seeks Settlement for Lawsuits, N.Y. TIMES, Dec. 14, 2013, at
B3, availableat http://www.nytimes.com/2013/12/14/business/toyota-seeks-settlement-for-lawsuits.
html; Ken Bensinger, Toyota Looks to Settle Sudden-Acceleration Lawsuits, L.A. TIMES (Dec. 12,
2013), http://articles.latimes.com/2013/dec/12/business/la-fi-toyota-settlement-20131213. This
settlement involves personal injury claims resulting from accidents allegedly caused by sudden
acceleration, which have been consolidated in In re Toyota MotorCorp., 2013 WL 5763178. These
settlement negotiations are potentially nearing a final resolution. Jaclyn Trop & Ben Protess, Toyota
in Talks on Final Settlements Over Car Recalls, N.Y. TIMES (Feb. 9. 2014),
http://www.nytimes.com/2014/02/10/business/toyota-in-talks-on-final-settlements-over-car-recalls.
html?hpw&rref=automobiles;see also Estate of Edward W. Knoster v. Ford Motor Co., 200 F.
App'x. 106, 114 (3d Cir. 2006) (applying New Jersey law and finding that section three of the
Restatement (Third) of Torts preserved the res ipsa loquitor inference so that sometimes when a
product fails, "'common experience' indicates it would not have done so absent a defect").
85. See, e.g., Bloomberg, Toyota Loses First Acceleration Lawsuit, Must Pay $3 Million,
AUTOMOTIVE NEWs (Oct. 24, 2013), http://www.autonews.com/article/20131024/OEM11/
131029935#axzz2r8ypeVU (reporting that an Oklahoma jury awarded $3 million in actual damages
and Toyota settled the punitive damage claim for an undisclosed amount in a sudden acceleration
case resulting in injury to the driver and the death of a passenger). The court presiding over the
multidistrict litigation involving personal injury claims applied this rationale in rejecting Toyota's
claim for summary judgment in one of the MDL cases (which may have influenced Toyota's
decision to settle all of these claims). The court said:
Toyota's Motion for Summary Judgment is premised on the uncontroverted fact that Plaintiff
has been unable to identify a precise software design or manufacturing defect and point to
physical or otherwise traceable evidence that the defect actually caused the Camry throttle to
open from an idle position to a much wider angle without analog input from the driver via the
accelerator pedal. To a lesser extent, it is also premised upon the fact that Plaintiff cannot
prove the actual failure of Toyota's fail-safe mechanisms in the Camry on the day of the
collision. As explained more fully below, Plaintiffs burden at the summary judgment stage is
not so onerous.
Essentially, Toyota asks the Court to conclude that the only reasonable inference that may be
drawn from the volumes of evidence proffered by the parties is that Mrs. St. John mistakenly
applied the accelerator pedal instead of the brake pedal. The Court cannot so conclude. As
Plaintiff points out, and as detailed by the Court more fully below, Mrs. St. John's testimony,
together with other evidence, much of it expert evidence, support inferences from which a
reasonable jury could conclude that the Camry continued to accelerate and failed to slow or
stop despite her application of the brakes.
In re ToyotaMotor Corp., 2013 WL 5763178, at *33.
WASHINGTON LAW REVIEW
untraceable to any defect, and inexplicable in that the vehicle's actions
are incompatible with the vehicle's decision-making design? Who, if
anyone, should bear liability then? To make the question more concrete,
suppose daredevil Mrs. Arnold again darts in front of a driver-less
vehicle. In this scenario, assume that the vehicle's designers have
programmed instructions that it should at all costs avoid striking a
human. And further assume that, in order to avoid hitting Mrs. Arnold,
the vehicle has only two choices: It can either (1) take evasive action by
making a sharp turn that would likely avoid any impact with Mrs.
Arnold, but would also likely result in the vehicle striking a brick wall
(thereby risking the safety of the vehicle's occupants), or (2) apply the
brakes with great force, even though, given the distance and vehicle's
speed, that course of action would not guarantee that the vehicle would
stop before hitting Mrs. Arnold; nor would it ensure that there would be
no collision between it and the car following closely behind. 86
Suppose that the vehicle does not adhere to its basic instruction to
avoid a collision with a human if at all possible and takes neither of
these courses of action. Instead, the car applies its brakes gently,
decreasing the car's speed, and avoiding the brick wall and a possible
collision with the car behind it, but nonetheless striking and injuring
Mrs. Arnold. As noted, the car was not "supposed" to make the choice it
did. The engineers and programmers who designed the car's autonomous
driving system "instructed" the car to avoid the possible collision with
Mrs. Arnold and the car behind it by braking and by steering itself (and
its passengers) into the brick wall, recognizing that the vehicle was
sufficiently crash-worthy to sustain the impact without unreasonable risk
to the safety of the car's occupants.
In some sense, this is the HAL question presented in a more benign
form. It may be that truly intelligent machines will learn to adapt the
instructions they initially receive from humans to circumstances not
directly forecast at the time of their creation. And perhaps these
machines will learn to internalize values that are not the ones their
creators tried to embed. HAL, for instance, was not programmed to
value self-preservation, but we know that he held that value dearly, and
placed it above human life.8
86. Of course, one might suppose that if the trailing vehicle were also driver-less, the two cars
would be communicating course, position, and speed to one another and the trailing vehicle would
thus leave sufficient room between the vehicles to stop safely even in an emergency. See also
Marcus, supra note 28 (using a slightly different hypothetical to demonstrate the moral questions
raised by driver-less vehicles).
87. In testing prototypes of war-fighting robots, one study cited as a potential threat modern war
robots that-like HAL could "turn on" their human creators, in part as a mode of self-
144 [Vol. 89:117
MACHINES WITHOUT PRINCIPALS
The intended lesson of this hypothetical is that at some point it will be
hard to conceptualize truly intelligent machines as mere agents or tools
of humans. Agency principles can take us only so far. A machine that
can define its own path, make its own decisions, and set its own
priorities may become something other than an agent. Exactly what that
may be, though, is not a question that the law is prepared to answer."
Nor does current product liability law yield answers that are
particularly satisfying. Consider the hypothetical sketched out above. In
that case, even though the car might have been capable of avoiding Mrs.
Arnold, her recovery would be uncertain, even if her claim were
stronger. First, we can rule out a manufacturing defect claim; the
mechanical features of the car functioned perfectly. To the extent that
Mrs. Arnold has a path to recovery, it lies with a design defect claim.
The consumer expectations test, even in those jurisdictions that have
crafted a hybrid form of consumer expectations/risk-utility test, is
unlikely to aid Mrs. Arnold. Whatever else this case may be, it is not an
"ordinary" case where it is fair to presume that jurors will have reasoned
expectations about how the vehicle should have performed.
Nor will Mrs. Arnold get much traction under the risk-utility theory,
which, in a case like our hypothetical, would likely place the burden of
showing a design alternative on the plaintiff. Again, the complexity and
sophistication of driver-less cars, and the complications that will come
with the fact patterns that are likely to arise, are going to make proof of
wrongdoing in any individual case extremely difficult. The question
whether there are alternative designs that would make driver-less cars
even safer and prevent one-off accidents will be so infused with complex
technical and economic questions that individual cases will be difficult
and expensive to try, and even harder for jurors to resolve.
This case illustrates the point. Even though the vehicle did not
perform in a way that was consistent with its designer's intentions, the
vehicle did make an objectively reasonable choice: The vehicle took
swift and decisive action to minimize the harm to Mrs. Arnold while
safeguarding the vehicle's passenger from possible harm. The jurors
may see that choice as a sensible one. And we know that Mrs. Arnold
cannot argue that the better design choice is a return to the status quo
ante, where cars were driven by humans. Even with today's safety
features-automatic braking, anti-lock brakes, traction control, airbags,
preservation. See Mick, supranote 23.
88. PAGALLO, supra note 17, at 152 53. See generally Solum, supranote 27.
2014] 145
WASHINGTON LAW REVIEW
backup cameras, "blind spot" cameras, and the like8 9-the rate of
accidents and injuries on the roads in the United States remains
frighteningly high.90 Whatever flaws may emerge in driver-less cars,
their introduction will almost certainly make a considerable dent in the
accident and injury rate in the United States.
In my view, in cases where driver-less cars fail and cause injuries to
persons or property and it would be unreasonable to attribute the failure
to the vehicle's manufacture or design, the law will need to fashion a
response that best serves the collective interests of the affected parties.
My proposal is to construct a system of strict liability, completely
uncoupled from notions of fault for this select group of cases. A strict
liability regime cannot be based here on the argument that the vehicles
are "ultra-hazardous" or "unreasonably risky" for the simple reason that
driver-less vehicles are likely to be far less hazardous or risky than the
products they replace. Indeed, it is precisely because these machines are
so technologically advanced that we expect them not to fail. For these
reasons, a true strict liability regime will be needed; one that does not
resort to a risk-utility test or the re-institution of a negligence standard
for the simple fact that those tests will be difficult, if not impossible, for
the injured party to overcome.
In this instance, a system of strict liability would, in effect, impose a
court-compelled insurance regime to address the inadequacy of tort law
to resolve questions of liability that may push beyond the frontiers of
science and technology. There are four strong policy reasons to establish
a strict liability regime for this category of cases.
First, providing redress for persons injured through no fault of their
own is an important value in its own right. The idea that individuals
should bear a loss that is visited upon them, even when the causal failure
is inexplicable, runs counter to basic notions of fairness, compensatory
justice, and the apportionment of risk in society.
Second, a strict liability regime is warranted because, in contrast to
the injured party, the vehicle's creators are in a position to either absorb
the costs, or through pricing decisions, to spread the burden of loss
widely. After all, it is not unreasonable that the costs of inexplicable
89. See, e.g., Mercedes Benz Safety, MERCEDES BENZ, http://www.mbusa.com/mercedes/
benz/safety (last visited Feb. 17, 2014) (describing various safety features available on Mercedes
automobiles).
90. Although the frequency of automobile crashes is slowly declining in the United States, there
were still more than 5.3 million crashes in 2011, resulting in more than 2.2 million injuries and
32,000 fatalities. Alcohol was a factor in 39 percent of the fatal crashes. See RAND REPORT, supra
note 30, at xiv.
146 [Vol. 89:117
MACHINES WITHOUT PRINCIPALS
accidents be borne, at least in part, by those who benefit from riskreducing,
innovative products.
Third, a strict liability regime will spare all concerned the enormous
transaction costs that would be expended if parties had to litigate
liability issues involving driver-less cars where fault cannot be
established. 91 The first law of litigation is as follows: As the complexity
of products rises geometrically, the cost of litigating products liability
cases increases exponentially. It is better to spend money compensating
the injured than paying lawyers and experts. Indeed, although Toyota
has not discussed this issue publicly, it is possible that Toyota decided to
settle the sudden acceleration cases not because it feared losing them in
court, but because it feared that litigating each case to judgment, even if
it won, would cost the company far more than settling the cases because
of the complexity of the cases. If so, Toyota would not be the first
company to settle a case to avoid transaction costs, regardless of the
case's merits.
And fourth, a predictable liability regime may better spur innovation
than a less predictable system that depends on a quixotic search for, and
then assignment of, fault. If driver-less cars deliver even a fraction of
their promised benefits of less carnage on the highways, more leisure
time for consumers, and reduced fuel consumption, they will provide a
substantial benefit to society. A liability system should not stifle
innovation; it should encourage responsible innovation. Stability,
coupled with a cost-spreading approach, would doubtlessly serve that
goal better that an uncertain fault-based liability system.
Assuming that the courts can be persuaded to adopt a strict liability
regime, the question then becomes, "Who bears the cost?" Should the
burden be placed on the operator, owner, the manufacturer, the
programmers, the designers, or all of them, or should the law simply
declare that the driver-less car itself is a legal "person" and require it to
insure itself against liability? Under prevailing products liability law, if
someone is injured by the failure of an automobile (and not the driver),
91. Lest there be any doubt, my argument is not based on notions of a "no-fault" liability system,
that is, a system that substitutes mandatory insurance and eliminates access to the judicial system.
My proposal is a strict liability regime implemented by the courts. Although the idea of "no fault"
systems took hold in the 1970s and 1980s, and was expected to drive down insurance costs by
limiting the transaction costs related to litigation, it is by now apparent that those systems have not
worked as envisioned. See, e.g., JAMES M. ANDERSON ET AL., THE U.S. EXPERIENCE WITH NoFAULT
AUTOMOBILE INSURANCE: A RETROSPECTIVE, at xiii (2010), available at
http://www.rand.org/pubs/monographs/MG860. It is likely, however, that the introduction of driverless
cars will shift liability from the "driver" to the manufacturer, and that shift may trigger a
resurgence of interest in "no fault" insurance regimes.
2014] 147
WASHINGTON LAW REVIEW
the manufacturer is the target of choice. It is the manufacturer who bears
front line responsibility for design and manufacturing defects, and is
generally the principal if not the only defendant in litigation. That
allocation of responsibility makes sense, because the manufacturer sets
the price for the vehicle, and so the manufacturer can build in an
"insurance premium" into the vehicle's sale price to offset expected
liability costs. 92
There are at least two concerns about making the manufacturer
shoulder the costs alone. One is that with driver-less cars, it may be that
the most technologically complex parts-the automated driving systems,
the radar and laser sensors that guide them, and the computers that make
the decisions-are prone to undetectable failure. But those components
may not be made by the manufacturer. From a cost-spreading standpoint,
it is far from clear that the manufacturer should absorb the costs when
parts and computer code supplied by other companies may be the root
cause. Second, to the extent that it makes sense to provide incentives for
the producers of the components of driver-less cars to continue to
innovate and improve their products, insulating them from cost-sharing
even in these kind, of one-off incidents seems problematic. The counterargument
would of course be that under current law the injured parties
are unlikely to have any claim against the component producers, and the
manufacturer almost certainly could not bring an action for contribution
or indemnity against a component manufacturer without evidence that a
design or manufacturing defect in the component was at fault.93 So
unless the courts address this issue in fashioning a strict liability regime,
the manufacturer, and the manufacturer alone, is likely to bear all of the
liability.
92. For that reason, it does not seem fair to saddle the vehicle's owner with the liability. The
scenario assumes that the owner's conduct has nothing at all to do with the accident. Nonetheless, if
the risk of accidents involving driver-less cars is as low as some experts forecast, then the costs of
insuring the vehicle may be so low that it is simply easier for all concerned to make the owner,
through mandatory insurance (as is the case in most states already), the responsible party. Moreover,
one could easily envision a system where private ownership of vehicles becomes a relic. Instead,
companies (or cities or smaller communities) would own fleets of driver-less cars and dispatch them
when requested, the way we now use cab services or Uber. This approach could also be used to
promote ride-sharing, thereby further reducing the demand for vehicles, fuel costs, and traffic
congestion.
93. See supra notes 37-39 and accompanying text. There is an additional point that might cut in
favor of having the manufacturer bear the costs. If one accepts the principles of "Moore's Law"that
computing power doubles every eighteen months-we can expect that the costs of the software
components will decrease or remain stable over time while the cost of the other components of the
vehicle will continue to rise. If this prediction is borne out, then the manufacturer will increasingly
be in the best position to pay, as the software will be an ever decreasing part of the vehicle's overall
cost.
148 [Vol. 89:117
MACHINES WITHOUT PRINCIPALS
One way-perhaps the only way under conventional law-to untie
this Gordian Knot would be to apply a variation on the settled doctrine
of "common enterprise" liability. Under that theory, "each entity within
a set of interrelated companies may be held jointly and severally liable
for the actions of other entities that are part of the group." 94 The
variations here would require a substantial deviation from the doctrine.
For one thing, the proposed liability theory would not require that the
companies function jointly; all that would be required is that they work
to a common end-to design, program, and manufacture a driver-less car
and its various component parts. For another, there is the question of
fault. The common enterprise doctrine is ordinarily applied to ensure
that once liability has been established, all of the participating
wrongdoers are held to account. Here, of course, there would be no
"wrongdoers." There would instead be an inference of liability drawn by
operation of law to protect a blameless party (the person who sustained
injured) by making others bear the cost. But the basic point is the same:
A common enterprise theory permits the law to impose joint liability
without having to lay bare and grapple with the details of assigning
every aspect of wrongdoing to one party or another; it is enough that in
pursuit of a common aim the parties engaged in wrongdoing. That
principle could be engrafted onto a new, strict liability regime to address
the harms that may be visited on humans by intelligent autonomous
machines when it is impossible or impracticable to assign fault to a
specific person.95
There is one other possible solution that avoids the legal fiction of
having the creators of the vehicle bear the cost even without a hint of
94. FTC v. Tax Club Inc., F. Supp. 2d , 2014 WL 199514, at *5 (S.D.N.Y. Jan. 17, 2014);
see alsoFTC v. Network Servs. Depot, Inc., 617 F.3d 1127, 1142-43 (9th Cir. 2010); SEC v. R.G.
Reynolds Enters., Inc., 952 F.2d 1125, 1130 (9th Cir. 1991). To determine whether a common
enterprise exists, the court considers factors such as common control; the sharing of office space
and officers; whether business is transacted through a maze of interrelated companies; the
commingling of corporate funds and failure to maintain separation of companies; unified
advertising; and evidence that reveals that no real distinction exists between the corporate
defendants. FTC. v. Grant Connect, LLC, 827 F. Supp. 2d 1199, 1216 (D. Nev. 2011) (quoting FTC
v. Nat'l Urological Grp., Inc., 645 F. Supp. 2d 1167, 1182 (N.D. Ga. 2008)).
95. There are other liability systems that are similar to the one suggested here. One form of
"enterprise liability" is the system underlying the National Childhood Vaccine Injury Act of 1986,
which sets up a no-fault system of compensation for children injured by certain vaccines, with
funding mainly from the companies that make and sell the vaccines. See 42 U.S.C. §§ 300aa-1-
300aa-34 (2006). The Act imposes a tax on the sale of vaccines to create a fund to pay for injuries
attributable to vaccines. See 26 U.S.C. § 9510 (2006) (establishing the Vaccine Injury
Compensation Trust Fund). If the plaintiff determines that the compensation offered by the program
is inadequate, the plaintiff may then file suit against the manufacturer, but most cases are resolved
without litigation. See generally Bruesewitz v. Wyeth LLC, -U.S.-, 131 S. Ct. 1068 (2011).
2014] 149
WASHINGTON LAW REVIEW
fault, and that is to have the vehicle itself bear the cost. Instead of suing
the manufacturer, let the injured party do what is now not possible-sue
the vehicle. As suggested earlier, at some point the courts will have to
wrestle with the underlying question of how to treat machines that are
agents of no one-i.e., machines without principals. One solution would
be to reconceptualize these autonomous, intelligent machines as entities
with the status of a "person" under the law. Conferring "personhood" on
these machines would resolve the agency question; the machines
become principals in their own right, and along with new legal status
would come new legal burdens, including the burden of self-insurance.
This is a different form of cost-spreading than focusing on the vehicle's
creators, and it may have the virtue of necessitating that a broader
audience-including the vehicle's owner-participate in funding the
insurance pool, and that too may be more fair.
CONCLUSION
The introduction of highly sophisticated autonomous machines may
be literally around the corner. Truly autonomous machines may be
driving cars through our neighborhoods or piloting drones that fly above
our heads sooner than we think. So long as we can conceive of these
machines as "agents" of some legal person (individual or virtual), our
current system of products liability will be able to address the legal
issues surrounding their introduction without significant modification.
But the law is not necessarily equipped to address the legal issues that
will start to arise when the inevitable occurs and these machines cause
injury, but when there is no "principal" directing the actions of the
machine. How the law chooses to treat machines without principals will
be the central legal question that accompanies the introduction of truly
autonomous machines, and at some point, the law will need to have an
answer to that question.
150 [Vol. 89:117