Could a machine think? In science fiction stories there are many examples of machines
which can think. But what would be involved? Is there any reason to believe that some
machines can already think? Computers seem the most plausible candidates. Those
who defend a position known as Strong AI (‘AI’ stands for artificial intelligence) believe
that artificial intelligence is a genuine possibility, not simply a metaphorical use of
language. In this ingenious paper John Searle (1932– ) concocts a thought experiment
designed to refute Strong AI. His conclusion is that computers can manipulate symbols,
but there is more to thinking than this. This is not to say that no machine can think:
we are in a sense machines ourselves.
*
What psychological and philosophical signiﬁcance should we attach to recent
efforts at computer simulations of human cognitive capacities? In answering
this question, I ﬁnd it useful to distinguish what I will call “strong” AI from
“weak” or “cautious” AI. According to weak AI, the principal value of the
computer in the study of the mind is that it gives us a very powerful tool. For
example, it enables us to formulate and test hypotheses in a more rigorous
and precise fashion. But according to strong AI, the computer is not merely
a tool in the study of the mind; rather, the appropriately programmed
computer really is a mind, in the sense that computers given the right
programs can be literally said to understand and have other cognitive states.
In strong AI, because the programmed computer has cognitive states, the
programs are not mere tools that enable us to test psychological explanations;
rather, the programs are themselves the explanations.
1111
2
3
4
5
6
711
8
9
10111
1
2
3
4
5
6
7
811
9
20111
1
2
3
4
5
6
7
8
9
30
1
2
3
4
5
6
7
8
911
40
1
2
3
4
4511
49
MINDS, BRAINS, AND
PROGRAMS
John R. Searle
From Behavioral and Brain Sciences, vol. 3, 1980
I have no objection to the claims of weak AI, at least as far as this article
is concerned. My discussion here will be directed at the claims I have deﬁned
as those of strong AI, speciﬁcally the claim that the appropriately programmed
computer literally has cognitive states and that the programs thereby explain
human cognition. When I hereafter refer to AI, I have in mind the strong
version, as expressed by these two claims.
I will consider the work of Roger Schank and his colleagues at Yale (Schank
and Abelson 1977), because I am more familiar with it than I am with any other
similar claims, and because it provides a very clear example of the sort of work
I wish to examine. But nothing that follows depends upon the details of
Schank’s programs. The same arguments would apply to Winograd’s SHRDLU,
Weizenbaum’s ELIZA, and indeed any Turing machine simulation of human
mental phenomena.
Very brieﬂy, and leaving out the various details, one can describe Schank’s
program as follows: The aim of the program is to simulate the human ability
to understand stories. It is characteristic of human beings’ story-understanding
capacity that they can answer questions about the story even though the
information that they give was never explicitly stated in the story. Thus, for
example, suppose you were given the following story: “A man went into a
restaurant and ordered a hamburger. When the hamburger arrived it was
burned to a crisp, and the man stormed out of the restaurant angrily, without
paying for the hamburger or leaving a tip.” Now, if you are asked “Did the
man eat the hamburger?” you will presumably answer, “No, he did not.”
Similarly, if you are given the following story: “A man went into a restaurant
and ordered a hamburger; when the hamburger came he was very pleased
with it; and as he left the restaurant he gave the waitress a large tip before
paying his bill,” and you are asked the question, “Did the man eat the
hamburger?” you will presumably answer, “Yes, he ate the hamburger.” Now
Schank’s machines can similarly answer questions about restaurants in this
fashion. To do this, they have a “representation” of the sort of information
that human beings have about restaurants, which enables them to answer such
questions as those above, given these sorts of stories. When the machine is
given the story and then asked the question, the machine will print out answers
of the sort that we would expect human beings to give if told similar stories.
Partisans of strong AI claim that in this question and answer sequence the
machine is not only simulating a human ability but also (1) that the machine
can literally be said to understand the story and provide the answers to questions,
and (2) that what the machine and its program do explains the human
ability to understand the story and answer questions about it.
Both claims seem to me to be totally unsupported by Schank’s work, as I
will attempt to show in what follows. I am not, of course, saying that Schank
himself is committed to these claims.
One way to test any theory of the mind is to ask oneself what it would be
like if my mind actually worked on the principles that the theory says all
minds work on. Let us apply this test to the Schank program with the following
John R. Searle404
Gedankenexperiment. Suppose that I’m locked in a room and given a large
batch of Chinese writing. Suppose furthermore (as is indeed the case) that I
know no Chinese, either written or spoken, and that I’m not even conﬁdent
that I could recognize Chinese writing as Chinese writing distinct from, say,
Japanese writing or meaningless squiggles. To me, Chinese writing is just so
many meaningless squiggles. Now suppose further that after this ﬁrst batch
of Chinese writing I am given a second batch of Chinese script together with
a set of rules for correlating the second batch with the ﬁrst batch. The rules
are in English, and I understand these rules as well as any other native speaker
of English. They enable me to correlate one set of formal symbols with another
set of formal symbols, and all that “formal” means here is that I can identify
the symbols entirely by their shapes. Now suppose also that I am given a third
batch of Chinese symbols together with some instructions, again in English,
that enable me to correlate elements of this third batch with the ﬁrst two
batches, and these rules instruct me how to give back certain Chinese symbols
with certain sorts of shapes in response to certain sorts of shapes given me in
the third batch. Unknown to me, the people who are giving me all of these
symbols call the ﬁrst batch a “script,” they call the second batch a “story,”
and they call the third batch “questions.” Furthermore, they call the symbols
I give them back in response to the third batch “answers to the questions,”
and the set of rules in English that they gave me, they call the “program.”
Now just to complicate the story a little, imagine that these people also give
me stories in English, which I understand, and they then ask me questions in
English about these stories, and I give them back answers in English. Suppose
also that after a while I get so good at following the instructions for manipulating
the Chinese symbols and the programmers get so good at writing the
programs that from the external point of view – that is, from the point of
view of somebody outside the room in which I am locked – my answers to
the questions are absolutely indistinguishable from those of native Chinese
speakers. Nobody just looking at my answers can tell that I don’t speak a word
of Chinese. Let us also suppose that my answers to the English questions are,
as they no doubt would be, indistinguishable from those of other native
English speakers, for the simple reason that I am a native English speaker.
From the external point of view – from the point of view of someone reading
my “answers” – the answers to the Chinese questions and the English questions
are equally good. But in the Chinese case, unlike the English case, I
produce the answers by manipulating uninterpreted formal symbols. As far
as the Chinese is concerned, I simply behave like a computer; I perform computational
operations on formally speciﬁed elements. For the purposes of the
Chinese, I am simply an instantiation of the computer program.
Now the claims made by strong AI are that the programmed computer
understands the stories and that the program in some sense explains human
understanding. But we are now in a position to examine these claims in light
of our thought experiment.
Minds, brains, and programs 405
1111
2
3
4
5
6
711
8
9
10111
1
2
3
4
5
6
7
811
9
20111
1
2
3
4
5
6
7
8
9
30
1
2
3
4
5
6
7
8
911
40
1
2
3
4
4511
1 As regards the ﬁrst claim, it seems to me quite obvious in the example
that I do not understand a word of the Chinese stories. I have inputs and
outputs that are indistinguishable from those of the native Chinese speaker,
and I can have any formal program you like, but I still understand nothing.
For the same reasons, Schank’s computer understands nothing of any stories,
whether in Chinese, English, or whatever, since in the Chinese case the
computer is me, and in cases where the computer is not me, the computer has
nothing more than I have in the case where I understand nothing.
2 As regards the second claim, that the program explains human understanding,
we can see that the computer and its program do not provide
sufﬁcient conditions of understanding since the computer and the program are
functioning, and there is no understanding. But does it even provide a necessary
condition or a signiﬁcant contribution to understanding? One of the
claims made by the supporters of strong AI is that when I understand a story
in English, what I am doing is exactly the same – or perhaps more of the same
– as what I was doing in manipulating the Chinese symbols. It is simply more
formal symbol manipulation that distinguishes the case in English, where I
do understand, from the case in Chinese, where I don’t. I have not demonstrated
that this claim is false, but it would certainly appear an incredible claim
in the example. Such plausibility as the claim has derives from the supposition
that we can construct a program that will have the same inputs and outputs
as native speakers, and in addition we assume that speakers have some level
of description where they are also instantiations of a program. On the basis
of these two assumptions we assume that even if Schank’s program isn’t the
whole story about understanding, it may be part of the story. Well, I suppose
that is an empirical possibility, but not the slightest reason has so far been
given to believe that it is true, since what is suggested – though certainly not
demonstrated – by the example is that the computer program is simply
irrelevant to my understanding of the story. In the Chinese case I have everything
that artiﬁcial intelligence can put into me by way of a program, and
I understand nothing; in the English case I understand everything, and
there is so far no reason at all to suppose that my understanding has anything
to do with computer programs, that is, with computational operations on
purely formally speciﬁed elements. As long as the program is deﬁned in terms
of computational operations on purely formally deﬁned elements, what the
example suggests is that these by themselves have no interesting connection
with understanding. They are certainly not sufﬁcient conditions, and not
the slightest reason has been given to suppose that they are necessary conditions
or even that they make a signiﬁcant contribution to understanding.
Notice that the force of the argument is not simply that different machines
can have the same input and output while operating on different formal
principles – that is not the point at all. Rather, whatever purely formal
principles you put into the computer, they will not be sufﬁcient for understanding,
since a human will be able to follow the formal principles without
John R. Searle406
understanding anything. No reason whatever has been offered to suppose that
such principles are necessary or even contributory, since no reason has been
given to suppose that when I understand English I am operating with any
formal program at all.
Well, then, what is it that I have in the case of the English sentences that
I do not have in the case of the Chinese sentences? The obvious answer is that
I know what the former mean, while I haven’t the faintest idea what the latter
mean. But in what does this consist and why couldn’t we give it to a machine,
whatever it is? I will return to this question later, but ﬁrst I want to continue
with the example.
I have had the occasions to present this example to several workers in artiﬁcial
intelligence, and, interestingly, they do not seem to agree on what the
proper reply to it is. I get a surprising variety of replies, and in what follows
I will consider the most common of these (speciﬁed along with their geographic
origins).
But ﬁrst I want to block some common misunderstandings about “understanding”:
In many of these discussions one ﬁnds a lot of fancy footwork about
the word “understanding.” My critics point out that there are many different
degrees of understanding; that “understanding” is not a simple two-place predicate;
that there are even different kinds and levels of understanding, and often
the law of excluded middle doesn’t even apply in a straightforward way to
statements of the form “x understands y”; that in many cases it is a matter
for decision and not a simple matter of fact whether x understands y; and so
on. To all of these points I want to say: of course, of course. But they have
nothing to do with the points at issue. There are clear cases in which “understanding”
literally applies and clear cases in which it does not apply; and these
two sorts of cases are all I need for this argument.1
I understand stories in
English; to a lesser degree I can understand stories in French; to a still lesser
degree, stories in German; and in Chinese, not at all. My car and my adding
machine, on the other hand, understand nothing: they are not in that line of
business. We often attribute “understanding” and other cognitive predicates
by metaphor and analogy to cars, adding machines, and other artifacts, but
nothing is proved by such attributions. We say, “The door knows when to
open because of its photoelectric cell. The adding machine knows how (understands
how, is able) to do addition and subtraction but not division,” and “The
thermostat perceives changes in the temperature.” The reason we make these
attributions is quite interesting, and it has to do with the fact that in artifacts
we extend our own intentionality;2
our tools are extensions of our purposes,
and so we ﬁnd it natural to make metaphorical attributions of intentionality
to them; but I take it no philosophical ice is cut by such examples. The sense
in which an automatic door “understands instructions” from its photoelectric
cell is not at all the sense in which I understand English. If the sense in which
Schank’s programmed computers understand stories is supposed to be the
metaphorical sense in which the door understands, and not the sense in which
I understand English, the issue would not be worth discussing. But Newell
Minds, brains, and programs 407
1111
2
3
4
5
6
711
8
9
10111
1
2
3
4
5
6
7
811
9
20111
1
2
3
4
5
6
7
8
9
30
1
2
3
4
5
6
7
8
911
40
1
2
3
4
4511
and Simon (1963) write that the kind of cognition they claim for computers
is exactly the same as for human beings. I like the straightforwardness of this
claim, and it is the sort of claim I will be considering. I will argue that in
the literal sense the programmed computer understands what the car and the
adding machine understand, namely, exactly nothing. The computer understanding
is not just (like my understanding of German) partial or incomplete;
it is zero.
Now to the replies:
1. THE SYSTEMS REPLY (BERKELEY)
“While it is true that the individual person who is locked in the room does
not understand the story, the fact is that he is merely part of a whole system,
and the system does understand the story. The person has a large ledger in
front of him in which are written the rules, he has a lot of scratch paper and
pencils for doing calculations, he has ‘data banks’ of sets of Chinese symbols.
Now, understanding is not being ascribed to the mere individual; rather it is
being ascribed to this whole system of which he is a part.”
My response to the systems theory is quite simple: Let the individual internalize
all of these elements of the system. He memorizes the rules in the
ledger and the data banks of Chinese symbols, and he does all the calculations
in his head. The individual then incorporates the entire system. There isn’t
anything at all to the system that he does not encompass. We can even get
rid of the room and suppose he works outdoors. All the same, he understands
nothing of the Chinese, and a fortiori neither does the system, because there
isn’t anything in the system that isn’t in him. If he doesn’t understand, then
there is no way the system could understand because the system is just a part
of him.
Actually I feel somewhat embarrassed to give even this answer to the
systems theory because the theory seems to me so implausible to start with.
The idea is that while a person doesn’t understand Chinese, somehow the connection
of that person and bits of paper might understand Chinese. It is not easy
for me to imagine how someone who was not in the grip of an ideology would
ﬁnd the idea at all plausible. Still, I think many people who are committed to
the ideology of strong AI will in the end be inclined to say something very much
like this; so let us pursue it a bit further. According to one version of this view,
while the man in the internalized systems example doesn’t understand Chinese
in the sense that a native Chinese speaker does (because, for example, he doesn’t
know that the story refers to restaurants and hamburgers, etc.), still “the man
as a formal symbol manipulation system” really does understand Chinese. The
subsystem of the man that is the formal symbol manipulation system for
Chinese should not be confused with the subsystem for English.
So there are really two subsystems in the man; one understands English,
the other Chinese, and “it’s just that the two systems have little to do with
each other.” But, I want to reply, not only do they have little to do with each
John R. Searle408