American Political Science Review Vo1. 89, No. 2 Tune 1995
THE IMPORTANCE OF RESEARCH DESIGN IN POLITICAL SCIENCE
GARY KING, ROBERT 0.KEOHANE, and SIDNEY VERBA Harvard University
Receiving five serious reviews in this syrnposium
is gratifymg and confirms our belief that
research design should be a priority for our
discipline. We are pleased that our five distinguished
reviewers appear to agree with our unified approach
to the logic of inference in the social sciences, and
with our fundamental point: that good quantitative
and good qualitative research designs are based fundamentally
on the same logic of inference. The reviewers
also raised virtuallyno objections to the main
practical contribution of our book--our many specific
procedures for avoiding bias, getting the most out of
qualitative data, and making reliable inferences.
However, the reviewsmake clear that although our
book may be the latest word on research design in
political science, it is surely not the last. We are taxed
for failing to include important issues in our analysis
and for dealing inadequately with some of what we
included. Before responding to the reviewers' most
direct criticisms, let us explain what we emphasize in
Designing Social Inquiry and how it relates to some of
the points raised by the reviewers.
WHAT WE TRIED TO DO
Designing Social Inquiry grew out of our discussions
while coteaching a graduate seminar on research
design, reflecting on job talks in our department, and
reading the professional literature in our respective
subfields. Although many of the students, job candidates,
and authors were highly sophisticated qualitative
and quantitative data collectors, interviewers,
soakers and pokers, theorists, philosophers, formal
modelers, and advanced statistical analysts, many
nevertheless had trouble defining a research question
and designing the empirical research to answer it.
The students proposed impossible fieldwork to answer
unanswerable questions. Even many active
scholarshad difficulty with the basic questions: What
do you want to find out? How are you going to find
it out? and, above all, How would you know if you
were right or wrong?
We found conventional statistical training to be
only marginally relevant to those with qualitative
data. We even found it inadequate for students with
projects amenable to quantitative analysis, since social
science statistics texts do not frequently focus on
research design in observational settings. With a few
important exceptions, the scholarly literatures in
quantitative political methodology and other social
science statistics fields treat existing data and their
problems as given. As a result, these literatures
largely ignore research design and, instead, focus on
making valid inferences through statistical corrections
to data problems. This approach has led to some
dramatic progress; but it slights the advantage of
improving research design to produce better data in
the first place, which almost always improves inferences
more than the necessarily after-the-fact statistical
solutions.
This lack of focus on research design in social
sciencestatisticsis as surprising as it is disappointing,
since some of the most historically important works
in the more general field of statistics are devoted to
problems of research design (see, e.g., Fisher (1935)
The Design of Experiments). Experiments in the social
sciences are relatively uncommon, but we can still
have an enormous effect on the value of our aualitative
or quantitative information, even withoui statistical
corrections, by improving the design of our
research. We hope our book will help move these
fields toward studying innovations in research de-
sim."
We culled much useful information from the social
science statistics literatures and qualitative methods
fields. But for our goal of explicatingand unifying the
logic of inference, both literatures had problems.
Social science statistics focuses too little on research
design, and its language seems arcane if not impenetrable.
The numerous languages used to describe
methods in qualitativeresearch are diverse, inconsistent
in jargon and methodological advice, and not
always helpful to researchers. We agree with David
Collier that asvects of our advice can be revhrased
into some of the languages used in the q;alitative
methods literature or that used by quantitative researchers.
We hope our unified logic and, as David
Laitin puts it, our "common vocabulary" will help
foster communication about these imvortant issues
among all social scientists. But we bekeve that any
coherent language could be used to convey the same
ideas.- - - - We
demonstrated that "the differences between
the quantitative and qualitative traditions are only
stylistic and are methodologically and substantively
unimportant" (p. 4). Indeed, much of the best social
science research can combine quantitative and qualitative
data, precisely because there is no contradiction
between the fundamental processes of inference
involved in each. Sidney Tarrow asks whether we
agree that "it is the combination of quantitative and
qualitative" approaches that we desire (p. 473). We
do. But to combine both types of data sources productively,
researchers need to understand the fundamental
logic of inference and the more specific rules
and procedures that follow from an explicationof this
logic.
Social science, both quantitative and qualitative,
seeks to develop and evaluate theories. Our concern
is less with the development of theory than theory
evaluation-how to use the hard facts of empirical
reality to form scientific opinions about the theories
and generalizationsthat are the hoped for outcome of
Review Symposium June 1995
our efforts. Our social scientist uses theory to generate
observable implications, then systematically applies
publicly known procedures to infer from evidence
whether what the theory implied is correct. Some
theories emerge from detailed observation, but they
should be evaluated with new observations, preferably
ones that had not been gathered when the theories
were being formulated. Our logic of theory
evaluation stresses maximizing leverage-explaining
as much as possible with as little as possible. It also
stresses minimizing bias. Lastly, though it cannot
eliminate uncertainty, it encourages researchers to
report estimates of the uncertainty of their conclu-
sions.
Theory and empirical work, from this perspective,
cannot productivelyexist in isolation. We believe that
it should become standard practice to demand clear
implications of theory and observations checking
those implications derived through a method that
minimizes bias. We hope that Designing Social lnquiy
helps to "discipline political science" in this way, as
David Laitin recommends; and we hope, along with
James Caporaso, that "improvements in measurement
accuracy, theoreticalspecification, and research
should yield a smaller range of allowable outcomes
consistent with the predictions made" (p. 459).
Our book also contains much specific advice, some
of it new and some at least freshly stated. We explain
how to distinguish systematic from nonsystematic
components of phenomena under study and focus
explicitly on trade-offs that may exist between the
goals of unbiasedness and efficiency (chap. 2). We
discuss causality in relation to counterfactualanalysis
and what Paul Holland calls the "fundamental problem
of causal inference" and consider possible complications
introduced by thinking about causal mechanisms
and multiple causality (chap. 3). Our
discussion of counterfactualreasoning is, we believe,
consistent with Donald Campbell's "quasi-experimental"
emphasis; and we thank James Caporaso for
clarifying this.'
We pay special attention in chapter 4 to issues of
what to observe: how to avoid confusion about what
constitutes a "case" and, especially, how to avoid or
limit selection bias. We show that selection on values
of explanatory variables does not introduce bias but
that selection on values of dependent variables does
so; and we offer advice to researchers who cannot
avoid selecting on dependent variables.
We go on in chapter 5 to show that while random
measurement error in dependent variables does not
bias causal inferences (although it does reduce efficiency),
measurement error in explanatory variables
biases results in predictable ways. We also develop
procedures for correcting these biases even when
measurement error is unavoidable. In that same
chapter, we undertake a sustained analysis of endogeneity
(i.e., when a designated "dependent variable"
turns out to be causing what you thought was
your "explanatory variable") and omitted variable
bias, as well as how to control research situations so
as to mitigatethese problems. In the final chapter, we
specifyways to increasethe informationin qualitative
studies that can be used to evaluate theories; we
show how this can be accomplished without returning
to the field for additional data collection.
Throughout the book, we illustrate our propositions
not only with hypothetical examples but with reference
to some of the best contemporary research in
political science.
This statement of our purposes and fundamental
arguments should put some of the reviewers' complaints
about omissions into context. Our book is
about doing empirical research designed to evaluate
theories and learn about the world-to make inferences-not
about generating theories to evaluate.We
believe that researchers who understand how to
evaluate a theory will generate better theories-theories
that are not only more internally consistent but
that also have more observable implications (aremore
at risk of being wrong) and are more consistent with
prior evidence.If, as Laitin suggests, our singlemindedness
in driving home this argument led us implicitly
to downgrade the importance of such matters as
concept formation and theory creation in political
science, this was not our intention.
Designing Social Inquiry repeatedly emphasizes the
attributes of good theory. How else to avoid omitted
variable bias, choose causal effects to estimate, or
derive observable implications? We did not offer
much advice about what is often called the "irrational
nature of discovery," and we leave it to individual
researchers to decide what theories they feel are
worth evaluating. We do set forth some criteria for
choosing theories to evaluate-in terms of their importance
to social science and to the real world-but
our methodological advice about research design
applies to any type of theory. We come neither to
praise nor to bury rational-choice theory, nor to make
an argument in favor of deductive over inductive
theory. All we ask is that whatever theory is chosen
be evaluated by the same standards of inference.
Ronald Rogowski's favorite physicist, Richard Feynman,
explainsclearly how to evaluate a theory (which
he refers to as a "guess"): "If it disagrees with [the
empirical evidence], it is wrong. In that simple statement
is the key to science. It does not make any
difference how beautiful your guess is. It does not
make any difference how smart you are, who made
the guess, or what his name is-if it disagrees with
[the empirical evidence] it is wrong. That is all there
is to it" (1965, 156).'
One last point about our goal:we want to set a high
standard for research but not an impossible one. All
interesting qualitative and quantitative research
yields uncertain conclusions. We think that this fact
ought not to be dispiriting to researchers but should
rather caution us to be aware of this uncertainty,
remind us to make the best use of data possible, and
energize us to continue the struggle to improve our
stockof valid inferencesabout the politicalworld. We
show that uncertain inferences are every bit as scientific
as more certain ones so long as they are accom-
American Political Science Review Vol. 89, No. 2
panied by honest statements of the degree of uncertainty
accompaning each conclusion.
OUR ALLEGED ERRORS OF OMISSION
The major theme of what may seem to be the most
serious criticism offered above is stated forcefullyby
Ronald Rogowski. He fears that "devout attention"
to our criteria would "paralyze, rather than stimulate,
scientific inquiry." One of Rogowski's arguments,
echoed by Laitin, is that we are too obsessed with
increasing the amount of information we can bring to
bear on a theory and therefore fail to understand the
value of case studies. The other major argument,
made by both Rogowski and Collier is that we are too
critical of the practice of selecting observations according
to values of the dependent variable and that
we would thereby denigrate major work that engages
in this practice. We consider these arguments in turn.
Science as a Collective Enterprise
Rogowski argues that we would reject several classic
case studies in comparative politics. We think he
misunderstands these studies and misses our distinction
between a "single case" and a collection of
observations. Consider two works that he mentions,
The Politics of Accommodation, by Arend Lijphart (1968),
and The Nazi Seizure of Power, by William Sheridan
Allen (1965). Good research designs are rarely executed
by individual scholars isolated from prior researchers.
As we say in our book, "A single observation
can be useful for evaluating causal explanations
if it is part of a research program. If there are other
observations, perhaps gathered by other researchers,
against which it can be compared, it is no longer a
single observation" (p. 211; see also secs. 1.2.1,4.4.4,
the latter devoted entirely to this point). Rogowski
may have overlooked these passages. If we did not
emphasize the point suf€iciently, we are grateful for
the opportunity to stress it here.
Lijphart: The Case Study that
Broke the Pluralist Camel's Back
What was once called pluralist theory by David Truman
and others holds that divisions along religious
and class lines make polities less able to resolve
political arguments via peaceful means through democratic
institutions. The specific causal hypothesis is
that the existence of many cross-cutting cleavages
increases the level of social peace and, thus of stable,
legitimate democratic government.
In The Politics of Accommodation, Arend Lijphart
(1968) sought to estimate this causal effe~t.~In addition
to prior literature, he had evidencefrom only one
case, the Netherlands. He first found numerous
observable implications of his descriptive hypothesis
that the Netherlands had deep class and religious
cleavages, relatively few of which were cross-cutting.
Then-surprisingly from the perspective of pluralist
theory-he found considerable evidence from many
levels of analysis that the Netherlands was an especially
stable and peaceful democratic nation. These
descriptive inferences were valuable contributions to
social science and important in and of themselves,
but Lijphart also wished to study the broader causal
question.
In isolation, a single study of the Netherlands,
conducted only at the level of the nation at one point
in time, cannot produce a valid estimate of the causal
effect of cross-cutting cleavages on the degree of
social peace in a nation. But Lijphart was not working
in isolation. As part of a community of scholars, he
had the benefit of Truman and others having collected
many prior observations. By using this prior
work, Lijphart could and did make a valid inference.
Prior researchers had either focused only on countries
with the same value of the explanatory variable
(many cross-cutting cleavages) or on the basis of
values of the dependent variable (high social conflict).
Previous researchers therefore made invalid
inferences. Lijphart measured social peace for the
other value of the explanatory variable (few crosscutting
cleavages) and, by using his data in combination
with that which came before, made a valid
inference.
Lijphart's classicstudy is consistent with our model
of good research design. As he stressed repeatedly in
his book, Lijphart was contributing to a large scholarly
literature. As such, he was not trylng to estimate
a causal effect from a single observation; nor was he
selectingon his dependent variable. Harvesting relevant
information from others' data, although often
overlooked, may often be the best way to obtain
relevant information.
By ignoring the place of Lijphart's book in the
literature to which it was contributing, Rogowskiwas
unable to recognize the nature of its contribution.
Rogowski's alternative explanation for the importance
of this book and the others he mentions-that
"(1) all of them tested, relied on, or proposed, clear
and precise theories; and (2) all focused on anomalies"
(p. 469)-suggests one of many possible strategies for
choosingtopics to research; but it is of almost no help
with practical issues of research design or ascertaining
whether a theory is right or wrong. Indeed, the
only way to determine whether something is an
anomaly in the first place is to follow a clear logic of
scientific inference and theory evaluation, such as
that provided in Designing Social Inquiry.
Allen: Distinguishing History From Social Science
The Nazi Seizure of Power is an account of life in an
ordinary German community during the Nazi seizure
of power. Allen is not a social scientist: In his book,
he proposes no generalization, evaluates no theory,
and does not refer to the scholarlyliteratures on Nazi
Germany; rather, he zeroes in on the story of what
happened in one small place at a crucial moment in
history, and he does so brilliantly. In our terms, he is
Review Symposium June 1995
describing historical detail and occasionally also conducting
very limited descriptive inference. We emphasize
the importance of such work: "Particular
events such as the French Revolution or the Democratic
Senate primary in Texas may be of intrinsic
interest: they pique our curiosity, and if they were
preconditions for subsequent events (such as the
Napoleonic Wars or Johnson's presidency) we may
need to know about them to understand those later
events" (p. 36).
In our view, social science must go further than
Allen. The social scientist must make descriptive or
causal inferences, thus seeking explanation and generalization.
Indeed, we think even Rogowski would
not accept Allen's classic work of history as a dissertation
in political science. Allen's work is, however,
not irrelevant to the task of explanation and generalization
that is of interest to us. In the hands of a good
social scientist, who could place Allen's work within
an intellectual tradition, it becomes a single case
study in the framework of many others. This, of
course, suggests one traditional and important way
in which social scientists can increase the amount of
information they canbring to bear on a problem:read
the descriptive case study literature.
The Perils of Avoiding Selection Bias
We agree with David Collier's observationthat, if our
arguments concerning selection bias are sustained,
then "a small improvement in methodological selfawareness
can yield a large improvement in scholarship"
(p. 461). Indeed, because qualitative researchers
generally have more control over the selection of
their observations than over most other features of
their research designs, selection is an especially important
concern (a topic to which we devote most of
our chap. 4).4
Rogowski believes that we would criticize Peter
Katzenstein's (1985) Small States in World Markets or
Robert Bates's (1981) Markets and States in Tropical
Afrrca as inadmissibly selecting on the dependent
variable. We address each book in turn.
Katzenstein: DistinguishingDescriptive
Inference from Causal Inference
Peter Katzenstein's (1985) Small States in World Markets
makes some important descriptive inferences.
For example, Katzenstein shows that small European
states responded flexibly and effectively to the economic
challenges that they faced during the 40 years
after World War 11; and he distinguishes between
what he calls "liberal and social corporatism" as two
patterns of response. But many of Katzenstein's
arguments also imply causal claims-that in Western
Europe "small size has facilitated economicopenness
and democratic corporatism" (p. 80), and that in the
small European states, weak landed aristocracies,
relatively strong urban sectors, and strong links between
country and city led to cross-classcompromise
in the 1930s, creating the basis for postwar corporatism
(chap. 4).
Katzenstein seeks to test the first of these causal
claimsby comparing economicopenness in smalland
large states (1985, table 1, p. 86). To evaluate the
second hypothesis, he compares cross-class compromise
in six small European states characterized by
weak landed aristocracies and strong urban sectors,
with the relative absence of such compromisein five
large industrialized countries and Austria, which had
different values on these explanatoryvariables. Much
of his analysis follows the rules of scientific inference
we discuss-selecting cases to vary the value of the
explanatory variables, specifying the observable irnplications
of theories, seeking to determine whether
the facts meet theoretical expectations.
But Katzenstein fudges the issue of causal inference
by disavowing claims to causal validity: "Analyses
like this one cannot meet the exacting standards
of a social science test that asks for a distinction
between necessary and sufficient conditions, a
weighting of the relative importance of variables,
and, if possible, a proof of causality" (p. 138). However,
estimating causal inferences does not require a
"distinction between necessary and sufficient conditions,
a weighting of the relative importance of variables,"
or an absolute "proof" of anything. Katzenstein
thus unnecessarily avoids causal language and
explicit attention to the logic of inference which
results. As we explain in our book, "Avoiding causal
language when causality is the real subject of investigation
either renders the research irrelevant or permits
it to remain undisciplined by the rules of scientific
inference" (p. 76).
Remaining inexplicit about causal inference makes
some of Katzenstein's clauns ambiguous or unsupported.
For example, his conclusion seems to argue
that small states' corporatist strategies are responsible
for their postwar economic success. But because of
the selection bias induced by his decision to study
only successful cases, Katzenstein cannot rule out an
important alternative causal hypothesis-that any of
a variety of other factors accounts for this uniform
pattern. For instance, the postwar international political
economy may have been benign for small,
developed countries in Europe. If so, corporatist
strategies may have been unrelated to the degree of
success experienced by small European states.
In the absence of variation in the strategies of his
states, valid causal inferences about their effects remain
elusive. Had Katzensteinbeen more attentive to
the problems of causal inference that we discuss, he
would have been able to claim causalvalidity in some
limited instances, such as when he had variation in
his explanatory and dependent variables (as in the
1930s analysis). More importantly, he would also
have been able to improve his research design so that
valid causal inferences were also possible in many
other areas.
Rogowski is not correct in inferring that we would
dismiss the significance of Small States in World Markets.
Its descriptions are rich and fascinating, it elab-
American Political Science Review Vol. 89, No. 2
orates insightful concepts such as liberal and social
corporatism, and it provides some evidence for a few
causal inferences.It is a finebook, but we believe that
more explicit attention to the logic of inference could
have made it even better.
Bates: How to Identify a Dependent Variable
Rogowski claims that Robert Bates's purpose in Marhts
and States was to explain economic failure in
tropical African states and that by choosing only
states with failed economies and low agricultural
production, Bates biased his inferences. If agricultural
production were Bates's dependent variable,
Rogowski would be correct, since (as we describe in
Designing Social lnquiy and as elaborated by Collier),
using-but not correctingfor-this type of case selection
does bias inferences. However, low agricultural
production was, in fact, not Bates's dependent variable.
Bates's book makes plain his two dependent variables:
(1)the variations in public policies promulgated
by African statesand (2)differencesin thegrouprelations
between the farmer and the state in each country.
Both variablesvary considerably across his cases. Bates
also proposed several explanatory variables, which he
derived from his preliminary descriptive inferences.
These include (1)whether state marketingboards were
founded by the producers or by alliances between
government and tradinginterests, (2)whether urban or
rural interests dominated the first postcolonial govemment,
(3)the degree of governmental committrnentto
spending programs, (4) the availability of nonagricultural
sources for governmental funds, and (5)
whether the crops produced were for food or export.
These explanatoryvariablesdo vary, and they helped
account for the variations in public policy and statefarmer
relations that Bates observed.
As such, Bates did not select his observations so
they had a constant value for his dependent variable.
Moreover, he did not stop at the national level of
analysis, for which he had a small number of cases
and relatively little information. Instead, he offered
numerous observable implications of the effects of
these explanatoryvariables at other levels of analyses
within each country. As with many qualitative studies,
Bates had a small number of cases but an immense
amount of information. We believe one of the
reasons Bates's study is-and should be-so highly
regarded is that it is an excellent example of a
qualitative study that conforms to the rules of scientific
inference. In sum, Rogowski says that Bates had
an excellent book that we would reject. If the book
were as Rogowski describes it, we very well might
reject it. Sinceit is not--and indeed is a good example
of our logic of research design-we join Rogowski in
applauding it.5
TRIANGULAR CONCLUSIONS
We conclude by emphasizing a point that is emphasized
both in Designing Social Inquiy and in the
reviews. We often suggest procedures that qualitative
researchers can use to increase the amount of information
theybring to bear on evaluatinga theory. This
is sometimesreferred to as "increasing the number of
observations." As all our reviewers recognize, we do
not expect researchers to increase the number of
full-blown case studies to conduct a large-n statistical
analysis: our point is not to make quantitative researchers
out of qualitativeresearchers. In fact, most
qualitative studies already contain a vast amount of
information. Our point is that appropriately marshaling
all the thick description and rich contextualization
in a typical qualitative study to evaluate a specific
theory or hypothesis can produce a very powerful
research design. Our book demonstrates how to
design research in order to collect the most useful
qualitative data and how to restructure it even after
data collection is finished, to turn qualitative information
into ways of evaluating a specific theory. We
explain how researchers can do this by collecting
more observations on their dependent variable, by
observing the same variable in another context, or by
observing another dependent variable that is an implication
of the same theory. We also show how one
can design theories to produce more observable implications
that then put the theory at risk of being
wrong more often and easily.
This brings us to Sidney Tarrow's suggestions for
using the comparativeadvantages of both qualitative
and quantitative researchers. Tarrow is interested
specifically in how unsystematic and systematic variables
and patterns interact, and seems to think that
principles could be derived to determine what unsystematic
events to examine. We think that this is an
interesting question for any historically-sensitive
work. Many unsystematic, nonrepeated events occur,
a few of which may alter the path of history in
significant ways; and it would be useful to have
criteria to determine how these events interact with
systematic patterns. We expect that our discussions
of scientific inferencecould help in identrfylngwhich
apparently random, but critical, events to study in
specific instances, and we are confident that our logic
of inference will help determine whether these inferences
are correct;but Tarrow or others may be able to
use the insights from qualitativeresearchers to specify
them more clearly. We would look forward to a
book or article that presented such criteria.
Another major point made by Tarrow is that all
appropriate methods to study a question should be
employed. We agree: a major theme of our book is
that there is a singleunified logic of inference. Hence
it is possible effectivelyto combinedifferentmethods.
However, the issue of triangulation that Tarrow so
effectively raises is not the use of different logics or
methods, as he argues, but the triangulation of diverse
data sources trained on the same problem. Triangulation
involves data collected at different places,
sources, times, levels of analysis, or perspectives,
data that might be quantitative, or might involve
intensive interviews or thick historical description.
The best method should be chosen for each data
Review Symposium June 1995
source. But more data are better. Triangulation, then,
is another word for refering to the practice of increasing
the amount of information to bear on a theory or
hypothesis, and that is what our book is about.
Notes
The table of contents, preface, and chapter 1 of Designing
Social Inquiry are available via Gopher from hdc-gopher.har-
vard.edu.
1. To clarify further, we note that the definition of an
"experiment" is investigator control over the assignment of
values of explanatory variables to subjects. Caporaso emphasized
also the value of random assignment, which is desirable
in some situations (but not in others, see pp. 124-8) and
sometimes achievable in experiments. (Random selection and
a large number of units are also desirable and also necessary
for relatively automatic unbiased inferences, but experimenters
are rarely able to accomplish either.) A "quasi4xperiment"
is an observational study with an exogenous explanatory
variable that the investigator does not control. Thus, it is
not an experiment. Campbell's choice of the word "quasiexperiment"
reflected his insight that observational studies
follow the same logic of inference as experiments. Thus, we
obviously agree with Campbell's and Caporaso's emphases
and ideas and only pointed out that the word "quasiexperiment"
adds another word to our lexicon with no
additional content. Its a line idea, much of which we have
adopted; but it is an unnecessary category.
2. Telling researchers to "choose better theories" is not
much different than telling them to choose the right answer:
it is correct but not helpful. Many believe that deriving rules
for theory creation is impossible (e.g., Popper, Feynman), but
we see no compelling justification for this absolutist claim. As
David Laitin correctly emphasizes, "the development of formal
criteria for such an endeavor is consistent with the
authors' goals."
3. Lijphart also went to great lengths to clarify the precise
theory he was investigating, because it was widely recognized
that the concept of pluralism was often used in conflicting
ways, none clear or concrete enough to be called a theory.
Ronald Rogowski's description of pluralism as a "powerful,
deductive, internally consistent theory" (p. 10) is surely the
first time it has received such accolades.
4. Selectionproblems are easily misunderstood. For example,
Caporaso claimsthat "if selectionbiases operate independently
of one's hypothesized causal variable, it is a threat to
internal validity; if these same selection factors interact with
the causal variable, it is a threat to external validity" (p. 9). To
see that this claim is false, note, as Collier reemphasizes, that
Caporaso's "selection factors" can also be seen as an omitted
variable. But omitted variables cannot cause bias if they are
independent of your key causal variable. Thus, although the
distinction between internal and external validity is often
useful, it is not relevant to selectionbias in the way Caporaso
describes.
5. Subsequently, Bates pursued the same research program.
For example, in Essays on the Political Economy of Rural
Africa he evaluated his thesis for two additional areas--
colonial Ghana and Kenya (1983, chap. 3). So Bates does
exactlywhat we recommend: having developed his theory in
one domain, he extracts its observable implications and
moves to other domains to see whether he observes what the
theory would lead him to expect.
Symposium References
Achen, Christopher H. 1982. Interpreting and Using Regression
Analysis. University Paper series on Quantitative Applications
in the Social Sciences, no. 29. Beverly Hills: Sage.
Achen, Christopher H. 1986. The Statistical Analysis of QuasiExperiments.
Berkeley: University of California Press.
Achen, Christopher H., and Duncan Snidal. 1989. "Rational
Deterrence Theory and Comparative Case Studies." World
Politics 41:143-69.
Allen, William Sheridan. 1965. The Nazi Seizure of Power: The
Experience of a Single German Town, 1930-1935. New York:
Watts.
Almond, Gabriel. 1990. A Divided Discipline. Newbury Park,
CA: Sage.
Arendt, Hannah. 1958. The Origins of Totalitarianism. Cleveland,
OH: World.
Ayer, A. J. 1946. Language, Tmth, and Logic. 2d ed. London:
Gollancz.
Bakhtin, M. M. 1986. Speech Genres and Other Late Essays.
Trans. Vern M. McGee. Austin: University of Texas Press.
Bates, Robert H. 1981.Markets and States in Tropical Africa: The
Political Basis of Agrarian Policies. Berkeley: University of
California Press.
Bates, Robert H. 1983. Essays on the Political Economy of Rural
Africa. New York: Cambridge University Press.
Bendix, Reinhard. 1963. "Concepts and Generalizations in
Comparative SociologicalStudies." American SociologicalReview
28:532-39.
Berkson, Joseph. 1946. "Limitations of the Application of
Fourfold Table Analysis to Hospital Data." Biometries Bulletin
2:47-53.
Blalock, Hubert M. 1964. Causal Inferences in Non-experimental
Research. Chapel Hill: University of North Carolina Press.
Blalock, Hubert M. 1984. Basic Dilemmas in the Social Sciences.
Beverly Hills, CA: Sage.
Bunce, Valerie. 1981. Do New Leaders Make a Difference?Exclusive
Successionand Public Policy under Capitalismand Socialism.
Princeton: Princeton University Press.
Bourdieu, Peirre. 1984. Distinction: A Social Critique of the
Judgement of Taste. Trans. Richard Nice. Cambridge: Harvard
University Press.
Campbell, Donald T. 1969. "Reforms as Experiments." American
Psychologist 24409-29.
Campbell, Donald T., and Julian C. Stanley. 1963. Experimental
and Quasi-Exuerimental Desims for Research. Boston:- .
~ o u ~ h t o nMifflin.
Collier. David. 1993. "The Comparative Method." In Political~ ~ --
Science: The State of the ~iscipkne11, ed. Ada W. Finifter.
Washington: American Political Science Association.
Cook, Thomas D., and Donald T. Campbell. 1979. QuasiExperimentation.
Chicago: Rand-McNally.
Durkheim, Emile. 1938. The Rules of SociologicalMethod. Trans.
Sarah A. Solovay and John H. Mueller. New York: Free
Press.
Eckstein, Harry. 1975. "Case Study and Theory in Political
Science." In Handbook of Political Science, vol. 7, Strategies of
Inquiry, ed. Fred I. Greenstein and Nelson Polsby. Reading,
MA: Addison-Wesley.
Fearon, James D. 1990. "Counterfactuals and Hypothesis
Testing in Political Science." World Politics 43:169-95.
Feynman, Richard Phillips. 1965. The Characterof Physical Law.
Cambridge: MIT Press.
Fisher, Sir Ronald Aylmer. 1935. The Design of Experiments.
Edinburgh: Oliver & Boyd.
Foucault, Michel. 1972. The Archaeology of Knowledge. Trans.
A. M. Sheridan Smith. New York: Pantheon Books.
Geddes, Barbara. 1990. "How the Cases You Choose Affect
the Answers You Get: Selection Bias in Comparative Politics."
In Political Analysis, vol. 2, ed. James A. Stimson. Ann
Arbor: University of Michigan Press.
Geertz, Clifford. 1973. "Thick Description: Toward an Interpretative
Theory of Culture." In his Interpretationof Cultures.
New York: Basic Books.
George, Alexander, and Timothy J. McKeown. 1985. "Case
Studies and Theories of Organizational Decision Making."
Advances in Information Processes in Organizations 221-58.
Gourevitch, Peter Alexis. 1978. "The International System
and Regime Formation: A CriticalReview of Anderson and
Wallerstein." ComparativePolitics 10:419-38.
American Political Science Review Vol. 89, No. 2
Griffin, Larry J. 1992. "Temporality, Events, and Explanation
in Historical Sociology:An Introduction." Sociological Methods
and Research 20:40>27.
Heberle, Rudolf. 1970. From Democracy to Nazism: A Regional
Case Study on Political Parties in Germany. New York: Grosset
& Dunlap.
Heberle, Rudolf. 1963. Landbevolkerung und Nationalsozialismus:
Eine soziologische Untersuchungder politischen Willensbildung
in Schleswig-Holstein 1918 bis 1932. Rev. ed. Stuttgart:
Deutsche VerlagsAnstalt.
Hempel, Carl, G. 1966. Philosophy of Natural Science. Englewood
Cliffs, NJ: Prentice-Hall.
Jewis, Robert. 1989. "Rational Deterrence: Theory and Evidence."
World Politics 41:18>207.
Katzenstein, Peter J. 1985. Small States in World Markets:
Industrial Policy in Europe. Ithaca, NY: Cornell University
Press.
Kendall, Maurice G., and Wiiam R. Buckland. 1960. A
Dictionary of Statistical Terms. 2d ed. New York: Hafner.
King, Gary. 1989. UnifyingPolitical Methodology: The Likelihood
Theoryof StatisticalInference. Cambridge: Cambridge University
Press.
King, Gary, Robert 0. Keohane, and Sidney Verba. 1994.
Designing Social Inquiry: Scientific Inference in Qualitative Research.
Princeton: Princeton University Press.
Kohli, Atul. 1987. The State and Poverty in India. New York:
Cambridge University Press.
Kornhauser, William. 1959. The Politics of Mass Society. New
York: Free Press of Glencoe.
Kriesi, Jan. 1994. New Social Movements in Western Europe.
Minneapolis: University of Minnesota Press.
Kubik, Jan. 1994. The Power of Symbols against the Symbols of
Power: The Rise of Solidarity and the Fall of State Socialism in
Poland. University Park: Pennsylvania State University.
Kuhn, Thomas. 1962. The Structure of Scientific Revolutions.
Chicago: University of Chicago Press.
Laba, Roman. 1991. The Roots of Solidarity:A Political Sociology
of Poland Working-classDemocratization. Princeton: Princeton
University Press.
Laitin, David. 1994. "The Tower of Babel as a Coordination
Game." American Political Science Review 88:622-34.
Lave, Charles, and James March. 1975. An Introduction to
Models in the Social Sciences. New York: Harper & Row.
Lederer, Emil. 1940. State of the Masses. New York: Norton.
Lijphart, Arend. 1971. "Comparative Politics and the Comparative
Method." American Political Science Review 65:682-
93.
Lijphart, Arend. 1975. The Politics of Accommodation: Pluralism
and Democracy in the Netherlands. Berkeley: University of
California Press.
Lipton, Michael. 1976. WhyPoor People Stay Poor: UrbanBias in
World Development. Cambridge: Harvard University Press.
McAdam, Doug. 1981. Freedom Summer. New York: Oxford
University Press.
Martin, Lisa L. 1992. Coercive Cooperation: Explaining Multilateral
Economic Sanctions. Princeton: Princeton University
Press.
Martin, Lisa L., and Kathryn Sikkink. 1993. "U.S. Policy and
Human Rights in Argentina and Guatemala, 1973-1980." In
Double-edged Diplomacy ed. Peter B. Evans, Harold K. Jacobson,
and Robert D. Putnam. Berkeley: University of California
Press.
Meehl, Paul E. 1967. "Theory-Testing in Psychology and
Physics: A Methodological Paradox." Philosophy of Science,
June, pp. 103-15.
Mill, John Stuart. 1974."Of the Four Methods of Experimental
Inquiry." In his A System of Logic. Toronto: University of
Toronto Press.
Moore, Barrington, Jr. 1967. Social Origins of Dictatorship and
Democracy. Boston: Beacon.
Moses, Lincoln E. 1968. "Truncation and Censorship." In
International Encyclopedia of the Social Sciences, vol. 15; ed.
David L. Sills. New York: Macmillan.
Perrot, Michelle. 1986. "On the Formation of the French
Working Class." In WorkingClass Formation, ed. Ira Katznelson
and Aristide Zolberg. Princeton: Princeton University
Press.
Porter, Michael E. 1990. The Competitive Advantage of Nations.
New York: Free Press.
Przeworski, Adam, and Fernando Limongi. 1992. "Selection,
Counterfactuals, and Comparisons." University of Chicago.
Typescript.
Przeworski, Adam, and Fernando Limongi. 1993. "Political
Regimes and Economic Growth." Journal of Economic Perspectives
F51-69.
Przeworski, Adam, and Henry Teune. 1970. The Logic of
ComparativeSocial Inquiry. New York: Wiley.
Putnam, Robert D. 1993. Making Democracy Work: Civic Traditions
in Modern Italy. Princeton: Princeton University Press.
Ragin, Charles C. 1987. The Comparative Method: Moving Beyond
Qualitative and Quantitative Strategies. Berkeley:University
of California Press.
Russell, Bertrand. 1969. The Autobiography of Bertrand Russell,
1914-1944. New York: Bantam Books.
Sartori, Giovanni. 1970. "Concept Misformation in Comparative
Politics." American Political Science Review 64:103>53.
Sartori, Giovanni. 1976. Parties and Party Systems: A Framework
for Analysis. Cambridge: Cambridge University Press.
Scuibba, Roberto, and Rossana Schiubba Pace. 1976. Le comunita
di base in Italia. 2 vols. Rome: Coines.
Skocpol, Theda. 1979. States and Social Revolutions: A Comparative
Analysis of France, Russia, and China. New York: Cambridge
University Press.
Skocpol, Theda, and Margaret Somers. 1980. "The Uses of
Comparative History in Macrosocial Inquiry." Comparative
Studies in Society and History 22174-97.
Smelser, Neil J. 1976.ComparativeMethods in the Social Sciences.
Englewood Cliffs, NJ: Prentice-Hall.
Stinchcombe, Arthur L. 1968.Constructing Social Theories. New
York: Harcourt, Brace & World.
Stolzenberg, Ross M., and Daniel A. Relles. 1990. "Theory
Testing in a World of Constrained Research Design: The
Significanceof Heclunan's Censored Sampling Bias Correction
for Nonexperimental Research." Sociological Methods
and Research 18:395-415.
Tarrow, Sidney. 1988a. "Old Movements in New Cycles of
Protest: The Career of an Italian ReligiousCommunity." In
From Structure to Action, ed. B. Klandermans, et al. International
Social Movement Research Series, no l. Greenwich,
CT: JAI
Tarrow, Sidney. 1988b. Democracy and Disorder: Protest and
Politics in Italy, 1965-1975. New York: Oxford University
Press.
Tarrow, Sidney. 1994. Power in Movement: Collective Action,
Social Movements, and Politics. New York: Cambridge University
Press.
Tilly, Charles. 1990. Coercion, Capital, and European States,
990-1990 A.D. Cambridge, MA: Blackwell.
Tilly, Charles. 1993. European Revolutions, 1492-1992. Oxford:
Blackwell.
Tilly, Charles. 1994. "State and Nationalism in Europe, 1492-
1992." Theory and Society 2313146.
Truman, David Bicknell. 1951. The Governmental Process: Political
Interest and Public Opinion. New York: Knopf.
Walker, Henry A., and Bernard P. Cohen. 1985. "Scope
Statements: Imperatives for Evaluating Theory." American
Sociological Review 50:288-301.
Wallerstein, Immanuel. 1974. The Modern World System. Vol.
1.New York: Academic.
Weber, Marianne. 1988. Max Weber:A Biography. Trans. Harry
Zohn. New Brunswick: Transaction.
Review: Disciplining Political Science
Reviewed Work(s): Designing Social Inquiry: Scientific Inference in Qualitative Research by
Gary King, Robert O. Keohane and Sidney Verba
Review by: David D. Laitin
Source: The American Political Science Review, Vol. 89, No. 2 (Jun., 1995), pp. 454-456
Published by: American Political Science Association
Stable URL: http://www.jstor.org/stable/2082440
Accessed: 08-03-2017 16:59 UTC
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted
digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about
JSTOR, please contact support@jstor.org.
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
http://about.jstor.org/terms
American Political Science Association is collaborating with JSTOR to digitize, preserve and extend
access to The American Political Science Review
This content downloaded from 193.157.119.248 on Wed, 08 Mar 2017 16:59:28 UTC
All use subject to http://about.jstor.org/terms
American Political Science Review Vol. 89, No. 2 June 1995
THE QUALITATIVE-QUANTITATIVE DISPUTATION: GARY KING,
ROBERT O. KEOHANE, AND SIDNEY VERBA'S DESIGNING SOCIAL
INQUIRY: SCIENTIFIC INFERENCE IN QUALITATIVE RESEARCH
Designing Social Inquiry: Scientific Inference in Qualitative Research. By Gary King, Robert 0. Keohane, and Sidney
Verba. Princeton: Princeton University Press, 1994. 238p. $55.00 cloth, $19.95 paper
G ary King, Robert 0. Keohane, and Sidney Verba (KKV) have provoked much impassioned
debate at conventions and over the information superhighway with a simple but controversial
argument: the logic of good qualitative and good quantitative research is essentially the same.
Their book shows how to design qualitative (small-n) studies so that they satisfy the canons of
scientific inference. We asked five senior scholars, each of whose work mixes qualitative and
quantitative data and methods, to evaluate the success of KKV's attempt to unify political science.
David Laitin is a skeptic who wonders whether anyone can "discipline" us unruly political scientists.
James Caporaso offers cautious reminders about the many varieties of qualitative research and the
many meanings of falsification. David Collier examines KKV's treatment of selection bias, arguing
that many of their recommendations correspond to conventional understandings that are already
well-established in the field of comparative method, and that qualitative researchers sometimes have a
different perspective on basic trade-offs involved in research design. Ronald Rogowski throws down the
gauntlet: political scientists who have a strong theory may properly ignore some of KKV's pet
"canons." Finally, Sidney Tarrow suggests that triangulating qualitative and quantitative approaches
involves much more than considerations of research design. In a rejoinder, KKV reaffirm
their belief that political scientists who slight design considerations ultimately hurt their own work.
They conclude with a message to the discipline: good design-assuming there is good theoryproduces
good qualitative and quantitative political science.
DISCIPLINING POLITICAL SCIENCE
DAVID D. LAITIN University of Chicago
If political science is ready to be disciplined, King,
Keohane and Verba's Designing Social Inquiry
(KKV) can do that disciplining. By this I mean that
the book contains a set of concepts, rules of inference,
and methodological precepts that apply to all researchers
who seek a generalized and systematic
understanding of politics. This does not mean that
we all should be doing the same sort of research.
Indeed, the rules elucidated in this book have relevance
to statistically minded scholars, formal modelers,
comparativists, thick describers, and interpretivists.
What it does mean is that we all must remain
conscious about the degree to which our own research
answers an important question, so that we can
accurately signal to fellow members of our discipline
how much of the picture we have filled in. If we all
share a common vocabulary and common standards
for evaluation of evidence in light of a theory, we can
become a community of scholars in common pursuit
of valid knowledge. More bluntly, if we could agree
upon standards of scientific inference, we could better
identify our colleagues who are guilty of scientific
malpractice-which, if regularly done, is a good
operational indicator of a discipline. We need not, as
Almond (1990) has suggested, eat at "separate tables"
any longer; it is now possible productively to
consume across cuisines.
Designing Social Inquiry is not itself a methodological
breakthrough. Very little in it will be new or
surprising to moderately well trained students in
political science. What is truly innovative about this
book is its catholicity. Its goal is not to exclude the
"soft" side of political science from a discipline controlled
by "hard-line" statisticians. Rather, its central
thesis is that at root, quantitative and qualitative
research in political science share a "unified logic"
(p. 3). With that viewpoint, KKV's critiques of the
methodological problems faced in actual qualitative
research show a generosity of spirit. The book has
high praise for qualitative work containing elements
of good scientific practice. It also has feasible suggestions
that would have improved other work that
failed to meet reasonable scientific standards. Indeed,
the primary goal of the book is to demonstrate to
those of us on the soft side that we can approximate
the standards of our brethren on the hard side if we
454
This content downloaded from 193.157.119.248 on Wed, 08 Mar 2017 16:59:28 UTC
All use subject to http://about.jstor.org/terms
American Political Science Review Vol. 89, No. 2 June 1995
make such an attempt. It must be noted that those
veterans in quantitative work who on principle ignore
"soft" research as unscientific will be disabused
from their narrow viewpoint. The achievement of this
book, then, is that it sets a reasonable disciplinary
standard without using the young A. J. Ayer's (1946)
tactic of calling all work on the soft side "metaphys-
ics."
But disciplining has its down side, as Foucault
insists in his analysis of those "discursive formations"
that transcend or deconstruct disciplines (1972,
178-81). Many political scientists eagerly entered our
field, perhaps from unfortunate or boring experiences
in the disciplines of economics or psychology,
precisely because we have been eclectic, undisciplined,
and willing to tolerate a multitude of discursive
formations. Brilliant young students who wish to
travel to exotic places, read the classics, work out a
personal utopia, or promote a political cause enter
political science programs. These students may find
the disciplining constraints imposed by rules of inference
to be an unnecessary burden. Sensitive colleagues
are willing to indulge these students, in large
part because they themselves were to some extent
attracted to political science because its lack of discipline
was so inviting.
Students of Bakhtin would make a complementary
critique. Bakhtin argued that linguistic assimilation,
which is part and parcel of disciplining, leads to the
emergence of "canonical" or "authoritative utterances,"
which themselves are capable of undermining
dissent (1986, 88). A common vocabulary, from a
Bakhtinian viewpoint, is never neutral. Accepting
KKV's "statistical" vocabulary as bedrock could consequently
lock us into a cultural framework. Indeed,
their call to engage in disciplinary discourse in a
language most qualitativists see as "foreign" is surely
the source of anger felt by many practitioners who
have read this book.
Although I am personally sympathetic with the
Foucauldians and Bakhtinian's amongst us, it must
be remembered that sharing a language promotes not
only effective communication but also focused debate
across subdisciplines. While the language of statistics
does have its biases, KKV provides a conceptual
apparatus that has referents in virtually all domains
of our discipline. Scholars writing an article for disciplinary
journals in the narrative mode, for example,
will be able to use KKV's apparatus to justify its
scientific merit in political science's division of labor
and thereby raise its chances of appearing in these
august and prestigious pages, with concomitant disciplinary
rewards. Even more: scholars who strongly
disagree with the statistician's bias will, after reading
this book, have the tools to show its limitations to
practitioners in all of political science. This book is
surely the icon that iconoclasts should lust after.
It could still be objected-(along the lines I argued
in Laitin 1994)-that maybe it will bring higher expected
utility if statisticians learned the language of
nonquantitative researchers, rather than the other
way around. To this I reply that as of now, there is no
contending universal vocabulary for ascertaining
whether our research findings are valid. However, I
would welcome a counterhegemonic project along
the lines of the present one, with an alternative
critical language of scientific evaluation that would be
applicable in all domains of our discipline. But my
welcome of alternatives in no way diminishes my
admiration of the three authors of this volume for
having centrally positioned their own hegemonic
design.
Causes and Concepts
KKV's hegemonic project is to highlight the making
of valid causal inferences as the highest goal for social
inquiry. To make such inferences, researchers need
to combine theory with observations in such a way as
to demonstrate a causal effect. With a disciplinary
division of labor, the search for valid causal inferences
invites participation of scholars on both sides of
our present disciplinary divide. On the one hand, the
discipline is open to pure describers. Historical and
anthropological interpretation are potentially fundamental
for us, just so long as researchers in this mode
seek to distinguish what is systematic-and what,
random-in the particular events they are interpreting.
Assessments of this nature will help other scholars
use those studies to construct more general theory.
On the other hand, the discipline must include
formal modelers, if only to demonstrate through the
use of mathematics the internal inconsistencies in
proposed theories. But within political science, these
modelers must subject their stories to systematic and
unbiased tests and alter assumptions or set parameter
conditions for their models when data do not confirm
their theories. Historians need not make general
theory; modelers need not collect systematic data; but
if both are members of a common discipline, they will
do their work in such a way that scholars on the other
side of the divide will be able to make reasonable and
productive use of their work to ensure that science
advances.
The summum bonum of political science, despite
KKV's admirable formulation, has never been valid
causal inferences. The founders of modern social
theory indeed thought otherwise. Max Weber has
suggested that the essence of social theory is in the
"creation of clear concepts" (Weber 1988, 278). And
Emile Durkheim (1938) was especially concerned
with the identification of "social facts" Indeed, concepts
such as "charisma" and "the division of labor"
have been longer-lasting than any valid claims about
the causal effects of these concepts. It is hard to think
about the political world without them, even if their
causal role in any political process remains obscure.
And many-other such concepts guide our thinking
and theorizing today, such as cross-cutting cleavages,
social mobility, prisoner's dilemma, exit/voice/loyalty, social
mobilization, political culture, median voter, and
hegemony. Such concepts are theoretical in the sense
that they combine discrete facts common to our daily
455
This content downloaded from 193.157.119.248 on Wed, 08 Mar 2017 16:59:28 UTC
All use subject to http://about.jstor.org/terms
Review Symposium June 1995
life into a category, helping us to see the confusing
universe in which we live in a more patterned way.
Although Designing Social Inquiry is at its weakest in
analyzing the role of concept formation in political
science, there is every reason to maintain that the
development of formal criteria for such an endeavor
is consistent with the authors' goals. Suppose, for
example, prisoner's dilemma captures elements of reality
in everyday life, in international relations, and in
congressional committees; but we can make no useful
inferences for what people will do-cooperate or
defect-if they find themselves in a prisoner's dilemma.
Does this mean that the concept has failed
and should not be included in our theories of conflict
and cooperation? Probably not. Implied by the framework
provided in KKV, what we should require of
researchers is that they set clear criteria to identify a
prisoner's dilemma and continue to search for regularities
in outcome depending on the context in which
the game is played. Compelling concepts need not be
part of a valid causal inference to be powerful; but to
remain powerful, these concepts must be part of a
research agenda that seeks to identify their systematic
implications, revealing their link on a causal
chain. KKV may have undervalued the crucial role of
conceptual formulation in social inquiry; but this by
no means is an argument to reject the disciplining
that their work demands.
Critiques from within the Discipline
At a symposium devoted to Designing Social Inquiry
held at the 1994 annual conference of the American
Political Science Association, leading scholars in our
discipline were far less enthusiastic than I in regard to
the success of this book. Larry Bartels pointed out
that the authors treated many statistical conventions
(which, in reality, cover over unresolved issues) as
solutions to complex epistemological problems. Reliance
on these conventions, Bartells inferred, is hardly
a solution to the related problems that qualitative
researchers have long been addressing with their
own conventions. Peter Lange argued that researchers
in the area-studies tradition do not seek generality
of explanation, because they hold that the "context"
in which politics get played out is highly determinative
of outcomes yet itself not subject to variable
analysis. And Ronald Rogowski argued that some of
the best work in the comparative field ignored KKV's
injunctions (e.g., on never choosing cases based on
codings on the dependent variable); and yet those
works' high scientific status can still be justified.
The criticisms made at the APSA convention have
merit. I believe two of the critiques are so fundamental
as to require future revision of the text. First, KKV
focus too much attention on selection criteria within a
single study and undervalue the scientific practice of
strategically choosing observations based upon
knowledge of cases from parallel studies. If the
community of scientists, rather than the individual
researcher, is the unit of evaluation, some of the
selection problems that King, Keohane, and Verba
identify in particular studies would be partially
washed away. Second, in undervaluing theory, they
do not address the issue that selection criteria may be
different when theory is strong as opposed to when
theory is weak.
The judgment of APSA panelists was harsh indeed.
But it ought to be remembered that the criticisms
came from scholars who share an understanding
of the bedrock concepts of our discipline that are
elaborated fully in KKV. This made the criticisms
powerful and interesting and allowed for focused
debate. Their critiques confirmed, rather than undermined,
the importance of this material for the construction
of a scientific discipline.
A Plea for Utopia
This review has become something of a plea, or to use
Henry Brady's label at the APSA symposium in
regard to KKV, a "homily". I would hope that all of
our political science curricula include the material
developed in Designing Social Inquiry. Assigning the
book in a required "logic of research" course is only
one route to this goal. An alternative is to present the
material in lectures, while assigning important articles
and books to the students, with the goal of
scrutinizing these studies to see how their authors
dealt with fundamental issues of descriptive or causal
inference. However presented, the concepts and precepts
outlined in this book ought to become part of
what Bourdiqu (1984) would call our intellectual habitus.
Mutual acknowledgement of work transcending
the quantitative and qualitative divide should ensue.
This can only spawn-and need not stifle-creativity.
And there are additional rewards for living in such
a habitus. Suppose it became common practice at job
talks, reviews for journals, and panels at disciplinary
meetings to ask authors how they addressed issues of
endogeneity, of multicollinearity, of possible missing
variable bias, of alternative observable implications of
their theory, or of their judgment concernin the
number of observations necessary for valid causal
inference. Such a disciplinary practice would impel
all researchers to think systematically about these
issues in the course of their research. They need not
follow all the rules in this book. KKV recognize that
in most real-world research environments, this
would be impossible. But all researchers must have
good scientific reasons for disregarding or modifying
a particular rule. And these reasons must be made
available to potential critics. The goal of making
political science a discipline seems utopian, but KKV
show that it is within our reach.
There is little reason, however, to be sanguine. The
reaction to this book at the APSA convention gives
me the impression that there is little interest in-and
great opposition to-our becoming a discipline. This
book will stand, then, as merely a useful exposition of
statistical solutions to epistemological questions for
those of us who are not statisticians. A pity that a
book with such potential will play such a limited role!
456
This content downloaded from 193.157.119.248 on Wed, 08 Mar 2017 16:59:28 UTC
All use subject to http://about.jstor.org/terms
Review: Bridging the Quantitative-Qualitative Divide in Political Science
Reviewed Work(s): Designing Social Inquiry: Scientific Inference in Qualitative Research by
Gary King, Robert O. Keohane and Sidney Verba
Review by: Sidney Tarrow
Source: The American Political Science Review, Vol. 89, No. 2 (Jun., 1995), pp. 471-474
Published by: American Political Science Association
Stable URL: http://www.jstor.org/stable/2082444
Accessed: 08-03-2017 17:19 UTC
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted
digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about
JSTOR, please contact support@jstor.org.
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
http://about.jstor.org/terms
American Political Science Association is collaborating with JSTOR to digitize, preserve and extend
access to The American Political Science Review
This content downloaded from 193.157.119.248 on Wed, 08 Mar 2017 17:19:01 UTC
All use subject to http://about.jstor.org/terms
American Political Science Review Vol. 89, No. 2 June 1995
BRIDGING THE QUANTITATIVE.QUALITATIVE
DIVIDE IN POLITICAL SCIENCE
SIDNEY TARROW Cornell University
In Designing Social Inquiry, Gary King, Bob Keohane
and Sidney Verba (KKV) have performed a
real service to qualitative researchers. I, for one,
will not complain if I never again have to look into the
uncomprehending eyes of first-year graduate students
when I enjoin them (pace Przeworski and Teune)
to "turn proper names into variables." The book is brief
and lucidly argued and avoids the weighty, musclebound
pronouncements that are often studded onto the
pages of methodological manuals.
But following KKV's injunction that "a slightly
more complicated theory will explain vastly more of
the world" (p. 105), I will praise them no more but
focus on an important weakness in the book. Their
central argument is that the same logic that is "explicated
and formalized clearly in discussions of quantitative
research methods" underlies-or should-the
best qualitative research (p. 4). If this is so, then they
really ought to have paid more attention to the
relations between quantitative and qualitative approaches
and what a rigorous use of the latter can
offer quantifiers. But while they offer a good deal of
generous (at times patronizing) advice to qualitatively
oriented scholars, they say very little about how
qualitative approaches can be combined with quantitative
research. Especially with the growth of choicetheoretic
approaches, whose users often illustrate
their theories with stories, there is a need for a set of
ground rules on how to make intelligent use of
qualitative data.
KKV do not address this issue. Rather, they use the
model of quantitative research to advise qualitative
researchers on how best to approximate good models
of descriptive and causal inference. (Increasing the
number of observations is their cardinal operational
rule.) But in today's social science world, how many
social scientists can be simply labeled "qualitative" or
-quantitative"? How often, for example, do we find
support for sophisticated game-theoretic models resting
on the use of anecdotal reports or on secondary
evidence lifted from one or two qualitative sources?
More and more frequently in today's social science
practice, quantitative and qualitative data are interlarded
within the same study. A recent work that
KKV warmly praise illustrates both that their distinction
between quantitative and qualitative researchers
is too schematic and that we need to think more
seriously about the interaction of the two kinds of data.
Marinating Putnam
In Robert Putnam's (1993) analysis of Italy's creation
of a regional layer of government, Making Democracy
Work, countless elite and mass surveys and ingenious
quantitative measures of regional performance are
arrayed for a 20-year period of regional development.
On top of this, he conducted detailed case studies of
the politics of six Italian regions, gaining, in the
process, what KKV recommend as "an intimate
knowledge of the internal political manoeuvering and
personalities that have animated regional politics
over the last two decades" (p. 5) and Putnam calls
"marinating yourself in the data" (Putnam 1993, 190).
KKV use Making Democracy Work to praise the virtues
of "soaking and poking," in the best Fenno tradition
(p. 38).
But Putnam's debt to qualitative approaches is
much deeper and more problematic than this; for
after spending two decades administering surveys to
elites and citizens in the best Michigan mode, he was
left with the task of explaining the sources of the vast
differences he had found between Italy's north-central
and southern regions. To find them, his quantitative
evidence offered only indirect evidence; and he
turned to history, repairing to the halls of Oxford,
where he delved deep into the Italian past to fashion
a provocative interpretation of the superior performance
of the northern Italian regional governments
vis-A-vis the southern ones. This he based on the civic
traditions of the (northern) Renaissance city-states,
which, according to him, provide "social capital" that
is lacking in the traditions of the South (chap. 5). A
turn to qualitative history (probably not even in
Putnam's mind when he designed the project) was
used to interpret cross-sectional, contemporary quantitative
findings.
Putnam's procedure in Making Democracy Work pinpoints
a problem in melding quantitative and qualitative
approaches that KKV's canons of good scientific
practice do not help to resolve. For in delving
into the qualitative data of history to explain our
quantitative findings, by what rules can we choose
the period of history that is most relevant to our
problem? And what kind of history are we to use; the
traditional history of kings and communes or the
history of the everyday culture of the little people?
And how can the effect of a particular historical
period be separated from that of the periods that
precede or follow it? In the case of Making Democracy
Work, for example, it would have been interesting to
know (as Suzanne Berger asked at the 1994 APSA
roundtable devoted to the book) by what rules of
inference Putnam chose the Renaissance as determining
of the North's late twentieth-century Italian civic
superiority. Why not look to its sixteenth-century
collapse faced by more robust monarchies, its nineteenth-century
military conquest of the South, or its
1919-21 generation of fascism (not to mention its
1980s corruption-fed pattern of economic growth)?
None of these are exactly "civic" phenomena; by
what rules of evidence are they less relevant in
"explaining" the northern regions' civic superiority
471
This content downloaded from 193.157.119.248 on Wed, 08 Mar 2017 17:19:01 UTC
All use subject to http://about.jstor.org/terms
Review Symposium June 1995
over the South than the period of the Renaissance
city-states? Putnam does not tell us; nor do KKV.
To generalize from the problem of Putnam's book,
qualitative researchers have much to learn from the
model of quantitative research. But their quantitative
cousins who wish to profit from conjoining their
findings with qualitative sources need, for the selection
of qualitative data and the intersection of the two
types, rules just as demanding as the rules put
forward by KKV for qualitative research on its own. I
shall sketch some useful approaches to bridging the
quantitative-qualitative gap from recent examples of
comparative and international research.
Tracing Processes To Interpret Decisions
One such rule that KKV cite favorably is the practice
of process tracing, in which the researcher looks closely
at 'the decision process by which various initial
conditions are translated into outcomes'" (p. 226,
quoting George and McKeown 1985, 35). But even
here, KKV interpret the advantages of process tracing
narrowly, assimilating it to their favorite goal of
increasing the number of theoretically relevant observations
(p. 227). As George and McKeown actually
conceived it, the goal of process tracing was not to
increase the number of discrete decision stages and
aggregate them into a larger number of data points
but to connect the phases of the policy process and
enable the investigator to identify the reasons for the
emergence of a particular decision through the dynamic
of events (George and McKeown 1985, 34-41).
Process tracing is different in kind from observation
accumulation and is best employed in conjunction
with it-as was the case, for example, in the study of
cooperation on economic sanctions by Lisa Martin
(1992) that KKV cite so favorably.
Systematic and Nonsystematic
Variable Discrimination
KKV give us a second example of the uses of qualitative
data but, once again, underestimate its particularity.
They argue that the variance between different
phenomena "can be conceptualized as arising
from two separate elements: systematic and nonsystematic
differences," the former more relevant to fashioning
generalizations than the latter (p. 56). For example,
in the case of conservative voting in Britain,
systematic differences include such factors as the
properties of the district, while unsystematic differences
could include the weather or a flu epidemic at
the time of the election. "Had the 1979 British elections
occurred during a flu epidemic that swept
through working-class houses but tended to spare
the rich," they conclude, "our observations might be
rather poor measures of underlying Conservative
strength" (pp. 56-57).
Right they are, but this piece of folk wisdom hardly
exhausts the importance of nonsystematic variables
in the interpretation of quantitative data. A good
example comes from how the meaning and extension
of the strike changed as systems of institutionalized
industrial relations developed in the nineteenth century.
At its origins, the strike was spontaneous, uninstitutionalized,
and often accompanied by wholecommunity
"turnouts." As unions developed and
governments recognized workers' rights, the strike
broadened to whole sectors of industry, became an
institutional accompaniment to industrial relations,
and lost its link to community collective action. The
systematic result of this change was permanently to
affect the patterns of strike activity. Quantitative
researchers like Michelle Perrot (1986) documented
this change. But had she regarded it only as a case of
"nonsystematic variance" and discarded it from her
model, as KKV propose, Perrot might well have
misinterpreted the changes in the form and incidence
of the strike rate. Because she was as good a historian
as she was a social scientist, she retained it as a crucial
change that transformed the relations between the
strike incidence and industrial relations.
To put this more abstractly, distinct historical
events often serve as the tipping points that explain
the interruptions in an interrupted time-series, permanently
affecting the relations between the variables
(Griffin 1992). Qualitative research that turns up
"nonsystematic variables" is often the best way to
uncover such tipping points. Quantitative research
can then be reorganized around the shifts in variable
interaction that such tipping points signal. In other
words, the function of qualitative research is not
only, as KKV seem to argue, to peel away layers of
unsystematic fluff from the hard core of systematic
variables but also to assist researchers to understand
shifts in the value of the systematic variables.
Framing Qualitative Research within
Quantitative Profiles
These two uses of qualitative data pertain largely to
aiding quantitative research. But this is not the only
way in which social scientists can combine quantitative
and qualitative approaches. Another is to focus
on the qualitative data, using a systematic quantitative
data base as a frame within which the qualitative
analysis is carried out. Case studies have been validly
criticized as being based on often dramatic but frequently
unrepresentative cases. Studies of successful
social revolutions often possess characteristics that
may also be present in unsuccessful revolutions,
rebellions, riots, and ordinary cycles of protest (Tilly
1993, 12-14). In the absence of an adequate sample of
revolutionary episodes, no one can ascribe particular
characteristics to a particular class of collective action.
The representativity of qualitative research can
never be wholly assured until the cases become so
numerous that the analysis comes to resemble quantitative
research (at which point the qualitative research
risks losing its particular properties of depth,
richness and process tracing). But framing it within
a quantitative data base makes it possible to avoid
generalizing on the occasional "great event" and
points to less dramatic but cumulative- historical
trends.
Scholars working in the "collective action event"
472
This content downloaded from 193.157.119.248 on Wed, 08 Mar 2017 17:19:01 UTC
All use subject to http://about.jstor.org/terms
American Political Science Review Vol. 89, No. 2
history tradition have used this double strategy with
success. For example, in his 1993 study of over seven
hundred revolutionary years in over five hundred
years of European history, Charles Tilly assembled
data that could have allowed him to engage in a
large-N study of the correlates and causes of revolution.
Tilly knows how to handle large time-series data
sets as well as anybody. But he did not believe that
the concept of revolution had the monolithic quality
that other social scientists had assigned to it (1993,
chap. 1). So he resisted the temptation for quantification,
using his data base, instead, to frame a series
of regional time-series narratives that depended as
much on his knowledge of European history as on
the data themselves. When a problem cried out for
systematic quantitative analysis (e.g., when it came
to periodizing nationalism), Tilly (1994) was happy to
exploit the quantitative potential of the data. But the
quantitative data set served mainly as a frame for
qualitative analysis of representative regional and temporal
revolutionary episodes and series of episodes.
Puffing Qualitative Flesh on Quantitative Bones
These examples are possibly exotic to the traditions of
much of American social science practice. But an
American sociologist, Doug McAdam, has shown
how social science can be enriched by combining
quantitative and qualitative approaches to the same
data base. McAdam's 1988 study of Mississippi Freedom
Summer participants was based on a treasuretrove
of quantifiable data-the original questionnaires
of the prospective Freedom Summer
volunteers. While some of these young people eventually
stayed home, others went south to register
voters, teach in "freedom schools" and risk the
dangers of Ku Klux Klan violence. Two decades later,
both the volunteers and the no-shows could be interviewed
by a researcher with the energy and the imagination
to go beyond the use of canned data banks.
McAdam's main analytic strategy was to carry out
a paired comparison between the questionnaires of
the participants and the stay-at-homes and to interview
a sample of the former in their current lives.
This systematic comparison formed the analytical
spine of the study and of a series of technical papers.
But except for a table or two in each chapter, the
texture of Freedom Summer is overwhelmingly qualitative.
McAdam draws on his interviews with former
participants, as well as on secondary analysis of other
people's work, to get inside the Freedom Summer
experience and to highlight the effects that participation
had on their careers and ideologies and their
lives since 1964. With this combination of quantitative
and qualitative approaches, he was able to tease a
convincing picture of the effects of Freedom Summer
activism from his data.
As I write this, I imagine KKV exclaiming, "But this
is precisely the direction we would like to see qualitative
research moving-toward expanding the number
of observations and respecifying hypotheses to allow
them to be tested on different units!" (see chap. 6).
But would they argue, as I am, that it is the combination
of quantitative and qualitative methods trained
on the same problem (not a move toward the logic of
quantitative analysis alone) that is desirable? Two
more ways of combining these two logics illustrate
my intent.
Sequencing Quantitative and Qualitative Research
The growth industry of qualitative case studies that
followed the 1980-81 Solidarity movement in Poland
largely took as given the idea that Polish intellectuals
had the most important responsibility for the birth
and ideology of this popular movement. There was
scattered evidence for this propulsive role of the
intellectuals; but since most of the books that appeared
after the events were written by them or by
their foreign friends, an observer bias might have
been operating to inflate their importance in the
movement vis-A-vis the working class that was at the
heart of collective action in 1980-81 and whose voice
was less articulate.
Solid quantitative evidence came to the rescue. In a
sharp attack on the "intellectualist" interpretation
and backed by quantitative evidence from the strike
demands of the workers themselves, Roman Laba
showed that their demands were overwhelmingly
oriented toward trade union issues and showed little
or no effect of the proselytizing that Polish intellectuals
had supposedly been doing among the workers of
the Baltic coast since 1970 (1991, chap. 8). This finding
dovetailed with Laba's own qualitative analysis of the
development of the workers' movement in the 1970s
and downplayed the role of the Warsaw intellectuals
who had been at the heart of a series of books by their
foreign friends.
The response of those who had been responsible
for the intellectualist interpretation of Solidarity was
predictably violent. But there were also more measured
responses that shed new light on the issue. For
example, prodded by Laba's empirical evidence of
worker self-socialization, Jan Kubik returned to the
issue with both a sharper analytical focus and better
qualitative evidence than the earlier intellectualist
theorists had employed, criticizing Laba's conceptualization
of class and reinterpreting the creation of
Solidarity as "a multistranded and complicated social
entity ... created by the contributions of various
people" whose role and importance he proceeded to
demonstrate (1994, 230-38). Moral: a sequence of
contributions using different kinds of evidence led to
a clearer and more nuanced understanding of the role
of different social formations in the world's first
successful confrontation with state socialism.
Triangulation
I have left for last the research strategy that I think
best embodies the strategy of combining quantitative
and qualitative methods-the triangulation of different
methods on the same problem. Triangulation is
particularly appropriate in cases in which quantitative
data are partial and qualitative investigation is
obstructed by political conditions. For example, Val-
473
This content downloaded from 193.157.119.248 on Wed, 08 Mar 2017 17:19:01 UTC
All use subject to http://about.jstor.org/terms
Review Symposium June 1995
erie Bunce used both case methodology and quantitative
analysis to examine the policy effects of leadership
rotation in Western and socialist systems. In
Do New Leaders Make a Difference?, she wrote, "I
decided against selecting one of these approaches to
the neglect of the other [the better] to test the impact
of succession on public policy by employing both
methodologies" (1981, 39).
Triangulation is also appropriate in specifying hypotheses
in different ways. Consider the classical
Tocquevillian insight that regimes are most susceptible
to a political opportunity structure that is partially
open. The hypothesis takes shape in two complementary
ways: (1) that liberalizing regimes are more
susceptible to opposition than either illiberal or liberal
ones; and (2) that within the same constellation of
political units, opposition is greatest at intermediate
levels of political opportunity. Since there is no
particular advantage in testing one version of the
hypothesis over the other, testing both is optimal (as
can be seen in the recent social movement study,
Kriesi et al. 1995).
My final example of triangulation comes, with
apologies, from my own research on collective action
and social movements in Italy. In the course of a
qualitative reconstruction of a left-wing Catholic
"base community" that was active in a peripheral
district of Florence in 1968, I found evidence that
linked this movement discursively to the larger cycle
of student and worker protest going on in Italy at the
same time (Tarrow 1988). Between 1965 and 1968, its
members had been politically passive, focusing
mainly on neighborhood and educational issues. But
as the worker and student movements exploded
around it in 1968, their actions became more confrontational,
organized around the themes of autonomy
and internal democracy that were animating the
larger worker and student movements around them.
Researchers convinced of their ability to understand
political behavior by interpreting "discourse"
might have been satisfied with these observations;
but I was not. If nothing else, Florence was only one
case among potential thousands. And in today's
global society, finding thematic similarity among different
movements is no proof of direct diffusion,
since many movements around the world select from
the same stock of images and frames without the least
connection among them (Tarrow 1994, chap. 11).
As it happened, quantitative analysis came to the
rescue to triangulate on the same problem. For a
larger study, I had collected a large sample of national
collective action events for a period that bridged the
1968 Florentine episode. And as it also happened,
two Italian researchers had collected reliable data on
the total number of religious "base communities" like
the Florentine one throughout the country (Sciubba
and Pace 1976). By reoperationalizing the hypothesis
cross-sectionally, I was able to show a reasonably
high positive correlation (R = .426) between the
presence of Catholic base communities in various
cities and the magnitude of general collective action
in each city (Tarrow 1989, 200). A longitudinal, local,
and qualitative case study triangulated with the results
of cross-sectional, national, and quantitative
correlations to turn my intuitive hunch that Italy in
the 1960s underwent an integrated cycle of protest
into a more strongly supported hypothesis.
KKV are not among those social scientists who
believe that quantification is the answer to all the
problems of social science research. But their singleminded
focus on the logic of quantitative research
(and of a certain kind of quantitative research) leaves
underspecified the particular contributions that qualitative
approaches make to scientific research, especially
when combined with quantitative research. As
quantitatively trained researchers shift to choice-theoretic
models backed up by illustrative examples
(often containing variables with different implicit
metrics), the role of qualitative research grows more
important. We are no longer at the stage when public
choice theorists can get away with demonstrating a
theorem with an imaginary aphorism. We need to
develop rules for a more systematic use of qualitative
evidence in scientific research. Merely wishing that it
would behave as a slightly less crisp version of
quantitative research will not solve the problem.
This is no plea for the veneration of historical
uniqueness and no argument for the precedence of
"interpretation" over inference. (For an excellent
analysis of the first problem, see KKV pp. 42-3 and of
the second, pp. 36-41.) My argument, rather, is that
a single-minded adherence to either quantitative or
qualitative approaches straightjackets scientific
progress. Whenever possible, we should use qualitative
data to interpret quantitative findings, to get
inside the processes underlying decision outcomes,
and to investigate the reasons for the tipping points
in historical time-series. We should also try to use
different kinds of evidence together and in sequence
and look for ways of triangulating different measures
on the same research problem.
KKV have given us a spirited, lucid, and well-balanced
primer for training our students in the essential
unity of social science work. Faced by the clouds of
philosophical relativism and empirical nominalism
that have recently blown onto the field of social
science, we should be grateful to them. But their
theoretical effort is marred by the narrowness of their
empirical specification of qualitative research and by
their lack of attention to the qualitative needs of
quantitative social scientists. I am convinced that had
a final chapter on combining quantitative and qualitative
approaches been written by these authors, its
spirit would not have been wildly at variance with
what I have argued here. As it is, someone else will
have to undertake that effort.
Notes
I wish to thank Henry Brady, Miriam Golden, Peter Katzenstein,
David Laitin, Peter Lange, Doug McAdam, Walter
Mebane, Robert Putnam, Shibley Telhami and Charles Tilly
for their comments on drafts of this review.
474
This content downloaded from 193.157.119.248 on Wed, 08 Mar 2017 17:19:01 UTC
All use subject to http://about.jstor.org/terms