14 The design of social and political research
Philippe Schmitter
Let us assume that you have an idea that has led you to identify a topic that you
believe to be of suﬃcient importance and of feasible execution to conduct
research on it. It may be a doctoral dissertation, or just a seminar exercise, but
regardless of length and complexity no topic can ‘research itself’. You will have
to translate it – via a series of strategic choices – into a project. It is this process
of translation from something problematic or puzzling into something on
which you can gather valid data and about which you can make compelling
inferences that constitutes your research design.
Granted, much social scientiﬁc research is not self-consciously designed –
it is not subject to a deliberate and critical process of choosing its components
and defending its overall conﬁguration. In many areas of inquiry, the design
is literally given along with the topic. So much research has already been conducted
on it that adding yet another case or extending it to yet another time
period does not seem to require a novel eﬀort of translation. Indeed, the universal
desire of all sciences to produce cumulative knowledge seems to militate
against continuously challenging and changing the standard way of doing
research. If you do propose a change in design – say, a reconceptualization of
the topic, a revised instrument for measuring variation, a diﬀerent way of
selecting relevant cases, or a novel method of testing for association – you will
risk confusing your reader-cum-critic. He or she may ﬁnd it diﬃcult to distinguish
whether eventual diﬀerences in data or inference are ‘really’ due to
the topic itself or ‘merely’ to your meddling with the established way of
researching it.
Most young social scientists, however, will not be choosing topics whose
research design is given. They will have to ﬁnd or invent an apposite design –
and they should be prepared to understand and defend the choices involved.
Moreover, if their immediate or eventual intent is comparative – if they anticipate
including more than one case or set of observations and drawing
inferences across them – then their choice of design will be even more crucial.
Making the right strategic choices will greatly enhance the value of the data
263
they gather and the inferences they can draw from them; neglecting these
choices or taking them for granted could result in idiosyncratic scraps of
information and inferences rooted in exceptional circumstances that make no
reliable or cumulative contribution to scientiﬁc knowledge.
Figure 14.1 is a schematic and idealized representation of the complete
‘social and political research cycle’. Each of its boxes involves an important set
of interrelated strategic choices, and its implication is that these should be
made in the displayed sequence, beginning with an idea that deﬁnes a topic at
12 noon and proceeding clockwise until the researcher arrives at an evaluation
of his or her ﬁndings that may or may not redeﬁne the original topic at midnight.
Inside the boxes lie a number of alternative courses of action. Choose
among them wisely, and you will do better research. Ignore them or fail to
grasp their signiﬁcance, and you risk accepting serious fallacies at each stage.
The most important message to keep in mind while proceeding through the
entire cycle is that there is no single best strategy or set of strategies for
researching all topics. Everything depends on your point of departure, the
initial substance you have decided to research. At the beginning of the cycle in
Figure 14.1, the range of options tends to be most extensive – and, hence, most
264 Philippe Schmitter
TOPIC
IDEA
SERENDIPITY
SELF ASSESSMENT
CAUSAL
INFERENCE
CASE
SELECTION
PROPOSAL
WRITING
TEST FOR
ASSOCIATION
VARIABLE
OPERATIONALIZATION
CONCEPTUALIZATION
HYPOTHESIS
GENERATION
NORMATIVE
CLARIFICATION
MEASUREMENT
OF INDICATORS
Fig. 14.1 The cycle of social and political research
confusing. Interesting topics clamour for equal attention; diﬀerent theories
and concepts can seem equally compelling. As one proceeds clockwise, the
successive choices are increasingly related to each other and the options
become more limited. At some point, you may well adopt or fall into an ‘established
disciplinary routine’. You can save yourself a lot of time and worry by
doing this but this will only beneﬁt you if your topic and, especially, its conceptualization
is suﬃciently isomorphic with the original – that is, it conforms
to the basic characteristics of the topic that has already been successfully
researched by others. Applying even the best established and/or most fashionable
design to the wrong topic can be a formula for disaster, especially
when it comes to drawing inferences.
Very few researchers really enter Figure 14.1 at noon and leave at midnight.
Most take shortcuts to get started in the process. Many social scientists begin
their research careers already knowing on which case or cases they intend to
work. Not infrequently, it happens to be the country from which they come
or in which they are trained. So-called ‘area specialists’ usually have some
prior personal commitment involving their knowledge of history, culture or
language, and this tends to aﬀect the topics they select. Others may have
picked up some novel statistical technique or measurement device that
they wish to show oﬀ – and they search about for an apposite topic to which
to apply it. Perhaps the most common (and, in my view, pernicious) point
of departure concerns theories or approaches that are currently fashionable
in sociology or political science. Imbued with the conviction that only
those espousing such a ‘paradigm’ will ﬁnd eventual employment, young
researchers are prepared to take up any topic – no matter how trivial or
obscure – if only to demonstrate their ﬁdelity to its assumptions and postu-
lates.
Do not presume that, once in the cycle, you will have to go all the way
around. As we shall see in the conclusion, there are many points of exit
that will still permit you to make an original and signiﬁcant contribution to
knowledge.
Wherever you have really begun your research and whatever your motives
for doing so, I recommend that you at least pretend that you are beginning at
the top of Figure 14.1, if only to help you clarify ex post the design choices you
should have taken deliberately or have already taken implicitly. Try to imagine
that it all began with an idea about a substantive topic that is important to you
and that only later did you place it in an explicitly conceptual context, elaborate
speciﬁc hypotheses about its occurrence, choose the cases to study, and so
forth.
265 The design of social and political research
Now, we can proceed to look sequentially into the ‘little black boxes’ in
Figure 14.1. Attached to each of them the reader will also ﬁnd a list of ‘possibly
optimizing’ choices and ‘potentially damaging’ fallacies.
Choice of topic
No one can predict where and when ideas will appear. With some knowledge
of the researcher’s personal and professional background, however, it may be
a bit easier to predict the conditions under which an idea becomes a topic –
that is, when someone will attach suﬃcient importance to a given thought and
place signiﬁcant boundaries around it to make it worthy of investing his or her
energy to explain how it came about or what its consequences might be. This
highly personal eﬀort at selection can be an important source of distortion
throughout the rest of the design and, especially, when it comes to drawing
inferences from whatever data distributions or associations are generated. The
very fact that you care enough to select some topic probably means that you
also value what it contributes or the eﬀect that it has. However subliminal the
thought may be, your values become embedded in the topic and can exert a
persistent inﬂuence on your choices as you make your way around the rest of
the research cycle. They may have an even greater impact when you decide to
make a ‘premature’ exit from the cycle.
It is often the case that one is attracted to a topic because the society or
polity also cares about it. Never is this more evident than when the subject
matter is in crisis or in fashion. As social scientists, we are attracted to
phenomena that call attention to themselves – whether they do so by creating
further problems or by providing novel solutions. Which is another
way of saying that our topics tend to be either failed experiences at the
end of their useful existence or recent successes that have yet to reveal
their complete impact. Rarely does one come across designs explicitly
focused on explaining social or political phenomena that are mediocre or
inconsequential.
Grosso modo, topics of research come in two guises: (1) projections, where
the researcher is conﬁdent that the existing approach and methods are adequate
and deserve to be applied to units or time periods that have not already
been covered or with greater precision to cases that only seem to be exceptional;
and (2) puzzles, where the researcher begins with the assumption that
something is deﬁcient in the way that the topic has been previously handled
and that the units or time periods to be examined will demonstrate the
266 Philippe Schmitter
existence of anomalies. Both projections and puzzles should be approached in
the same ‘critically rational’ manner, but the perspective of the researcher
diﬀers. If the topic selected is regarded as a projection, he or she has the intent
(at least, initially) of conﬁrming established wisdom and will take more seriously
the obligation to make a cumulative contribution to knowledge within
a speciﬁc discipline or paradigm. The perspective when tackling a puzzle leads
one to seize on anomalies that seem to expose deﬁciencies in how the topic has
been conceptualized, measured or reported, and that is more likely to lead the
researcher to alternative concepts and methods – frequently by drawing on
other disciplines. Needless to say, both are capable of making valid contributions;
both are needed by all social science disciplines.
Possibly optimizing choices
1 Choose a topic that you care enough about to be willing to spend the time
to complete the project.
2 Choose a topic (and make an argument) that interests other social scientists
(even those outside your ﬁeld); the better it is, the more it will interest those
working in adjacent ﬁelds and disciplines.
3 Specify the temporal, spatial and, if necessary, cultural boundaries of the
topic in a way that makes the research feasible, but does not make it trivial
or ‘unique’.
4 Acknowledge your initial source of inspiration for the topic and your personal
preference about its outcome, without apologizing for them.
5 Never justify your selection only on the grounds that it has been ‘underexplored’,
and, especially, do not ignore, trivialize or dismiss what has
already been written on the topic.
6 Try to reach as far back as possible in social and political theory to ﬁnd
grounds for the relevance of your topic and avoid being manipulated by
academic fad and fashion.
7 By all means, listen to your advisor and your peers, but be absolutely certain
that, regardless of who ﬁrst suggested it, the topic ‘belongs’ to you.
Potentially damaging fallacies
1 ‘Fad-ism’: Your topic (or method or theory) is being very much and
very favourably discussed right now in your ﬁeld, so that if you
adopt it your work will be less criticized and you will be more likely to ﬁnd
a job.
267 The design of social and political research
2 ‘Wishful thinking’: Your topic has already produced well-publicized and
promising results for the society or polity; therefore, if you conduct
research on it, your ﬁndings will be taken more seriously and favourably.
3 ‘Ambulance-chasing’: Because the topic of your research is presently in
crisis, you will have greater access to data and the public will be more interested
in whatever you ﬁnd out.
4 ‘Presentism’: The assumption that whatever you ﬁnd associated with some
topic in the present must have been there in the past and will probably
remain there in the future.
5 ‘Standing on the shoulders of the past giants’: This might apparently allow
you to see further and to avoid being distracted by the squabbles among
contemporary pygmies – yet those giants might not have been looking at
the same thing or in the same direction.
Conceptualization
Almost all substantive matters emerge ‘pre-conceptualized’ in the strict sense
that they can only be recognized by the potential researcher and shared with
others if they are expressed in some intelligible language. The idea may come
initially as a shape or a colour or an emotion, but words are the indispensable
way in which it acquires factual speciﬁcity and shared signiﬁcance. The complication
for research resides in the high probability that the words initially
involved will be those of the social or political actors involved – which implies
that their words could bear many diﬀerent meanings and be attached to a wide
range of contrasting assumptions.
Conceptualizing a topic involves translating the words that surround it in
‘real-existing’ societies or polities into variables (although see Della Porta, ch.
11, and Bray, ch. 15). These are not just fancy academic labels applied to a
speciﬁc event or process. They should identify analogies, generic conditions
that are shared by a distinctive set of events or processes and can take on
diﬀerent values over time – whether these are quantitatively or qualitatively
observed. They acquire their peculiar status as causes or eﬀects according to the
way they are connected to other variables by theories. Once these variables have
been assembled, whether from the same or varying theories, they constitute
your provisional argument concerning the topic you have chosen to explain.
Which brings us to the ‘Elephant-in-the-Room’ that is so rarely mentioned
but so frequently the source of confusion at each stage of designing research.
Even the most elementary and frequently used concepts – such as class, status,
gender, age, region and religiosity for explaining voting behaviour – derive
268 Philippe Schmitter
their meanings from being inserted into a more comprehensive (and presumably
coherent) matrix of concepts (see Kratochwil, ch. 5, and Mair, ch. 10).
Their deﬁnitions may sound the same and, as we shall see later, operationalization
of these variables may even be identical, but their role depends on prior
assumptions and contingent relations that diﬀer according to the theory,
paradigm, approach or framework that is being applied. And no single piece
of research can possibly specify what these are. If you tried to do this, there
would be no time or space left for your analysis. In other words, all social and
political research is part and parcel of ‘the state of theory’ prevailing at the
moment it is conducted. No research can be conceptualized ex novo without
reference to what has been produced already on that and related topics. This
applies just as much to those who are trying to solve puzzles as to those who
are ‘merely’ trying to make projections.
Choosing one’s concepts is only the ﬁrst step. Making them into variables
means assigning a status to them, and this is where their embeddedness in
theory most saliently enters into the research design. The most important task
is to distinguish between those that are regarded as operative with regard to the
chosen topic and those that are inoperative. The former are expected to play
some discernible role in the explanation of outcomes – either as an explicans
(that which does the explaining) or as an explicandum (that which is to be
explained). The more elaborate the prior theory and, hence, the conceptualization
derived from it, the more it may be possible to assign diﬀerent statuses
to the operative variables, for example, by distinguishing between primary
and secondary ones (according to their explanatory power), direct and intervening
ones (according to how near the eﬀect is to the cause), continuous and
episodic ones (according to how constant in time their eﬀect is), and so forth.
Needless to say, all these initially assigned roles can be inverted, especially
where and when the objective is to explain a relatively long-term sequence of
social or political processes. Inoperatives are variables that are present and can
be expected to take on diﬀerent values during the subject matter being
researched, but whose eﬀect is not expected to produce a discernible or
signiﬁcant diﬀerence. Of course, when it comes to making eventual inferences,
allegedly inoperative variables may turn out to be an important potential
source of spuriousness. Even constants, variables that were present but not
thought to vary during the research period and, hence, a priori considered not
capable of contributing to variation in the outcome, may gain eventually in
importance – especially when it becomes evident that the impact of operative
variables was contingent on slight modiﬁcations or even simple reinterpretations
of such background factors. Hopefully, irrelevant variables – those whose
269 The design of social and political research
variation cannot conceivably be logically or empirically associated with the
topic under investigation – will remain that way.
Possibly optimizing choices
1 As much as possible, avoid references to speciﬁc persons, countries or
cultures with ‘upper-case’ names by using only ‘lower-case’ variables to
describe them and their prospective eﬀects in your argument.
2 There is nothing wrong with using a ‘hunch’ as your starting point in conceptualization
– the world surrounding most interesting topics is usually
full of them – but try as soon as possible to identify the more generic theory
in which this hunch is embedded, switch to its language, and explore its
axioms or presumptions before going further.
3 Try to avoid ‘multicollinearity’ – clusters of variables that are closely associated
with each other – and simplify by only using the dominant variable
in such clusters or providing it with an ideal-type connotation that captures
as precisely as possible the nature of the cluster.
4 Make as explicit as possible not only the operative but also the inoperative
variables and the constants, those characteristics that do not vary, in your
argument – and be prepared to change their status in the course of conducting
the research.
5 When using classiﬁcation systems (see Mair’s chapter), make sure that the
categories are both inclusive of all observations and exclusive in their assignment
of every single observation – and that all of them are potentially relevant
to explaining outcomes, including those that are vacant for the moment.
6 Specify as soon and as explicitly as possible the universe to which your conceptualization
is intended to apply in both time and space.
7 Exercise caution when using concepts and variables across long periods of
time or diﬀerent cultural contexts, since their meaning to actors and, hence,
their eﬀect may change.
8 Strive for parsimony by eliminating double-dealing or superﬂuous variables,
but without resorting to excessive simpliﬁcation. One way of doing
this is to restate your argument several times and to make it more concise
each time.
Potentially damaging fallacies
1 ‘Obscurantism’: If you cloak your conceptualization in highly abstract
terms or ﬁt all of your observations into some complicated classiﬁcation
270 Philippe Schmitter
scheme, no one will notice that all you are doing is describing what hap-
pened.
2 ‘Attribute-ism’: The more deﬁnitional attributes or analogous properties
you attach to a given concept, the more signiﬁcant it is likely to be in
explaining the outcomes you want to understand.
3 ‘Concept stretching’: A concept used successfully to identify an analogy
among events in one time and place must be equally valid when applied to
other times or places.
4 ‘Isolation’: Your preferred variable plays such an important role in explaining
your topic that it can be conceptualized, measured and manipulated
alone, without regard for the network of other variables surrounding it and
the prior axioms upon which it rests.
5 ‘Novelty at any price’: Because existing concepts are so embedded in (old)
theories, by inventing and using novel ones, you will be credited with
greater originality in your research.
6 ‘Arbitrariness’: Since all concepts are basically arbitrary – a function of
unpredictable practical uses and/or theoretical fashions – it will make no
diﬀerence which ones you use, provided that your public and peers come to
accept them.
7 ‘Consensual-ism’: If everyone in your discipline is using some concept and
seems to agree on its meaning, as well as its explanatory relevance, you
should feel safe to do so.
Formation of hypotheses
Not all research designs involve the formation (or the testing) of explicit
hypotheses. There exists a very broad range of social and political topics for
which it is possible to conceptualize the variables that may contribute to an
explication, but not to assign any sort of provisional ‘if . . . then . . .’ status to
their relationships. For these topics, the apposite research logic is one of discovery
and not of proof. The purpose is to improve one’s conceptualization of
a topic, probe its plausibility against a range of data and eventually generate
hypotheses among its conclusions, but it would be premature to expect them
as a pre-condition for conducting the research itself.
The determining factor is again that ‘Elephant-in-the-Room’, the prevailing
state of theory on a given topic. Substantive matters that are of recent
occurrence, that are only characteristic of a small number of cases, that incite
strong emotions or political controversies, or that fall between diﬀerent social
science disciplines are obvious candidates for ‘discovery’ status. The potential
271 The design of social and political research
researcher is reminded that this should not be taken as a sign of inferiority.
Somewhere behind all social scientiﬁc research that today routinely follows
the logic of proof, there must have been a glorious moment in the past when
someone launched a voyage of discovery. Unfortunately, behind the facade of
increased professionalism and standardization of techniques, this message has
been suppressed. Only the most intrepid of young scholars will accept the
challenge of trying to make sense out of alternative conceptualizations of the
same topic; or piecing together potentially coherent and general arguments by
‘process tracing’ on the basis of speciﬁc cases; or admitting that, in instances
of highly interdependent and complex social or political systems, it may never
be possible to distinguish between independent and dependent variables,
much less to express them in terms of a ﬁnite set of bivariate relationships.
Possibly optimizing choices
1 Ensure that the assumption of any ‘if . . . then . . .’ relationship is
suﬃciently precise that it speciﬁes its ‘micro-foundation’, the functional
dependence, structural mechanism or intentional logic that is supposed to
connect its variables and, where possible, introduce an independent
measure of its presence.
2 Do not assume ex ante that only individual human beings are capable of
laying ‘micro-foundations’, when the ‘real-existing’, historical world is chock
full of social and political units that have acquired the capacity to act collectively
in ways that cannot be reduced to individual intentions and choices.
3 Ensure that the presumed cause is independent of the presumed eﬀect, and
not parallel or convergent manifestations of the same social or political
process.
4 Where possible, specify explicitly the existence of intervening conditions or
prevailing constants that must always be present for the hypothesized relation
to produce its eﬀect – even if these contextual factors do not vary
during the research.
5 An ideal research situation can emerge when you ﬁnd yourself in a ‘tworing
circus’ – when two rival versions of the same hypothesized relation are
plausible and would explain diametrically diﬀerent outcomes based on
diﬀerent theoretical assumptions.
6 Be prepared to recognize and deal with ‘equiﬁnalities’, similar outcomes
that are produced by diﬀerent sequences or mechanisms, when they
emerge, and therefore to test diﬀerent sets or, better, ‘strings’ of hypotheses
– not just isolated ones.
272 Philippe Schmitter
7 Remember that you always have three hypotheses to test, namely, the ones
that suggest a positive or a negative relation, and the null hypothesis that no
‘if . . . then . . . ’ relationship exists. The latter should be regarded as the
most probable in occurrence. Everything may be related to everything else
in our complex environment, but not always in a predictable direction or
to a signiﬁcant degree.
8 Try ex ante when elaborating hypotheses to diﬀerentiate between variables
that you think are ‘necessary’(always likely to be present when the outcome
is present), ‘suﬃcient’ (always and only present) and merely ‘helpful’
(sometimes present, sometimes not). Never assume that your set of variables
is going to be both ‘necessary and suﬃcient’ and, therefore, make
space for the inevitable ‘error term’.
9 Since most research projects consist of ‘clusters’ and ‘chains’ of related
hypotheses that contribute to explaining a selected outcome, it is often
useful to draft a ‘model’ of these simultaneous and sequential relations
using time and space as co-ordinates.
Potentially damaging fallacies
1 ‘Scientism’: If your variables are not organized into hypotheses with clearly
diﬀerentiated independent and dependent variables, your research will not
be scientiﬁc.
2 ‘Fear of failure’: If your hypothesis or hypotheses are disproved, you will
have made no contribution to knowledge.
3 ‘Inﬁnite regress’: All hypotheses about variable relationships in the social
sciences are preceded by a potentially inﬁnite historical chain of causality
and consequence, therefore, it makes no diﬀerence when you choose to
break into that chain.
Selection of cases
For all but a few projects, the potential number of societies or polities aﬀected
by the chosen topic will exceed the researchers’ capability for gathering data,
testing for associations and drawing inferences. It is, therefore, normal that
only some subset of these units will enter into your analysis. One of the most
prominent of the strategic choices you will have to make involves the number
and the identity of those to be included and the criteria you impose to select
them. This can vary from one unit (the single case or person) to as many as
273 The design of social and political research
are apposite (the universe of those aﬀected); but there is a fairly inescapable
trade-oﬀ between the quantity of variables that have been included in your
initial conception of the topic and the number of units for which you will be
able to gather data. Including more cases probably also means poorer quality
data, more missing observations and greater problems of conceptual equivalence.
Inversely, the more narrowly you have deﬁned and operationalized
those variables – that is, the lower they are on the ladder of abstraction – the
less likely they are to be relevant in a wide range of cases.
Case selection may have its practical side when it comes to gathering data
and, especially, making one’s own detailed observations; but its real payoﬀ is
analytical. Manipulating the identity of cases provides most sociologists and
political scientists with their closest equivalent to experimentation. It ‘simulates’
the introduction of control variables. By ‘holding constant’ across the
sample such potentially relevant conditions as cultural identity, geographic
location, level of development and temporal proximity, the researcher can at
least pretend that variation in them is unlikely to have produced the outcome
one is looking at. Granted that the controls can be somewhat approximate
and that there still will remain many potential sources of ‘contaminating’
diﬀerentiation among units in the sample – still, this is the best design instrument
that he or she has available. It should, therefore, be wielded with deliberation
– and caution.
Strictly speaking, the researcher does not select individual cases but
‘conﬁgurations of variables’that co-habit the same unit and may even co-vary
in a unique or distinctive fashion within that unit. But one cannot analyse
‘France’ as such and compare it with, say, ‘Spain’ or ‘Italy’. There are simply
too many diﬀerent (and potentially relevant) conditions within each of these
countries with regard to almost any topic you choose to work on. This holds
even when comparing micro-units within the same country, where the
number of variables can be more reliably controlled because of common constraints
at the nation-state level. So-called ‘holistic’ research is, therefore,
largely an illusion in social and political research and, when tried, it usually
amounts to little more than a detailed or ‘thick’ description of one case (or of
parallel ones if more units are covered) (but see Bray, ch. 15, for a diﬀerent
view).
This is not to say that there are not signiﬁcant diﬀerences between designs
that are driven by the eﬀort to isolate a small number of variables and test
exclusively for their association with other variables across a larger number of
units, and designs that begin with a large number of interrelated variables
(often combined via ideal-type constructs) within one country and then seek
274 Philippe Schmitter
to ﬁnd signiﬁcant and persistent connections across a few, carefully selected
units of an allegedly comparable nature. But in either strategy, what you are
usually comparing are variables – one or many, alone or in clusters – not units.
This brings us to the second aspect of case selection, which has long been
taken for granted and yet has recently become of growing concern. For a unit
of observation to be a valid case for analysis, it must possess identical or, at
least, comparable degrees of freedom with regard to the topic under investigation.
A design that drew inferences – descriptive or causal – from a sample
of units composed of Brazilian municipalities, Mongolian provinces, Spanish
comunidades autónomas and the permanent members of the United Nations
Security Council about the eﬃcacy of taxation systems would not attract
much attention. Much as its author might (correctly) protest that this ‘sample’
embodies a ‘most-diﬀerent systems design’, critics would (rightly) object that
actors in these units did not have remotely equivalent powers to make or
enforce their decisions on taxation.
The usual formula for getting around this problem was to select only units
that were at the same level of aggregation and enjoyed the same formal status
within the world social and political system. This presumably explains why so
many comparative research projects have been based on nation-state units or,
to a lesser degree, on relatively autonomous subnational units within federal
or confederal systems. The reductio ad absurdum of this strategy has been
reached with large-N comparisons containing all the members of the UN for
which data can be obtained – despite the blatant fact that these so-called sovereign
states have radically divergent capabilities for governing their respective
populations or even satisfying their most elementary needs.
Since Donatella della Porta has contributed an entire chapter (ch. 11) to this
volume that deals extensively with the issues involved in case selection, I have
little more to add. I will, however, provide a pedagogic device that I have found
useful in explaining to students what their options are at this point in the
research cycle (Figure 14.2).
Where researchers are committed to producing scientiﬁc knowledge
(deﬁned here as causal inference), the preferred case selection strategy should
usually be the experimental one, choosing the units of observation randomly
and introducing some element of change in a subset of them while holding
variation constant for the others. Unfortunately, most social and political scientists
have to operate in ‘real-existing’settings, where this is not possible. And
even when they are permitted to engage in experimentation, the topics tend
to be so trivial and the settings so artiﬁcial that projecting inferences based on
such ﬁndings to more ‘realistic’ contexts is very hazardous.
275 The design of social and political research
Quasi-experiments may be second-best, but they oﬀer some interesting
advantages, with regard to both the eﬃciency of research and the credibility
of inferences. The case-base can be as low as one, although it is better to replicate
the quasi-experiment in several other settings, if possible, within the same
time frame. They are, however, limited to real-world situations where the
independent variable is highly discrete and temporally circumscribed and
where data-gathering over a suﬃcient period of time has been consistent and
reliable. Assessing the eﬀect of a new public policy or the impact of some
unexpected social or natural event tends to ﬁt this narrow bill of particulars,
but only if nothing else is happening to the unit or units at the same time. This
is also a strategy of case selection that is especially vulnerable to diﬀusion or
contagion eﬀects, if the units involved know of each other’s behaviour.
276 Philippe Schmitter
DESIRE TO PRODUCE
SCIENTIFIC KNOWLEDGE
ABOUT SOCIETY/POLITY
POSSIBILITY OF ARTIFICIALLY
EXERTING CONTROL OVER
DOPERATIVE VARIABLE(S)
YES
PURE EXPERIMENTS
NO
POSSIBILITY OF ‘SIMULATING’
EXPERIMENT BY ISOLATING NATURAL
TREATMENT AND ITS EFFECT(S)
YES
QUASI-EXPERIMENTS
NO
POSSIBILITY OF MAKING
RELIABLE CROSS-UNIT
OBSERVATIONS
YES
NATURAL COMPARISONS
SELECTED
CASES
ALL CASES
DESIGN
METHOD OF
CONCOMITANT
VARIATION
SINGLE
CASE
DESIGN
CONFIGURATIVE – IDIOGRAPHIC
CONFIGURATIVE – NOMOTHETIC
HEURISTIC
PLAUSIBILITY PROBES
CRUCIAL/CRITICAL
DEVIANT
COUNTERFACTUAL THEORETICAL
RHETORICAL
NO
THOUGHT EXPERIMENTS
MOST-
DIFFERENT
CASES DESIGN
MOST-
SIMILAR
CASES DESIGN
METHOD OF
AGREEMENT
METHOD OF
DIFFERENCE
JOINT METHOD
TYPES OF
CASE
STUDY
Fig. 14.2 A schematic for the selection of cases
Most sociologists and political scientists will have to settle for the study of
variations in their subject matter that appear ‘naturally’, whether within a
single case or across diﬀerent numbers of them. Della Porta (ch. 11) explores
the implications of making these choices.
There are, I would add, a number of other alternative strategies that are not
usually included in texts on research methods or design – presumably because
their scientiﬁc status is dubious. They typically arise in contexts in which it is
risky or impossible to observe and record the behaviour of ‘real-existing’social
or political units. All involve what Max Weber once called ‘thought experiments’.
The best-known goes under the rubric of counter-factualism and
involves the researcher in an eﬀort to imagine what would have happened to
the topic if some condition, person, event or process had not been present.
Usually, this focuses on a single country – for example, ‘How would Germany
have evolved politically if Hitler had not been “available” in the early 1930s?’
It can also be applied to a sample or even to the universe of cases – for example,
‘What would be today the level of international insecurity in Europe if the EU
did not exist?’ Or, ‘How many people in the world would know how to speak
English if the Americans had lost their Revolutionary War?’ If this sounds
‘exotic’ and somewhat ‘ﬂaky’, you should remember that every time that you
invoke the famous and indispensable Latin phrase, ceteris paribus, before
advancing a hypothesis, you are being a counterfactualist.
Moving even further from social scientiﬁc orthodoxy, one ﬁnds a vast
number of seminar exercises, MA papers and PhD dissertations that are essentially
rhetorical, theoretical or normative. These certainly deal with topics –
often more important ones for ‘real-existing’ societies and polities than those
chosen by empiricist-cum-positivists – but their purpose is to follow the
development of concepts or discourses over time, or to examine the logical
consistency of particular arguments, or to promote the ethical acceptance of
speciﬁc forms of human behaviour. Such projects do indeed involve research.
In a ‘heuristic’ and ‘nomothetic’ way they have inﬂuenced empirical inquiry.
Just think of the impact of recent works by John Rawls, Jürgen Habermas and
Jon Elster upon how even the most ‘hardcore’ empiricists select and conceptualize
a wide range of topics (see Bauböck, ch. 3).
Possibly optimizing choices
1 If you are not trying to cover the entire universe, consider the possibility of
selecting a sample of cases randomly and how that would aﬀect your
project.
277 The design of social and political research
2 As a rule, when randomization is excluded and you have to be purposively
selective, choose your cases based on their relation to the independent variable
or variables rather than the dependent variable or variables.
3 And when you make this choice, try to ensure that the cases chosen ‘represent’
as wide a range as possible of scores on those independent variables.
4 When your topic will not permit this, when you are motivated to research
something precisely because it involves a compelling, arresting or extreme
outcome and you therefore have to select on the dependent variable, remember
this as a potential source of bias when it comes to drawing inferences.
5 Keep in mind that you do not have to use only one strategy of case selection
and that so-called ‘nested strategies’, where you start with a large N of cases
and relatively few crudely measured variables and, subsequently, shift to a
small N with a much more detailed ‘battery’ of variables, can give you the
advantages of both strategies when it comes to drawing inferences.
6 Always prefer the lowest level of spatial or functional aggregation that is
compatible with the actor behaviour presumed by your conceptualization,
since you can subsequently reassemble your research upwards – but not
downwards – in scale.
7 No matter which or how many cases you initially select, some may prove to
be ‘decomposable’, in that you may be able to generate additional cases by
dividing up the initial ones, but only provided these subunits possess some
and the same degree of autonomy.
8 Before selecting the number and identity of the cases for which you intend
to gather data, make sure that you are aware of the criteria that you originally
used for classifying your topic and ask ﬁrst: ‘What is this a case of?’
Only after satisfying that demand will you know what units are ‘eligible’ for
inclusion, and you can proceed to exclude some of them for good reason.
Potentially damaging fallacies
1 ‘Notoriety’: Just because a particular case has been prominent in public discussion,
it will be more interesting to research, and others will pay more
attention to your research.
2 ‘Numbers’: It is always advantageous to have a larger number of cases, even
when, by adding them, you are compelled to attenuate their relation to the
topic or to use less valid indicators.
3 ‘Cruciality’: Because a given unit is an outlier according to your criteria of
case selection, it will be a crucial case whose conformity or non-conformity
provides a deﬁnitively signiﬁcant test of causal association.
278 Philippe Schmitter
4 ‘The illusion of control’: Selecting cases because they seem to share certain
general cultural, locational or structural characteristics necessarily controls
for their relevance – when it is still possible that minor or qualitative variations
in ‘controlled variables’ could be aﬀecting variation in what you are
trying to explain.
5 ‘Contemporaneousness’: In units chosen for comparison within the same
time frame, the actors must have similar (or at least suﬃcient) awareness of
the relevance of common variables and be capable of acting upon them
simultaneously – when these units may be at diﬀerent points in longer
cycles or simply on diﬀerent time schedules.
6 ‘Imitation’: When actors in the selected units are acutely aware of having to
deal with some topic within the same time frame, they will also be sensitive
to what others are doing about it and will learn from each other’s successes
and failures – in fact they may be quite ignorant of what the others are doing.
Writing the proposal
This stage in the research cycle is ‘optional’, although highly desirable.
Diﬀerent graduate programmes place greatly diﬀerent emphasis on the
importance of defending a formal proposal. Some require it before allowing
the candidate to ‘go into the ﬁeld’. My personal experience suggests that the
greater the plurality of approaches or paradigms surrounding a given topic
and present in a particular institution, the greater will be the emphasis on
writing and defending your proposal. In scholastic contexts dominated by a
single theoretical or disciplinary orientation, the eﬀort may be eschewed completely.
The reigning orthodoxy favours problems rather than puzzles and may
even dictate in considerable detail how topics should be conceptualized and
operationalized. At the extreme, there is no ‘ﬁeld’ to go into, no speciﬁc cases
to select and no measurement details to discuss. What matters at this stage is
the normative or logical consistency of the ‘argument’, of one’s conceptualization
of the topic and how well it conforms to prevailing orthodoxy. The
number and identity of cases are relatively unimportant, if not irrelevant, to
the extent that both prior axioms and subsequent expectations are believed to
be universal. The data can be simulated or assembled from the usual sources
for illustrative purposes. The eventual inferences are usually predictable and
in line with original expectations. The fellow members of your ‘research club’
will enthusiastically congratulate you on your cumulative contribution to
knowledge. Practitioners of other disciplines and members of other clubs
279 The design of social and political research
within your discipline will yawn and tell you that you have ‘rediscovered the
wheel’or produced something utterly trivial. In other words, there are costs as
well as beneﬁts in belonging to an established research tradition.
Another condition aﬀecting the utility of proposal-writing is its potentially
critical role in obtaining research funding. Where such support is assured or
not subject to competitive pressures, the researcher may content him- or
herself with a brief statement of intention. Otherwise, your ability to summarize
coherently and justify convincingly the design choices that you have made
up to this point could make all of the diﬀerence in determining whether you
will be able to carry out your project at all. Although it is not frequently discussed
openly, this ‘commercial’ aspect of proposal-writing can also be a
source of distortion when the preferences of the sponsor come to be anticipated
in the proposal itself and the researcher ﬁnds him- or herself pandering
to them by modifying the topic, changing its conceptualization, restricting the
range of hypotheses and even selecting diﬀerent cases in an eﬀort to please the
prospective sponsor. More experienced researchers soon learn how to ‘ﬁnetune’
their proposals to get support from donors and then go on to follow the
course of inquiry they think will lead to the most compelling inferences.
Fortunately, national or supranational sponsors rarely control for conformity
between proposals and the research actually performed. At most, they may be
interested in whether or not the policy implications drawn from such research
conform to their preferences.
The ‘real’ purpose of writing a proposal should be to give the researcher a
chance to sit back and reﬂect critically on the strategic choices he or she has
made – and to exchange these reﬂections with supervisors and peers before
plunging into the inevitably messy and absorbing process of gathering data
and trying to make sense out of them. There may be subsequent moments for
self-criticism and changes – see the remarks below on the importance of
serendipity – but writing and defending a proposal at this stage oﬀers a unique
opportunity to ‘rewrite’ and ‘resubmit’ before becoming irrevocably locked
into a course of action.
Operationalization of variables
In principle, the conceptualization of variables should be carried out beforehand
and without regard for how they will be converted into indicators and
eventually measured. There is a good reason for this. What is of paramount
theoretical importance is to specify clearly the condition or factor that is
280 Philippe Schmitter
supposed to be present in order to produce some anticipated eﬀect – alone or
in conjunction with other variables. Having previously and independently
conceptualized the projection or puzzle in such a fashion should provide a
strong incentive subsequently to specify the observations that need to be made
in order to verify the presence, magnitude, direction, or persistence of that
variable. During the early stages of research, this means that you should adopt
the attitude that all social and political variables can potentially be operationalized
– and later be prepared to compromise when you start looking for
indicators in the real world.
In practice, unfortunately, anticipations of such diﬃculties do tend to
intrude and can even inhibit scholars from using concepts that are known to
be ‘impossible’ to operationalize. Just think of such indispensable political
properties as power, authority and legitimacy; or of such social ones as esteem,
respect and trust. For none of them is there a standard and easily accessible set
of measures. Even elaborate (and expensive) attempts to operationalize them
based on ‘reputational’ criteria from public opinion surveys have been problematic.
And criticisms of these eﬀorts become more insistent the more such
indicators are stretched across countries and over time.
Another way of putting this dilemma is that there are bound to be trade-oﬀs
that have to be made at this stage in the research cycle. The higher one’s concepts
are on the ladder of abstraction – and, presumably, the wider their
prospective range of application – the more diﬃcult it is going to be to make
convincing observations about their presence in a speciﬁc case or set of cases.
Increase the number of units in your study – either of persons or of organizations
– and you are almost bound to run into problems with missing data and
misleading indicators. Do not be afraid to make these trade-oﬀs, but do so
self-consciously. Tell your reader-cum-critic when you are settling for a less
satisfactory indicator or a less speciﬁc level of observation. Be prepared when
necessary even to eliminate cases, but also be sensitive to how this may distort
your eventual capacity to draw inferences. Those research sites where operational
requirements are most diﬃcult to satisfy are usually places where social
and political behaviour is the least ‘normal’ and their exclusion from the
design will probably narrow the range of variation and reduce the eventual
strength of association.
The theme that haunts all aspects of this stage of the research cycle is validity.
Do the observations you propose to make accurately reﬂect and, hence, capture
the meaning of the concepts you have chosen to bear the burden of explanation?
Nomatterhowaccuratetheobservations,howcomparabletheyareacrossunits,
how replicable they turn out to be when another scholar makes them, if they are
281 The design of social and political research
not valid, your research will have broken down at one of its most vulnerable
points. You may well have discovered something important and the associations
revealed by your indicators might be incontrovertible, but you have not proved
(or disproved) what you started out with. Your ﬁndings are irrelevant in the
strictest sense. They have told you nothing about the topic you announced that
you intended to work on – unless you are prepared to rely on serendipity (see
below) and reconceptualize your entire project from its very origins.
Possibly optimizing choices
1 Pay close and critical attention to the correspondence between your initial
concepts and their proposed indicators or assessments by comparing them
to research by others on the same or related topics.
2 Be wary of variable speciﬁcations and of empirical indicators that have been
applied routinely over time and across units to measure diﬀerent concepts.
3 Make sure that the concept and its indicator(s) are applied to the same level
of analysis and are as close as possible in level of abstraction.
4 When available, use alternative operationalizations and multiple potential
indicators and, where necessary, rely on ‘triangulation’ among them to
resolve disparities and to improve validity.
5 All things being equal (although they never are), you are better oﬀ using
unobtrusive rather than obtrusive indicators, since the actors whose behaviour
is being observed will have less of an opportunity to respond strategically,
ethically or emotionally to your request for information.
6 Remember that there are various ways of assessing the validity of indicators,
ranging from consensus among independent respondents to co-variation
between diﬀerent ‘internal’ measures and, least reliably, correlation with
other hypothesized ‘external’ outcomes.
Potentially damaging fallacies
1 ‘Availability’: This indicator exists and has been used successfully by others;
therefore, it must be valid when applied to your topic.
2 ‘Operationalism’: You decide to include in your analysis only variables for
which you know that a valid (or consensually accepted) indicator already
exists.
3 ‘Mimetism’: ‘X’ got away with using data on this to indicate a concept
similar to yours, even when drawing upon a diﬀerent theory; therefore, you
can safely use it for the same purpose.
282 Philippe Schmitter
4 ‘Ignorance of the uncertainty principle’: If you operationalize a variable by
intruding on the ‘real-existing’world of your respondent, you can nonetheless
ignore the possibility that his or her answer will be contaminated by
prevailing norms of correctness or strategic calculations of interest, or that
you will be creating rather than measuring variation.
Measurement
At this point in the cycle, your choices will be more or less dictated by the ones
you have already made – whether you did so consciously in relation to the
speciﬁcity of your problem or puzzle (as I hope was the case) or whether you
settled into an established research tradition – whether quantitative or qualitative
– and obediently followed its dictates. Moreover, there is a good reason
why you should let yourself ‘go with the ﬂow’at this point. Using existing techniques
of observation and indicators for variables not only saves you a lot of
time and anxiety, but can also provide you with an element of internal ‘quality
control’ – provided that the measures used are valid, that is they capture the
characteristics of the variable that you are relying upon for an eventual explanation.
When it occurs, successful replication of previous research is a very
desirable result – and one that can be personally very reassuring. Should you
decide to invent and apply a new indicator or, worse, battery of indicators –
especially to measure some frequently used variable – you will have to make
an especially strong eﬀort at justiﬁcation. Otherwise, you will run the risk at
the inference stage of confounding the reader: ‘Is this seemingly compelling
ﬁnding really novel, or is it only due to some change in measurement?’
The discussion on measurement tends to be dominated by the distinction
between quantitative and qualitative indicators – with a marked bias in favour
of the former. There is no reason to be surprised by this, since most methods
texts are written by quantiﬁers and they have convincing arguments in their
favour. Numerical data are said to be more reliable, i.e. more likely to provide
agreement among independent observers, more accurate, i.e. more likely to
produce agreement across units, and more useful, i.e. more compatible with
diﬀerent ways of testing for association. Certainly, the social science disciplines
have tended to assign greater ‘scientiﬁc status’ to quantitative than to
qualitative research – and to reward its practitioners accordingly.
This is unfortunate for at least three good reasons: (1) it has encouraged
researchers to attach numbers to variables when the validity of their connection
with the designated concept was dubious; (2) it has resulted in the
283 The design of social and political research
exploitation of standard numerical indicators whose multiple components are
often theoretically disputable and whose weighted combinations are poorly
understood by those who use them; and (3) it has discouraged the innovative
use of more direct and imaginative techniques of observation – precisely to
capture qualities inherent in complex and contingent relations. You can assign
a number to anyone and anything; but nothing guarantees that the assignment
will produce relevant information. If these qualities are diﬀerences in kind
(nominal) rather than in magnitude (cardinal or ordinal), then – whatever the
rule of their assignment – the number could well be a worthless piece of disinformation.
What matters is how you have conceptualized your topic, not the
allegedly superior virtues of one over another form of measurement.
Of all of the stages in the cycle, this is probably the one that is best suited
for serendipity, for learning from the research process itself in ways that can
feed back to your previous choices and lead you to introduce improvements
in them before ‘path dependence’ has completely taken over. At last, you are
back in touch with the ‘real-existing’ subjects/agents of your topic – having
spent much time wandering around making abstract ‘disciplinary’ decisions.
If you are lucky, they will talk to you directly about their intentions and perceptions,
and they may even have some opinions about what you are asking
them and intend to do with their answers. Even if your research relies exclusively
on secondary or publicly available sources, there can be ‘voices’ in such
documents that can speak in ways you have not anticipated. Of course,
there will be a lot of sheer ‘noise’ generated by the data you are gathering, and
that can be very confusing when juxtaposed to the relatively parsimonious
approach you have been applying to the topic. Nevertheless, keep your eyes,
ears and mind open for subtleties and surprises, and be amenable to introducing
‘course corrections’– even some that go all the way back to the boundaries
you initially placed around the topic or key aspects of your original
argument.
Possibly optimizing choices
1 Routinely test for the reliability of indicators, if possible by using alternative
sources of data and/or alternative persons to score the data.
2 If validity requirements can be satisﬁed, opt for quantitative over qualitative
measurement, since the technical advantages are considerable and
because you can more easily move from the former to the latter.
3 Always opt for the highest, most informative level of measurement possible
(given the nature of the variable), since it will later be possible to shift to a
284 Philippe Schmitter
lower level. Cardinal data can always be made ordinal, and virtually anything
can later be dichotomized or ﬁled away in nominal categories – but
you cannot move in the reverse direction.
4 Make your instructions – even if only for your own use – concerning the
assignment of quantitative scores or qualitative labels as transparent and
complete as possible so that the measurement operations can be replicated
by you or someone else in the future.
5 Especially when working on the macro level of a complex society or polity,
most variables will contain multiple components and be indicated by composite
measures – which should obligate the researcher to devote concerted
attention to how such ‘scales’ are aggregated.
6 Especially when gathering information over time about social or political
processes, make sure to check that changes are not due to modiﬁcations
of the instruments of observation rather than to changes in actual behav-
iour.
7 Many measurement devices are calibrated to pick up only relatively largescale
and consequential changes in variables, which means that they may
systematically fail to capture more modest and gradual ones. Social and
political ‘revolutions’ are always recognized; ‘reforms’ are more often
underreported, until their eﬀects have accumulated suﬃciently to draw
attention to them.
8 Try to estimate before actually gathering the data where the error sources
are most likely to come from and how they will aﬀect your ﬁndings. Worry
less about random errors (they will attenuate possible associations) than
about systematic ones (they will bias the direction of your ﬁndings).
9 Try to catch yourself before adjusting the data or correcting for errors in
them in ways that make these data ﬁt better the general expectations or
speciﬁc hypotheses with which you started.
Potentially damaging fallacies
1 ‘Composite-ness’: Many concepts are complex and multidimensional in
nature and, therefore, can only be measured by similarly complex and
multidimensional indicators – regardless of variation in their internal
structures and, hence, the probability that identical scores will be assigned
to quite diﬀerent clusters of variation.
2 ‘Longevity’: It is always better to use an indicator that has been around for
some time, used in a variety of research settings, and can provide the
researcher with a longer time perspective – despite the likelihood that
285 The design of social and political research
during this period the techniques for measurement will have changed and
the meanings of items for actors may not be the same.
3 ‘Clarity’: It is always preferable that each variable be given a speciﬁc and
unambiguous score – even if the nature of its conceptualization and theoretical
status is calculatedly ‘fuzzy’ or ‘radial’.
4 ‘Reiﬁcation’: What you are measuring is identical to what you have conceptualized
which, in turn, is identical to the way in which actors perceive
‘it’ – regardless of how much is lost in translation as the researcher moves
from one realm to another.
Test for association
By now, the researcher may have momentarily lost almost all strategic control
over his or her project and, at best, should consult one among many texts on
methodology to discover which among all of the verbal or mathematical, symbolic
or numerical, parametric or non-parametric, deterministic or probabilistic
devices available for testing for association best ﬁts the data that he or
she has gathered.
Variables can be associated with each other in diﬀerent ways. Typically,
the social scientist will be interested in direction, or whether the ﬁt is positive,
negative or null; strength, or how much one variable aﬀects another; and
signiﬁcance, or the likelihood that the ﬁt could simply have been due to chance.
Since his or her research will almost inevitably be ‘historical’, the time, timing
and sequence of how they ﬁt to each other should also be important – indeed,
these chronological dimensions often provide the basic orientation to how
one’s ﬁndings are presented and defended.
The reason for this is that the most powerful means of testing for the ﬁt
among variables and, therefore, for presenting one’s ﬁndings has long been to
tell a believable story in chronological order. Perhaps, within some highly professionalized
niches in sociology and political science, storytelling is no longer
regarded as acceptable. The occupants of these niches – not infrequently,
Americans or those trained in America – have forgotten that their disciplines
are profoundly and irrevocably historical. What counts is not just what
happens, but when it does and in relation to what else has already happened
or is simultaneously happening. Moreover, the actors themselves are not just
passive recipients of scores, but active and reﬂexive keepers of the score. They
remember what they and their ancestors did in the past, and their preferences
in the present are conditioned by this knowledge. In my opinion, no means of
286 Philippe Schmitter
testing for such associations has yet been invented that can supplant or even
surpass the chronological narrative in capturing these subtleties of time and
timing, and in bringing simultaneously into focus the multitude of variables
involved in the sheer complexity of most social and political phenomena. The
narration of your ﬁndings can, no doubt, be considerably bolstered in credibility
by inserting quantitative tests about speciﬁc associations into the basic
narrative. Cross-tabulations, rank-orderings, regression equations, factor- or
small-space analyses, even mathematical models, can often be helpful, but primarily
when analysing topics that are heavily circumscribed in time and space
and that can be separated into relative simple and repetitive components.
Even social and political scientists relying exclusively on quantitative data
may ﬁndit occasionally usefultotella plausiblestory that placestheassociations
they calculate and the inferences they draw in some chronological order.
Narration can also serve to ﬁll in the gaps between cause and eﬀect by providing
a verbal description of the mechanisms involved – especially when mathematical
formulae and formal models typically treat such exchanges as taking
place within impenetrable ‘black boxes’ (see Héritier, ch. 4, and Vennesson,
ch. 12). The ﬁndings of hard-core quantiﬁers often circulate only among small
groups of cognoscenti and are incomprehensible to outsiders; but whenever sociologists
or political scientists aspire to enlighten and inﬂuence wider publics,
they will either have to learn how to narrate their ﬁndings or hire someone else
to translate the esoteric results of their tests into more intelligible stories.
Possibly optimizing choices
1 Never forget the ‘inter-ocular impact test’ that consists in simply eyeballing
the data – scatterplots are especially useful for this – and forming your own
visual impression of what is going on among the variables and across the
cases.
2 Always try to apply diﬀerent tests of ﬁt and only try the more demanding
ones once you have experimented with simpler ones.
3 If possible (and it will be much more possible with quantitative designs),
manipulate the number by eliminating one or two, and/or by dividing the
sample into subsamples – say, by size or location – and do not be discouraged
if this shakes up their ﬁt, but try to discover what variables may have
intervened to produce such diﬀerent results.
4 Remember that most tests for association – quantitative for sure, qualitative
for some – are exceedingly sensitive to extreme cases, so that you may
be well advised to eliminate them in order to ﬁnd out how persistent or
287 The design of social and political research
signiﬁcant is the association among variables when only more ‘normal’
units are included in the analysis.
5 Remind yourself of the time dimension and test whether successive crosssections
through the data – say, at ten-year intervals – produce equally
strong associations. If they do not, reﬂect on what intervening or contextual
variables might be responsible for the new ﬁndings.
6 Your tests for association will be all the more convincing, the more eﬀort
you put into falsifying initial hypotheses, rather than merely seeming to
verify them by grasping at all favourable distributions of data.
7 The treatment of ‘deviant’ cases that do not ﬁt the general pattern of association
is often taken as an indicator of how seriously the researcher accepts
the task of falsiﬁcation. Ignoring them (or transforming their scores) suggests
that you are excessively concerned with veriﬁcation; embrace them,
exploit their contrariness and try to determine the extent to which they call
into question the hypothesis and you will gain favour as a ‘falsiﬁcationist’.
Potentially damaging fallacies
1 ‘Spuriousness’: You have found a close association between two variables
and you report this ﬁnding – without considering that if you were to introduce
a third variable, it might explain variation in both of the original ones.
2 ‘Contingency’: The associations you ﬁnd are strong and signiﬁcant, but
only if and when certain, usually unspeciﬁed, contextual variables are
present.
3 ‘Curve-ﬁtting’: Since there is always ‘noise’ and ‘error’ in the data, it is permissible
to ‘smooth’ distributions by transforming the raw data or eliminating
outliers and this will usually result in a more ‘satisfactory’ ﬁt.
4 ‘Anachronism’: Whatever are the associations that satisfy your test criteria
and the time period covered by your research, the ﬁndings they generate
will be valid whenever.
5 ‘Ad-hocracy’: At some level of abstraction and measurement, each case can
be uniquely identiﬁed and used to ‘explain away’ any and all observed deviations
from the outcome predicted by those variables included in the study.
Causal inference
This is by far the most hazardous – and the most rewarding – of the stages in
the research cycle. It is the one in which you will have the least disciplinary
288 Philippe Schmitter
or academic guidance and, hence, the widest range of discretionary choices to
make.
Many social researchers will have exited the process before arriving here.
They will have made their accurate observations, published their empirical
descriptions and gone home. Others will have stopped even earlier, before
having gathered any data, and left satisﬁed that they have advanced further the
plausibility of an argument or helped to specify the universe to which it can
be applied. Some will have gone further and proﬀered tests – numerical and
narrative – illustrating how frequently and strongly variables have been associated
with each other. But they will have prudently refrained from trying to
answer two further questions: (1) the retrospective one of why and how these
variables combined to produce the outcome that was the topic of the research
in the ﬁrst place; and (2) the prospective one of what the consequences of this
will be in the future and when these consequences will happen.
Consider, as an example, the current controversies over climate research.
Do you think that if climatologists and other scientists had merely ﬁled
reports demonstrating that temperatures were rising across the planet and
that various chemical substances have been accumulating in its atmosphere,
there would have been much of a reaction? As far as I know, these facts were
accepted by all as uncontroversial. It was only when these researchers correlated
these indicators and drew the inference that increases in them masked
a causal relation that could not be due to chance or fate that things became
controversial. When they attributed primary causation to factors related to
human intervention and, even more, when they began to advance threatening
projections about what will happen in the future, then all hell broke loose!
Without even hinting that all social scientists have a responsibility for generating
such controversy, they should feel a more modest responsibility for
exploiting their data to the fullest extent possible; that almost inevitably
commits them to drawing retrospective and (sometimes) prospective inferences.
Just think back to the number of occasions when you have read a report
on extensive and expensive research and still found yourself asking the ‘why’
and ‘how’ question at the end. This could be regarded as favourable by
younger researchers, since it means that there is a very considerable amount
of unexploited data out there just waiting for ‘secondary analysis’ at low cost.
Nevertheless, it is lamentable when the scholars who initially chose the topic,
conceptualized it, selected the cases and gathered the data do not go as far as
they could in drawing ‘grounded’ inferences about the causality it might
reveal. Manuals for sociology and political science are full of sage advice concerning
the limits of doing this. Not infrequently, teachers of graduate courses
289 The design of social and political research
and dissertation advisors will revel in providing the student with egregious
examples of researchers who exceeded the conﬁnes of their data or ignored the
contribution of other variables, and made what proved to be erroneous statements
about causality or consequence.
The controversy that tends to dog most discussions about inference is generalizability.
A cautious researcher who draws inferences from his or her ﬁndings
that are restricted to the cases investigated and the time period covered is
unlikely to face much criticism – or to generate much attention. Specialists on
the topic will, no doubt, have something to say about the validity of indicators,
the accuracy of measurements and the appropriateness of tests for association –
but it is not until you dare to generalize across temporal, spatial or cultural contexts,
until you trample on someone else’s turf, that you will be seriously challenged.
No one likes to be told that his or her topic can be diﬀerently explained
by someone intervening from another theoretical or disciplinary perspective.
And there are good reasons for this. Although they may seem arbitrary or
anachronistic (and some no doubt are), the lines of specialization built into
diﬀerent social science disciplines have served to enforce professional standards
and preside over the accumulation of knowledge. Generalizations that are
based on alternative conceptualizations and/or novel methods should be especially
carefully scrutinized. Nevertheless, this is where the real scholarly excitement
lies – this is where ‘seminal’contributions are to be made – provided the
researcher is well prepared to face his or her critics.
The strategy of case selection will play an especially signiﬁcant role. Singlecase
studies are rarely a convincing basis for generalization – even the so-called
‘crucial’ ones. Large-N studies should be less objectionable, were it not for the
fact that many of their cases are dubious in terms of their (alleged) common
capacity to act and the probability that behind any associations found in the
whole universe there are bound to be subsets of cases where the ﬁt diﬀers considerably
– and may even reverse itself. Middle-size samples based on controlling
for the ‘usual suspects’ (geographic location, development, size,
religion, cultural area) by their very nature inhibit generalization, unless they
are replicated for diﬀerent samples. Indeed, replication can be a powerful
weapon – and not just to the extent that other cases or periods produce the
same direction, magnitude and signiﬁcance of association. If you can show
that a reliable pattern holds at diﬀerent levels of aggregation within the same
sample, you will have made the inference that it is more likely to hold elsewhere
considerably more compelling.
The other critical factor will come from accusations of researcher bias, often
alleged to be the product of the national or disciplinary context in which the
290 Philippe Schmitter
researcher operates. It is only human to prefer to discover what you thought
was there in the ﬁrst place and, then, to extend that ﬁnding to other places
about which you know less. Most often this can be attributed to a natural tendency
to ‘over-observe’ what you expected to see and to ‘under-observe’ variation
that you were less prepared to encounter. Along with this ‘type I
conﬁrmation bias’, one has to mention that type II errors also exist. In this
instance, for some perverse reason, the researcher prefers to reject his or her
original hypothesis and, thereby, underestimates the degree of association that
actually exists. Whether the peculiarities of national cultures or academic disciplines
have anything to do with either of these typical errors seems dubious
to me, but there is no doubt that both exist.
The most secure way of guaranteeing enduring respect for the inferences
you have drawn from your research – and of securing your place in the
Pantheon of Notable Social Scientists – is to place them under the protection
of a covering law. Such a law oﬀers an explanation for a much broader range
of social or political phenomena, for example the Darwinian ‘Law of the
Fittest’. It should be widely, if not universally, accepted by the Notables who
have preceded you and, ideally, it should not be derived from the theory you
started with. But do not worry if you do not make it to the Pantheon. Your
contribution to knowledge can still be signiﬁcant and your career as a social
scientist still very rewarding.
Possibly optimizing choices
1 Add alternative explanatory variables suggested by other cases or experiences
(if available without conducting an entirely new piece of research) to
discover whether the original ﬁt within your sample is maintained.
2 Probe your data by subtracting subsets of cases from the initial sample to
see how robust the ﬁndings based on it were, especially when you think you
are dealing with the entire universe but have reason to suspect ‘regional’
variations.
3 Be careful not to ‘anchor’ your inferences by relying too much on a single
prominent association among variables at the expense of lesser (and less
expected) ones.
4 When assembling a batch of inferences from a research project, do not privilege
or attach greater signiﬁcance to ﬁndings that were easier to document
or closer to your own experience.
5 It will be risky, but try on the basis of your inferences from a given sample
to predict what analogous behaviours have been in a diﬀerent sample of
291 The design of social and political research
persons or places that you know nothing about; and (even more risky) to
apply the inferences you have drawn to predicting the future performance
of the units you have studied.
Potentially damaging fallacies
1 ‘Triumphalism’: You have made a signiﬁcant ﬁnding; therefore, your work
is over – even though it could be the result of some variable you forgot to
include and that may be very prominent in other cases or samples.
2 ‘Pago-Pago-ism’: Whenever you think you have found something that
applies everywhere, there will always be some place that you do not know
(or have not even heard of) where the ﬁnding does not ﬁt – and there will
always be a scholar who knows the place and will inform you of your error.
3 ‘Exceptionalism’: You chose to study a particular topic only in a particular
country because you considered that the context was exceptional and, then,
you turn around and claim that your ﬁndings are universal.
4 ‘Cross-level replicability’: Associations among variables that have been
found to be consistent in direction, strong in magnitude and signiﬁcant at
one level of analysis will replicate themselves at other – lower or higher –
levels of aggregation within the same sample.
5 ‘Cognitive dissidence’: If variables that simply ‘should’ not go together still
seem to be associated, this must be due either to some unidentiﬁed
measurement error or conceptual confusion, so that you are justiﬁed when
drawing any inference by excluding the case or withdrawing the variable
from your analysis.
6 ‘Temporal proximity’: You choose to give greater prominence and to attach
greater importance to associations of variables that have occurred more
recently and to presume that earlier associations (or dissociations) should
be ‘discounted’.
Self-assessment
Once you have arrived at whatever stage in the research cycle you have chosen
as your point of exit, your objective should be quite simple: make yourself into
the best possible critic of your own work. Anticipate all of the potential objections
at each of the previous stages. Where possible, return and enter appropriate
corrections. Since this is often impossible, given the numerous and
irreversible ‘path dependencies’ built into the research cycle, signal to your
292 Philippe Schmitter
reader-cum-critic that you are aware of the defect and have tried your best not
to be misled by it or to magnify its impact. Above all, remind yourself right
from the start that no research is perfect and all researchers make mistakes.
Inscribe above your desk (or on your screen saver) the Latin phrase Errare est
humanum – ‘to make mistakes is to be human’– and recognize that to be a
human being studying human behaviour is to be doubly vulnerable to this
maxim.
My overarching purpose in writing this chapter has been to help you to
become your own best critic.
Conclusion
Social and political research is characterized by the diversity of its concepts,
theories, designs – and logics. Only a few will work ‘around the clock’in Figure
14.1 and conclude with empirically grounded inferences about causal relations
among variables. Many will choose a topic for which this would be premature
or inappropriate, given the existing state of his or her discipline or his
or her purpose in selecting a particular topic. They may exit the cycle relatively
early, sometime between 1 and 3 p.m. – hopefully, with an improved understanding
of the generic relations involved and, possibly, with a more elaborate
set of hypotheses for future research. Still others will be interested in drawing
out the ethical and normative implications of these relationships, perhaps by
exploring analogies with previous experiences or prior philosophic assumptions.
In Figure 14.3, I have labelled this point of exit as the ‘logic of discovery’,
the idea being that those who take it will have made their original
contribution by discovering empirical or normative relationships previously
ignored or distorted by existing wisdom. The chapters in this volume by Zoe
Bray, Alessandro Pizzorno, Sven Steinmo and Rainer Bauböck should be especially
useful to those who choose to leave the cycle at this point.
From 3 to 6 p.m. fewer social and political researchers will be leaving the
cycle.1
Their distinctive contribution will have been to identify the apposite
universe surrounding the topics selected, to select cases that represent
speciﬁed distributions of key variables and to have invented new ways of
deﬁning these variables and embedding them in more comprehensive theories.
Most importantly, they will have carried further and in greater detail the
existing conceptualization of the relationships surrounding their topic –
hence, the notion that they have followed a ‘logic of explication’. Donatella
della Porta, Peter Mair, Christine Chwaszcza and Friedrich Kratochwil have
293 The design of social and political research
contributed chapters that should be of particular interest to them, but all of
them presuppose that explication is not an end in itself, but only a necessary
pre-condition for passing to the next stage which involves the speciﬁcation of
indicators and the gathering of data.
Many more social and political scientists will exit after 6 p.m. and before
9 p.m. They will have produced research that is fundamentally descriptive in
nature. Here, the preoccupation is with the validity of their measurements and
the accuracy of their observations. They will have gone into the ﬁeld – even if
it is in their own backyard – and generated new data about social and political
phenomena. They are also most likely to have contributed to the development
of better instruments of observation and more reliable indicators. Mark
Franklin’s chapter deals primarily with this ‘logic of description’.
The chapter by Adrienne Héritier is the one that comes closest to tackling
head-on the issues involved in the ‘logic of proof’, although virtually all of the
authors touch at least peripherally on the very controversial objective of
making empirically grounded inferences about causal relations in the political
and social sciences. This may be where the ultimate payoﬀ lies – and certainly
294 Philippe Schmitter
TOPIC
IDEA
SERENDIPITY
SELF ASSESSMENT
CAUSAL
INFERENCE
CASE
SELECTION
PROPOSAL
WRITING
TEST FOR
ASSOCIATION
VARIABLE
OPERATIONALIZATION
LOGIC
OF
ACCURACY
LOGIC
OF
EXPLICATION
LOGIC
OF
PROOF
LOGIC
OF
DISCOVERY
CONCEPTUALIZATION
MEASUREMENT
OF INDICATORS
HYPOTHESIS
GENERATION
NORMATIVE
CLARIFICATION
Fig. 14.3 The cycle of social and political research and its four logics
where the highest disciplinary status is usually awarded – but only a select few
make it to this stage in the cycle, and even their conclusions are always contingent
upon eventual replication by other scholars.
The reader should not be discouraged by this. To do original research on a
topic about which you care is an adventure. It can take you in diﬀerent directions
and end in diﬀerent places. A lot will depend on your point of departure,
but you will also be inﬂuenced at every turn by your professors and peers –
not to mention the fads and fashions of your discipline. The most important
thing is to be conscious and conﬁdent of the choices you will be making, and
then to know when and where to exit from the cycle. Hopefully, this and the
other chapters in this volume will help you to make the voyage easier and, ultimately,
more rewarding.
NOTE
11 Unfortunately, many of them will be so-called ABDs (all but dissertations) who come up
with a design for a previously conceptualized piece of research, could have written a proposal
and may even have given some thought to operationalizing its variables, but never
managed to actually ﬁnd the time, resources or energy to gather the relevant data – much
less to write them up.
295 The design of social and political research