Qualitative methods Andrew Roberts Favorite qualitative papers •What are some of your favorite qualitative papers? •What method do they use? •How do they persuade you? Qualitative research •Small-N: often 1-10 cases •Usually nominal or ordinal level data •Focus on time sequences, controlled comparisons, smoking guns •Sources: historical archives, interviews, observation • • • Qualitative methods in political science •Qualitative methods are still the dominant form in comparative politics –>60% of articles in a recent survey of CPS, CP, and WP –Though trend towards quantitative •However standards are changing –Harder to publish single-country qualitative studies –Reviewers more demanding on methods Characteristics •Explain individual cases: why WWI, why revolution in France, China & Russia –Typically causes of effects •Often deterministic causation •Often distinct pathways, equifinality •Focus on crucial cases •Focus on creating concepts • Benefits of fewer cases •Less worry about causal heterogeneity: Do X & Y have same relation in all cases –Does oil have same effect on politics in Venezuela and Norway? •Easier to see pathways, sequences •Key outcomes are often rare: revolutions, wars –They would get lost with random selection or buried in negative cases – Kinds of evidence •Historical archives – events, meetings, news reports •Unstructured or semi-structured interviews with policy-makers, protesters, etc. •Participant observation Quantitative versus Qualitative Templates Two cultures? •Do qualitative and quantitative methods have different logics? –KKV: single logic for all inference –Mahoney and Goertz: two different logics Approaches and concepts Quantitative Qualitative Large # of cases Usually continuous measures Small # of cases Often ordinal or cardinal measures Goal = estimate size of average effects of independent variables on dependent variable, often effects of causes Goal = fully explain causes of individual cases, typically causes of effects Probabilistic conception of cause •Deterministic conception of cause: necessary and sufficient conditions •Equifinality: multiple discrete causal paths More qualitative specificities •Case selection focused on specific, positive cases –Negative cases don’t tell us as much –Some cases more important than others (eg, WWII) •Each case should be explained correctly –Because causation is deterministic •Few cases => avoid causal heterogeneity •Sequences and pathways are important •More attention to concepts, revision of concepts • • • Issues with qualitative template •Is social world deterministic or probabilistic? –How many phenomena where cause is necessary or sufficient for outcome? –“No bourgeoisie, no democracy” •Often simply historical or descriptive •Can it predict? –Small scope conditions mean that inferences don’t apply outside of particular cases •Is it useful for making interventions? –Can it tell us what we should do? Main types •Case study •Process tracing –Path dependence & critical junctures •Structured, focused comparison •Qualitative comparative analysis •Conceptual/typological analysis •Ethnographic methods •Mixed methods & case selection 1. Case study Small-n problem •Statistical techniques work best when number of cases is large •Don’t work at all if number of cases is too small –If number of cases is less than or equal to number of independent variables, we can fit an infinite number of different regression lines Small-n problem Small-n problem Larger-n Making Democracy Work by Robert Putnam - Summary and Notes · Rufus Pollock Online Solutions to small-n problem •Case study to generate hypotheses or concepts –Then test it with more cases •Case study relative to previous knowledge •Turn one case into multiple cases –Evolution of case over time: process tracing –Break case into more cases: subnational •Small-n comparisons: structured Hypothesis generating case study: Japan’s developmental state •Chalmers Johnson studies Japan’s incredible period of growth •Singles out government agency – Ministry of International Trade & Industry – which selectively encourages particular industries –Highly meritocratic agency not captured by special interests –Provides subsidies, loans, trade protection Can we prove that these actions cause growth? •Firms supported by MITI prosper •Plausible mechanism: infant industry argument •Try to rule out alternatives –Unique national culture: but why only now? –Distinctive institutions like lifetime employment, shopfloor management: they predate period of growth Problem of case selection •How do we know that if this case reflects a general trend or is unique? –Problem of cherry-picking •Probably want to confirm hypothesis in other cases –South Korea, Taiwan seem to show similar trends –But what about developmental state in Brazil, Nigeria? 2. Process tracing Process tracing •The main form of qualitative analysis •Focus on sequences and mechanisms –Do events and processes fit those predicted by alternative theories? –Similar to detective solving a crime or doctor diagnosing an illness • spying-detectives-mumbai-india-pune.jpg (380×309) Causal process observations (CPOs) •CPOs are key form of evidence –Versus dataset observations: spreadsheet •“An insight or piece of data that provides information about context, process, or mechanism and that contributes distinctive leverage in causal inference” Examples •Nuclear taboo: Is there cultural aversion to using nuclear weapons? –Documentary evidence that key leaders thought about & discussed taboo •Why do leftist Latin American leaders switch to neoliberalism? –Documentary evidence that leaders like Fujimori or Menem spoke with international investors and got scared Theory testing with CPOs •Independent variable CPOs: is cause actually present? –Theory may imply that cause should be present at certain time and place •Mechanism CPOs: presence of intervening event posited by theory •Auxiliary outcome CPOs: presence of events that should occur if theory is true • Brady on 2000 election •Lott does a regression analysis showing that Bush lost votes in western time zone of Florida because media report that race is over •Brady shows this is highly unlikely –# of potential voters from 7-8 PM –% of those who would vote for Bush –% of those who heard the media reports –% of those who decided not to vote fltz.gif (430×297) Types of tests •Hoop test: necessary conditions –Is suspect in state on day of crime? –Good for eliminating hypos –But passing test doesn’t help much •Smoking gun: sufficient conditions –Is suspect holding a recently fired gun? –Good for confirming hypos –But failing test doesn’t help much 1270745475-smoking-gun.jpg (400×500) Pension privatization in CZ and PL •I hypothesized that privatization is due to responsive policy making •If this is true, we should see… –Public opinion differences between privatizers and non-privatizers –High sign-up rates in privatizers –Consultation with and support from interest groups –Reform only where all parties support –Reform passed just before elections • • Equifinality •Multiple pathways to the same outcome •Moore: How do countries become modern? –Democratic: bourgeois revolution –Fascist: revolution from above –Communist: peasant revolution •Quantitative analysis doesn’t discover pathways – only influence of individual variables and interactions – – My advice •Specify the theory very clearly –Try to outline the steps: X=>X1=>X2=>X3=>Y –Microfoundations: individual actions lead to outcome •Think of as many observable implications as possible –If X1=>X2, then we should see… •Show evidence that implications actually occurred •BEWARE: Not just a narrative or description More information •Check Lindsay Mayka’s video on process tracing on youtube • 3. Path dependence & critical junctures Path dependence •History matters – arbitrary past decisions determine present outcomes •Small initial advantages lead to lock-in and large later advantages •Consequences –Suboptimal outcomes may win –Multiple outcomes possible Polya urn •Urn with one red and one black ball •Pull one out and return with another ball of same color •Do this 100 times •What happens? –Many outcomes possible –Ultimately reach an equilibrium –Sequence & initial draws matter a lot http://www.generationaldynamics.com/ww2010/polya.gif Qwerty keyboard •Initially chosen so that keys don’t jam •But once people get used to it and companies start manufacturing typewriters, no one wants to change • QWERTY: Milwaukee's Typing Legacy | Wisconsin Life Mechanisms of path dependence •Large set-up costs but then returns to scale: need to be the first •Learning effects: get better at technology more that you use it •Coordination effects: technology more popular if more people using it •Adaptive expectations: picking the wrong horse has big costs Welfare state •Once we set up a state welfare program, it can get locked in –Interest groups grow up around the policy –Private actors (firms, charities) stop providing –People come to expect benefits •“Don’t take away my medicare!” Critical junctures •If path dependence important, we can isolate critical points where path gets locked-in •Key moments of openness, when different paths are possible •But due to contingencies, a certain path gets chosen and locks in Path dependence State Building and Agricultural Policy in 19th Century Political Regimes in 20th Century Military Authoritarianism Traditional Authoritarianism Progressive Democracy Source: Mahoney 2001, p. 230 How to do it? •Identify critical juncture where many choices are possible •Show the lock-in effects of this choice –How does it lead actors to coordinate on this outcome or increase costs of switching 4. Structured, focused comparison Structured, focused comparison •Comparison of small number of cases •Structure: general question and systematic gathering of equivalent data across cases •Focused: only certain aspects of cases Mill’s methods •Similarity –Very similar cases with different outcomes –Isolate what makes them different •Difference –Very different cases with similar outcomes –Isolate what they have in common • Most similar analysis Most different analysis How to do it? •Typically use most similar analysis –Similar cases with different outcomes –Helps to reduce potential causes •Try to identify key causal difference •Often combine with process tracing to show how cause leads to outcome My take •Still the workhorse of qualitative methods •Becoming less popular –Depends a lot on case selection –Worries about generalizability –Quantitative methods are approaching: matching •Probably best to combine with process tracing or quantitative methods •Or use to generate & eliminate hypotheses 5. Qualitative comparative analysis QCA •Use of set theory and Boolean logic to show that certain conditions are necessary and sufficient for an outcome •Explaining outcomes in specific cases rather than average effects of particular causes Increasing popularity of QCA Truth tables Some deterministic relations in PS •No bourgeoisie, no democracy •No famine in democracies •No wealthy democracies transition to authoritarianism •Every suicidal campaign has as its goal coercing a foreign state to leave its territory •Note: Are these causal claims? Fuzzy set analysis •Worry that QCA depends on dichotomous variables and deterministic causation •Fuzzy set analysis as a solution –Calibration of variables between 0 & 1: Degree of membership –Non-deterministic causation: eg, cause is 90% necessary Usefulness •Helps in identifying deterministic causation – necessary & sufficient conditions •Helps when many interactions between variables Some bad news •Recent tests of QCA with simulated data –QCA has difficulty recovering the correct pathways –Regressions do better •For larger-N fuzzy set, not clear what is added to regression-style analysis •Maybe useful because of closer attention to cases, concepts, and measures 6. Concepts and typologies Useful typologies & concepts •Typologies & concepts aren’t right or wrong, but useful and not useful –Don’t contradict common usage –Offer enough detail to differentiate cases –Avoid unnecessary detail –Help us understand world and causality –Can be measured easily Typologies •Often useful to create typologies •Typically 2X2 tables but can be larger screen-capture-104.png (751×667) Basics •Overarching concept •Row and column variables •Matrix (2x2 or different size) •Types – give each meaningful labels –Nominal: no scale –Partially ordered: A > B & C > D –Ordinal: A > B = C > D Issues •Typology should be mutually exclusive and collectively exhaustive –Categories don’t overlap –Categories cover all possibilities (though only in reference to particular cases) •Typologies can be outcomes or explanatory factors Three-level concepts •Basic level: Democracy •Secondary level: political rights & civil liberties •Indicator level: voter turnout, bans on political parties, restrictions on free speech, etc. Clear indicators and rules of aggregation Aggregation •Do you need all characteristics: necessary conditions, AND –Must have inclusiveness and competitiveness to be democratic •Or characteristics substitute for each other: sufficient conditions, OR –Need any 3 of 4 to be a welfare state –Pensions, healthcare, unemployment insurance, housing What is an ethnic group? •Family resemblance structure: not all characteristics need to be present –Membership reckoned mainly by descent –Members are conscious of group membership –Members share cultural features –These features held as valuable by most members of group –Group has homeland or remembers one –Group has shared history Spikes at extremes •Necessity of extending the scale http://www.systemicpeace.org/polity/P4Cases1800to2006.jpg http://www.phdcomics.com/comics/archive/phd090307s.gif 7. Ethnographic methods Participant observation •Researcher spends time with subjects, gets to know them well, watches them at work –Fenno refers to “soaking and poking” & “just hanging around” •New insights and concepts but hard to show causality and generality Fenno – Home style •Fenno attaches himself to Congresspeople in their home districts •Findings –Different constituencies: Geographic, reelection, primary, and intimates –Different types: Homefolks, issue-articulating activist, errand boy, local leader –Not just concerned with reelection –Give same account of themselves to different groups 8. Mixed methods Integration versus Triangulation •Triangulation: ask same question with 2 different methods –Is the question really the same? How to confirm a regression estimate of 0.21 with a case study? •Better: Integration –Use each method for what it is good at it –One method to produce final inference –Other method to design, test, or refine analysis Varieties of combinations •Start with regression and choose cases –Outliers to probe other possible causes –Typical cases to probe mechanism •Start with case studies to develop theory –Then use results to conduct larger test •Small-N comparison combined with subnational regressions •Iteration between cases and large-N How can case study add to statistical analysis •Test measurement quality •Evaluate plausibility of causal pathways •Search for evidence of omitted variables •Causal process observations may show that large effect likely or unlikely Types of case selection Method Use Typical: fits relationship between X & Y Help discover mechanisms that may confirm or disconfirm theory Diverse: exemplifies diverse values of X, Y, or X/Y Explore new causes or confirm theory Extreme: extreme values of X or Y Explore new hypotheses Deviant: deviates from relationship between X & Y – outliers Probe new explanations How does statistics add to case studies •Test generalizability of finding –What is proper domain? •Quantify effect sizes •Add evidence about steps in process tracing –Survey evidence, embedded experiment 9. Final issues Replicability •Increasing belief that qualitative work should be replicable •You should archive field notes, oral interviews •Codebook that describes measurement of all variables • When to choose qualitative template? •Good place to start your exploration – not sure what matters –Immerse yourself in some important cases and develop hypotheses •You care a lot about some very important cases •Unsure about how to conceptualize outcomes or causes • • Advantages of qualitative research •Focus on particular cases and explaining them correctly –Avoid causal heterogeneity •Necessary & sufficient conditions and multiple discrete paths (equifinality) •Better development of concepts and use of data to revise concepts •Lots of evidence about causal process can’t be reduced to spreadsheet • Disadvantages •Weaker generalizability •Quantitative judgments (where data exists) beat qualitative ones •Often deteriorates into history and description •How deterministic is social world? •Prediction and interventions? Exercises •Pick one of the following: –Choose one article from Collier’s piece on process tracing and answer questions –Choose several questions from Goertz’s long list on concepts –Something else? Paper on a specific method? Analysis of a particular qualitative article? Discussion questions • Small-n comparisons •Can we learn much from small-N (structured, focused) comparisons? •What can we conclude from a comparison of the Czech Republic, Hungary, and Poland? •Yes, they are similar enough, but can we really identify causality or produce general knowledge? •Can you name any papers that use structured, focused comparisons to produce strong conclusions? Process tracing •Just about any qualitative paper that traces a case over time gets called “process tracing”. Is there really a distinctive method here? •In the lectures I struggled to say exactly what you should do (except for tests of necessary & sufficient conditions). What are the key steps? •Again, are any of your favorite qualitative papers examples of process tracing? How do they do it? Concepts and typologies •Does political science already have too many concepts and typologies? In what areas, do we need more concepts/typologies? •Should we focus more on measures and new data than on new conceptions of, say, democracy/regime or the welfare state? •Where might we find new conceptions? When and where? •When are qualitative methods most useful and what can they provide? •When should we avoid them? •Should we usually combine them with quantitative methods? How should we do this? •Is qualitative data always necessary to supplement quantitative work?