Experiments GLCb1008 Introduction to Methodology of Social Sciences Aim of this lecture •Logic of experiments • •Experimental designs • •Strengths and weaknesses of experiments • •Practical issues Testing theories •Limits of observational studies • •We cannot save and load reality • •Ask a political party to go back in time and do a different campaign • •Experiments help us to overcome this impossible scenario • • Experiment - basics •Main role: •To estimate effect of independent variable on dependent variable •Almost perfect control for effects of all other variables • •The researcher manipulates with the independent variable • •Participants are split into at least two groups: •Each group obtains a different treatment (or no treatment at all) • •The analysis compares results for each group to estimate the effect of IV • •Many modifications • • A simple two group design Experimental group (Group 1) Control group (Group 2) Article about corruption (treatment) Waiting in a room (no treatment) Questionnaire: Trust in politics (0-10) A three group design Experimental group 1 (Group 1) Article about tiny corruption (treatment) Waiting in a room (no treatment) Questionnaire: Trust in politics (0-10) Experimental group 2 (Group 2) Control group (Group 3) Article about large corruption (treatment) Groups •Theoretically unlimited number of groups • •Beware of too complicated designs (especially if you have less experience) • •Costs of more groups (each groups needs participants) • •Control group is not a necessary feature • •Control group may receive a meaningless treatment (article about weather) •Does corruption decrease trust in elected representatives? • •Laboratory experiment, 2 groups •Group 1: Article with corruption of a political official •Group 2: Article with daily duties of a political official • •Sample of 100 people • •How to divide the sample into groups? How to divide participants into groups? 100 people 50 men (blue dots), 50 women (white dots) Wrong Solution •Experimental group: Control group: • 50 M, 0 W 0 M, 50 W Post experimental questionnaire •On a scale from 0 to 10 how do you trust the political official? • •Experimental group (50 M, 0 W): •Result: 4.2 • •Control group (0 M, 50 W): •Result 6.7 • •The result – reading an article on corruption lowers trust to political officials by 2.5 points Post experimental questionnaire •On a scale from 0 to 10 how do you trust the political official? • •Experimental group (50 M, 0 W): •Result: 4.2 • •Control group (0 M, 50 W): •Result 6.7 • •The result – reading an article on corruption lowers trust to political officials by 2.5 points What if the article has no effect but men simply have lower trust in politicians? Correct Solution •Experimental group: Control group: • 26 M, 24 W 24 M, 26 W Correct Solution •Experimental group: Control group: • 26 M, 24 W 24 M, 26 W All differences (that can affect the results, e.g., age, income, education) are eliminated. The only difference between the groups is our treatment (the article). If the article has an effect, we will see it in different scores of groups and vice versa. Random assignment vs. random sampling •Never confuse these two concepts • •Random sampling: •Procedure of selection of participants •Typically a random choice (not always from the whole population, but only from students, locals etc.) • •Random assignment: •Procedure of assignment of participants into groups •Random assignment is a critical feature that defines experiments Types of experiments •Based on conditions: •Lab (laboratory) •Field •Survey • •Do not take the names literally: •Lab does not mean you need a laboratory, but controlled conditions •Field experiments do not (usually) take place at fields •Survey experiments can be held in laboratories • •Conditions (and not the place) decide Lab Experiment Field experiment Is It (Finally) Time for a New PC? - Tabula IT Services Institution 1 (short email) • Institution 2 (short email) • Institution 3 (long email) • Institution 4 (long email) • Natural experiments •Experiments occur naturally: •Randomly assigned people who get basic income •Towns selected for high-speed Internet •Natural disaster that affects a part of a country • •Researchers do not provide a treatment, they “only” collect the data • •Assignment to groups is either random or it should be close to random • Přehled chytré inspirace pro všechny | iQLANDIA Transparency at the Local Level •Topic: Reactive provision of information by local governments • •Effect of Freedom of Information (FOI) requests on responsiveness of local governments compared to less formal requests • •Experimental study on 2,926 municipalities in Slovakia • •Requests on information about local elections • Preparation of Experiment •Random assignment of municipalities into three groups • •All municipalities obtained an information request • •Control group – baseline version •Exp group 1 – baseline + moral paragraph •Exp group 2 – baseline + FOI paragraph • •All requests sent via e-mail on Monday Results Group N No response (%) Response (%) Control 961 78.0 22.0 Moral Appeal 922 77.3 22.7 Legal FOI 943 56.1 43.9 Validity •Internal: •Conviction that the findings are the result of experimental treatment •Main threats – noncompliance and attrition •How to support • •External: •Ability to generalize the results beyond the group of participants •Experiments often conducted on student samples •Hawthorne effect – reaction to being observed •What to do – replications, different ways of measurement, better samples (population experiments) Ethics •Experiments are often confronted with ethical challenges • •Risk to current and later experiments • •Solution: •Good organization, ethical boards •Natural experiments Experiment •Strengths: •Effective isolation of other variables •Reliability, internal validity • •Weaknesses: •External validity (vs. experiments based on population sample) •Not always applicable in social sciences •Not all independent variables are subject of manipulation (age, values) •Need of replication