Intro Design Admin Data collection Data analysis External validity Examples References ooo oooooooo ooooooooo ooo ooo oooo ooooooo ooo Experimental methodology Ondrej Krcal DXhLMETl, MUNI 23.-24. 11. 2023 Some sources: James Tremewan's course thought at MUNI in May 2022, Handbook of Research Methods and Applications in Experimental Economics and Handbook of Experimental Economic Methodology Intro Admin Data collection •oo ooooooooo ooo Data analysis OOO External validity oooo Examples References ooooooo ooo Are you using experimental methods in your research? Intro Admin Data collection Data analysis External validity Examples •oo ooooooooo ooo ooo Are you using experimental methods in your research? Some kind of experimental design necessary for causal inference. Intro Admin Data collection Data analysis External validity Examples •OO Are you using experimental methods in your research? Some kind of experimental design necessary for causal inference. You can look for experimental designs in observational data (Stepán Mikula). Intro Admin Data collection Data analysis External validity Examples •oo ooooooooo ooo ooo Are you using experimental methods in your research? Some kind of experimental design necessary for causal inference. You can look for experimental designs in observational data (Stepán Mikula). Or you can produce your own experimental data - popular in many fields: • Gender and labor market - huge literature in the lab and field • Enviromental economics - common lab and field studies • Management/adaptation - common experiments: (1, 2) • DSGE - lab studies: (1, 2) Intro Admin Data collection Data analysis External validity Examples References o«o oooooooo ooooooooo ooo ooo oooo ooooooo ooo Overview Economic experiments (as compared to surveys) have two main features: * randomization • incentivization (exceptions: survey experiment, some lab studies) Intro Admin Data collection Data analysis External validity Examples o«o ooooooooo ooo ooo Overview Economic experiments (as compared to surveys) have two main features: • randomization • incentivization (exceptions: survey experiment, some lab studies) Running experiments (as opposed to using experiments in the data): • Design • informed by theory • to be able to analyze data efficiently (power, avoid assumptions) • Admin • Pre-registration • Ethics approval Data collection (depends on the type of the experiment) • Data analysis Design Admin Data collection Data analysis External validity Examples oooooooo ooooooooo ooo ooo oooo ooooooo MU Experimental Economics Laboratory (MUEEL) We offer research infrastructure: • experimental laboratory - hroot, otree programmers, support (lab manager, payments) • access to software and equipment (Qualtrics, Veyon, Air Quality, ...) • experience with lab/field/survey experiments and agent-based models • network of experts - local (CERGE-EI, VSE, WU) and broader Design Admin Data collection Data analysis External validity Examples •ooooooo ooooooooo ooo ooo Questionnaires in the lab Scientific disciplines are as precise as the measurements of basic concepts. Main example in economics: preferences (consistency, stability, values/distributions) • risk preferences • time preferences: discount rate, present bias • social preferences: e.g. altruism, trust, reciprocity Design Admin Data collection Data analysis External validity Examples •ooooooo ooooooooo ooo ooo Questionnaires in the lab Scientific disciplines are as precise as the measurements of basic concepts. Main example in economics: preferences (consistency, stability, values/distributions) • risk preferences • time preferences: discount rate, present bias • social preferences: e.g. altruism, trust, reciprocity Global preference survey: b r i q Institute on Behavior & Inequality niriated by Deuhttie Post Foundation global preferences survey elicit risk, time, and social preferences home about maps rankings downloads publications faq contact GPS Homepage Welcome to the website of the Global Preferences Survey, a globally representative dataset on risk and time preferences, positive and negative reciprocity, altruism, and trust - At*. Design Admin Data collection Data analysis External validity Examples o«oooooo ooooooooo ooo ooo Theory and experiments (1/2) Economic theory provides structure for the examination of how people behave in economic situations. Experiments not motivated by theory may lead to duplication of effort. Design Admin Data collection Data analysis External validity Examples o«oooooo ooo oooo Theory and experiments (1/2) Economic theory provides structure for the examination of how people behave in economic situations. Experiments not motivated by theory may lead to duplication of effort. Experiments are best at: • testing theories - make comparative-static predictions (about direction and not about size/parameters of the model) • testing assumptions field work is based on Design Admin Data collection Data analysis External validity Examples OO0OOOOO ooooooooo ooo ooo Theory and experiments (2/2) Economic theory • is strong and works: e.g. market experiments • is strong and does not work: e.g. expected utility theory • is weak: e.g. ultimatum game • selfish preferences are not theory • experience Design Admin Data collection Data analysis External validity Examples OO0OOOOO ooooooooo ooo ooo Theory and experiments (2/2) Economic theory • is strong and works: e.g. market experiments • is strong and does not work: e.g. expected utility theory • is weak: e.g. ultimatum game • selfish preferences are not theory • experience Adjustments to make theory more useful, e.g. • other-regarding preferences (Fehr & Schmidt, 1999; Bolton & Ockenfels, 2000) • quantal response equilibrium (McKelvey & Palfrey, 1995) • one-one shot interactions - level-k theory (Stahl & Wilson, 1995) Design Data collection Data analysis External validity Examples References ooo ooo«oooo ooooooooo ooo oooo ooooooo ooo Design choices: treatments Random assignment to treatment —>> causal inference Treatments should differ only in one thing! Intro Design Admin Data collection Data analysis External validity Examples References OOO OOO0OOOO ooooooooo ooo ooo oooo ooooooo ooo Design choices: treatments Random assignment to treatment —» causal inference Treatments should differ only in one thing! Sometimes experiments involve multiple factors, e.g. communication (with/without) and stake-size (low/high) in a dictator game. Full factorial design would lead to 2 x 2 = 4 designs. Design Admin Data collection Data analysis External validity Examples OOO0OOOO ooooooooo ooo ooo Design choices: treatments Random assignment to treatment —> causal inference Treatments should differ only in one thing! Sometimes experiments involve multiple factors, e.g. communication (with/without) and stake-size (low/high) in a dictator game. Full factorial design would lead to 2 x 2 = 4 designs. Rule of thumb: Use the minimum number of treatments possible to test your hypothesis! • Why? • More treatments - problem of multiple testing. • Fewer treatments = more observations per treatment • Avoid • intermediate treatment unless interested in non-linearities. • full factorial design unless interested in the interaction term. Design Admin Data collection Data analysis External validity Examples 0000*000 Design choices: within/between subject design Within-subject design: Each subject in more than one treatments • Advantages: • Effect on individual • Greater power • Disadvantages: • Subjects may want to choose the same to be consistent • Experimental demand effect = changes in behavior due to cues about what constitutes appropriate behavior • Potential order effects = order of tasks affects decisions Design Admin Data collection Data analysis External validity Examples OOO 0000*000 ooooooooo ooo ooo oooo ooooooo ooo Design choices: within/between subject design Within-subject design: Each subject in more than one treatments • Advantages: • Effect on individual • Greater power • Disadvantages: • Subjects may want to choose the same to be consistent • Experimental demand effect = changes in behavior due to cues about what constitutes appropriate behavior • Potential order effects = order of tasks affects decisions Between-subject design: Each subject only in one treatment • Advantages - solves problems of demand effects and consistency • Disadvantages - power and subjective judgment (Birnbaum, 1999) Design Admin Data collection Data analysis External validity Examples ooo ooooo«oo ooooooooo ooo ooo oooo ooooooo ooo Design choices: incentives Transparent (minimum payment in lab experiments) Incentive compatible Pay all vs. random incentive system (RIS) (Clot et al., 2018) Design Admin Data collection Data analysis External validity Examples OOOOOO0O ooooooooo ooo ooo Other design choices • One-shot vs. repeated play of a game/situation • advantages: allow learning, interested in learning effects • distadvantages: dilutes incentives (random lottery incentive mechanism to avoid wealth effects) Design Admin Data collection Data analysis External validity Examples OOOOOO0O ooooooooo ooo ooo Other design choices • One-shot vs. repeated play of a game/situation • advantages: allow learning, interested in learning effects • distadvantages: dilutes incentives (random lottery incentive mechanism to avoid wealth effects) • Partner vs. stranger matching • Partner matching - play with the same subjects each time • Stranger matching - randomly rematched every time in a given matching group (perfect stranger matching if played always with someone else) Design Admin Data collection Data analysis External validity Examples OOOOOO0O ooooooooo ooo ooo Other design choices • One-shot vs. repeated play of a game/situation • advantages: allow learning, interested in learning effects • distadvantages: dilutes incentives (random lottery incentive mechanism to avoid wealth effects) • Partner vs. stranger matching • Partner matching - play with the same subjects each time • Stranger matching - randomly rematched every time in a given matching group (perfect stranger matching if played always with someone else) • Direct response vs. strategy method in sequential games: • direct-response: PI makes a decision, P2 is informed of the decision then makes their own decision. • strategy method: PI makes a decision, and simultaneously P2 decides for all possible choices of PI. The strategies of the two players are combined and the outcome determined. Design Data analysis External validity Examples ooooooo« ooo Power calculations Power is the probability of finding an effect that is really there. Underpowered studies (too few subjects): • less likely to detect real treatment effects. • more likely to have a large effect, even when there is no real effect Design Admin Data collection Data analysis External validity Examples 0000000« ooooooooo ooo ooo Power calculations Power is the probability of finding an effect that is really there. Underpowered studies (too few subjects): • less likely to detect real treatment effects. • more likely to have a large effect, even when there is no real effect Useful exercise. Problem: you need to know the effect size. Especially useful to get some idea about number of subjects when interested in more complicated effects (e.g. interaction effect) Admin Data analysis External validity •oooooooo ooo Replication crisis • Psychology: 36/100 studies replicated, effect sizes 50% of original effect (Open Science Collaboration, Science, 2015) • Experimental economics: 11/18 (61%) studies in AER or QJE replicated, effect sizes 66% of original effect (Camerer et al, Science, 2016) • Social sciences: 13/21 (62%) studies in Nature or Science, effect sizes 50%) of original effect (Camerer et al, Science, 2018) Admin Data analysis External validity •oooooooo ooo Replication crisis • Psychology: 36/100 studies replicated, effect sizes 50% of original effect (Open Science Collaboration, Science, 2015) • Experimental economics: 11/18 (61%) studies in AER or QJE replicated, effect sizes 66% of original effect (Camerer et al, Science, 2016) • Social sciences: 13/21 (62%) studies in Nature or Science, effect sizes 50%) of original effect (Camerer et al, Science, 2018) Reasons: • Publication bias • The way tests work (sample size, a, reasonable hypotheses) • P-hacking • Testing multiple hypotheses Admin Data analysis External validity Examples O0OOOOOOO ooo ooo P-hacking • Selectively removing outliers • Running multiple tests - reporting lowest p-value • Running multiple regressions specifications and reporting only the one that works. • Including pilot data to the analysis • Stopping gathering data as soon as p < 0.05 Admin Data analysis External validity OO0OOOOOO ooo Multiple testing Problem when using multiple simultaneous tests (common in experimental economics - multiple outcomes/subgroups/treatments). Family-wise error rate (FWER): Probability of at least one false rejection for k statistically independent tests at a level: FWER = 1 - (1 - a)k For a = 0.05 and k = 2, 3, 4, .... FWER = 0.098, 0.143, 0.185,. Admin Data analysis External validity OO0OOOOOO Multiple testing Problem when using multiple simultaneous tests (common in experimental economics - multiple outcomes/subgroups/treatments). Family-wise error rate (FWER): Probability of at least one false rejection for k statistically independent tests at a level: FWER = 1 - (1 - a)k For a = 0.05 and k = 2, 3, 4, .... FWER = 0.098, 0.143, 0.185,... There are methods used for adjusting for multiple hypotheses: • Bonferroni • Holm-Bonferonni • List (2019) Admin Data analysis External validity OOO0OOOOO When to adjust? cw5e acne! SCIENTISTS! INVESTIGATE! 3UT ljc'äe Piling We RX)MON0 UNK, BOUEEM ACHE ( p > &0fr} that serriES that X HEAR ITS ONLV ft CERrairt Color TiW closes it: I 9vr H'llltfKftMrT! If Admin Data analysis External validity OOO0OOOOO ooo When to adjust? WEFOjNüNÖ UNK BOUEDJ FORPUE JtliY (p>0.0?). UNK BCTJEIN 5ALMOM XUV (p>0.Ofr). ; UNKCOUELN 0WWW JEüy WE FOUND NO (p>o,o?} WERXJ^DNO ÜMK BDVJEEN GEJPWSWDAOC (pXW), WERW4ON0 TuftÖUOISE JEUY Wt FCOT4DMÖ UNK ßOUEBJ (p> 0.059. WE RXnC>NO UNKßOUEEN (p> 0.057 WE FOUND NO UNK GOVJEE*J (p>0,0?)r y WE FOUND NO UNK GOVJQW VEUJOV 3ELiy (p>0,0?). V Admin Data analysis External validity OOO0OOOOO ooo When to adjust? WE POUND NO GREY JEUY / WE FOUND NO UNKGÜVEEN ßDGE JEUV (P>o.o?) WE FOUND NO UNK, GOUEIM TAH JEUY WE FOUND NO UNKGCWEIN I WE1KXM4DN0 UNK0DUS>i CYAN XUV AMD flCWE CP > 0,0?), WERXJMONO UNK GEWEW (p>0,Of), WE FOUND ft UNK GOVJEEN GREEN JEUr (p<0,C*). WE FOUND NO UNK OOVEEN PBKH (P>0.O?). WE FOUND NO LiNK GOVJEEM (p>0,05> WE FOUND NO UNKOGUEIN (P>0.OS}r y Admin Data collection Data analysis External validity OOO^OOOOO ooo ooo When to adjust? GREEN CJEUY BEPNS LINKED To/V»El 955{C«Hf«06KE *- 5£COhiC|0EMCt! Sao*r> one-sided • competing theories (direction is unclear) —> two-sided Admin Data collection Data analysis External validity Examples OOO OOOOOOOO OOOOOOOOO OOO «00 oooo ooooooo ooo Hypotheses and p-values Tests: • null (Hq) vs. alternative hypothesis (Hi) • one-sided vs. two-sided test • strong theoretical reasons one-sided • competing theories (direction is unclear) —> two-sided Interpret p-values carefully: • It is not the probability the null hypothesis is true • Never "accept the null": p > 0.1 is not evidence the null is true • A small p-value does not mean an effect is important Admin Data collection Data analysis External validity Examples ooo oooooooo ooooooooo ooo o«o oooo ooooooo ooo Types of tests Treatment tests: • Non-parametric tests do not assume that data comes from a particular probability distribution, e.g. normal distribution. • important because experimental data often has small sample size and is non-normally distributed • At the cost of stronger (typically untestable) assumptions about true distribution of data, parametric methods allow us to do more interesting stuff, e.g.: • regression analysis (controls, non-independent data) • treating with heterogenous types and zeros • structural models (estimating utility functions) Admin Data collection Data analysis External validity Examples ooo oooooooo ooooooooo ooo o«o oooo ooooooo ooo Types of tests Treatment tests: • Non-parametric tests do not assume that data comes from a particular probability distribution, e.g. normal distribution. • important because experimental data often has small sample size and is non-normally distributed • At the cost of stronger (typically untestable) assumptions about true distribution of data, parametric methods allow us to do more interesting stuff, e.g.: • regression analysis (controls, non-independent data) • treating with heterogenous types and zeros • structural models (estimating utility functions) • Exact tests calculate the p-value exactly (true asymptotically) • Non-exact tests are OK for a big-enough sample (but "big enough" depends on the true distribution of the data, which is unknown) Admin Data collection Data analysis External validity Examples ooo oooooooo ooooooooo oo« oooo ooo Overview of statistical methods used2 Treatment tests • looking for least assumptions (non-parametric) and most power • depends on type of data (binary/multi-valued) and comparison (within/between subject) Moffatt, P. (2020). Experimetrics: Econometrics for experimental economics. Bloomsbury Publishing. Admin Data collection Data analysis External validity Examples ooo oooooooo ooooooooo oo« oooo ooo Overview of statistical methods used2 Treatment tests • looking for least assumptions (non-parametric) and most power • depends on type of data (binary/multi-valued) and comparison (within/between subject) Regression analysis (parametric) • testing effect of more than one treatment simultaneously • controlling for additional variables • accounting for dependence between observations Moffatt, P. (2020). Experimetrics: Econometrics for experimental economics. Bloomsbury Publishing. Admin Data collection Data analysis External validity Examples ooo oooooooo ooooooooo ooo oo« oooo ooooooo ooo Overview of statistical methods used • Treatment tests • looking for least assumptions (non-parametric) and most power • depends on type of data (binary/multi-valued) and comparison (within/between subject) • Regression analysis (parametric) • testing effect of more than one treatment simultaneously • controlling for additional variables • accounting for dependence between observations • Structural modelling = estimating parameters of a utility function Moffatt, P. (2020). Experimetrics: Econometrics for experimental economics. Bloomsbury Publishing. Admin Data collection Data analysis External validity Examples OOOOOOOOO OOO «000 ooo What is external validity External validity (ecological validity, generalizability) - validity of applying the conclusions of a scientific study outside the context of that study (across other situations, people, stimuli, and times). Internal validity - reflects how well the study is conducted and hence to what extent do the observed results describe in the population we are studying. Admin Data collection Data analysis External validity Examples OOOOOOOOO OOO 0900 ooo Levitt and List (2007) Levitt and List (2007) - behavior in the lab (when measuring social preferences) is influenced by factors other than monetary incentives: • the presence of moral and ethical considerations • the nature and extent of scrutiny of one's actions by others • the context in which the decision is embedded • self-selection of the individuals making the decisions • the stakes of the game Admin Data collection Data analysis External validity Examples ooooooooo ooo oo«o ooo Camerer's reply to Levitt and List Camerer (2011) comes with three arguments against the criticism: 1. The goal of experimental economics is to establish a general theory linking economic factors to behaviour. Generalizability from the lab to the field is not a primary concern in a typical experiment. 2. The factors listed by Levitt and List are not essential for all lab experiments (except for obtrusiveness, because of human subjects protection) and there is little evidence that typical lab features undermine generalizability. 3. Economics experiments designed to test lab-field generalizability show that laboratory findings could be generalized to comparable field settings. Admin Data collection Data analysis External validity Examples ooooooooo ooo oo«o ooo Camerer's reply to Levitt and List Camerer (2011) comes with three arguments against the criticism: 1. The goal of experimental economics is to establish a general theory linking economic factors to behaviour. Generalizability from the lab to the field is not a primary concern in a typical experiment. 2. The factors listed by Levitt and List are not essential for all lab experiments (except for obtrusiveness, because of human subjects protection) and there is little evidence that typical lab features undermine generalizability. 3. Economics experiments designed to test lab-field generalizability show that laboratory findings could be generalized to comparable field settings. Comparing lab and field: Laboratory experiments are more easily replicated whereas field experiments are less obtrusive. Admin Data analysis External validity Examples ooo oooooooo ooooooooo ooo ooo ooo« ooooooo Lab experiments focus on qualitative effect Kessler and Vesterlund (2015) argue that • the debate concentrates around a straw-man version of external validity that quantitative results are externally valid • for most laboratory studies it is only relevant to ask whether the qualitative results (= direction or the sign of the estimated effect) are externally valid • laboratory studies are conducted to identify general principles of behavior and therefore promise to generalize Admin Data collection Data analysis External validity Examples ooooooooo ooo ooo« ooo Lab experiments focus on qualitative effect Kessler and Vesterlund (2015) argue that • the debate concentrates around a straw-man version of external validity that quantitative results are externally valid • for most laboratory studies it is only relevant to ask whether the qualitative results (= direction or the sign of the estimated effect) are externally valid • laboratory studies are conducted to identify general principles of behavior and therefore promise to generalize Different methodologies are not in competition. They are complementary. Admin Data collection Data analysis External validity Examples OOOOOOOOO OOO «000000 ooo Case: Are bankers cheaters? (1/6) Cohn et al. (Nature, 2014) run a lab-in-the-field experiment with 128 bankers • Priming professional identity with seven questions • T: about their professional background (e.g. "At which bank are you presently employed?" or "What is your function at this bank?") • c: that were unrelated tot heir profession (e.g. "How many hours per week do you watch television on average?") • Playing a cheating game (10 coin throws earning $20 or 0 each) + paid only if higher than earnings of a randomly drawn person from a pilot • Manipulation check - converting word fragments into meaningful words (—oker = broker or smoker) Admin Data collection Data analysis External validity Examples OOOOOOOOO OOO «000000 ooo Case: Are bankers cheaters? (1/6) Cohn et al. (Nature, 2014) run a lab-in-the-field experiment with 128 bankers • Priming professional identity with seven questions • T: about their professional background (e.g. "At which bank are you presently employed?" or "What is your function at this bank?") • c: that were unrelated tot heir profession (e.g. "How many hours per week do you watch television on average?") • Playing a cheating game (10 coin throws earning $20 or 0 each) + paid only if higher than earnings of a randomly drawn person from a pilot • Manipulation check - converting word fragments into meaningful words (—oker = broker or smoker) Binomial 0 20 40 60 80 100 120 140 160 180 200 Earnings (US$) 0 20 40 60 80 100 120 140 160 180 200 Earnings (US$) Intro Admin Data collection Data analysis External validity Examples References OOOOOOOO OOOOOOOOO OOO OOO OOOO O0OOOOO Case: Are bankers cheaters? (2/6) Several channels though which priming may have worked: • Competitive behaviour in banking is intrinsically desirable (no evidence) • Salience of competitive incentive schemes (no difference between core and support units) • Norm obedience - what I should do/what others do (no difference in beliefs what others will do) • Evidence on materialistic values below Admin Data collection Data analysis External validity Examples ooooooooo ooo o«ooooo ooo Case: Are bankers cheaters? (2/6) Several channels though which priming may have worked: • Competitive behaviour in banking is intrinsically desirable (no evidence) • Salience of competitive incentive schemes (no difference between core and support units) • Norm obedience - what I should do/what others do (no difference in beliefs what others will do) • Evidence on materialistic values below Control Professional Materialism Materialism identity < median > median Admin Data collection Data analysis External validity Examples ooooooooo ooo oo«oooo ooo Case: Are bankers cheaters? (3/6) Other tests and robustness: • other industries beliefs Intro Design Admin Data collection Data analysis External validity OOO oooooooo ooooooooo ooo ooo oooo Examples oo«oooo ooo Case: Are bankers cheaters? (3/6) Other tests and robustness: • other industries beliefs Admin Data collection Data analysis External validity Examples ooooooooo ooo ooo*»ooo ooo Case: Are bankers cheaters? (4/6) Criticism: • Vranka and Houdek (FrontPsy, 2015) criticize interpretation: • stereotype threat instead of social norms • what is treatment? control group primed by leisure activities - call for multiple control groups • priming bankers with money instead of professional identity • why not replicate when they had bankers giving expectations? • Hupe (2018) criticize the analysis (not published) propose a different HO: cheating the same in all six groups • criticizes the use of MW test (ties) Admin Data collection Data analysis External validity Examples OOOOOOOOO OOO OOOO0OO ooo Case: Are bankers cheaters? (5/6) Replications: Rahnwan et al. (Nature, 2019) do not replicate the result (n = 768) • Cohn et al. (Nature, 2019) why the replication by Rahnwan not valid: • Media attention —> selection (2 out of 27 banks) • Disclosed purpose at the beginning (demand effect) • Manipulation check did not work in Middle East • Basic retail services vs. investment/private banking • Study tests whether "prevailing scandals involving fraud and unethical behaviour in the banking industry were partly the result of a problematic business culture, rather than, for example, the employment of dishonest people in the banking industry." instead of "the prevailing business culture in the banking industry weakens and undermines the honesty norm" • Huber and Huber (JEBO, 2020) Admin Data collection Data analysis External validity Examples OOOOOOOOO OOO OOOOO0O ooo Case: Are bankers cheaters? (6/6) Huber and Huber (JEBO, 2020): • Lying game (T: 31; F: 37) in which individual lying is observed • Framing: • Abstract: Imagine there are two possible states of nature and one of your tasks is to report the current state. • Neutral: Imagine you are a security clerk at a museum and one of your tasks is to inform the manager each week about the average number of visitors in the preceding week. • Financial: Imagine you are the Chief Executive Officer (CEO) of a publicly listed company and one of your tasks is to inform shareholders each quarter about the course of business and the earnings per share. Admin Data collection Data analysis External validity Examples OOOOOOOOO OOO OOOOO0O ooo Case: Are bankers cheaters? (6/6) Huber and Huber (JEBO, 2020): • Lying game (T: 31; F: 37) in which individual lying is observed • Framing: • Abstract: Imagine there are two possible states of nature and one of your tasks is to report the current state. • Neutral: Imagine you are a security clerk at a museum and one of your tasks is to inform the manager each week about the average number of visitors in the preceding week. • Financial: Imagine you are the Chief Executive Officer (CEO) of a publicly listed company and one of your tasks is to inform shareholders each quarter about the course of business and the earnings per share. PROF stud 100 w i— o Cl £ 75 to Q c o f 50 o CD CT ta c 25 CD 2 4 6 Economic cost of honesty 100 o CD 75 in CD c o f 50 O CD cn CO c 25 CD 0 1 s x V ^^^^^ k 0 12 3 4 FIN —> NEU ABS Economic cost of honesty — FIN — NEU ABS Admin Data collection Data analysis External validity Examples ooooooooo ooo oooooo* ooo Some more examples A good strategy is to combine experiments with observational data: • Bursztyn, Leonardo, Thomas Fujiwara, and Amanda Pallais. '"Acting Wife': Marriage Market Incentives and Labor Market lnvestments."/4mer/car? Economic Review 107.11 (2017): 3288-3319. • Kuziemko, I., Buell, R. W., Reich, T., Norton, M. I. (2014). "Last-place aversion": Evidence and redistributive implications. The Quarterly Journal of Economics, 129(1), 105-149. Admin Data collection Data analysis External validity Examples ooooooooo ooo oooooo* ooo Some more examples A good strategy is to combine experiments with observational data: • Bursztyn, Leonardo, Thomas Fujiwara, and Amanda Pallais. '"Acting Wife': Marriage Market Incentives and Labor Market lnvestments."/Arner/car? Economic Review 107.11 (2017): 3288-3319. • Kuziemko, I., Buell, R. W., Reich, T., Norton, M. I. (2014). "Last-place aversion": Evidence and redistributive implications. The Quarterly Journal of Economics, 129(1), 105-149. Some Czech examples: • Bartos, V., Bauer, M., Cahlfkova, J., Chytilova, J. (2022). Communicating doctors' consensus persistently increases COVID-19 vaccinations. Nature, 1-8. • Vranka, M., Frollova, N., Pour, M., Novakova, J., Houdek, P. (2019). Cheating customers in grocery stores: A field study on dishonesty. Journal of Behavioral and Experimental Economics, 83, 101484. • Krcal, O., Peer, S., Stanek, R., Karlinova, B. (2019). Real consequences matter: Why hypothetical biases in the valuation of time persist even in controlled lab experiments. Economics of Transportation, 20, 10013 Admin Data collection Data analysis External validity Examples References ooo oooooooo ooooooooo ooo ooo oooo ooooooo «oo Literature (1/3) • Birnbaum, M. H. (1999). How to show that 9>221: Collect judgments in a between-subjects design. Psychological Methods, 4(3), 243. • Bolton, G. E., &. Ockenfels, A. (2000). ERC: A theory of equity, reciprocity, and competition. American economic review, 90(1), 166-193. • Butler, D. M., &. Desposato, S. (2022). Proposing a Compensation Requirement for Audit Studies. Political Studies Review, 20(2), 201-208. • Camerer, C. (2011). The promise and success of lab-field generalizability in experimental economics: A critical reply to Levitt and List. Available at SSRN 1977749. • Camerer, C. F., Dreber, A., Forsell, E., Ho, T. H., Huber, J., Johannesson, M., ... & Wu, H. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433-1436. • Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T. H., Huber, J., Johannesson, M., ... & Wu, H. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637-644. • Clot, S., Grolleau, G., &. Ibanez, L. (2018). Shall we pay all? An experimental test of Random Incentivized Systems. Journal of Behavioral and Experimental Economics, 73, 93-98. • Cohn, A., Fehr, E., & Marechal, M. A. (2014). Business culture and dishonesty in the banking industry. Nature, 516(7529), 86-89. • Cohn, A., Fehr, E., & Marechal, M. A. (2019). Selective participation may undermine replication attempts. Nature, 575(7782), E1-E2. Admin Data collection Data analysis External validity Examples References ooooooooo ooo o«o Literature (2/3) • Fehr, E., & Schmidt, K. M. (1999). A theory of fairness, competition, and cooperation. The quarterly journal of economics, 114(3), 817-868. • Frechette, G. R., & Schotter, A. (Eds.). (2015). Handbook of experimental economic methodology. Handbooks of Economic Methodol. • Henrich, J., Boyd, R., Bowles, S., Camerer, C, Fehr, E., Gintis, H., ... &. Tracer, D. (2001). Economic man in cross-cultural perspective: Behavioral experiments in fifteen small-scale societies (No. 01-11-063). • Huber, C., & Huber, J. (2020). Bad bankers no more? Truth-telling and (dis) honesty in the finance industry. Journal of Economic Behavior &. Organization, 180, 472-493. • Hupe, J. M. (2018). Shortcomings of experimental economics to study human behavior: a reanalysis of Cohn et al. 2014, Nature 516, 86-89, "Business culture and dishonesty in the banking industry". • Levitt, S. D., &. List, J. A. (2007). What do laboratory experiments measuring social preferences reveal about the real world? Journal of Economic perspectives, 21(2), 153-174. • List, J. A., Shaikh, A. M., & Xu, Y. (2019). Multiple hypothesis testing in experimental economics. Experimental Economics, 22(4), 773-793. • List, J. A. (2020). Non est disputandum de generalizability? A glimpse into the external validity trial (No. w27535). National Bureau of Economic Research. • Milgram, S. (1963). Behavioral study of obedience. The Journal of abnormal and social psychology, 67(4), 371. • McKelvey, R. D., & Palfrey, T. R. (1995). Quantal response equilibria for normal form games. Games and economic behavior, 10(1), 6-38. Admin Data collection Data analysis External validity Examples References ooooooooo ooo oom Literature (3/3) • Mujcic, R., &l Frijters, P. (2021). The colour of a free ride. The Economic Journal, 131(634), 970-999. • Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. • Ortmann, A., &. Hertwig, R. (2002). The costs of deception: Evidence from psychology. Experimental Economics, 5(2), 111-131. • Radner, R. (1980). Collusive behavior in noncooperative epsilon-equilibria of oligopolies with long but finite lives. Journal of economic theory, 22(2), 136-154. • Rahwan, Z., Yoeli, E., & Fasolo, B. (2019). Heterogeneity in banker culture and its influence on dishonesty. Nature, 575(7782), 345-349. • Schram, A., & Ule, A. (Eds.). (2019). Handbook of research methods and applications in experimental economics. • Stahl, D. O., & Wilson, P. W. (1995). On players models of other players: Theory and experimental evidence. Games and Economic Behavior, 10(1), 218-254. • Vranka, M. A., & Houdek, P. (2015). Many faces of bankers' identity: How (not) to study dishonesty. Frontiers in Psychology, 6, 302.