Experimental Methods in Economics Research
Prof. Maroš Servátka
MGSM Vernon Smith Experimental Economics
Laboratory
I. What is experimental economics?
• Methodology in which laboratory and field economies are created in
order to conduct economic experiments.
– Data collection (decisions of real people) in a controlled, specifically
designed environment in order to address economic research
questions.
Motivation for econ experiments
• Economic theories devised to explain:
1) market activity between many people
2) strategic interaction between few people
3) individual decision making
• How can we tell how successful a theory is predicting subsequent
outcomes?
• For example, you want to study generosity/charitable giving:
– Research question: Do transaction costs reduce charitable giving?
Motivation for econ experiments
• Why not use readily available data?
• Example: “A change in policy variable X will cause a change in
economic outcome Y.”
ΔX → ΔY
• Problem: if we collect real world economic data on X and Y using
surveys of households or firms, there will be other variables that
changed at the same time X did, e.g. Z. The truth may be that
ΔZ → ΔX and ΔZ → ΔY, and there is no causal link between X and Y at
all!
Motivation for econ experiments
• Traditional Solution
– Collect survey data on as many Z variables as thought might be
relevant, and use econometric techniques to test for whether
historical variation in X can predict variation in Y while controlling
for variations in other Z variables.
• Complementary Solution
– Create a decision environment that simulates the real world
environment of interest, and randomly assign people between
treatments in that environment where X is varied. Structure the
design so that Z factors are either held constant across treatments,
or else “average out” between treatments due to random
assignment. See if Y varies across treatments as theory predicts.
• So why not run a survey to study generosity?
• Do people tell the truth? How do we know?
• Economists skeptical of what people say – let’s see what they do!
– “Put your money where your mouth is!”
• General idea: Observe incentivized behavior in controlled
conditions
• How do you control conditions? Go to the lab!
6
Economists skeptical about surveys
• We recruit volunteers and randomly allocate them to roles.
• Participants read instructions and solve control questions.
• Anonymous interaction in a market or strategic situation.
• No deception!
• We pay participants according to their decisions in cash.
7
What is an economics experiment?
Brief history of experimental econ
• Edward Chamberlin conducted the first lab experiment at Harvard in 1948 with graduate students.
• Divided a group of students into buyers and sellers. Buyers were each given a card that listed a dollar
amount like $3. Sellers were each given a card that listed a dollar amount like $1. If a buyer could buy a
card from a seller for less than $3, he would earn the difference. If a seller could sell a card to a buyer for
more than $1, he would earn the difference.
• Buyers and sellers allowed to mill about the room and make deals. Chamberlin tested whether overall
trades converged to the competitive equilibrium predictions of p*, q* and maximized (buyer plus seller)
surplus.
• Vernon Smith was a graduate student subject in this experiment, and has gone on to spend a lifetime
conducting experiments regarding the performance of various market and auction mechanisms. He
altered Chamberlin’s market design slightly to create the “Double Auction” mechanism that is extremely
robust at achieving the competitive equilibrium predictions.
P S
1
. 1 Q
P
3
D
. 1 Q
P S
p*
D
Q
q*
Brief history
• Experiments testing non-cooperative game theory as a way or predicting the
outcomes of strategic interactions between a small number of people began in the
1950’s regarding the famous “Prisoner’s Dilemma.” (Many by sociologists and social
psychologists)
• Sauerman and Selten (1959) began testing the Cournot duopoly application of the
Prisoner’s Dilemma. Relabel “Confess” as “Produce a lot” and “Don’t Confess” as
“Produce a little.” (Reinhard Selten later won a Nobel Prize in 1994, but for theory.)
• Increasingly complex game theory structures tested since in industrial organization
and bargaining, but also simple games to test for non-selfish social preferences
(altruism, fairness, trust, reciprocity, envy etc.)
• Experiments testing individual decision making began with tests of the Expected
Utility theory of von Neumann and Morgenstern (1947). One of the 3 axioms
underlying EU theory failed in experiments, causing some theorists to develop more
general descriptive theories of decision making under uncertainty. Example:
Prospect Theory of Kaheman and Tversky (1979).
Nobel Prize in Economics 2002
• Daniel Kahneman for having integrated insights from
psychological research into economic science, especially
concerning human judgment and decision-making under
uncertainty
• Vernon L. Smith for having established laboratory experiments
as a tool in empirical economic analysis, especially in the study
of alternative market mechanisms
• Experimental economics became mainstream
MGSM Vernon Smith Experimental
Economics Laboratory
II. Objectives of experiments
• Testing theories
• Searching for facts
• Testing institutions and environments
• Whispering in the ears of princes: policy advice and wind tunnel
experiments
• Teaching experiments
12
Testing theories
• Testing theories under precisely controlled and/or measured
conditions that are typically unavailable in field data.
• Is a theory empirically valid?
– E.g. testing theory of tournaments
• Three ingredients of a theory (Smith AER’82, JEP’89)
– Environment
– Institution
– Behavior
13
Environment
• Collection of all agents’ characteristics
– Tastes and technology, represented by utility or preference
functions, resource endowments and technology or cost functions
• In reduced form these characteristics are the individual demand
(willingness-to-pay) and supply (willingness-to-accept)
schedules
• Controlled by using monetary rewards (Induced value theory,
Smith 1976)
14
Institution
• Defines the language (possible messages or actions) of
communication
– E.g. bids by buyers, asks by sellers, acceptance by either
• Specifies formally or informally by tradition:
– Order in which agents move, or that there is no order
– Rules under which the messages become contracts and thus
allocations
• Information conditions
• Framing (story)
15
Behavior
• Agents’ choices and messages that lead to allocations
– Messages are not outcomes
– Messages translate into outcomes depending upon the allocation
and cost imputation rules of the institution
• Theories introduce assumptions about agents’ behavior
– E.g. (expected) utility and profit maximization, common information
yields common expectations, agents make choices as if they were
risk averse, transaction costs (costs of thinking, deciding, acting)
are negligible…
• Theories of behavior make predictions about messages
– E.g. bids in a sealed bid auctions, price posted by an oligopolist,
reservation price below which a price searching agent will buy…
16
What does it mean to test a theory?
• Directly test a theory and behaviorally discriminate between
theories
– Theory provides basis for experimental abstraction and
experimental design
– Implement the conditions of the theory (e.g. preference, technology,
institutional assumptions)
– Compare the prediction with the experimental outcome
• Theory is often silent on institutional assumptions
• Ex 1: Supply-demand model
• Ex 2: An Empirical Study of Decentralized Institutions of
Monopoly Restraint (Smith, 1981)
• Note: Experimental test is always a joint test concerning all
assumptions (in particular also induced value theory)17
What if the theory fails?
• Explore the causes of a theory’s failure
– First, reexamine the design
– If design OK, find out when the theory fails and when it succeeds
– Design proper control treatment that allows causal inferences about
why the theory fails
Searching for facts
• To establish empirical regularities as a basis for new theories
– E.g. behavior in gift-exchange games → models of reciprocity
• To direct theorists’ effort in developing empirically relevant
theories
• In the presence of multiple equilibria experiments might help to
select the relevant ones
• Allow to go beyond the current state-of-the-art in theory
– E.g. continuous double auction
19
Testing institutions and environments
• Compare environments within the same institution
– How robust are the results across different environments?
– Objective: To stress the theory with extreme environmental
conditions under which an institution’s established properties may
begin to break down
• E.g. in common value auctions the Nash model performs better
when there are 3-4 bidders than 6-7 bidders
20
Testing institutions and environments
• Compare institutions within the same environment
– To learn about efficiency properties of institutions it is not
necessary to have a full theory that explains and predicts behavior
• Experiments allow for welfare comparisons of institutions
• Welfare measure: total money earnings of all subjects in the
experiment divided by the total possible earnings
• E.g. comparison of English, Dutch, first and second price
sealed bid auctions, comparison of posted (retail) pricing with
double auction trading
• E.g. various institutions within the VCM framework – allowing
for punishment/reward, taxes, competition, etc.
21
Policy advice & wind tunnel
experiments
• Evaluate policy proposals
– Effect of reduction of entry barriers on aggregate welfare
– How should airport slots be allocated?
– Do emission permits allow efficient pollution control?
– Remedies to decrease (or eliminate) insurance fraud
• Wind tunnel experiments
– Fine tune policies (e.g. UI bonus)
– Test-bed for institutional design
– What are the distributional and welfare consequences of incentive
compatible mechanisms?
22
Teaching experiments
• Better understanding of economic phenomena through
participation in experiments
– Markets, bargaining, social dilemma
• Own experience important
– Failing to behave as rational model predicts
– Anomalous behavior
• Starting to think about economic questions differently
• Many websites w/ computerized experiments: EconPort,
Veconlab, Market.Econ
III. Advantages of experiments
• Scientific progress relies crucially on the testing of theories
• Given the richness of happenstance of field data (data on
uncontrolled naturally occurring economic activity), why should
we bother creating our own data and performing experiments?
– Experiments are time consuming and costly… so what can we
gain?
24
What do we gain by doing experiments?
• Control over the economic environment and data generating
process
– Experimenter controls the conditions under which evidence is
generated
• Possibility of implementing truly exogenous ceteris paribus
changes
• Precise replicability of experimental evidence
– Those who question the result can replicate the experiment
– Historical data is not replicable.
25
In an experiment we can…
• Reproduce the structure of theoretical models
– E.g. in the area of International Trade, can’t typically find a “two
country world” in field data, but can be created in the laboratory
• Observe variables not observable in field data
– E.g. autarky price (no trade) data are typically not available in the
field
– E.g. insurance fraud unobservable by nature, but we can observe it
in the lab
• Control and manipulate variables
– E.g. we have the ability to double the number of demanders in a
market
26
IV. Limitations and objections
• Subject pool bias
– Students recruited because of convenience: quickly understand the
rules, low opportunity costs
– Non-student subject pools are used increasingly (Cooper et al.
1999, List 2003, Fehr & List 2003)
– There are interesting behavioral differences across subject pools.
However, the studies show that different subject pools do not
behave in fundamentally different ways.
IV. Limitations and objections
• Stake size
– Higher stake levels reduce behavioral variance (Camerer &
Hogarth 1999), i.e., subjects become more focused
– However, central behavioral tendencies are rarely overturned by
changes in stake levels.
– Exception Holt & Laury’s 2002 finding on risk aversion
– No or little impact in fairness games
IV. Limitations and objections
• Small sample
– Small sample comparing to field happenstance data.
– Answers to the objection:
• 1. It is always possible to increase the sample size by running
more experiments.
• 2. One can conduct experiments that are representative for
whole countries (Bellemare & Kroeger 2003, Knowles, Servátka
& Sullivan 2015)
• Internal validity
– Concerns the question whether the data gathered in an experiment
permits causal inferences
• Proper experiment control, sensible environment design, and
correct data analysis
IV. Limitations and objections
• External validity
– Possibility to generalize inferences from lab to the field.
– External validity holds only if the principle of induction holds –
behavioral regularities will persist in new situations as long as the
relevant underlying conditions remain substantially unchanged. We
are never sure that experimental results are replicable but this is
true for any empirical finding.
– Do experiments capture the essential conditions that prevail in
reality? If a skeptic argues that experiment is not externally valid
and not realistic, one should implement the neglected conditions.
– The simplicity of a model and an experiment is often a virtue
because it enhances the understanding of interaction of relevant
variables. Moreover, whether realism is important or not depends
on the purpose of the experiment.
– It is possible to add realism – e.g. real effort labor experiments
(Falk & Ichino 2006 and others)
IV. Limitations and objections
• Limits of experiments
– Control is never perfect
• Weather, lab conditions
• Self-selection – who takes part in experiments?
– Experiments comparing to theory
• Experiments are never general, just an example
– Experiments comparing to field studies
• Can all preferences be induced? For example time preference
(an experiment lasts 1-2 hours)
General remark
• Whether the conditions implemented in the laboratory are also
present in reality will probably always be subject to some
uncertainty
• Therefore, laboratory methods are not a substitute for
– The analysis of happenstance data
– For the conduct and the analysis of field experiments
– For survey data
• This calls for a combination of all these empirical methods
PhD/MRes opportunities
• Interested in doing a PhD/MRes in
Experimental/Behavioral Economics at MGSM in
Sydney?
• Cotutelle scholarships available
– Pay tuition in Australia, $27,000/year allowance, round-trip
airfare to Sydney, $10,000 research expenses.
• Contact me at maros.servatka@mgsm.edu.au
 Established in 1969, one of the first
business schools created in
Australia.
 1st tier Business School in Australia.
 Current students from 50 countries.
 Over 16,000 alumni, across 135
countries.
 AACSB accredited.
 Consistent high ranking
#1 in Australia*
#3 in Asia
# 28 in the World**
* Financial Times 2015 and 2016 Global MBA
rankings
** The Economist Which MBA? 2015
Some terminology
• Experiment: a collection of treatments testing related hypotheses of
interest
• Treatment: the collection of decisions under a specific set of conditions
and parameters
• Session: the sequence of decisions involving a given set of subjects on the
same day
• Between-subject design: when different subjects are assigned to two or
more treatments
• Within-subject design: when the same subjects are sequentially put
through two or more treatments
• Round/Period/Stage: the name assigned to a decision subjects make when
they make decisions repeatedly within a session.
Some terminology
• Treatment Cell: the set of sessions with identical treatment conditions.
• Example of Between-Subject Design:
Experiment
Treatment I Treatment II Treatment III
Session 1 Session 2 Session 3
Session 4 Session 5 Session 6
• Example of Within-Subject Design:
Experiment
Session 1: Treatment I, II, III
Session 2: Treatment III, II, I
Session 3: Treatment II, I, III
Session 4: Treatment I, III, II
Session 5: Treatment III, I, II
Session 6: Treatment II, III, I
Further readings
Metodology
• V. Smith, "Microeconomic Systems as an Experimental Science," American Economic Review, December
1982, pp. 923-955.
• V. Smith, "Experimental Methods in Economics," in J. Eatwell, et al. (eds.), The New Palgrave: A
Dictionary of Economics (New York: The Stockton Press, 1987).
• V. Smith, "Theory, Experiment, and Economics," Journal of Economic Perspectives, Winter 1989, pp 151-
169.
• J. Cox and M. Isaac, "Experimental Economics and Experimental Psychology: Ever the Twain Shall
Meet?", in A.J. MacFadyen and H.W. MacFadyen (eds.), Economic Psychology: Intersections in Theory
and Application (New York: North-Holland, 1986).
• V. Smith, "Rational Choice: The Contrast between Economics and Psychology," Journal of Political
Economy, 99, 1991, pp. 877-897.
• V. Smith, “Experimental Economics: Induced Value Theory, American Economic Review (P&P), 66, 274-
279, 1976
• G. Harrison and J. A. List, "Field Experiments," Journal of Economic Literature, XLII (December): 1013-
1059, 2004
• A. Falk and E. Fehr, “Why Labour Market Experiments,“ Labour Economics 10 (2003), 399-406.
• A. Falk and J. J. Heckman, “Lab Experiments Are a Major Source of Knowledge in the Social Sciences”
Science 23 October 2009: Vol. 326 (5952), 535 - 538.
• S. Levitt, and J. List, “What do Laboratory Experiments Tell Us About the Real World.” Journal of
Economic Perspectives 21 (2), 2007, 153-174.
• C. Camerer
Further readings
Textbook
• Friedman, D and S. Sunder, Experimental Methods – A Primer for Economists, Cambridge University
Press, 2004
Handbooks
• Davis, D. and C. Holt, Experimental Economics, Princeton University Press, 1993.
• Kagel, J. and A. Roth (eds.), The Handbook of Experimental Economics, Princeton University Press, 1995
• Plott, C. and V. Smith (eds.), Handbook of Experimental Results, New York: Elsevier Press, forthcoming.
• Bibliography
• The Y2K Bibliography of Experimental Economics and Social Science by Charles A. Holt:
http://people.virginia.edu/~cah2k/y2k.htm
• Field Experiment Bibliography by John List and Dean Karlan: http://www.fieldexperiments.com/