ECO331 Behavioural & Experimental Economics - Syllabus Topic 2: Experiment Design
University of Toronto Department of Economics - ECO331: Behavioural & Experimental Economics
Introduction to Syllabus Topic 2: Experiment Design
Class is divided into two distinct sections:
First Half: Overview of Experimental Methods
Second Half: Assessing Internal and External Validity
1. Overview of Experiment Methods
The overview of experimental methods will be brief.
Notable Supplemental Readings:
Roth (1995): Provides a high-level view on the role of experimental economics in the profession.
Friedman and Sunder (1994): Classic reference for the fundamental aspects of experiments (not required due to its length).
Required Reading: Gazzale, Jacobson, and Linardi (2018): Offers a shorter, updated perspective including online experiments from Amazon Mechanical Turk.
Recommendation: Engaging with Gazzale et al. is advised as the first hour focuses on highlights from this reading.
2. Assessing Internal & External Validity
Objectives for Second Hour:
Develop skills to assess internal and external validity using Gneezy, Niederle, and Rustichini (2003) (GNR).
Students should read GNR closely and come prepared with thoughts regarding its internal and external validity.
2.1 The Theory
Effort Selection in Piece Rate:
Effort, including cognitive effort, is variably costly, with output being concave in relation to effort.
The optimal effort level equates the marginal cost of effort to the marginal benefit (represented as change in tasks completed multiplied by piece rate).
Effort Selection in a Tournament:
More complex than piece rate.
Assuming risk neutrality:
Optimal effort level equates the marginal cost of additional effort to the marginal benefit (depicted as change in likelihood of winning the tournament multiplied by tournament prize).
Risk Aversion Analogy:
A tournament can be likened to a lottery, with effort seen as the purchase of lottery tickets.
With constant expected value, increased risk aversion leads to lower willingness to pay for lottery tickets, hence reduced effort.
Homo economicus would thus select effort in accordance with these principles in experiments.
2.2 Internal Validity
Internal validity assesses whether the authors’ preferred explanation of results (causal relationship) is the most plausible.
Review Process for Internal Validity:
Describe the economic environment generally.
Example from GNR: Individuals pay an upfront cost (effort) for uncertain outcomes.
Create List 1: Reasons for different actions within the economic environment.
Analyze why some individuals exert high effort (buy many lottery tickets) while others exert low/no effort (few or no tickets).
Closely examine the experiment design.
Think from the perspective of an experiment subject about additional influences on choice.
Add these influences to List 1.
Ensure the authors’ preferred explanation is included in List 1.
Create List 2: Identify incidental differences between treatment conditions.
Example: A treatment condition includes a 5-minute video versus a baseline direct transition to the second choice.
Analyze each item in List 1.
Identify how the experiment controls for each explanation and evaluate the strength of those controls.
Assess each item in List 2.
Evaluate how plausible it is that incidental differences would influence results similarly to the treatment.
Fewer “not-well-controlled-for” items in List 1 improves internal validity; fewer plausible items in List 2 also improves it.
Note on robustness of internal validity critiques:
Recognize that alternative explanations are inherent in empirical research.
The ability to propose alternative explanations does not diminish the strength of causal claims unless the alternative's plausibility and size are significant.
2.3 External Validity
External validity evaluates the applicability of experiment results to real-world contexts.
Key questions for assessing external validity:
Identify real-world environments for which the study's results can be generalized.
Evaluate the importance of those environments.
For instance, one might find GNR’s results relevant exclusively to women self-selecting into male-dominated professions.
Approach 1 to External Validity:
Describe features of the economic environment broadly.
Example: Rather than noting specific group sizes, describe them as “relatively small groups.”
Analyze subject pool characteristics similarly.
Itemize environmental features and assess:
Plausibility of noted characteristics affecting outcomes in line with treatment.
Determine the strength of characteristic effects relative to treatment effects.
Can the evidence indicate that environmental characteristics are either weaker or stronger than treatment effects?
Create an “external-validity map” using the features identified to distinguish reliable extrapolations from less sound applications.
Evaluate economic significance of the identified environments.
Approach 2 to External Validity:
Conceptualize a scenario where treatment effects are unlikely to hold.
Identify minimum changes needed to likely disrupt treatment results.
Retain perspective that external validity critiques arise in all empirical research, assessing how plausible or large these characteristic effects are.
References
Friedman, Daniel, and Shyam Sunder. "Experimental Methods: A Primer for Economists." Cambridge University Press, 1994.
Gazzale, Robert S., Sarah Jacobson, and Sera Linardi. "Experiment Nuts and Bolts." July 2018. Unpublished manuscript.
Gneezy, Uri, Muriel Niederle, and Aldo Rustichini. "Performance in Competitive Environments: Gender Differences." Quarterly Journal of Economics, August 2003, 118(3), 1049–1074.
Roth, Alvin E. "Introduction to Experimental Economics." In The Handbook of Experimental Economics, edited by John H. Kagel and Alvin E. Roth. Princeton, NJ: Princeton University Press, 1995. pp. 1–23.