1/104
Comprehensive vocabulary flashcards covering randomized experiments, natural experiments, quasi-experimental designs, threats to internal and external validity, econometric models, and landmark empirical replications.
Quasi-experiment
A research design used to estimate a treatment effect based on a credible comparison when treatment is not randomly or as-if randomly assigned.
Natural experiment
An observational study where a process outside the researcher's control randomly or as-if randomly assigns units to conditions.
As-if random assignment
A condition in which treatment is not actively randomized by a researcher, but unit characteristics do not influence which group a unit is assigned to.
Comparison group
A group of untreated units used for evaluation in non-randomized studies, commonly used in place of 'control group' when assignment is not randomized.
Discrete treatment
A treatment condition where units receive one of a fixed number of conditions, either two (e.g., pill or placebo) or several distinct alternatives.
Continuous treatment
A treatment condition where units receive varying numerical quantities of an intervention, such as inches of rainfall, years of education, or months of prison sentence.
Dose-response relationship
The mathematical or empirical pattern describing how an outcome changes as the amount or intensity of a continuous treatment increases.
Exogenous treatment
A treatment determined by factors external and unrelated to other causes of the outcome, ensuring treated and untreated units remain comparable.
Endogenous treatment
A treatment determined by factors related to other causes of the outcome or the outcome itself, resulting in non-comparable treated and untreated units.
Selection bias (source of endogeneity)
A threat to causal inference occurring when individuals or units choose treatment conditions for reasons that also correlate with the outcome.
Reverse causality
A form of endogeneity where the outcome variable influences or causes the treatment status, such as a state passing a gun law because gun violence rose.
Confounder
An extraneous variable that simultaneously causes both the treatment and the outcome, creating a spurious statistical association.
Resume audit study (assignment mechanism)
A randomized field experiment where a researcher randomly assigns White- or Black-sounding names to resumes sent to employers to measure hiring discrimination.
Boston charter school lotteries
An institutional natural experiment where oversubscribed public charter schools admit students through random lottery drawings.
New Jersey and Pennsylvania minimum wage policy (1992)
A natural experiment setting where New Jersey raised its minimum wage in 1992 while neighboring Pennsylvania did not, allowing cross-border comparison.
Legal drinking age threshold (age 21)
A sharp legal assignment mechanism where legal access to alcohol begins precisely at age 21, generating an exogenous rule-based boundary.
Kindergarten school entry cutoff (September 1)
An administrative rule assigning children with birthdays before September 1 to start school at age 5, while those born after start at age 6.
Election Day rainfall assignment
A natural assignment mechanism where atypical precipitation on Election Day serves as an exogenous factor reducing voter turnout.
Denmark DNA database policy (May 2005)
A policy change starting in May 2005 where individuals charged with the equivalent of a felony in Denmark could be added to a national DNA database.
Institutional lotteries
Formal procedural lotteries run by organizations—such as the Oregon Medicaid lottery or Vietnam draft lottery—that randomly allocate conditions.
Institutional procedures (natural experiments)
Standard operating procedures of organizations—such as randomly assigning probationers to officers or residents to jury pools—that create as-if random assignment.
Anwar et al. (2012)
A natural experiment investigating the impact of jury race in criminal trials using court software that randomly summons and assigns eligible residents to cases.
Quasi-random (Anwar et al., 2012)
The term used by Anwar et al. because they relied on court software randomization and non-random case scheduling could potentially correlate pool composition with case traits.
Noncompliance in Anwar et al. (2012)
The empirical condition where a Black juror was seated in only 40% of trials with one or more Black pool members, making pool analysis like an ITT with 60% noncompliance.
Exogenous shock
An unanticipated event or process external to a causal model (e.g., natural disasters, policy shifts, or weather) that impacts an independent variable.
Excludability (of an exogenous shock)
The requirement that an exogenous shock affects the outcome strictly through the targeted independent variable, with nothing else changing simultaneously.
Kirk (2009)
A study exploiting Hurricane Katrina as an exogenous shock to examine the causal impact of residential relocation on recidivism among parolees.
Selection on observables
The identifying assumption in regression, matching, and weighting that treatment is as-if randomly assigned once all measured characteristics are accounted for.
Matching
A non-experimental method pairing treated and untreated units with identical or similar measured characteristics to balance observed covariates.
Coarsened exact matching (CEM)
A matching method that temporarily bins continuous covariates into discrete categories to find exact matches between treated and comparison units.
Propensity score matching
A technique that estimates the conditional probability of receiving treatment given observed covariates and matches units on this single predicted score.
Inverse probability of treatment weighting (IPTW)
A statistical method that estimates propensity scores and weights units in a regression model rather than matching them directly.
Within-person fixed effects
A longitudinal modeling strategy that removes all time-invariant differences between units, while still assuming no time-varying confounders and no feedback loops.
Cerda et al. (2012)
A research study cited in lecture titled 'Reducing violence by transforming neighborhoods.'
Interrupted time series (ITS)
A quasi-experimental design using longitudinal data on a single unit before and after an intervention, using the pre-intervention trend as the counterfactual.
Comparative interrupted time series (CITS)
An interrupted time series design that adds an untreated comparison unit to contrast projected trends and control for contemporaneous historical shocks.
Intercept change (in ITS)
The immediate, abrupt level shift in the outcome variable observed at the precise moment an intervention is introduced.
Slope change (in ITS)
The gradual alteration in the trajectory or rate of change of the outcome variable over time following an intervention.
Pennsylvania mandatory minimums study
A comparative ITS study showing that the striking down of mandatory minimums by the Pennsylvania Supreme Court had no statistically significant impact on violent crime compared to Ohio.
Difference-in-differences (DiD)
A quasi-experimental design that calculates treatment effects by comparing the over-time change in a treated group to the over-time change in an untreated comparison group.
Parallel trends assumption
The key identifying assumption in DiD that, in the absence of treatment, the treated and comparison groups would have followed identical outcome trajectories over time.
Average treatment effect on the treated (ATT)
The causal effect estimated by difference-in-differences, representing the mean intervention impact specifically on the units that received treatment.
First difference (in DiD)
The contrast of outcomes before versus after treatment within a group, which nets out time-invariant, stable characteristics of the units.
Second difference (in DiD)
The contrast between the treated group's change and the comparison group's change, which nets out secular time shocks common to all units.
Third difference (Triple Differences)
A design extension comparing a targeted outcome or subgroup to an untargeted one within the same units to remove unit-specific contemporaneous shocks.
DiD numerical estimate calculation
A lecture demonstration where treated change was +73.7 (67.4 to 141.1) and comparison change was +23.6 (27.4 to 51.0), yielding an effect of 73.7−23.6=50.1.
Synthetic control method (SCM)
A data-driven quasi-experimental method that creates an optimal comparison group by taking a weighted average of untreated units to match a treated unit's pre-treatment trend.
Synthetic control weights
Non-negative weights assigned to comparison units chosen to minimize the discrepancy in pre-intervention outcomes and covariates between the synthetic and treated unit.
Proposition 99 (California)
A 1988 California tobacco control initiative evaluated via SCM against a synthetic control constructed from states like Utah (0.334), Nevada (0.234), and Montana (0.199).
In-time placebo test
A falsification check in SCM or quasi-experiments that artificially re-assigns the intervention date to a pre-treatment period to ensure no false effect appears.
In-space placebo test
A falsification test that iteratively applies the synthetic control method to untreated comparison units to see if the treated unit's estimated effect is an anomaly.
Regression discontinuity (RD)
A quasi-experimental design estimating treatment effects by comparing units located immediately above and below an arbitrary administrative cutoff on a continuous running variable.
Running variable
The continuous rating, score, or covariate (X) used by a formal rule to assign treatment status based on whether it falls above or below a cutoff.
Sharp regression discontinuity
An RD design where treatment status is a strictly deterministic function of the running variable, jumping from probability 0 to 1 precisely at the threshold.
Fuzzy regression discontinuity
An RD design where crossing the cutoff induces a sharp discontinuity in the probability of treatment rather than deterministic assignment, functioning as an instrumental variable.
Bunching (in RD)
An abnormal concentration of observations clustered just above or below a threshold, signaling that individuals or staff manipulated the running variable.
Latin honors study (RD)
An RD application examining whether employers reward Latin honors (e.g., cum laude) by comparing earnings of graduates immediately on either side of GPA cutoffs.
Tahamont (2019)
A regression discontinuity study evaluating the causal impact of prison security classification levels on serious rules violations in California prisons.
Instrumental variables (IV)
A causal identification method using an exogenous variable (Z) to isolate variation in an endogenous treatment (D) that is unrelated to outcome confounders.
Instrument (Z)
A variable that is correlated with the causal variable of interest (D) but has no direct relationship with the outcome (Y) other than through D.
Exclusion restriction (in IV)
The fundamental untestable assumption that an instrument (Z) affects the dependent variable (Y) solely through its impact on the treatment variable (D).
Independence assumption (in IV)
The instrumental variable requirement that the instrument (Z) is as-if randomly assigned and completely independent of all unobserved confounders (U).
Same-sex sibling instrument
An instrument exploiting parental preference for mixed-sex children: parents whose first two children share the same sex are more likely to have a third child.
Weak instrument
An instrument that has a low statistical correlation with the endogenous treatment variable (D), leading to biased IV estimates and large standard errors.
Monotonicity assumption
The IV condition that an instrument shifts units in only one direction relative to treatment—assuming there are compliers, always-takers, and never-takers, but no defiers.
Compliers
The subpopulation of units who take the treatment if encouraged or assigned to do so by the instrument, and who do not take it if not encouraged or assigned.
Local average treatment effect (LATE)
The causal effect estimated by an instrumental variable, representing the average treatment effect specifically for the subgroup of compliers.
Intention-to-treat (ITT) estimate
The estimated causal impact of being offered or assigned to a treatment condition across the entire randomized sample, regardless of actual compliance.
Angrist (2006)
A methodological reference demonstrating how random treatment assignment (Z) serves as an instrument for treatment received (D) under noncompliance.
Loeffler (2013)
An empirical study investigating the causal impact of imprisonment on recidivism and employment using random judicial case assignment as an instrumental variable.
Placebo tests (probing quasi-experiments)
Diagnostic checks assessing outcomes, dates, or cutoffs that could not plausibly be impacted by the intervention to detect hidden confounding.
Sensitivity analyses (in quasi-experiments)
Statistical assessments calculating how strong an unmeasured confounder would have to be to completely explain away an observed treatment effect.
Angrist & Pischke (2009)
Authors of 'Mostly Harmless Econometrics,' a textbook referenced for learning empirical causal inference and quasi-experimental techniques.
Cunningham (2021)
Author of the textbook 'Causal Inference: The Mixtape,' an introductory econometrics text covering quasi-experimental methods.
Random assignment (experimental design)
The experimental procedure that ensures baseline equivalence across treatment conditions on all observed and unobserved pre-treatment confounders by design.
Sample size (experimental ingredient)
A critical design element where larger numbers of units shrink chance pre-treatment baseline imbalances and minimize uncertainty around the estimated ATE.
Treatment effect heterogeneity
The variation in the direction or magnitude of an intervention's effect across different demographic subgroups, individuals, or environmental settings.
Unit compliance
An experimental condition where study participants actually take and adhere to the exact treatment condition to which they were assigned.
Differential attrition
An internal validity threat where participant dropout rates or the characteristics of dropouts differ systematically between treatment and control groups.
Excludability (experimental validity)
The requirement that assignment to a treatment group influences the outcome solely through receipt of the assigned treatment, not through ancillary factors.
Hawthorne effect
An excludability threat where participants alter their behavior simply because they are aware they are being actively studied or observed.
John Henry effect
An excludability violation where control group members exert compensatory effort to compete with the treated group after learning they were denied treatment.
Interference (or contamination)
An experimental breakdown occurring when units assigned to one experimental condition interact with or receive components of another condition.
Marques et al. (1994)
A study where closer parole supervision of treated sex offenders caused higher detection of reoffending rather than true higher recidivism, violating excludability.
Stable Unit Treatment Value Assumption (SUTVA)
The assumption that one unit's treatment assignment does not affect the potential outcomes of other units (no interference) and that treatment dosage is identical.
Compromised randomization
An internal validity threat where experimental assignments are subverted, altered, or bypassed by implementers to determine which subjects receive treatment.
Implementation failure
A breakdown where an intervention is delivered poorly or with inadequate dosage, weakening the treatment without necessarily creating systematic group differences.
Time-varying confounders (in experiments)
Events or external environmental changes occurring during an experiment that differentially affect one experimental condition more than another.
Treatment carryover effects
A validity threat in within-subject designs where exposure to an earlier treatment condition alters or contaminates a subject's response to later treatments.
Order effects
A threat in within-subject experiments where the sequence in which conditions are administered influences the observed outcome.
Floor effect
A measurement limitation where an outcome is so rare or measurement threshold so high that scores cluster at the minimum, obscuring treatment effects.
Ceiling effect
A measurement problem where scores cluster at the maximum possible value, preventing researchers from detecting positive treatment differences.
Sampson (2010)
A study highlighting that nonprobability samples paired with treatment effect heterogeneity cause sample ATEs to diverge from population ATEs.
Ethical dilemma of experimental withholding
The ethical tension between denying a promising treatment to a control group versus the risk of administering untested, potentially harmful interventions.
Staggered approach (experimental rollout)
A deployment strategy for treatments that cannot be delivered simultaneously, where conditions are phased in across randomized cohorts over time.
Excess demand (experimental condition)
A real-world setting where demand for an intervention exceeds capacity, making lotteries or randomized waitlists ethically and practically viable.
Replication failure (in program scaling)
The failure of interventions shown to be effective in single experiments to reproduce benefits when implemented across diverse sites or expanded to larger scales.
Bundled treatments
Multi-component interventions evaluated as a composite package, leaving uncertainty about which specific component drove the observed effect.
Hawaii's HOPE Probation program
A probation program emphasizing swift, certain, and proportionate sanctions; an initial randomized trial found strong positive effects, but multisite replications failed to replicate them.
Judge Alm and Honolulu HOPE implementation
The original judge and setting whose exceptional leadership and master's-level probation staff constituted part of the unique treatment condition that proved difficult to replicate.