1/115
A comprehensive set of 100 vocabulary flashcards covering experimental methods, causal inference, assignment mechanisms, validity threats, statistical estimation, and real-world social science field experiments.
Causality (Three Requirements)
The three criteria required to establish a causal relationship: the cause and effect covary, the cause occurs before the effect in time, and all alternative explanations for the relationship can be ruled out.
Covariation
A condition for causality in which the proposed cause and effect change together—when one variable changes, so does the other.
Temporal Precedence
The requirement in causal inference that the independent variable (cause) must occur prior in time to the dependent variable (effect).
Ruling Out Alternative Explanations
The requirement in causal inference to eliminate plausible rival hypotheses, third variables, or confounders that could account for an observed association.
Longitudinal Study (re: Causality)
A study that follows subjects over time, noted as particularly helpful for identifying covariates and establishing the proper temporal order of cause and effect.
Internal Validity
The extent to which a study design can rule out alternative explanations for an observed association between variables.
Counterfactual World
A hypothetical, identical parallel world where the proposed cause does not change, used as the benchmark to determine what would have happened to the outcome in the absence of the treatment.
Approximating the Counterfactual
The process by which experimental researchers use a control group combined with random assignment to simulate the unobservable counterfactual world.
Fundamental Problem of Causal Inference
The methodological barrier that a researcher cannot simultaneously observe the same exact unit (e.g., a family) receiving a treatment and not receiving that treatment at the exact same time.
Experiment
An empirical investigation in which the independent variable is under the direct control of the investigator.
Randomized Controlled Trial (RCT)
An experimental investigation that uses random assignment to allocate units to different values or conditions of the independent variable.
Observational Study
A study design where the allocation of individuals to different values of the independent variable is entirely outside of the investigator's control.
Laboratory Experiment
An experiment carried out in an artificial, highly controlled setting created by the researcher.
Field Experiment
An experiment carried out in a natural, real-world setting where units typically live or work.
Simulated Experiment
A computational research approach where researchers construct a computer model of a system and systematically test conditions without randomly assigning human subjects.
Random Assignment
The application of a random chance procedure to assign units to different values, conditions, or treatments of the independent variable.
Treatment Group
The set of units in an experiment that receives a specific intervention, active condition, or dosage of the independent variable.
Control Group
The group of units in an experiment that serves as the baseline standard of comparison to assess the impact of treatments, representing the best estimate of the counterfactual.
Placebo Group
A control group that receives an inert substance or inactive condition to benchmark against active treatment groups.
Known, Nonzero Probability
The property of random assignment guaranteeing that every unit in the study has an identified probability greater than zero of receiving each possible value of the independent variable.
Equivalence in Expectation
The theoretical principle that random assignment distributes all unit differences—both observed and unobserved—equally across treatment conditions on average.
Pre-treatment Factors
Baseline unit characteristics and background variables that exist prior to the administration of an experimental treatment, which random assignment balances in expectation.
Chance Differences
Random imbalances between experimental groups that occur by bad luck in any single experiment, which become smaller and less probable as the sample size increases.
Self-Selection
A non-experimental process where individuals choose for themselves whether to participate in or receive an intervention, introducing confounding variables.
Confounder
A third variable that is associated with both the independent variable (e.g., enrollment) and the dependent variable (e.g., job placement), distorting the apparent causal relationship.
Motivation (as a Confounder)
An unobserved characteristic in the job training example that simultaneously increases an individual's likelihood of enrolling in training and their likelihood of securing employment.
Selection Bias
A systematic difference between groups arising because individuals who select into a treatment differ fundamentally from those who do not, independent of the treatment effect.
Selection Bias Gap (Job Training Example)
The 45 percentage point difference (70% vs. 25%) observed under self-selection, where only 15 points are caused by the program and 30 points stem from motivation bias.
Baseline Equivalence
A state achieved under random assignment where both treatment and control groups share identical initial expected outcomes (e.g., both starting at 40% job attainment without the program).
Average Treatment Effect (ATE)
The average change in the dependent variable caused by the independent variable (treatment) across all units in the experimental sample.
Difference-in-Means ATE Estimator
The standard calculation of the ATE obtained by subtracting the average outcome of the control group from the average outcome of the treatment group.
High Dosage vs. Placebo ATE (Blood Pressure Example)
The calculated average treatment effect of 82−126.5=−44.5, indicating the high dose pill lowered blood pressure by 44.5 points on average compared to placebo.
Low Dosage vs. Placebo ATE (Blood Pressure Example)
The calculated average treatment effect of 98−126.5=−28.5, indicating the low dose pill lowered blood pressure by 28.5 points on average compared to placebo.
High Dosage vs. Low Dosage ATE (Blood Pressure Example)
The calculated average treatment effect of 82−98=−16, demonstrating that randomized designs permit direct causal comparisons between different active treatment conditions.
Unbiased Estimate
An experimental estimate produced when the treatment is the only systematic difference between the experimental groups, centering on the true value across repeated trials.
Bias
A systematic difference between groups, other than the treatment, that pushes the treatment effect estimate in the same direction every time an experiment is repeated.
Imprecision (in ATE Estimates)
Estimation error caused by random chance differences scattering single experiment results around the true bullseye, which shrinks as the sample size increases.
Confidence Interval (ATE)
A statistical range around the estimated treatment effect that quantifies the degree of imprecision and sampling uncertainty.
Nonprobability Sampling (Sampson 2010)
A sampling approach (e.g., relying on volunteers, court defendants, or prisoners) that likely biases the ATE for the broader population when treatment effects vary across units.
Between-Unit Study
A study design that estimates treatment effects by comparing outcomes across different groups of units—those assigned to receive the treatment versus those assigned to control.
Within-Unit Study
A study design where each individual unit receives multiple conditions over time, eliminating the confounding impact of stable unit differences because each unit serves as its own control.
Lasting Treatment Effects
Outcomes that permanently alter participants (e.g., job training skills, moving neighborhoods, or therapy) and cannot be undone, requiring a between-unit design.
Washout Period
A designated period of time left between successive conditions in a within-unit design to ensure the effects of an earlier treatment completely dissipate before the next begins.
Carryover Effect
A primary risk in within-unit studies where exposure to an earlier treatment condition alters or confounds the participant's response to subsequent conditions.
Participant Fatigue
A threat in within-unit designs where subjects become tired, bored, or less engaged as they undergo multiple consecutive experimental conditions.
Condition Order Randomization
A methodological solution in within-unit studies where the sequential order of experimental conditions is randomized across subjects to mitigate order effects.
Assignment Mechanism
The specific procedural rule or process that determines which experimental units receive which values of the independent variable.
Bernoulli Trial
A randomized assignment mechanism where each unit is assigned independently with a fixed probability (often 0.5), like a coin flip, resulting in variable group sizes.
Dynamic Arrival Setting
A research context where units arrive over time rather than being available all at once in a pre-existing sampling frame, frequently utilizing Bernoulli trials.
Completely Randomized Experiment
An experimental assignment mechanism where a fixed total number of units from an available pool is assigned to receive treatment or control with equal probability.
Propensity Score
The probability that a given unit is assigned to the treatment condition, which is uniform (Ntreatment/N) for all units in a completely randomized experiment.
Stratified / Block Randomized Experiment
An experimental design where units are first partitioned into homogeneous subgroups (strata/blocks) based on covariates, followed by a completely randomized experiment within each block.
Strata / Blocks
Subsets of experimental units grouped together prior to randomization based on shared baseline characteristics (e.g., biological sex, age) that predict the outcome.
Blocking Variable
A pre-treatment covariate that strongly predicts the outcome measure and is used to partition units into blocks to eliminate chance imbalances.
Paired Randomized Experiment
An assignment design where units are matched into blocks of two based on predictive covariates, and exactly one unit in each pair is randomly assigned to treatment.
Small Sample Imbalance (in Bernoulli Trials)
The substantial probability in small samples of obtaining severe group size splits by chance (e.g., flipping 10 coins yields an 8/2 split approximately 9% of the time and a 7/3 split approximately 23% of the time).
Pairwise Attrition Cost
A vulnerability of paired randomized designs where the dropout of one unit may force the researcher to discard its matched partner, substantially reducing sample size.
Clustering
The practice of aggregating individual units into larger social or geographic groups (e.g., classrooms, lines, neighborhoods) and assigning treatment at the group level.
Cluster-Level Analysis
The statistical requirement in cluster-randomized trials to analyze data at the cluster level, recognizing that the total number of clusters dictates precision more than unit count.
Precision Loss in Clustering
The reduction in statistical power and precision associated with cluster-randomized designs compared to individual-level random assignment due to correlated unit behaviors.
Multi-Level Experiment
An experimental design in which random assignment occurs across multiple hierarchical tiers, such as assigning lines to procedures and then assigning individual passengers within those lines.
Hierarchical Interaction
The statistical interaction estimated in multi-level experiments examining whether the effect of an individual-level treatment depends on the cluster-level treatment condition.
Treatment Effect Heterogeneity
Variations in how different experimental units or demographic subgroups respond to the identical treatment intervention.
Hidden Treatment Effects
A situation where an aggregate Average Treatment Effect (ATE) of zero obscures substantial positive effects for some participants and negative effects for others.
Subgroup Analysis
The analytical practice of estimating the ATE separately within distinct categories of participants (e.g., comparing males vs. females).
Observational Nature of Subgroups
The methodological limitation that subgroup membership is not randomly assigned, meaning differing subgroup ATEs could be driven by underlying confounders rather than the subgroup trait.
Factorial Design
An experimental design that randomly assigns units to every possible combination of two or more treatments to test separate main effects and interaction effects simultaneously.
3×3 Factorial Experiment (Slide Example)
A specific design testing medication dose (high dosage, low dosage, placebo) crossed with exercise intensity (high intensity, low intensity, no exercise).
Causal Interaction
A finding derived from a factorial design demonstrating that the causal impact of one randomly assigned variable depends upon the assigned level of another variable.
Waitlist / Stepped-Wedge Design
A design where all units begin in the control condition and cross over into the treatment group at staggered, randomly assigned calendar intervals.
Secular Trends (in Stepped-Wedge Designs)
Outside real-world events or time-related historical changes that coincide with treatment roll-out and can easily be mistaken for treatment effects.
Adequate Sample Size
A necessary ingredient for experiments ensuring that chance differences average out and statistical uncertainty surrounding the ATE is sufficiently minimized.
Rare Outcomes Challenge
A methodological issue where an outcome occurs very infrequently, requiring a massive sample size to detect statistically meaningful differences between groups.
Compliance (as an Ingredient)
The requirement that experimental participants actually receive and adhere to the specific treatment condition to which they were randomly assigned.
Interference / Contamination
A breakdown in experimental isolation occurring when units in one treatment group experience or receive aspects of another group's assigned condition.
Differential Attrition
The systematic or unequal dropout of participants across treatment arms that undermines baseline equivalence and biases the ATE estimate.
Excludability
The foundational assumption that assignment to a treatment group affects the dependent variable solely through the intended active mechanism of the assigned treatment.
Hawthorne Effect
An excludability violation where subjects alter their natural behavior simply because they know they are being observed or evaluated by researchers.
John Henry Effect
An excludability violation where control group members alter their behavior to work harder or overcome their perceived disadvantage relative to the treatment group.
Staff Expectancy Violation
An excludability violation occurring when researchers or program staff inadvertently treat subjects differently because they know their assigned group conditions.
Time-Varying Confounder
An extraneous historical event occurring during the experimental window that impacts one experimental condition more than another.
Measurement Consistency
The practice of using valid, reliable, and standardized outcome instruments administered in the identical manner across all experimental arms.
Noncompliance
A condition where units assigned to a specific value of the independent variable end up receiving an alternative condition instead.
One-Sided Noncompliance
A scenario where noncompliance is restricted to the treatment arm, such that assigned treatment units can refuse or quit, but control units cannot access the treatment.
Two-Sided Noncompliance
A scenario where units assigned to the control group manage to obtain the treatment and units assigned to the treatment group fail to receive it.
Intention-to-Treat (ITT) / Treatment as Assigned
An analytical approach that estimates the treatment effect by analyzing subjects strictly within the groups to which they were originally randomized, regardless of whether they complied.
Treatment as Delivered
An analysis comparing outcomes based on the actual condition subjects received rather than what they were assigned, which destroys randomized equivalence and introduces confounding.
Dilution of Effect (in ITT)
The downward attenuation of the estimated treatment effect under ITT analysis caused by including noncompliant participants who never actually received the treatment.
Policy Relevance of ITT
The practical value of an ITT estimate in reflecting the real-world net impact of offering an intervention to a general population where adherence is not universal.
Stable Unit Treatment Value Assumption (SUTVA)
The core assumption that each unit's outcome depends strictly on its own treatment assignment, and that all units within a treatment group receive identical treatment.
SUTVA Part 1 (No Interference)
The specific requirement that one unit's experimental outcome is completely unaffected by the treatment assignments or actions of any other units.
SUTVA Part 2 (Uniform Treatment Delivery)
The specific requirement that every unit within a given treatment group receives the exact same dosage and standardized implementation of the intervention.
Spillover Bias Toward Zero
The tendency of SUTVA violations to underestimate treatment efficacy because program benefits leaking to control units make both groups appear more similar.
Displacement of Harms
A SUTVA violation where an intervention diverts negative behaviors (e.g., crime or accidents) into adjacent control zones, artificially exaggerating the perceived treatment effect.
Buffer Areas
Geographic safety zones deliberately left between treatment and control territories to prevent physical spillover and protect SUTVA.
Random Attrition
Participant dropout that occurs purely by chance and at equal rates across groups, causing a loss of sample size and precision without inducing systematic bias.
Oversampling (for Attrition)
Recruiting and randomizing more participants than mathematically required based on expected dropout rates to safeguard post-attrition statistical power.
Bounding / Trimming (Attrition Strategy)
Analytic techniques that impute extreme potential values or trim sample distributions based on baseline covariates to calculate the upper and lower bounds of the true ATE.
Administrative Records Tracking
Using official, non-reactive records (e.g., arrest logs, tax records) to measure outcomes directly without relying on participant contact, mitigating attrition.
Implementation Evaluation
An assessment conducted alongside an experiment to confirm that treatments were delivered with fidelity across sites/staff and to identify time-varying confounders.