Study Notes on Causal Inference and Experimental Design

Causal Inference

Pre & Post-Test Experimental Design

• Focuses on understanding the true effects of treatments in research settings.

Field Experiments Over Time

• Field experiments may face various external factors impacting outcomes.

• Example: Testing two types of promotions that require a time span of 1–4 weeks:
    - During this time, external factors can shift consumer purchase decisions.

• Importance of classifying these extraneous factors in order to evaluate effectiveness accurately.

Threats to Experiments

Extraneous Variables

• Definition: Factors that can confound the treatment effect over time, potentially skewing results.

Types of Extraneous Variables

1. History (H)
      - Definition: Specific events outside of the experimental control occurring during the experiment.
      - Examples: Changes in economic conditions, natural disasters occurring concurrent with the study timeline.

2. Maturation (MA)
      - Definition: Changes in test units resulting from growth or temporal effects over time.
      - Examples: Aging, mood changes, or other psychological shifts affecting participants during the study.

Comparison: History vs. Maturation

• History
    - Involves changes in the external environment that may affect all groups.

• Maturation
    - Involves changes within the test units themselves.

Testing Effects

• Related to the act of measuring participants before and after treatment.
    - Participants exposed to products prior to the test may create a consistent opinion affecting post-experiment responses.

• Main Testing Effect (MT): Prior observations influence later observations in both treatment and control groups.

• Interaction Testing Effect (IT): This effect only occurs in the treatment group as a consequence of the treatment itself.

More Extraneous Variables

1. Instrumentation (I)
      - Definition: Changes in the measurement tools or observers during the experiment can skew results.

2. Mortality (MO)
      - Definition: Loss of participants while the experiment is ongoing.
      - Participants dropping out can introduce bias, particularly if the dropout is related to the treatment.

3. Selection Bias (SB)
      - Definition: Concerns arise when non-random assignment occurs for test units across treatment and control conditions.
      - Groups must not differ significantly in traits or motivations; otherwise, results may be invalid.
      - Importance of randomization in eliminating selection bias.

Extraneous Variables Summary

• H: History - External events during the experiment.

• MA: Maturation - Changes in test units over time.

• MT: Main Testing - A prior observation affecting a later one (both groups).

• IT: Interaction Testing - Testing effect only present in the treatment group.

• I: Instrumentation - Changes in measurement tools during the study.

• MO: Mortality - Loss of test units during the experiment.

• SB: Selection Bias - Non-random assignment to groups.

True Experimental Design

Pre & Posttest Control Group Design

• Pretest-Posttest Design: Structure for measuring treatment effects in controlled environments.
     - Notation:
     - Treatment Group (TG): $R ext{ } O_1 ext{ } X ext{ } O_2$
     - Control Group (CG): $R ext{ } O_3 ext{ } O_4$
     - Key Elements:
       - R = Randomization
       - O = Observation
       - X = Treatment
     - Random assignment of test units to either treatment or control conditions helps eliminate selection bias.
     - Measuring Treatment Effect:
       $(O_2 - O_1) - (O_4 - O_3)$.

Difference-in-Differences (DiD)

• Treatment effect measured as:
  $(O_2 - O_1) - (O_4 - O_3)$

• Breakdown of Results:
    - $O_2 - O_1 = TE + H + MA + MT + IT + I + MO$
    - $O_4 - O_3 = H + MA + MT + I + MO$

• Note: Interaction Testing effect (IT) not fully eliminated, leading to residual contamination in the estimate.

The Remaining Problem: Interaction Testing (IT)

• Variables Potentially Eliminated by DiD:
    - History (H): Yes
    - Maturation (MA): Yes
    - Main Testing (MT): Yes
    - Instrumentation (I): Yes
    - Mortality (MO): Yes
    - Selection Bias (SB): Yes (via randomization)
    - Interaction Testing (IT): No (still present).

Solution: The Placebo

• Placebo Use: Assign control group a placebo treatment.
    - Ensures both groups are exposed to similar testing conditions, allowing the interaction testing effect (IT) to cancel out during analysis.
    - Allows for accurate calculation of treatment effect:
  $(O_2 - O_1) - (O_4 - O_3) = TE$.

In-Class Exercise

• Hands-on Analysis:
     - Experiment with a pre-post dataset (experiment_(pre post).xlsx).
     - Task:
       - Apply the Pretest-Posttest Control Group Design.
       - Assess the randomization balance using pre-treatment characteristics.
       - Estimate treatment effect utilizing the Difference-in-Differences methodology.

Comparison of Experimental Designs

Post-Test Only vs. Pretest-Posttest Design

• Post-Test Only Design:
    - Involves analyzing one period of data (post-treatment results).
      1. Evaluate significant differences in characteristics between control group (CG) and treatment group (TG).
      2. If not significant, assess average outcome differences.
      3. Significant results suggest treatment effectiveness; if not significant, treatment is ineffective.

• Pretest-Posttest Design:
    - Incorporates two time periods (pre- and post-treatment).
      1. Evaluate differences in pre-treatment characteristics between CG and TG.
      2. If not significant, calculate the difference-in-differences (post–pre).
      3. Significant outcomes validate treatment effectiveness. If not significant, treatment is ineffective.

Key Assumption

SUTVA

• SUTVA: Stable Unit Treatment Value Assumption

• Fundamental requirement that outcomes of participants should be independent of others.

• Potential Violation Scenario:
    - Example: Vaccination randomized controlled trial (RCT).
      - If vaccinated participants reduce disease transmission, control participants benefit from spillover effects, violating SUTVA’s independence assumption.