causal interference 2

Causal Inference

Pre & Post-Test Experimental Design

Field Experiments Over Time

• Overview: Field experiments can be affected by various factors when treatment takes time to take effect.

• Example: Testing the effectiveness of two types of promotions.
    - A promotion may require a duration of 1–4 weeks (or longer) to show results.
    - During this time, other factors can influence consumer purchase decisions.

• Classification Needed: It's crucial to classify these extraneous factors that can confound results.

Threats to Experiments

Extraneous Variables

• Definition: Variables that can confound the treatment effect over time.

• Categorized as sources of bias in experimental design.

Specific Extraneous Variables

History (H)

• Definition: Events that occur outside the experiment during its duration that may impact outcomes.

• Examples: Economic shifts, natural disasters, or any significant external events coinciding with the experiment.

Maturation (MA)

• Definition: Changes that occur within the test subjects over time, unrelated to the treatment itself.

• Examples: Changes in age, emotional state, mood shifts, etc.

Comparison of History and Maturation

• History: Relates to external changes occurring during the experiment.

• Maturation: Pertains to internal changes within the subjects over time.

Testing Effects

Overview

• Definition: Effects caused by the experimentation process itself, particularly from pre and post treatment measurements.

• Importance of understanding these effects for accurate results.

Types of Testing Effects

1. Main Testing Effect (MT)
      - Description: Affects both control and treatment groups wherein prior observations influence later observations.

2. Interaction Testing Effect (IT)
      - Description: Affects only treatment group respondents; these individuals’ observations are influenced solely due to their exposure to treatment.

More Extraneous Variables

Instrumentation (I)

• Definition: Refers to changes in measurement tools, observers, or scores during the experiment.

Mortality (MO)

• Definition: Loss of test units or participants over the course of the experiment.
    - Concern: Participant dropout can introduce bias if it correlates with the treatment condition.

Selection Bias (SB)

• Definition: Occurs when there is a non-random assignment of test units to treatment conditions.
    - Consequences: If treatment and control groups differ in characteristics or motivations, all results become invalid.

• Importance of Randomization: Critical to eliminate selection bias and ensure fairness in group comparisons.

Summary of Extraneous Variables

Key Variables

• H: History - External events during experiment.

• MA: Maturation - Changes in test units over time.

• MT: Main Testing - Prior observation affects subsequent observation (both groups).

• IT: Interaction Testing - Testing effects limited to treatment group.

• I: Instrumentation - Changes in measurement tools.

• MO: Mortality - Loss of test units during the experiment.

• SB: Selection Bias - Non-random group assignment.

True Experimental Design

Pre & Posttest Control Group Design

Pretest-Posttest Design

• Notation:
    - Treatment Group (TG): $R <br>ightarrow O₁ <br>ightarrow X <br>ightarrow O₂$
    - Control Group (CG): $R <br>ightarrow O₃ <br>ightarrow O₄$
    - Where: $R$ = Randomization, $O$ = Observation, $X$ = Treatment.

• Process: Random assignments for treatment/control ensure selection bias is eliminated.

• Treatment Effect Measurement: $(O₂ - O₁) - (O₄ - O₃)$ summarizes treatment impact.

Difference-in-Differences (DiD)

Treatment Effect Measurement

• Formula: $(O₂ - O₁) - (O₄ - O₃)$

• Breakdown:
    - $O₂ - O₁ = TE + H + MA + MT + IT + I + MO$
    - $O₄ - O₃ = H + MA + MT + I + MO$

• Conclusion: $(O₂ - O₁) - (O₄ - O₃) = TE + IT$, indicating IT remains uncontrolled.

The Remaining Problem: Interaction Testing (IT)

Evaluation of Eliminated Variables

Solution: The Placebo Effect

Implementation

• Definition: Introduce a placebo to the control group to mimic treatment conditions.
    - Purpose: Ensures interaction testing effects are experienced across both groups.

• Formula Post Placebo: $(O₂ - O₁) - (O₄ - O₃) = TE$,
    - Assures accurate measurement of the treatment effect by neutralizing IT.

In-Class Exercise

Hands-on Analysis

• Task: Open the file experiment_(pre post).xlsx and apply the Pretest-Posttest Control Group Design.
    - Check randomization balance on pre-treatment characteristics.
    - Estimate the treatment effect using Difference-in-Differences methodology.

Comparison of Designs

Post-Test Only vs. Pretest-Posttest Design

Post-Test Only Design

1. Collect post-treatment data only.

2. Check for differences in characteristics between treatment group (TG) and control group (CG).

3. If differences not significant, calculate average outcome difference.

4. Determine treatment effect based on the outcome.
      - If significant → treatment is effective.
      - If not significant → treatment is ineffective.

Pretest-Posttest Design

1. Collect data at two periods (before and after treatment).

2. Assess pre-treatment characteristics between CG and TG.

3. If differences not significant, calculate the difference in differences (post-pre).

4. Evaluate treatment effectiveness based on the outcome.
      - If significant → treatment is effective.
      - If not significant → treatment is ineffective.

Key Assumption: SUTVA

Stable Unit Treatment Value Assumption

• Definition: Outcomes of the participants remain independent of each other during the treatment phase.

• Violation Example: In a vaccination RCT, if vaccinated individuals mitigate disease transmission, it benefits the control group, thus violating independence in outcomes.

Final Concepts

Validity in Experimentation

Internal Validity

• Definition: Does the treatment actually cause the observed effects?

• Requirement: Control of extraneous variables is necessary; lab experiments often achieve high internal validity.

External Validity

• Definition: Can findings be generalized to different populations, settings, and times?

• Risk: High in lab settings due to potentially unrealistic conditions; field experiments can achieve high external validity, provided SUTVA holds.

Key Takeaways

• Extraneous Threats: Time-based experiments are impacted by history, maturation, testing effects, instrumentation, mortality, and selection bias.

• Design Utilization: The Pretest-Posttest design employs Difference-in-Differences to account for most extraneous variables.

• Placebo Necessity: Address remaining IT effects through the use of a placebo.

• Core Assumption: SUTVA ensures the absence of spillovers for valid RCT results.

• Validity Assessment: Internal validity concerns causal relationships; external validity addresses generalizability of findings.