Evaluating Causal Claims

Tools for Evaluating Causal Claims

Chapter Overview

  • Focus on common threats to internal validity in experimental designs.

Common Threats to Internal Validity

  • Selection Effects: Systematic differences between groups in a between-subjects design.
    • Example: Effectiveness of intensive therapy for Autism can be confounded by caregiver involvement.
  • Order Effects: Impact of a prior condition on subsequent conditions in within-person designs.
    • Example: In a depression study, participants might recall past answers affecting their current responses.
  • Design Confounds: Another variable varies alongside the independent variable (IV), affecting the results.
    • Example: In a pasta experiment, bowl size may affect perceived appetiteness rather than actual serving size.

Additional Threats to Internal Validity - Mnemonic** M**RS HRIM PDOS

  • Maturation threat: Changes over time that could affect the dependent variable (DV).
  • Regression threat: Extremes in initial measurements may regress towards the mean in subsequent measurements.
    • Example: Recruiting participants at peak depression may yield skewed results.
  • Selection Bias: Non-random selection impacts contrived group differences.
  • History Threat: External events affecting all participants between tests.
  • Researcher Bias: Influences on data interpretation due to the researcher's expectations.
  • Instrumentation Threats: Changes in measurement tools that affect results over time.
  • Mortality (Attrition): Participants dropping out can alter study dynamics.
  • Placebo Effects: Perceived treatment due to belief in its effectiveness.
  • Demand Characteristics: Participants adjusting behavior based on perceived expectations.
  • Observer Bias: Observers may record data influenced by their expectations.
  • Situation Noise: External distractions affecting the results.

Maturation Threat

  • Definition: Changes not due to the experimental manipulation, but natural progression.
  • Example: Measuring aggression in individuals at multiple time points.
  • Solution: Utilize a comparison group to control for maturation effects.

History Threat

  • Definition: External events impacting study participants systematically.
  • Example: Mood changes in the U.S. during COVID-19.
  • Solution: Ensure groups experience identical conditions aside from the IV.

Regression Threat

  • Definition: When extreme scores at the beginning lead to less extreme scores over time (regress to the mean).
  • Example: In a depression study, highs may lead to lows in follow-up measures.
  • Solution: Use a comparison group and verify patterns of results.

Attrition Threat (Mortality)

  • Definition: Participants leaving the study before it concludes could bias results.
  • Example: High dropout rates in lengthy or intensive interventions.
  • Solution: Check differences between those who drop out versus those who remain.

Testing Threats

  • Definition: Changes in participant scores due to prior testing experiences (e.g., practice or fatigue).
  • Example: Improved scores on a standardized test due to familiarity.
  • Solution: Utilize a post-test only design or alternate testing forms.

Instrumentation Threats

  • Definition: Changes in measurement tools can skew results.
  • Example: Variability in IQ test scoring criteria over time.
  • Solution: Use stable measuring tools and train personnel consistently.

Combined Threats

  • Selection-History Threat: An external event affects only one group level of the IV.
  • Selection-Attrition Threat: Different attrition rates across experimental groups.

Researcher Bias/Observer Bias

  • Definition: Researcher's personal biases affect data interpretation.
  • Example: Data trimming or selective perceived outcomes.
  • Solution: Double-blind studies that block bias influence.

Demand Characteristics

  • Definition: Changes in behavior due to participants speculating the study's hypothesis.
  • Solution: Implement double-blind designs where both participants and researchers are unaware of group assignments.

Placebo Effects

  • Definition: Individuals experience changes due to belief in treatment efficacy.
  • Example: Placebo use in psychiatric studies showing significant effectiveness rates.
  • Solution: Conduct double-blind placebo-controlled studies.