Observational vs Experimental Studies and Experimental Design Types

Observational vs Experimental Studies

  • Observational studies
    • We cannot eliminate lurking variables in observational studies.
    • Because of lurking variables, observational studies can never establish causation (only association).
    • They are more prone to bias (nonresponse/undercoverage, lying responses, leading questions via wording).
    • Even with care, accuracy is not as high as in experiments.
    • Ethical and practical limitations (e.g., using animals vs humans) can affect generalizability to humans.
  • Experiments
    • Experiments can be more accurate but come with their own issues (ethics, monitoring adherence, long time horizons for effects, potential long-term side effects or environmental ramifications).
    • Both approaches have flaws; there is no perfect study design.
  • Key takeaway: distinction between observational studies (no imposed treatment) and experiments (imposed treatment with attempt to control for other factors) underpins causal inference.

Four big types of experimental designs

  • Completely randomized design (CRD)
    • The simplest design.
    • Randomly assign units to treatment groups based on the explanatory variable.
    • Then measure the response.
    • Example: random assignment to practice test vs no practice test, then compare exam scores.
  • Block design
    • Recognizes that there are lurking variables that could confound results if not accounted for.
    • Step 1: form homogeneous blocks (preexisting groups that are similar on potential confounders).
    • Step 2: randomize within each block.
    • This is analogous to stratified sampling (group by a factor, then sample within strata).
    • Examples of blocking variables: gender, race, health factors, etc.
    • Rationale: block on variables that could influence response, to reduce variability due to those factors.
  • Matched-pairs design
    • A special case of blocking where subjects are paired with a very similar counterpart on key variables.
    • Then randomize within each pair to receive different treatments.
    • Benefits: higher accuracy because treatment groups are closely matched on confounders; treatment effects can be estimated from within-pair differences.
    • Practical challenges: finding highly similar pairs can be difficult; twin studies are one workaround.
  • Crossover design (special case of paired)
    • Each subject serves as their own control by receiving multiple treatments across different periods.
    • Treatments are administered in random order with a washout period between periods to remove carryover effects.
    • Examples:
    • Two teaching methods: each student experiences both methods in random order; order is varied to balance carryover.
    • Two body parts on the same person measured under different treatments.
    • Two drugs with a washout period between administrations.
    • Key caveat: need to randomize the order to balance order effects; washout is critical to avoid carryover.

Important terminology and concepts

  • Factors (explanatory variables)
    • Categorical or quantitative but discrete (finite number of levels).
    • Continuous factors (infinite levels) are not suitable for a straightforward fixed-effects design.
  • Levels
    • The categories or discrete quantities of a factor.
    • Example: water amount with levels {1/2 cup, 1 cup}; sunlight exposure with levels {4 hours, 8 hours}.
  • Treatments
    • Combinations of levels across all factors in the experiment.
    • Example with two factors:
    • If Factor A has 2 levels and Factor B has 2 levels, there are 2 × 2 = 4 treatments (A1B1, A1B2, A2B1, A2B2).
    • In the agricultural example discussed, a block of two factors is described as: tillage type {A, B} and pesticide application schedule {1, 2, 3}, yielding treatments: A1, A2, A3, B1, B2, B3 (6 total).
  • Randomization
    • Purpose: to ensure the only systematic difference between groups is the treatment, by balancing unknown and known confounders.
  • Control
    • A baseline or placebo-like condition to neutralize placebo effects or external influences.
  • Replication
    • Having several subjects per treatment to balance natural variation and gain reliable estimates of treatment effects.
  • Blinding (masking)
    • Single blind: subjects do not know which treatment they received to reduce bias.
    • Placebo controls are especially important in clinical trials to prevent dropout or differential behavior.
  • Placebo effect
    • Improvement due to the perception of being treated rather than the treatment itself.
  • Confounding
    • When a lurking variable is entangled with the treatment effect, making it hard to separate the treatment impact from the confounder (e.g., time of day vs advertisement content).
  • Blocking variable
    • A preexisting factor used to form homogeneous groups before randomization to reduce variability.
  • Observational study vs experimental study (summary):
    • Observational: no treatment is imposed; more prone to lurking variables and bias; cannot establish causation.
    • Experimental: treatment is imposed with randomization, blocking, or pairing to isolate the treatment effect.

Practical examples from the transcript

  • Example 1: Practice test vs no practice test (CRD)
    • Randomize students to receive a practice test or not; measure exam scores; analyze difference in outcomes to infer effect of practice testing.
  • Example 2: Plant growth with water and sunlight factors
    • Factor 1: water amount with levels {0.5 cup, 1 cup}
    • Factor 2: sunlight exposure with levels {4 hours, 8 hours}
    • Treatments are combinations of levels (e.g., 0.5 cup with 4 hours, 0.5 cup with 8 hours, 1 cup with 4 hours, 1 cup with 8 hours).
    • Purpose: understand how factors jointly affect plant production.
  • Example 3: Blood pressure drug studies (two studies discussed in class)
    • Study 2 is considered better because it includes randomization, which helps balance confounding variables and reduce bias.
    • Discussion of control groups and replication in clinical trials.
  • Example 4: Shoe company advertising experiment (marketing study)
    • Likely observational or non-randomized; no true randomization; potential confounding due to time of day and other factors.
    • Issues: no randomization, potential confounding (advertising content vs time of day), impractical to blind the study.
    • Outcome: demonstrates how confounding can undermine causal inference in real-world experiments.
  • Example 5: Twin studies and crossover design as solutions to matching challenges
    • Twin studies offer a way to control for genetic and early-life differences.
    • Crossover design allows each subject to receive multiple treatments, mitigating between-subject variability.

Connections to core principles and real-world relevance

  • Randomization is the key to causal inference in experiments; it aims to ensure that the only systematic difference between groups is the treatment.
  • Blocking and matching reduce confounding by balancing known and unknown covariates, increasing precision of estimated treatment effects.
  • Replication helps account for natural variability among subjects and supports generalizable conclusions.
  • Blinding and control groups reduce bias and placebo effects, increasing reliability.
  • In practice, ethical considerations and feasibility heavily influence design choice (e.g., ethical concerns with certain human experiments, use of animal models with generalizability caveats).
  • Understanding design types helps in critiquing real-world studies (e.g., medical trials, educational interventions, marketing experiments).

Quick recap of the core formulas and ideas (LaTeX)

  • Estimating treatment effect in CRD (difference in means):
    τ^=Yˉ<em>t−Yˉ</em>c\boxed{\hat{\tau} = \bar{Y}<em>{t} - \bar{Y}</em>{c}}
    where (\bar{Y}{t}) and (\bar{Y}{c}) are the average responses in the treatment and control groups, respectively.
  • Block design (within-block comparison):
    τ^=1B∑<em>b=1B(Y</em>t,b−Yc,b)\boxed{\hat{\tau} = \frac{1}{B} \sum<em>{b=1}^{B} \left( Y</em>{t,b} - Y_{c,b} \right)}
    where (B) is the number of blocks.
  • Matched-pairs (within-pair difference):
    τ^=1n∑<em>i=1n(Y</em>i,treatment−Yi,control)\boxed{\hat{\tau} = \frac{1}{n} \sum<em>{i=1}^{n} \left( Y</em>{i, \text{treatment}} - Y_{i, \text{control}} \right)}
  • Crossover design (within-subject difference):
    τ^=1n∑<em>i=1n(Y</em>i,A−Yi,B)\boxed{\hat{\tau} = \frac{1}{n} \sum<em>{i=1}^{n} \left( Y</em>{i,A} - Y_{i,B} \right)}
  • Key design relationships:
    • Treatments = all combinations of factor levels.
    • Randomization balances known and unknown confounders.
    • Blocking reduces within-group variability and improves precision.
    • Replication ensures results are not driven by random chance.

Quick study tips based on the lecture

  • Distinguish between observational and experimental designs at a glance:
    • Observational: no imposed treatment; look for associations; beware lurking variables.
    • Experimental: impose treatments; use randomization, blocking, pairing, or crossover to isolate effects.
  • When evaluating a study, check for these design elements:
    • Is there randomization? If yes, benefits include reduced bias and balanced confounders.
    • Are there blocking or matching steps? If yes, note what variables drive the blocks or pairs.
    • Is blinding used? If yes, this supports reliability.
    • Is there a clear control or placebo? If yes, helps assess the treatment effect.
    • Is replication present? If yes, supports precision and generalizability.
  • Remember: there is no one-size-fits-all design; practical and ethical constraints often shape the final approach.