Notes on Testing Theories Lecture 5

Key Idea

  • The fragment indicates the core aim of empirical validation: to subject theories to tests that can support or falsify them.
  • It implies testing predictions against data through observation or experimentation, with updates to theory based on results.

Core Concepts

  • Theory: a well-substantiated explanation of phenomena that makes testable predictions.
  • Hypothesis: a specific, testable prediction derived from a theory.
  • Prediction: an outcome expected if the theory is correct under given conditions.
  • Test: an empirical evaluation designed to assess a hypothesis or prediction.
  • Evidence: data collected to evaluate predictions and refine theories.

Methods of Testing Theories

  • Observational studies and experiments are used to test predictions.
  • Replication validates results and strengthens confidence.
  • Controls, randomization, and blinding reduce bias and confounding factors.
  • Preregistration and transparency improve credibility of findings.

Experimental Design Elements

  • Variables: independent variable (IV) manipulated; dependent variable (DV) measured; controls kept constant.
  • Control group provides a baseline for comparison.
  • Randomization assigns subjects to conditions to reduce selection bias.
  • Blinding reduces observer and participant bias.
  • Sample size and power considerations affect the ability to detect true effects.
  • Operational definitions clarify what is being measured.

Statistical Inference

  • Null hypothesis H0: no effect or no difference; alternative hypothesis H1: an effect or difference.

  • Significance level: α, commonly set at 0.05.

  • p-value: probability of observing data as extreme as the observed under H0.

  • Confidence intervals estimate the range where the true parameter lies with a specified probability.

  • Common test statistics:

    • t-statistic: t=xˉ−μ0s/nt = \frac{\bar{x} - \mu_0}{s / \sqrt{n}}
    • z-statistic: z=θ^−θ0SE(θ^)z = \frac{\hat{\theta} - \theta_0}{SE(\hat{\theta})}

  • For mean estimation with known variance: z=Xˉ−μ0σ/nz = \frac{\bar{X} - \mu_0}{\sigma / \sqrt{n}}

Theoretical Foundations

  • Falsifiability (Popper): theories must be testable and refutable.
  • Induction vs deduction: testing uses empirical data to revise general conclusions.
  • The iterative nature of theory refinement: tests can support, modify, or discard theories.

Practical and Ethical Considerations

  • Reproducibility: others should be able to reproduce results.
  • Transparency: preregistration, data sharing, and open methods.
  • Ethical treatment of participants and responsible reporting.
  • Real-world relevance: aims to improve understanding and societal outcomes.

Examples and Metaphors

  • Metaphor: a theory is a map; tests identify and correct errors to improve the map.
  • Example: testing a drug efficacy theory with randomized controlled trials.

Connections to Foundational Principles

  • The scientific method steps: observe, hypothesize, predict, test, revise.
  • Link to prior lectures on experimental design, statistics, and ethics.

Formulas and Notation

  • Hypothesis framing: H<em>0:θ=θ</em>0vsH<em>1:θ≠θ</em>0H<em>0: \theta = \theta</em>0 \quad vs \quad H<em>1: \theta \neq \theta</em>0
  • p-value definition: p-value=P(Test statistic≥observed value∣H0)p\text{-value} = P( \text{Test statistic} \ge \text{observed value} \mid H_0)
  • Confidence interval for mean (known variance): Xˉ±z1−α/2⋅σn\bar{X} \pm z_{1-\alpha/2} \cdot \frac{\sigma}{\sqrt{n}}
  • Confidence interval for mean (unknown variance): Xˉ±tn−1,1−α/2⋅sn\bar{X} \pm t_{n-1, 1-\alpha/2} \cdot \frac{s}{\sqrt{n}}

Pitfalls and Caveats

  • Misinterpretation of p-values; failure to reject H0 is not evidence for H1.
  • Publication bias against null results.
  • Overgeneralizing findings beyond the data.