Understanding Sample Size and Power Analysis

Power Analysis Assumptions and Sample Size

Initial Parameters

  • The initial power analysis used the following parameters:
    • Power: 80%
    • Test: Two-sided
    • Alpha: 0.05
    • Effect Size: Cohen's D = 0.5 (medium)
  • Resulting sample size: 64 people per group, totaling 128 people.

Impact of Changing Power

Increasing Power

  • Scenario: Increasing power from 80% to 90%, while keeping other parameters constant.
  • Observation: Increasing power requires a larger sample size.
  • Result: The required sample size increased from 128 to 172 subjects.
  • Lesson: Increasing sample size increases power.

Impact of Changing Effect Size (Cohen's D)

Increasing Effect Size

  • Scenario: Changing Cohen's D from 0.5 to 0.8, while keeping other parameters constant.
  • Observation: A larger expected difference (larger effect size) reduces the required sample size.
  • Result: The required sample size decreased from 128 to 52 subjects.
  • Lesson: With a bigger expected effect, fewer observations are needed.

Decreasing Effect Size

  • Scenario: Changing Cohen's D from 0.5 to 0.2, while keeping other parameters constant.
  • Observation: A smaller expected difference (smaller effect size) increases the required sample size significantly.
  • Result: The required sample size increased from 128 to 788 subjects (394 per group).
  • Lesson: Detecting small effects requires a much larger sample size.

Implications for Research Design

  • Small effects, though clinically important, are harder to detect and require larger sample sizes.
  • Grant reviewers are critical of study plans proposing small effect sizes with small samples.
  • It's crucial to be realistic about the detectable effect size given the available or affordable sample size.

Summary

  • Changing assumptions in a power analysis affects sample size estimates.
  • Increasing power increases the required sample size.
  • Increasing the expected effect size decreases the required sample size, while decreasing the expected effect size increases the required sample size substantially.