chapter 8 justifying sample size

a-priori power analysis

A method used before data collection to determine the required sample size to detect an expected effect size with a specified power and alpha level.

goal of a-priori power analysis

To plan for detecting a specified effect size (often a SESOI), not the true unknown effect size.

Smallest Effect Size of Interest (SESOI)

The smallest effect size that is still considered theoretically or practically meaningful.

benefit of planning for precision

You can draw informative conclusions even if the effect is zero, by ruling out large or meaningful effects.

hardest part of planning for precision

Deciding what width of the confidence interval is "precise enough"—this is often subjective.

heuristics for sample size justification

Simple rules like "use 30 participants per group"; problematic because they ignore study-specific context, power, and goals.

explicitly stating ‘no justification’

Encourages transparency, avoids misleading assumptions, and improves scientific integrity—even if it's uncomfortable.

best justification approach (if available)

Using a SESOI, because it ties the study design directly to meaningful, interpretable effects.

why not use effect size from meta-analysis?

They may be inflated due to publication bias and may not generalize to your study context.

why not use effect size from a single study?

Often unreliable and overestimated; high sampling error and context-specific.

compromise power analysis

Used when sample size is fixed; calculates alpha and beta based on a desired error ratio and expected effect size.

post-hoc (retrospective) power analysis

Not informative—just a mathematical function of the p-value; tells you nothing new.

within-subject design advantage

Higher statistical power due to reduced error variance; more efficient than between-subjects design.

sensitivity power analysis

Calculates the smallest effect size that can be detected with a given sample size, alpha, and power.

increase power without increasing sample size

Use within-subjects design, reduce measurement error, strengthen manipulations, use one-sided tests, or include covariates.