Power and Effect Size Concepts in Statistical Analysis

Power in Statistics

• Definition of Power

  • Power is the probability of correctly rejecting a false null hypothesis ($H_0$).
  • It is calculated as:
    Power = 1 - \beta
  • Where:
  • $\beta$ = probability of a Type II error (failing to reject a false null hypothesis).

• Factors Affecting Power

  • Alpha Level ($\alpha$): The probability of a Type I error (rejecting a true null hypothesis).
  • Effect Size: The degree to which the phenomenon is present in the population.
  • If $H_0$ is true, Effect Size = 0.
  • If $H_0$ is false, Effect Size > 0.
  • Sample Size ($N$): Larger samples generally provide more power.
  • Variability in Data: Less variability in data increases power.

Outcomes of Statistical Tests

• Outcomes of hypothesis testing can lead to four possible results:
  • Fail to reject $H_0$ when it is true: Correct Decision
  • Fail to reject $H_0$ when it is false: Type II Error
  • Reject $H_0$ when it is true: Type I Error
  • Reject $H_0$ when it is false: Correct Decision

Effect Size

• Understanding Effect Size

  • Indicates how much of a difference there is between groups.
  • For example, using GRE scores:
  • Population mean ($\mu$) = 587; standard deviation ($\sigma$) = 152.
  • After training, sample mean ($M$) = 595.

• Cohen’s $d$: A common measure of effect size calculated as:
  d = \frac{\mu{treatment} - \mu{control}}{\sigma}

  • For the example provided:
    d = \frac{595 - 587}{152} = 0.052
  • Indicates a small effect since it's only 0.05 standard deviations.

Examples of Power Analysis

1. Example 1:

   • Population: $\mu = 80$, $\sigma = 10$; Sample Size ($n$) = 25; Expected Effect = 8 points ($\mu = 88$).
   • Effect Size: Calculated to be $d = 0.8$.
   • Power calculation resulting in approximately 97.93% chance of detecting the 8-point effect.

2. Example 2:

   • Same population, $\mu = 80$, $\sigma = 10$; Sample Size ($n$) = 25; Expected Effect = 4 points ($\mu = 84$).
   • Lower power calculated at approximately 51.60% for detecting the 4-point effect.

3. Example 3:

   • Population: $\mu = 80$, $\sigma = 10$; Sample Size ($n$) = 50; Expected Effect = 4 points.
   • Effect size again calculated as $d = 0.4$.
   • Resulted in higher power at approximately 81.06% chance of detecting the 4-point effect due to increased sample size.

A Priori Power Analysis

• Purpose: To estimate the required sample size to achieve a specified power with a particular effect size and alpha level.

• Strategies to increase power:

  1. Increase the strength of manipulation of the independent variable.
  2. Increase the number of participants.
  3. Reduce variability through more trials per participant or obtaining more accurate measurements.

• Use of Software: G*Power can be utilized for complex power calculations, allowing better planning in study designs.