Module 13 – Errors, Effect Size, Power

Errors, Effect Size, & Power

Errors

• When conducting a null hypothesis test, there are two types of errors:
  • Type I error: Rejecting the null hypothesis when it is actually true.
  • Type II error: Failing to reject the null hypothesis when it is actually false.

Type I Errors

• A Type I error occurs when the null hypothesis is rejected, but it's actually true.
• Alpha ($\alpha$) represents the probability of rejecting the null hypothesis when it is true.
• Setting $\alpha$ to 0.05 means accepting a 5% chance of making a Type I error.
• Increasing $\alpha$ increases the chance of a Type I error.

Type II Errors

• A Type II error occurs when the null hypothesis is false, but we fail to reject it.
• Being more conservative with $\alpha$ increases the likelihood of a Type II error.
• Example: Using $\alpha = 0.001$ reduces the chance of a Type I error but increases the chance of a Type II error.
• With Type I Errors, the sample is said to come from a different distribution but it really came from the comparison distribution.
• With Type II Errors, the sample is said to come from the comparison distribution, but it really came from a different distribution.

Adjusting Alpha

• The level of $\alpha$ depends on the type of error you're more willing to make.
  • To avoid Type I errors, set a small level of $\alpha$. Example: Falsely telling someone they have a disease.
  • To avoid Type II errors, set a higher level of $\alpha$. Example: Telling someone to take aspirin when they need immediate medical treatment.
• Lowering $\alpha$ is sometimes done when conducting many individual tests to reduce the risk of finding something significant by chance.
• Example: A neuroimaging study conducting 100,000 null hypothesis tests.

Four Options

• There are four possible outcomes in null hypothesis testing:
  • Reality: Null hypothesis is true
    • Decision: Reject the null hypothesis: Type I Error ($\alpha$)
    • Decision: Fail to reject the null hypothesis: Correct Decision (1 - $\alpha$)
  • Reality: Null hypothesis is false
    • Decision: Reject the null hypothesis: Power (1 - $\beta$)
    • Decision: Fail to reject the null hypothesis: Type II Error ($\beta$)

Correct Decisions

• Correctly failing to reject the null hypothesis when it is true.
  • We decide that $\mu<em>1 = \mu</em>2$ when $\mu<em>1 = \mu</em>2$ is true.
  • This result means there is no difference between the sample's mean and the population.
• Rejecting the null hypothesis when it is false.
  • This supports the research hypothesis.

Power

• Power is the probability that the study will yield a significant result if the research hypothesis is true.
• It's the power of detecting a significant difference.
• Mathematically, Power = 1 - $\beta$

Visualization of Outcomes

• Distribution H0 represents the sampling distribution of the mean when the null hypothesis is true ($\mu<em>1 = \mu</em>2$).
• Distribution H1 represents the distribution if the null hypothesis is false ($\mu<em>1 \neq \mu</em>2$).
• The darker shaded right tail of the H0 distribution represents $\alpha$ (in a one-tailed test).
• The lighter shaded left tail of the H1 distribution represents Type II errors.
• The unshaded area of the H1 distribution is power.
• The unlabeled and unshaded area of the H0 distribution is the