l6- Understanding Type I and Type II Errors and Statistical Power

The fundamental framework of hypothesis testing involves addressing the potential for error that arises from a mismatch between the statistical conclusion reached and the actual underlying reality.
Definition of Reality: In this context, reality refers to whether the null hypothesis is actually true or false in the population.
- Null Hypothesis is True: This represents the state where there is naturally no effect, no difference between groups, no correlation, and no relationship between variables.
- Null Hypothesis is False: This represents the "alternative reality" where some form of effect, difference, or relationship truly exists.
Scientific Dichotomy: Researchers always structure the null hypothesis to create a strict dichotomy. This ensures that the state of the world fits into one of two mutually exclusive categories: the null is either false or true.
The Epistemological Limitation: While a reality exists, researchers never truly know it. They can only reach a conclusion by running a statistical test on a sample.
The Two Possible Conclusions:
- Reject the Null Hypothesis: This is the decision that sufficient evidence exists to suggest an effect is present.
- Fail to Reject the Null Hypothesis: This is the decision that there is not enough evidence to support the presence of an effect, essentially concluding that there is no effect.

Accuracy in statistical analysis is achieved when the conclusion aligns perfectly with the underlying reality. There are two scenarios where a conclusion is considered correct:
- Correct Rejection: The researcher rejects the null hypothesis when the null is, in fact, false (the effect is real).
- Correct Retention: The researcher fails to reject the null hypothesis when the null is, in fact, true (no effect exists).
High Probability Goal: The primary objective of experimental design is to maximize the probability of these two outcomes occurring. Researchers do not want to conduct exhaustive analyses only to reach an incorrect conclusion.

Definition: A Type I error occurs when a researcher incorrectly rejects the null hypothesis when it is actually true. This is conceptually known as a "false positive."
The Significance Level ( $\alpha$ ): In the hypothesis testing framework, the researcher explicitly sets the rate of Type I error. This value is referred to as the significance level or the confidence threshold, denoted by the Greek letter alpha ( $\alpha$ ).
Standard Benchmarks in Biology:
- In biological sciences, an alpha level of $\alpha = 0.05$ is typically utilized.
- This setting means that, mathematically, the researcher expects to encounter a false positive approximately $5\%$ of the time due to random chance.
Controlling Type I Error:
- Researchers can choose to reduce the probability of a Type I error by selecting a lower significance threshold.
- Examples of lower thresholds include $\alpha = 0.01$ or $\alpha = 0.0001$ .
- While reducing $\alpha$ is beneficial for preventing false positives, it carries a significant side effect: it increases the probability of committing a Type II error.

Definition: A Type II error occurs when a researcher fails to reject the null hypothesis even though it is false. This is essentially missing a real effect, denoted by the Greek letter beta ( $\beta$ ).
The Trade-off Mechanism: There is a mathematical tension between the two error types. By reducing the probability of incorrectly rejecting the null (lowering $\alpha$ ), the researcher inadvertently increases the probability of failing to detect a real effect (increasing $\beta$ ).
Rationale for $0.05$ : The common use of the $\alpha = 0.05$ threshold is a result of balancing this trade-off, providing a standardized limit that maintains reasonable protection against false positives without excessively inflating the risk of false negatives.

Definition of Power: Statistical power is the probability of correctly rejecting the null hypothesis when it is indeed false. It represents the probability of detecting a "true positive."
Mathematical Relationship: Power is inversely related to Type II error; if a study has a low probability of Type II error ( $\beta$ ), it is said to have high power.
Functional Definition: Power is the probability that a random sample taken from a population will, when subjected to analysis, lead to the rejection of a false null hypothesis.
General Principle: All other factors being equal, a study with higher statistical power is always superior because it minimizes the risk of missing real phenomena.

Interactive visualization tools allow students to manipulate variables to see how error rates shift.
Scenario Observations:
- Effect of Large Differences: When there is a massive discrepancy between the null hypothesis and reality (the alternative), the distributions of the two scenarios move far apart. If they do not overlap, power can reach $100\%$ .
- Overlap and Power: If a scientist observes a large difference where only a small portion of the distributions overlap, power might be approximately $89\%$ . This means the researcher will correctly detect the false null hypothesis $89\%$ of the time.
- Sample Size Interaction: Even if the effect size (the difference) is relatively low, power can be increased by increasing the sample size. Conversely, a low effect size combined with a small sample size results in very low power and a high Type II error rate.
- Shifting the Threshold: Decreasing the significance threshold ( $\alpha$ ) shifts the critical values for the test. This movement directly alters both the Type I and Type II error regions within the probability distributions.
Conclusion on Experimental Design: When designing experiments, the most effective strategy for minimizing total error rates and maximizing the reliability of the findings is to ensure a sufficiently large sample size.