Post Hoc Pairwise Comparisons of Means

8.2 Post Hoc Pairwise Comparisons of Means

This lecture focuses on post hoc pairwise comparisons, which are essential for understanding the results of a one-way between-subjects ANOVA when a significant result is obtained.
In the previous lecture (8.1), a significant ANOVA result was obtained and this lecture will address the next steps.

Pairwise Comparison: Comparing two means (e.g., imagery group vs. modeling group).
Setwise Comparison: Comparing a combination of means against another (e.g., average of imagery and modeling groups vs. average of all self-talk groups).
This lecture will focus on pairwise comparisons. Setwise comparisons are more advanced and will be covered later in the course.

When conducting multiple pairwise comparisons, the alpha level is altered, which increases the Type I error rate (false alarms).
With more than two groups, it's more likely to incorrectly conclude there's a significant difference when there isn't one.
Example: With 4 groups, there are 6 possible pairwise comparisons. This increases the alpha level from 0.05 to 0.27, meaning a 27% chance of making a Type I error due to inflated false alarms.
With 5 means, there are 10 comparisons.
Making many comparisons increases the likelihood of finding a significant result by chance.
Statistical adjustments are needed to account for this inflated error rate.

ANOVA: A significant ANOVA result is required before moving on to pairwise comparisons or post hoc tests.
Post Hoc: Means "after the fact." These tests are conducted after a significant ANOVA finding.
Example: A significant ANOVA effect was found that the F value was significant (less than 1 in 1000 chance).

Null Hypothesis: All means are equal.
Alternate Hypothesis: At least one mean is different.
Descriptive statistics can provide insights into which groups might be different. (e.g., If the instructional self-talk group has the lowest score and the imagery group has the highest, differences may be between these two groups).
If no significant difference between two groups, it's unlikely find a significant difference between other comparisons.

If we had 4 groups, that would mean we'd have 6 comparisons.
The alpha rate could explode up to 26.5 \%, which is much higher than the 0.05 \%. That means it is more than 5 times higher than the 0.05 \%.
If comparisons are not independent (e.g., pre-test vs. post-test), the experiment-wise error rate increases further.
With 10 groups, the 0.05 comparison rate could increase to a 50% chance of rejecting the null hypothesis.
The experiment-wise error rate increases with many comparisons and increases even further if the comparisons are not independent.

Student's t-test: Used only when comparing two means.
Student-Newman-Keuls (SNK) test / Tukey test: Needed when comparing more than two means to control for the inflated error rate.