Follow-up Comparisons and ANOVA

Follow-up Comparisons and ANOVA

• Post Hoc Comparisons

Omnibus F and Follow-Up Comparisons

• A significant F test (assuming more than 2 levels to an IV) indicates statistically significant differences among groups, but doesn't specify which groups differ.
• Follow-up comparisons are necessary to examine the differences between the means of the groups of interest.
• The reason for using the omnibus test before multiple t-tests is to control for family-wise Type I error.

Family-wise Type I Error

• The alpha rate for significance tests in psychology is conventionally set at 0.05, meaning there's a 5% chance of making a Type I error for each test.
• When conducting multiple tests on the same data, the risk of committing a Type I error increases.

Family-wise Type I Error Example

• If comparing three diet groups (Fruit, Veggie, Donut) with three t-tests (Fruit vs. Veggie, Fruit vs. Donut, Veggie vs. Donut) without correction, the Type I error rate would approximate to 0.15.
• This elevated error rate increases the likelihood of incorrectly identifying an effect as real.

Multiple Comparisons

• It's crucial to account for family-wise error (FWE) when performing multiple comparisons.
• Two main approaches exist to address this issue:
  • Post Hoc Procedures
  • Planned Comparisons

Post Hoc Tests

• Post Hoc procedures are appropriate when there is no a priori theoretical basis for expecting specific group differences.
  • For example, one might expect that diet affects happiness without knowing how.
• The atheoretical nature of post hoc tests is acceptable and can be advantageous.
• However, post hoc tests tend to be more conservative than planned comparisons, because they are not guided by theory.

Post Hoc Tests - Options

• Various post hoc procedures are available, generally falling into two categories:
  • Adjusting the Type I error rate to accommodate multiple comparisons.
  • Calculating a new, more conservative test statistic.
• Both approaches are more conservative compared to planned comparisons.

Post Hoc Tests – Bonferroni Correction

• The Bonferroni correction adjusts the alpha value for multiple comparisons.
• Formula: αB = {α{FWE} \over c}
  • α_B is the corrected alpha level.
  • α_{FWE} is the desired family-wise error rate (usually 0.05).
  • c is the number of comparisons.

Post Hoc Tests – Bonferroni Correction Example

• After calculating α_B, perform regular between-groups t-tests.
• Instead of using 0.05 as the p-value cutoff, use α_B.
• In the diet example (3 comparisons), α_B = 0.05 / 3 = 0.016.

Post Hoc Tests – Bonferroni Correction Significance

• Example:
  • Fruit vs. Veggie: t = -0.658, p = 0.514
  • Fruit vs. Donut: t = -2.840, p = 0.007
  • Veggie vs. Donut: t = -2.386, p = 0.022
• With the Bonferroni correction (αB = 0.016), only the Fruit vs. Donut comparison is significant (p = 0.007 < 0.016).

Post Hoc Tests – Tukey (Tukey HSD)

• Tukey's HSD test calculates a new test statistic representing the minimum mean difference required for statistical significance.
• Assumes all means are being compared.
• SPSS can calculate this.

SPSS output example

Dependent Variable: Happy

• The mean difference is significant at the 0.05 level.

Post Hoc Tests – Wrap Up

• Post Hoc tests are conservative, making it more difficult to achieve statistical significance.
  • It involves a trade-off between Type I and Type II error.
• The level of conservativeness varies among different post hoc tests.
• Post Hoc tests are best suited for situations with a significant omnibus F-test but no specific predicted differences.
• Their conservative nature is appropriate in such cases. In other words, post hoc tests are perfect when you use them for what they were designed:
  • Significant omnibus F, but no specific differences predicted.
  • Appropriate to be conservative.