PSYU2248 Lecture 6: Multiple Comparisons in One-way ANOVA

Multiple Comparisons

Follow-up tests after a significant omnibus ANOVA to pinpoint group differences.

Omnibus F-Test

Tests if any group means significantly differ, but doesn't specify which groups.

Multiple Comparisons

Follow-up tests after a significant omnibus ANOVA to pinpoint group differences.

Omnibus F-Test

Tests if any group means significantly differ, but doesn't specify which groups.

Family-wise Error Rate

Probability of committing at least one Type I error across multiple tests.

Per-comparison Error Rate

Probability of committing a Type I error in an individual test

Pairwise Comparisons

Simple comparisons involving exactly two group means.

Linear Contrast

Uses assigned weights (contrast coefficients) to compare group means

Contrast Coefficients (Weights)

Numbers assigned to group means in a linear contrast, summing typically to zero.

Bonferroni Adjustment

• Adjusts alpha by dividing by the number of comparisons (α / number of comparisons).

• Suitable for planned comparisons, small number of tests.

Tukey’s Honestly Significant Difference (HSD)

• Ideal for all pairwise comparisons.

• Controls family-wise error rate effectively.

Scheffé’s Method

• Most conservative method, controls error across all possible contrasts.

• Useful for complex comparisons.

T-Statistic for Pairwise Comparisons

Similar calculation to independent t-test: mean difference / pooled standard deviation.

F-Statistic in Linear Contrast

Calculated by partitioning variance using assigned contrast weights.

Effect Size (Cohen’s d)

mean difference divided by square root of MSwithin

Interpretation guidelines of cohen’s D

0.2 (small), 0.5 (medium), 0.8 (large).

Relationship of T and F

F = t²; t = square root of F

Basic ANOVA command

anova DV IV

Pairwise Comparisons (with adjustments) command

pwmean DV, over(IV) effects mcompare(bon|tukey|sch)

Linear Contrasts command

contrast {group w1 w2 w3}, effects

Bonferroni with Linear Contrast command

contrast group, effects mcompare(bon)

Tukey HSD (pairwise only) command

pwmean DV, over(IV) effects mcompare(tukey)

Scheffé’s Method command

contrast group, effects mcompare(sch)

Identify Group Coding command

label list

codebook IV

Common Issues of multiple comparisons

• Running many independent tests without adjustments increases Type I error.

• Avoid "p-hacking" by selecting one adjustment method beforehand.

Tips for Contrasts in Stata

• Ensure weights sum to zero.

• Whole numbers simplify interpretation.

• Adjust reference categories with: contrast rb3.group, effects