Post-hoc tests

Purpose of post-hoc tests (2, linked)

• Determine which groups/conditions differ significantly from each other

• without accumulating familywise error

So how do we determine which groups/conditions differ significantly?

Post-hoc tests

• Conduct t-tests each possible pair of groups/condition

• But with smaller alpha level (to prevent accumulation of Type-I error)

2 types of post-hoc tests

• Bonferroni tests

• Holm’s stepdown

Bonferroni alpha formula

Bonferroni alpha = alpha level (e.g. .05) / number of comparisons

Bonferroni strength + limitation

• Simple

• Very conservative (increases risk of type II error)

Alternative since Bonferroni can be too conservative (1)

• Holm’s stepdown: more liberal alpha than Bonferroni

Holm’s stepdown process

1. Conduct t-tests all possible combination of pairs

2. Order obtained p-values (not t-values) from lowest to highest

3. Starting with the first test (lowest p-value, most significant), calculate a Holm’s modified alpha level

4. Compare obtained p-value 1 to Holm’s modified alpha level 1

5. If p-obtained is smaller than Holm’s modified alpha, then move onto next test, and keep doing this until you p-obtained is larger than Holm’s modified alpha (non-significant)

Holm’s modified alpha formula, w/ example for first t-test