Follow-up Comparisons and ANOVA: Planned Comparisons

Follow-up Comparisons and ANOVA

Planned Comparisons

• A second approach to following up a significant omnibus F is to conduct planned comparisons.
• They are 'planned' in the sense that they should be guided by theory.
• Does the fact that they're planned get us around the family-wise error problem?
  • Not necessarily.
  • However, there is a difference between fishing for differences using post-hoc tests and predicting what the outcome of an analysis should be in advance.
• The number of planned comparisons you have is generally small relative to the number of conditions you have.
  • If you only have a few, a priori comparisons, family-wise error is less of a concern.
  • If you have many, you still need to be concerned with family-wise error.
• The process usually runs as follows:
  • Your experiment has many conditions.
  • Several may be included as controls.
  • Others are of theoretical importance.
  • The experiment yields a significant F.
  • You then test the differences between a few key groups that are of theoretical significance.
• If only a few comparisons are made, family-wise error is generally seen as less of a concern.
  • There is no rule for how many comparisons are considered few.
• There are two types of planned comparisons we can do when we have a one-way ANOVA:
  • Pairwise comparisons – Analyze the simple difference between two means.
  • Complex comparisons – Analyze the differences between sets of means.

Pairwise Comparisons

• Doing pairwise comparisons is relatively simple.
• Let’s say we wanted to see if the fruit and veggie groups differed significantly.
• To do the comparisons there are a few new statistics we will need to generate:
  • $\psiˆ = \bar{X}<em>1 - \bar{X}</em>2$
    • Where 1 and 2 refer to the groups you want to compare.
  • $SS_{comp} = \frac{(\psiˆ)^2}{n}$
    • Where n is the number of people in each of your groups.
    • Assumes equal sample sizes across groups.
• The statistics, continued:
  • $MS<em>{comp} = SS</em>{comp}$
    • Note here, there is always 1 degree of freedom, so $MS<em>{comp}$ will always equal $SS</em>{comp}$.
  • $F<em>{comp} = \frac{MS</em>{comp}}{MS_{Within}}$
    • Important Note: We use the error term (i.e., our Within Groups Mean Squares) from our omnibus test here.

Complex Comparisons

• Sometimes we want to compare a condition mean to two other conditions means.
• E.g., \psiˆ = \bar{X}{Avg.Fruit&Veggie} - \bar{X}{Donut}
• In order to construct our comparisons, we have to come up with contrast weights.
• We need these weights to incorporate the means we want, in the way we want, into our comparison.
• Imagine we wanted to demonstrate that the donut group had a higher mean than the fruit and veggie groups combined.
• Another way to express this would be: \psiˆ = \bar{X}{Avg.Fruit&Veggie} - \bar{X}{Donut}
  • $\psiˆ = \frac{X<em>{Fruit} + X</em>{Veggie}}{2} - X_{Donut}$
• Still another way would be to express these differences as the sum of weighted means:
  • $\psiˆ = \frac{X<em>{Fruit} + X</em>{Veggie}}{2} - X_{Donut}$
  • $\psiˆ = (+.5)(X<em>{Fruit}) + (+.5)(X</em>{Veggie}) + (-1)(X_{Donut})$
• Thus, each sample mean we want to compare is weighted by a coefficient.
  • Coefficients will determine which sample means are compared.
  • We could also exclude means from our comparisons by giving a sample mean a weight of zero.
• Though this seems complicated (it is!), it ends up being highly flexible.
  • $\psiˆ = (+.5)(X<em>{Fruit}) + (+.5)(X</em>{Veggie}) + (-1)(X_{Donut})$
• Now that we are incorporating coefficients in our comparisons, the formulas change slightly.
  • **Because there is only 1 df, MS=SS for these comparisons.
  • $\psiˆ = (c<em>1)(X</em>1) + (c<em>2)(X</em>2) + (c<em>3)(X</em>3) + \dots$
  • $SS<em>{comp} = \frac{(\sum c</em>i X<em>i)^2}{\sum \frac{c</em>i^2}{n}}$
  • $MS<em>{comp} = SS</em>{comp}$
• Though we can use SPSS to generate the contrasts we need in a Between Groups One-Way ANOVA, there will times when you may need to do them by hand.
  • More complex versions require different error terms and SPSS does not do them automatically (or at least generate them in an easy way).
  • If you calculate $F<em>{comp}$ or $t</em>{comp}$ by hand, you can still compute exact p-values using on-line calculators.
• Regardless, you still have to provide contrast coefficients.

Comparison Coefficients

• Field specifies several rules to generate comparison coefficients (also known as weights).
  • Choose sensible comparisons.
  • Groups with positive weights will be compared to those with negative weights.
  • The sum of the weights should always be zero.
  • Groups not involved in a comparison always get a coefficient equal to zero.
• If we follow these rules, the contrasts we do should always be “orthogonal” (i.e., independent).
• Our comparisons were orthogonal.
• This is important because the between groups variance estimates we were comparing were uncorrelated.
  • The outcome of one comparison is unrelated to the outcome of the other.
  • This helps with the family-wise error issue.

• Though considered best practice, it is not always required that we have orthogonal comparisons.
  • Most important that comparisons make theoretical sense.
  • That said, if you do comparisons they need to be orthogonal, or you must justify why they are not.

Final Thought on Planned Comparisons

• What if we developed a design to test the differences between two specific conditions, but the omnibus test was non-significant?
• Would it be appropriate to run our planned comparison?
  • Yes.
  • In most cases you will have a significant omnibus F-ratio, but this may not always happen (e.g., p = .057)
• This is where planned comparisons offer a huge advantage over post hoc tests.