Independent Measures ANOVA: Post-Hoc Tests and APA Formatting

Transitioning to the Six-Step Procedure for ANOVA

• In previous lessons on Chapter 12 and ANOVA (Analysis of Variance), the traditional four-step hypothesis testing procedure was utilized.
• Starting today, the procedure expands to a six-step process to accommodate complex post-calculation requirements.
• The two primary additions to the workflow are the construction of an APA Table and the execution of post hoc tests.
• Revised Step Workflow:
  • Steps 1-3: Remain essentially the same (Hypothesis formulation, setting criteria, and calculating the F-ratio).
  • Step 4: Simplified to making a statistical decision ($H_0$ decision) and a brief work-around conclusion.
  • Step 5: Includes calculations for effect size ($\eta^2$), the APA sentence, and the new APA Table summarizing Means ($M$) and Standard Deviations ($SD$).
  • Step 6: Reserved for post hoc tests, specifically Tukey's HSD (Honestly Significant Difference), to determine which specific groups differ from one another.
• Reference locations in the course packet:
  • Page 129: Outlines the steps of the procedure.
  • Page 130: Contains the formulas for Chapter 12 independent measures ANOVA.

The Limitations of Rejecting the Null Hypothesis in ANOVA

• Researchers generally conduct experiments with the hope of rejecting the null hypothesis ($H_0$), as they believe the factor or treatment being tested will have an effect.
• However, in ANOVA, rejecting $H_0$ is only a starting point rather than a definitive conclusion because the result is too broad.
• The Range of "Some Overall Effect":
  • Rejecting $H_0$ indicates that the factor has "some overall effect," but it does not specify the magnitude or the nature of that effect.
  • For instance, in a study for a pain medication, knowing the pill has "some effect" is insufficient if that effect is infinitesimally small or, worse, if it actually increases pain.
• The Issue of Variability vs. Pattern:
  • ANOVA measures the variability among group means; it is insensitive to the order or direction of those means.
  • Example: Consider three doses—Low Dose (LD), Medium Dose (MD), and High Dose (HD).
    • Pattern A (Success): Means of $30$, $20$, and $10$ miles/hour (pain level) show a steady decrease as dose increases.
    • Pattern B (Failure/Paradox): Means of $10$, $20$, and $30$ show a steady increase (making pain worse).
    • Pattern C (Random): Means of $10$, $30$, and $20$ (jumping around randomly).
  • Because variance is calculated without regard to order, all three patterns could yield the same variance and the same F-ratio. ANOVA would reject $H_0$ for all three, even though Pattern B represents a failure of an anti-pain drug and Pattern C suggests meaningless "noise."

Step 5: The APA Table and Pattern Detection

• The APA (American Psychological Association) Table is a standardized formatting tool used to organize results so patterns can be detected quickly by researchers.
• Formatting Requirements:
  • All numerical values should be rounded to two decimal places.
  • The mean is labeled as $M$ (capitalized), and Standard Deviation is labeled as $SD$ (capitalized, usually without periods).
  • Rows typically represent the statistics ($M$ and $SD$), while columns represent the conditions or levels of the factor.
• Case Study: Eyewitness Suggestibility (Elizabeth Loftus, UC Irvine):
  • Wording is the factor with three levels: "Bumped," "Hit," and "Smashed."
  • These are increasingly violent verbs used to ask participants how fast cars were going in a video of an accident.
  • Hypothetical Sample Data Table:
    • Bumped: $M = 31.00$, $SD = 4.22$
    • Hit: $M = 42.00$, $SD = 5.10$
    • Smashed: $M = 57.00$, $SD = 6.15$
  • Interpretation: The table shows a steady increase in speed recall as the violence of the verb increases ($31 \rightarrow 42 \rightarrow 57$). This confirms the pattern the researcher predicted. If the numbers jumped from $31$ to $57$ then back to $32$, the pattern would be inconsistent with the theory of suggestibility.
• Calculations for the Table:
  • Mean ($M$): $M = \frac{T}{n}$. For Level 1, $M_1 = \frac{T_1}{n_1}$.
  • Standard Deviation ($SD$): Calculated by taking the square root of the variance ($s^2$).
  • $s^2 = \frac{SS}{n - 1}$
  • $SD = \sqrt{\frac{SS}{n - 1}}$
  • Note: Sum of Squares ($SS$) for each treatment condition is always provided in this course's problems.

Step 6: Post Hoc Tests (Tukey's HSD)

• Definition: "Post hoc" means "after the fact." These tests are only conducted after a significant F-ratio is found (rejecting $H_0$).
• Purpose: To perform pairwise comparisons and determine exactly which pairs of means are significantly different from one another without inflating Type I error.
• While there are dozens of post hoc tests (e.g., Scheffe's test), Tukey's Honestly Significant Difference (HSD) is the primary "all-purpose" test taught, used in approximately 95% of relevant research scenarios.
• The Minimum Effective Dose Example:
  • In drug research, scientists look for the minimum effective dose to avoid side effects of higher doses.
  • If LD = $30$, MD = $12$, and HD = $10$, ANOVA might show an overall effect. However, a post hoc test might reveal that while LD and MD are significantly different ($30$ vs $12$), MD and HD are not statistically different ($12$ vs $10$). This identifies MD as the minimum effective dose.

Calculating and Applying Tukey's HSD

• The Tukey HSD procedure consists of two parts: calculating the cutoff value and using that value to test mean differences.
• The Formula:
  • $HSD = q \times \sqrt{\frac{MS_{within}}{n}}$
  • $n$: This represents the number of scores in each treatment. Tukey's HSD requires equal sample sizes across all groups. If groups are equal, the subscript is dropped.
  • $MS_{within}$: This is the denominator from the original ANOVA F-ratio.
  • $q$: This is the "Studentized Range Statistic," found in Table B5.

Navigating Table B5 (The Q Table)

• Table B5 is located on page 580 of the textbook and page 231 of the course packet.
• It requires three pieces of information:
  • Alpha ($\alpha$): Usually $0.05$ (light face type) or $0.01$ (bold face type).
  • $k$: The number of treatment conditions (not $k - 1$). This is located at the top of the table. If there are 3 groups (Low, Medium, High), search for $k = 3$.
  • $df_{denominator}$: The table labels this as "df for error term," but it is identical to $df_{within}$ from the ANOVA.
• Common Error Warning: In the same problem, you will use the F-table with $k - 1$ ($df_{numerator}$) and the Q-table with $k$. Be careful not to confuse the two.

Using the HSD Cutoff Rule

• Once the HSD value is calculated, it acts as a hurdle for the difference between any two means.
• Decision Rule: If the absolute value of the difference between two means is greater than the HSD value, the difference is significant.
  • $|M_1 - M_2| > HSD \rightarrow \text{Significant Difference}$
• Example Comparison ($HSD = 8.21$):
  • Means: $M_1 = 30$, $M_2 = 12$, $M_3 = 10$
  • Pair 1 ($M_1$ vs $M_2$): $|30 - 12| = 18$. Since $18 > 8.21$, it is significant.
  • Pair 2 ($M_1$ vs $M_3$): $|30 - 10| = 20$. Since $20 > 8.21$, it is significant.
  • Pair 3 ($M_2$ vs $M_3$): $|12 - 10| = 2$. Since $2 < 8.21$, it is not significant.

Questions & Discussion

• Q: Do sample sizes always have to be the same for ANOVA?
• A: ANOVA itself can function with unequal sample sizes (e.g., $n_1 = 4, n_2 = 4, n_3 = 5$). However, the specific version of the Tukey HSD formula provided only works when all group sizes are equal. If sample sizes are unequal, different post hoc methods are required.
• Q: What if the pattern in the APA table is significant but in the opposite direction predicted?
• A: This is why the APA table is vital. If a memory pill shows a steady decrease in memory scores, you have rejected the null hypothesis and found a pattern, but it proves the pill is a failure for its intended purpose. The table provides context that the F-ratio alone cannot.