Factorial ANOVA: Comprehensive Study Guide

Introduction to Factorial ANOVA

• Factorial ANOVA is considered the most complex statistical test covered in this course.
• While One-way ANOVA (discussed previously) handles a single categorical predictor with multiple groups, Factorial ANOVA is used when there are multiple categorical predictors.
• It allows researchers to group participants by several variables simultaneously (e.g., by the type of pet they own, the city they live in, and their highest level of tertiary education).
• The test serves to examine the effects of each grouping variable individually as well as the interactions between them.

Determining the Appropriate Statistical Test

• Continuous Outcome: All tests discussed this week require exactly one continuous outcome measure. If this condition is not met, ANOVA should not be used.
• Categorical Predictors:
  • If there is one categorical predictor with only two groups, a $t$-test is used.
  • If there is one categorical predictor with three or more groups, a One-way ANOVA is used.
  • If there are two categorical predictors, a Factorial ANOVA is used.
• Technically, researchers can add any number of categorical predictors (three, four, or seven), but this course focuses on cases with exactly two predictors to maintain conceptual clarity.

Naming Conventions and Levels

• Terminology by Predictor Count:
  • The term "Factorial ANOVA" refers to any analysis with two or more categorical grouping variables.
  • A "Two-way Factorial ANOVA" indicates there are exactly two categorical grouping variables.
• Terminology by Levels:
  • Predictors are also described by their "levels" (the number of groups within that variable).
  • Example 1: If the first predictor is "Pet Owned" (with 4 options) and the second is "City" (comparing 2 cities), it is a $4 \times 2$ Factorial ANOVA.
  • Example 2: A $2 \times 2$ Factorial ANOVA indicates two predictor variables, each with two levels.
  • Example 3: A $3 \times 2 \times 3$ Factorial ANOVA indicates three predictor variables with three, two, and three levels respectively.

Main Effects vs. Interactions

• One-way ANOVA (Recap): This is an omnibus test that asks one question: "Are there differences among the groups of the single factor?" result tables show one row of numbers for this question.
• Factorial ANOVA: It allows for the investigation of three distinct questions (in a two-way design):
  1. Main Effect of Factor 1: Is there an overall effect of the first predictor variable?
  2. Main Effect of Factor 2: Is there an overall effect of the second predictor variable?
  3. Interaction between Factor 1 and Factor 2: Does the effect of one factor depend on the level of the other factor?

Definitions

• Main Effect: A significant main effect occurs when different groups within a single factor have significantly different means, regardless of the other factors. For example, if city of residence affects average income, there is a main effect of "City."
• Interaction: A significant interaction occurs when the effect of one factor changes depending on the level of another factor.
  • Real-world metaphor: In sociology, the effect of profession on income might depend on the city. Farmers in Dunedin might earn more than dentists, while in Auckland, farmers might earn less than dentists. In this case, the effect of "Profession" differs based on "City."

Hypotheses in Factorial ANOVA

• Factorial ANOVA is an omnibus test with multiple parts. The hypotheses are fixed and not determined by the researcher.
• Null Hypothesis ($H_0$): All group means within the first factor are equal; all group means within the second factor are equal; and there is no interaction between the factors.
• Alternative Hypothesis ($H_1$): At least one group mean is different from the others, or there is a significant interaction between the factors.

Assumptions for Factorial ANOVA

• The assumptions are identical to those of a One-way ANOVA:
  1. Independence of Observations: Data points must come from different people within each subgroup.
  2. Normality of Residuals: Residuals (the distance between each data point and its subgroup mean) must be normally distributed. This is checked using the Shapiro-Wilk test (which should be non-significant).
  3. Homogeneity of Variance: The spread or variance of data points should be roughly equal across all subgroups. This is checked using Levene's test (which should be non-significant).
• Balanced Design: While not a strict assumption, Factorial ANOVA is more robust and reliable if there are an equal number of participants in each subgroup.

Case Study: Brainflex Training App

• Research Question: Does brain training make you better at solving challenging mental puzzles?
• Factor 1: Training Type (3 Levels):
  1. Brainflex Training: Participants use the app for $30$ minutes daily.
  2. Sudoku Training: Participants do Sudoku puzzles for $30$ minutes daily (comparison treatment).
  3. No Training: Control group following their usual routine.
• Factor 2: Expertise (2 Levels):
  1. Expert: Participants who regularly do Sudoku puzzles.
  2. Novice: Participants with no previous expertise.
• Outcome Measure: Time (in minutes) taken to solve a set of Sudoku puzzles.
• Participants: $120$ total ($40$ per training group; $20$ per subgroup for a balanced design).
• Rationale for Design: Including a comparison treatment (Sudoku) helps determine if the app is more effective than standard practice. Including expertise allows researchers to see if the app's benefits depend on the population (e.g., if it helps novices more than experts).

Interpreting the Result Table

• The output table (from software like R or Jamovi) provides three rows for a two-way ANOVA, each with its own statistics:
  1. Row 1 (First Factor): Provides the $F$-value and $p$-value for the main effect of training. It conceptually performs a One-way ANOVA by ignoring expertise and comparing the three training column means.
  2. Row 2 (Second Factor): Provides the $F$-value and $p$-value for the main effect of expertise. It compares the row means (Expert vs. Novice) while ignoring training.
  3. Row 3 (Interaction): Labeled as "Expertise:Training" or "Expertise * Training." It tests if the effect of one variable depends on the other.
• Degrees of Freedom ($df$): Each row has its own degrees of freedom. To report a result, you need the $df$ from that specific row and the residual (error) $df$ from the bottom of the table.
• F-Statistic: A ratio of variability between groups to variability within groups. Higher $F$-values are less likely by chance.
• Note on Tails: There is no concept of one-tailed or two-tailed tests in ANOVA; the test always looks for high $F$-values in the top $5\%$ (the critical region) of the distribution.

Reporting and Follow-up Procedures

• Factorial ANOVA results must be reported systematically for all three parts, regardless of significance.
• Omnibus Report Template: "A two-way factorial ANOVA with a significance level of $\alpha = 0.05$ revealed a main effect of prior expertise ($F(df1, df2) = F_{value}, p < 0.01$), a main effect of training type ($F(df1, df2) = F_{value}, p < 0.01$), and a significant interaction between expertise and training ($F(df1, df2) = F_{value}, p < 0.01$)."
• Follow-up Tests: Because the ANOVA is an omnibus test, it does not specify which groups differ. If a main effect or interaction is significant, a post hoc test—specifically Tukey's Honest Significant Differences (HSD) test—is required to perform pairwise comparisons.
• Example Findings from Case Study:
  • Main effect of expertise: Experts solved puzzles significantly faster than novices ($4.87$ minute mean difference).
  • Main effect of training: Sudoku training was more effective than no training or Brainflex, but Brainflex showed no significant difference compared to no training.
  • Interaction: The difference between experts and novices was much smaller in the Sudoku training group than in the other two groups.

Questions & Discussion

• Quiz Question 1: A study measures stats grades across two universities (Us and AUT) and three degrees (BA, BSc, BEng). What test should be used?
  • Answer: A $2 \times 3$ Factorial ANOVA. It is factorial because there are two grouping variables, and the levels are $2$ and $3$.
• Quiz Question 2: Which of these things could not be claimed on the basis of a significant $F$-value alone in a Factorial ANOVA?
  1. Income differs between New Zealand cities.
  2. Income differs between professions.
  3. Engineers earn more than dentists.
  4. The effect of income/city depends on profession.
  • Answer: Option 3 ("Engineers earn more than dentists"). A significant $F$-value only indicates that some differences exist among professions (the omnibus result); to claim one specific profession earns more than another, a follow-up pairwise test is mandatory.