Two-Way ANOVA: Calculating F-ratios and Null Hypothesis Significance Testing (NHST)

Course Announcements and Logistics

Exam 2: Grades will be posted after today's lecture. Exams can be viewed during office hours.
Exam 3: Scheduled for next Wednesday (7/29). Details and a review sheet will be provided by Friday.
Schedule Update: No lecture on Tuesday (7/28) to allow for study time. Additional office hours will be held from 10:00 am to 12:00 pm at Psychology 1327.
Lab: Lab sessions will still occur next week; students should aim to complete Assignment 6 during their section.
Course Evaluations: Students are requested to complete their course evaluations.

Therapy Dogs Study Background

Participants: 60 students total were recruited for the study.
Groups by Predisposition:
- $n = 30$ students who are afraid of dogs.
- $n = 30$ students who are not afraid of dogs.
Experimental Assignment (Random Assignment): Half of each predisposition group is assigned to one of two conditions ( $n = 15$ per cell):
- Dog Therapy: Interacting with a dog in a room for 30 minutes.
- No Dog Therapy: Sitting in a room for 30 minutes without a dog.
Measurement: Stress levels were measured after the 30-minute session on a scale of 0–100. Higher scores indicate higher levels of stress.

The Four Steps of Null Hypothesis Significance Testing (NHST)

State the Hypotheses: State the null ( $H_0$ ) and alternative ( $H_1$ ) hypotheses. Consider whether the alternative hypothesis should be directional or non-directional.
Choose Criteria: Select an alpha level ( $\alpha$ ) and determine associated critical values. Consider the consequences regarding Type I and Type II errors and the choice between one-tailed and two-tailed tests.
Calculate Test Statistic: Use sample data to calculate the statistic. Understand what the resulting $p$ -value indicates about the sample test statistic.
Make a Decision: Reject or fail to reject the null hypothesis. Results must be written in APA style, including multiple F-ratios and post-hoc tests where applicable.

Partitioning Variance in Two-Way ANOVA

Total Variability is divided into two primary categories:
- Within-groups variance: Represents error or unsystematic variability. It remains the same regardless of which effect is being tested.
- Between-groups variance: Represents systematic variability, which is further partitioned into three specific effects:
  1. Main Effect of Factor A: Fear of Dogs.
  2. Main Effect of Factor B: Therapy Participation.
  3. Interaction Effect ( $A \times B$ ): Fear of Dogs $\times$ Therapy Participation.
F-Ratio Structure:
- $F_{\text{Fear of Dogs}} = \frac{MS_{\text{Fear}}}{MS_{\text{Within}}}$
- $F_{\text{Therapy}} = \frac{MS_{\text{Therapy}}}{MS_{\text{Within}}}$
- $F_{\text{Interaction}} = \frac{MS_{\text{Interaction}}}{MS_{\text{Within}}}$

Calculating Sum of Squares (SS) and Degrees of Freedom (df)

Total Variability ( $SS_T$ ):
- Formula: $SS_T = \sum(x - GM)^2$ , where $GM$ is the Grand Mean.
- $df_T = N - 1$ . For this study, $N = 60$ , so $df_T = 59$ .
Within-Groups Variability ( $SS_W$ ):
- This is the sum of the Sum of Squares within each individual cell.
- Data points from the study:
  - Cell 1 SS: $4187.82$
  - Cell 2 SS: $2041.9$
  - Cell 3 SS: $2400.72$
  - Cell 4 SS: $3434.2$
- Total $SS_W = 4187.82 + 2041.9 + 2400.72 + 3434.2 = 12,064.64$ .
- $df_W = N - k$ , where $k$ is the number of cells. $df_W = 60 - 4 = 56$ .
Between-Groups Variability ( $SS_B$ ):
- Formula: $SS_B = SS_T - SS_W$ .
- Data calculation result: $SS_B = 5,702.27$ .
- $df_B = df_T - df_W = 59 - 56 = 3$ .
Main Effect and Interaction SS Breakthrough:
- $SS_{\text{Fear}} = 3,631.70$ with $df_{\text{fear}} = 1$
- $SS_{\text{Therapy}} = 7.14$ with $df_{\text{therapy}} = 1$
- $SS_{\text{Interaction}} = 2,063.42$ with $df_{\text{interaction}} = df_{\text{fear}} \times df_{\text{therapy}} = 1 \times 1 = 1$

Calculating Mean Squares (MS) and F-Ratios

Mean Square Formulas: $MS = \frac{SS}{df}$
- $MS_{\text{within}} = \frac{12,064.64}{56} = 215.44$
- $MS_{\text{fear}} = \frac{3,631.70}{1} = 3,631.70$
- $MS_{\text{therapy}} = \frac{7.14}{1} = 7.14$
- $MS_{\text{interaction}} = \frac{2,063.42}{1} = 2,063.42$
F-Ratio Calculations:
- $F_{\text{fear}} = \frac{3,631.70}{215.44} = 16.86$
- $F_{\text{therapy}} = \frac{7.14}{215.44} = 0.03$
- $F_{\text{interaction}} = \frac{2,063.42}{215.44} = 9.58$

Two-Way ANOVA Summary Table

Source	SS	df	MS	F
Between Groups	$5,702.27$	$3$
Fear	$3,631.70$	$1$	$3,631.70$	$16.86$
Therapy	$7.14$	$1$	$7.14$	$0.03$
Interaction	$2,063.42$	$1$	$2,063.42$	$9.58$
Within Groups (Error)	$12,064.64$	$56$	$215.44$
Total	$17,766.91$	$59$

Significance Decisions and Critical Values

Alpha Level: $\alpha = .05$
Critical Value ( $F_{crit}$ ): With $df_{\text{numerator}} = 1$ and $df_{\text{denominator}} = 56$ , the $F_{crit} = 4.02$ .
Note on Critical Values: The $F_{crit}$ is the same for all three effects in this example because the degrees of freedom for each effect ( $df = 1$ ) and the within-groups degrees of freedom ( $df = 56$ ) are identical. If the factors had different numbers of levels, the critical values would change.
Statistically Significant Findings:
- Fear of Dogs: 16.86 > 4.02 (Reject $H_0$ ).
- Therapy Participation: 0.03 < 4.02 (Fail to reject $H_0$ ; since F < 1.00, it is automatically non-significant).
- Interaction: 9.58 > 4.02 (Reject $H_0$ ).

Post-Hoc Testing and Interaction Deconstruction

Tukey’s HSD (Honestly Significant Difference):
- Main effects do not require Tukey's HSD if there are only two levels per factor.
- A significant interaction requires follow-up post-hoc tests to determine where specific differences lie among the means.
- In this sample, the HSD value was calculated as $|10.61|$ . Group mean differences must be greater than this value to be significant at $\alpha = .05$ .
Interaction Comparisons (Deconstruction):
- Comparison 1: $48.45 - 36.02 = 12.43$ (Significant, as 12.43 > 10.61).
- Comparison 2: $52.27 - 63.31 = -11.04$ (Significant, as |11.04| > 10.61).
- Comparison 3: $48.45 - 52.27 = -3.82$ (Not significant).
- Comparison 4: $36.02 - 63.31 = -27.29$ (Significant).
Rule of Comparison: When interpreting an interaction, always compare one level of a variable with the other level of that same factor. Do not compare diagonally. An interaction occurs when the effect of one factor depends on the level of the other factor.

Interpretation and Visualization

Qualifying Main Effects: A significant interaction "qualifies" a main effect. This means the main effect is present but does not tell the whole story; the results only make full sense when viewing the interaction.
Visualization: Interactions are easiest to interpret via line graphs. Non-parallel lines (converging, diverging, or crossing) indicate a potential interaction.

Reporting Results in APA Style

Example Text: "A two-way between-subjects Analysis of Variance (ANOVA) was conducted. The factors were therapy participation (i.e., whether someone had dog therapy or not) and fear of dogs (i.e., whether someone was scared of dogs or not). The dependent variable was stress level. The ANOVA revealed that there was not a significant effect for therapy participation, F(1, 56) < 1.00, but there was for fear of dogs, $F(1, 56) = 16.86$ , $p = .0001$ . People who were afraid of dogs ( $M = 57.79$ , $SD = xx$ ) had higher stress levels than those who were not afraid of dogs ( $M = 42.24$ , $SD = xx$ ). However, this main effect was qualified by a significant interaction between fear of dogs and therapy participation, $F(1, 56) = 9.58$ , $p = .003$ ."
Effect Size: Eta-squared ( $\eta^2$ ) should be included for all F-ratios. Cohen’s $d$ should be reported for post-hoc tests.
Conclusion Interpretation: The effect of dog therapy depends on how afraid a person is of dogs. For those not scared of dogs, therapy significantly lowers stress. For those scared of dogs, therapy participation actually results in significantly higher stress level. Therefore, dog therapy is recommended only for those who do not fear dogs.

Insight Activities & Discussion

Student Prompt: Describe the steps to calculate three F-ratios for a two-way ANOVA using SS, df, and MS.
Follow-up Question: Why would the $F_{crit}$ be the same for all effects?
- Answer: It is the same because the $df_{\text{effect}}$ and $df_{\text{within}}$ are identical for all tests in this specific design ( $1, 56$ ). This changes if factors have different numbers of levels.
Follow-up Question: Do we conduct Tukey’s HSD for main effects with 2 levels?
- Answer: No, post-hoc tests are only required if a factor has more than two levels.