Week 9: Factorial ANOVA III

Week 9 Overview: Factorial ANOVA

• This week's lecture covers:
  • Factorial ANOVA II recap
  • Follow-up on a significant interaction
  • Introduction to simple effect
  • Simple effect SS (Sum of Squares)
  • Simple effect in Stata
  • Simple effect summary
  • Follow-up on a significant main Effect (in Stata)
  • Results Report
  • Conclusions

Factorial ANOVA II Recap

• Concept + Omnibus Tests

Create Marginal and Cell Mean Table in Stata

• Stata command: tab IV1 IV2, summarize(DV)
  • Example: Using the 3 Costume Type × 2 Age example.
    • Marginal mean calculations (e.g., 11.2 = (5 + 12.4 + 16.2) / 3).
    • Main effect of age.
    • Main effect of costume type.
    • Interaction (e.g., (5 - 4.6) vs. (12.4 - 7.4) vs. (16.2 - 10.2)).

Generate Line and Bar Graphs in Stata

• Using the 3 Costume Type × 2 Age Example:
  • Line graph Stata commands:
    • anova DV IV1##IV2 (e.g., anova freq cos##age)
    • margins IV1#IV2 (e.g., margins cos#age)
    • marginsplot (marginsplot)
  • Bar graph Stata commands:
    • cibar DV, over(IV1 IV2) (e.g., cibar freq, over(age cos))
    • To place "cos" on the x-axis, position "cos" second in cibar DV, over(IV1 IV2) but first in margins IV1#IV2.
  • Only one graph is needed; choose the most appropriate one based on the research example.

Test Factorial ANOVA Assumptions in Stata

• Using the 3 Costume Type × 2 Age Example:
  1. Numeric DV.
  2. Independence of observations (between & within).
  3. Normality of DV for all combined levels of IVs:
     • Stata command: by IV1 IV2, sort: swilk DV
     • Check if all p-values are greater than 0.05.
     • Stata command: histogram DV, by(IV1 IV2)
  4. Equal variances for all combined levels of IVs:
     • Stata command: egen new_var = group(IV1 IV2), label
     • Stata command: robvar DV, by(new_var)
     • Check if p-value is greater than 0.05.

Running Factorial ANOVA in Stata

• Using the 3 Costume Type × 2 Age Example:
  • Stata command: anova DV IV1##IV2
  • Statistical hypotheses:
    • Main effect of age: H0: \mu1 = \mu_2
    • Main effect of costume: H0: \mu1 = \mu2 = \mu3
    • Interaction: H0: \mu{diff1} = \mu{diff2} = \mu{diff3}
  • Statistical significance:
    • F statistic + p-value
  • Effect size:
    • \eta^2 (eta-squared): \eta^2 = \frac{SS{effect}}{SS{total}}
    • \etap^2 (partial eta-squared): \etap^2 = \frac{SS{effect}}{SS{effect} + SS_{res}}
    • Stata command: estat esize
  • Degrees of freedom:
    • Costume (a levels): a-1
    • Age (b levels): b-1
    • Interaction: (a-1)(b-1)
    • Total: df_{total} = N - 1
  • F statistic: F = \frac{MS(effect)}{MS(residual)}

Analysis Steps in a Factorial ANOVA

• Similar to follow-ups of one-way ANOVA.
  • Omnibus F test.
  • Significant interaction?
    • Yes: Simple effect analysis (effect of the primarily interested IV at levels of the other IV).
    • No: Significant main effects?
      • Yes, #levels = 2: Interpret the main effect.
      • Yes, #levels > 2: Further comparing means (like week 6).
      • No: Stop.

Introduction to Simple Effect

Follow-up on a Significant Interaction

• When to conduct: After obtaining a significant interaction.
• What is a simple effect?
  • The effect of one IV at one level of the other IV.
    • The effect: Differences between the group means of one IV.
  • Remember interaction (in the 4 college year x 2 gender example)?
    • Assuming primary interest is gender.
    • RQ: Is the gender effect (or the difference between males’ and females’ mean confidence) the same across four college years?
    • Mathematically, comparing the four differences between male and female groups, i.e., (100-100) vs. (110-100) vs. (115-100) vs. (125-120)
    • A significant gender x college year interaction implies the four differences are not the same.
    • We need to dive deeper and find out whether there is a gender effect for each college year: freshmen, sophomores, juniors, and seniors – the simple effect of gender.
• The simple effect of ‘gender’ is the gender effect at each college year; RQ asked:
  • Gender@Freshman – Is there an effect of gender on freshmen (Mathematically, comparing 100 vs. 100)?
  • Gender@Sophomore – Is there an effect of gender on sophomores (Mathematically, comparing 100 vs. 110)?
  • Gender@Junior – Is there an effect of gender on junior (Mathematically, comparing 110 vs. 115)?
  • Gender@Senior – Is there an effect of gender on seniors (Mathematically, comparing 120 vs. 125)?
  • AKA breaking down / teasing apart the interaction.
• Using last week’s costume example, obtained a significant costume type × age interaction.
  • This interaction reflects that the age effect (the difference in injury between 2-4 and 5-8 years old) varies across the three levels of costume.
  • Mathematically, diff.(5-4.6), diff.(12.4-7.4), diff.(16.2-10.2) are statistically different.
  • The simple effect of age is the effect of age for children wearing each costume type:
    • Age@Mickey – Does age affect injuries for children wearing Mickey costumes? (Are 5 and 4.6 significantly different?)
    • Age@Superman – Does age affect injuries for children wearing Superman costumes? (Are 12.4 and 7.4 significantly different?)
    • Age@Batman – Does age affect injuries for children wearing Batman costumes? (Are 16.2 and 10.2 significantly different?)
  • We could also examine the simple effect of costume:
    • Costume@2-4 years – Is there a costume effect for younger children (2-4 years)? (Are 5, 12.4, and 16.2 significantly different?)
    • Costume@5-8 years – Is there a costume effect for children who are 5-8 years of age? (Is there a significant difference between 4.6, 7.4, and 10.2?)
  • Simple effect analysis breaks down / teases apart the interaction.
  • NB. In actual research, we don’t need to conduct simple effect analyses both ways (i.e., we don’t need to examine the simple effects of age AND simple effects of costumes).
  • Typically, examine the simple effect of the IV of primary interest.

Simple Effect Sum of Squares

The Source of Simple Effects

• DV: One numerical DV – children’s number of injuries in three months.
• IVs / Factors: 1) costume type; 2) age.
• Design: 3 (costume type: Superman, Batman, Mickey) × 2 (age: 2-4 years; 5-8 years) factorial design.
• RQ: Do costume type and age influence the frequency of children getting injured over three months?
  • Main effect of the costume type:
    • Does the frequency of children getting injured change as a function of the costumes they wear, ignoring their age?
    • H0: \mu{Mickey} = \mu{Superman} = \mu{Batman}
  • Main effect of age:
    • Does the frequency of injury between children of 2-4 years and 5-8 years differ, regardless of their costume?
    • H0: \mu{2-4} = \mu_{5-8}
  • Costume × Age interaction:
    • Is the age effect the same or different for children who wear different costumes?
    • H0: \mu{2-4.Mickey} - \mu{5-8.Mickey} = \mu{2-4.Superman} - \mu{5-8.Superman} = \mu{2-4.Batman} - \mu_{5-8.Batman}

Calculating the Sum of Squares

Simple Effect of Age

• 3 × 2 factorial design.
• SS{age@cos} = n\Sigma(\bar{x}{age.i@cos.j} – \bar{x}_{cos.j})^2
  • Where:
    • n = cell sample size
    • \bar{x}_{age.i@cos.j} = cell mean of age at a given costume level
    • \bar{x}_{cos.j} = marginal mean of a given costume level
  • SS_{age@Mickey} = 5 \times ((5.0 - 4.8)^2 + (4.6 - 4.8)^2) \approx 0.4
  • SS_{age@Superman} = 5 \times ((12.4 - 9.9)^2 + (7.4 - 9.9)^2) \approx 62.5
  • SS_{age@Batman} = 5 \times ((16.2 - 13.2)^2 + (10.2 - 13.2)^2) \approx 90.0

Additivity of Simple Effects

Simple Effect of Age

• 3 A × 2 B factorial design
• Sum(SS{age@cos.j}) = SS{age} + SS_{cos \times age} = 152.9

Calculating Sum of Squares II

Simple Effect of Costume

• SS{cos@age} = n\Sigma(\bar{x}{cos.j@age.i} – \bar{x}_{age.i})^2
  • Where:
    • n = cell sample size
    • \bar{x}_{cos.j@age.i} = cell mean of costume for a given age group
    • \bar{x}_{age.i} = marginal mean of a given age group
  • SS_{cos@2-4} = 5 \times ((5.0 - 11.2)^2 + (12.4 - 11.2)^2 + (16.2 - 11.2)^2) \approx 324.4
  • SS_{cos@5-8} = 5 \times ((4.6 - 7.4)^2 + (7.4 - 7.4)^2 + (10.2 - 7.4)^2) \approx 78.4
• NB. Typically, we only examine the simple effect of the primary interested IV.

Additivity of Simple Effects II

Simple Effect of Costume

• Sum(SS{cos@age.i}) = SS{cos} + SS_{cos \times age} = 402.8

Simple Effect Analysis in Stata

• Two common commands in Stata are used:
  • contrast IV1@IV2, effects mcompare(method)
  • pwcompare IV1#IV2, effects mcompare(method)
• Using Last Week’s Example Again … 3 COSTUME TYPE × 2 AGE
  • DV: One numerical DV – children’s number of injuries in three months
  • IVs / Factors: 1) costume type; 2) age
  • Design: 3 (costume type: Superman, Batman, Mickey) × 2 (age: 2-4 years; 5-8 years) factorial design
  • RQ: Do costume type and age influence the frequency of children getting injured over three months?
    • Main effect of the costume type
      • Does the frequency of children getting injured change as a function of the costumes they wear, ignoring their age?
      • H0: \mu{Mickey} = \mu{Superman} = \mu{Batman}
    • Main effect of age
      • Does the frequency of injury between children of 2-4 years and 5-8 years differ, regardless of their costume?
      • H0: \mu{2-4} = \mu_{5-8}
    • Costume × Age interaction
      • Is the age effect the same or different for children who wear different costumes?
      • H0: \mu{2-4.Mickey} - \mu{5-8.Mickey} = \mu{2-4.Superman} - \mu{5-8.Superman} = \mu{2-4.Batman} - \mu_{5-8.Batman}

Simple Effect in Stata: contrast

• contrast IV1@IV2, effects mcompare(method)
  • Examine the effect of IV1 at every level of IV2.
  • IV1 represents the factor of primary interest (in our overall research question).
  • Need to control the type I error rate for simple effect analysis since more than one set of comparisons/contrasts is conducted.
  • 'method' can be 'bon' or 'sch' – the principles for using the method are similar to one-way ANOVA follow-up analysis.

Simple Effect of Age

• contrast age@cos, effects mcompare (bon)
  • When only two group means are compared (or single df comparison).
  • Same information in F and t tables.

Simple Effect of Age Contd.

• contrast age@cos, effects mcompare (bon)

Simple Effect of Age Contd.

• Putting these together:
  • Simple effect of age on Mickey: No
  • Simple effect of age on Superman: Yes, 5-8 yrs < 2-4 yrs
  • Simple effect of age on Batman: Yes, 5-8 yrs < 2-4 yrs

Effect Size

• Contrast, or difference between the two means From the omnibus F-test table
  • Here, we also use Cohen’s d for the effect size of a simple effect
  • For simple effect: Cohen’s d = (\frac{x1 - x2}{\sqrt{MS_{Within}}})
  • Example: the effect size of the simple effect of age at Batman level: Cohen’s d = \frac{-6}{\sqrt{6.4666667}} \approx -2.36
  • In Stata: display -6/sqrt(6.4666667)
  • Cohen’s (1988) effect size rule-of-thumb for Cohen’s d
    • 0.2 (small effect); 0.5 (medium effect); 0.8 (large effect)

Simple Effect of Costume

• contrast cos@age, effects mcompare (bon)
  • When more than two group means are compared (df = 2 in the F table)
  • Different information in F and t tables

Simple Effect of Costume Contd.

• contrast cos@age, effects mcompare (bon)
  • Akin to two one-way ANOVA
  • Can we interpret these F-statistics directly?
    • They inform if there are significant differences among the three different costume types for each age group, without providing further comparisons.
  • Is there a simple effect for costumes for 2-4-year-olds? – Compare the means of the three cells (5 vs. 12.4 vs. 16.2)
  • Is there a simple effect for costumes for 5-8-year-olds? – Compare the means of the three cells (4.6 vs. 7.4 vs. 10.2)

Simple Effect of Costume Contd.

• contrast cos@age, effects mcompare (bon)
  • Two sets of comparisons for each age group
  • Compare every other group to the base (the lowest level); However, it doesn’t compare Superman vs. Batman
  • The t-statistics table conducts additional comparisons (like the secondary comparison after an omnibus F)

Simple Effect of Costume Contd.

ALTERNATIVE CONTRASTS

• contrast a.cos@age, effects mcompare (bon)
  • Comparing two adjacent levels, i.e., 1 vs. 2, 2 vs. 3

Simple Effect of Costume Contd.

CUSTOMIZED CONTRASTS

• Assume we want to compare: (1) Superhero (Batman + Superman) vs. Mickey; (2) Batman vs. Superman for each age group

Simple Effect in Stata: pwcompare

• pwcompare IV1#IV2, effects mcompare(method)
  • Compare all possible pairs of cell means
  • Can use it to look at the effect of IV1 at each level of IV2, and the effect of IV2 at each level of IV1
  • Generally, conduct more sets of comparisons than the ‘contrast’ command
  • Need to control for the type I error rate for these multiple sets of comparisons
  • 'method' can be 'tukey', 'bon' or 'sch' – principles for use are similar to one-way ANOVA follow-up analysis

Pairwise Comparisons

• pwcompare cos#age, effects mcompare (tukey)
  • A@B + B@A compare every two cell means; #comparisons= k(k-1)/2
  • P-values were adjusted to control the type I error rate for all 15 comparisons
  • Simple effect of age for Mickey group ?
  • Simple effect of cos (i.e., Batman vs. Mickey) for 2-4-yrs group

Pairwise Comparisons Contd.

• pwcompare cos#age, effects mcompare (tukey)

Simple Effect Summary

• Analysis steps in a factorial ANOVA
• Simple effect: the effect of one IV at one level of the other IV
• Typically, only look at the simple effect of one IV (of primary interest) at the levels of the other IV
• In Stata: ‘contrast’ or ‘pwcompare’

A Research Example

• A person’s perceived distance from a location can be affected by the actual spatial distance to that location, but by psychological factors as well.
• Maglio and Polman (2014, Study 1) sought to examine whether a person’s spatial orientation (facing toward a location vs. facing away from it) would also affect its perceived distance.
• 2 (orientation: toward, away from) × 4 (station: Spadina, St. George, Bloor-Yonge, Sherbourne) design

A Research Example Contd.

RESULTS

• We carried out a 2 (orientation: toward, away from) × 4 (station: Spadina, St. George, Bloor- Yonge, Sherbourne) analysis of variance (ANOVA) on closeness ratings, which revealed no main effect of orientation, F < 1, and a main effect of station, F(3, 194) = 24.10, p < .001, \eta_p^2 = .27.
• This main effect was qualified by the predicted interaction between orientation and station, F(3, 194) = 16.28, p < .001, \eta_p^2 = .20.
• We decomposed this interaction by comparing the subjective-distance ratings between participants traveling east and west for each of the four subway stations.
• Westbound participants rated the stations to the west of Bay Street as closer than did eastbound participants; this effect was obtained for both the station one stop to the west (St. George, p < .001, \etap^2 = .28) and the station two stops to the west (Spadina, p = .001, \etap^2 = .20).
• The opposite pattern held true for stations to the east of Bay Street. Eastbound participants rated the stations to the east of Bay Street as closer than did westbound participants; this effect was obtained for both the station one stop to the east (Bloor-Yonge, p = .053, \etap^2 = .08) and the station two stops to the east (Sherbourne, p < .001, \etap^2 = .24).
• Figure 1 summarizes these results.

## Analysis steps in a factorial ANOVA

• Very similar to the follow-ups of one-way ANOVA

Follow up on a sig. main effect

IF THE FACTOR OF THE MAIN EFFECT HAS >2 LEVELS

• Two common commands in Stata:
  • contrast IV, effects mcompare(method)
  • pwcompare IV, effects mcompare(method)

Main Effect Follow-up: contrast

• contrast IV, effects mcompare (method)
  • Compare every other group to the base (the lowest level); However, it doesn’t compare Superman vs. Batman

Main Effect Follow-up: contrast Contd.

• contrast IV, effects mcompare (method)
  • Compare every other group to the base (the lowest level); However, it doesn’t compare Superman vs. Batman

Main Effect Follow-up: contrast

ALTERNATIVE

• contrast a.IV, effects mcompare (method)
  • Comparing two adjacent levels, i.e., 1-2, 2-3

Main Effect Follow-up: contrast

CUSTOMIZED CONTRASTS

• We compare:
  • (1) Superhero (Batman + Superman) vs. Mickey;
  • (2) Batman vs. Superman

Follow-up on the main effect: pwcompare

• pwcompare cos, effects mcompare(tukey)

Follow-up on the main effect: effect size

• Calculate Cohen’s d for contrast effect size
  • For Batman vs. Superman
  • Cohen’s d = (\frac{3.3}{\sqrt{6.5}}) \approx 1.29
  • In Stata: display 3.3/sqrt(6.5)
  • Cohen’s (1988) effect size rule-of-thumb (for Cohen’s d)
    • 0.2 = small effect; 0.5 = medium effect; 0.8 = large effect

Summary & Results Reporting

• Back to Our Research Example
• Suppose we are interested in whether the effect of age differs for children who wear different types of costumes
• Sometimes, the follow-up for a main effect is omitted in the results report if it is not part of the effect of interest to researchers.

Conclusions

• Simple effect follows a significant interaction (“tease apart” the interaction)
• Simple effect refers to the effect of one factor at one level of the other factor
• Is there an effect (difference of cell means) of one factor at each level of the other factor
• E.g., is there an effect of A at B1, B2, or Bj level, respectively?
• E.g., is there an effect of B at A1, A2, or Ai level, respectively?
• Although we can perform and interpret the simple effects from both perspectives (as mentioned above), typically, only one perspective is needed
• When our IVs only have two levels, we can interpret the results straight away
• When our IVs have more than two levels, a follow-up analysis comparing pairs of means is needed
• There are different ways of conducting simple effect analysis in Stata; choose how you perform the analysis wisely (based on the research question)

Lecture Learning Outcomes

• After this week’s lecture, you know:
  • Main effects and interaction in factorial ANOVA
  • Simple effect in factorial ANOVA
  • How to perform the simple effect analysis and interpret its results
  • How to interpret the main effect of a factor with only two levels
  • How to perform the secondary analysis for a sig. main effect of a factor with more than two levels and interpret the results
  • The analysis procedure (when and how to run a follow-up analysis for the interaction and main effects)
• In Stata, you should be able to:
  • Create a summary table summarizing cell and marginal means
  • Test assumptions of factorial ANOVA
  • Run a multiple-way ANOVA analysis
  • Create a line/bar graph to help you interpret the effects
  • Conduct the simple effect analysis using various Stata options provided
  • Perform a follow-up analysis (comparisons of means) for a significant main effect with IV > 2 levels using various Stata options provided
  • Create and save a .do file for your commands (syntax)