Mixed Factorial ANOVA Notes

Learning Outcomes

• Describe the purpose of mixed factorial ANOVA and circumstances for its selection.
• Distinguish between interactions and main effects in mixed factorial ANOVA.
• Interpret simple effects (post-hoc tests) to describe interactions between IVs.
• Describe the assumptions of mixed factorial ANOVA, methods of evaluation, and how to identify if each assumption is satisfied or violated.
• Conduct a mixed factorial ANOVA in Jamovi and interpret the output.
• Interpret key statistics for mixed factorial ANOVA (omnibus tests, post-hoc comparisons, simple effects plots).
• Report results of mixed Factorial ANOVA in APA format.

Overview of Mixed Factorial ANOVA

• Mixed factorial ANOVA is a statistical test used to examine the effects of two or more IVs on a DV, where at least one IV is a between-groups factor and at least one is a repeated measures factor.

Between-Group IV

• Different groups of participants are exposed to different levels of this factor.
• Each participant experiences only one level of this IV.
• Example: Randomly allocating one group to CBT and another to general counseling.

Repeated Measures IV

• All participants are exposed to all levels of the factor.
• Example: Measuring all participants at three time points: baseline, after two weeks, and after six weeks.

Mixed Factorial Designs

• Combine both between-groups and repeated measures IVs.
• Example: Randomly allocate participants to CBT or general counseling (between-groups IV), and measure anxiety symptoms at baseline (Time 1), after 2 weeks (Time 2), and after 6 weeks (Time 3).

Benefits of Mixed Factorial Designs (and ANOVA)

• Allows us to measure the effect of a between-groups IV and to test if different groups change in the same way across repeated-measures conditions.
• Example: A new education intervention might increase student grades compared to a traditional curriculum.
• Test as a between-groups factor by randomly allocating students to the new intervention or traditional curriculum.
• Measuring student grades at multiple time points allows observation of changes in performance over time (repeated-measures factor).
• Determine not only if the new intervention is more effective overall but also if its impact develops differently throughout the year compared to the traditional curriculum.

Main Effects and Interactions in Mixed Factorial ANOVA

• Allows us to test both the main effect of our IVs and the interaction between them.

Main Effects

• Between-Groups IV: Tells us whether there is a significant difference in mean scores between the groups, ignoring the repeated measures IV.
  • Takes the mean of the DV for each group, averaging across all levels of the repeated measures IV, and tests if there are differences between the groups.
  • Example: Group 1 different to Group 2.
• Repeated Measures IV: Tells us whether there is a significant difference in mean scores between repeated measures conditions, ignoring the between-groups IV.
  • Takes the mean of the DV for each repeated measures condition, averaging across all levels of the between-groups IV, and tests if there are differences between conditions.
  • Example: Are Time 1, Time 2, and Time 3 different?
• If significant main effects are found, post-hoc tests can identify where the differences are.
  • Conducted if specific hypotheses require testing or if the interaction isn't significant.
  • Significant interaction can make the main effects uninterpretable.

Interaction Effect

• Tells us if the effect of one of the IVs is different at different levels of the other IV.
• If a significant interaction is found, follow up with simple effects analyses to identify where the differences are.
  • Driven by your hypotheses.
  • Two ways to think of the interaction:
    • Does the effect of the repeated measures IV differ at different levels of the between-groups IV?
      • Does the dependent variable change across repeated measures conditions (e.g., across time points)?
      • Is the pattern of change over time the same for each group? (comparing scores across the rows)
    • Does the effect of the between-groups IV differ at different levels of the repeated measures IV?
      • Are there differences between groups?
      • Are these differences the same at each time point? (comparing down the columns)

Mixed Factorial ANOVA - Study Notes

Introduction to Mixed Factorial ANOVA

• Combines between-groups and repeated measures analyses.
• Researchers can examine both group differences and changes over time within a single design.
• Valuable when separating participants into different groups while also measuring them at multiple time points.

Review: Between-Groups vs. Repeated Measures ANOVA

• Between-Groups ANOVA
  • Involves separating participants into distinct groups and testing differences in mean scores between these groups.
  • Example: Anxiety treatment study with a control condition, CBT treatment, and generalized counseling.
    • Group 1: Control condition
    • Group 2: CBT treatment
    • Group 3: Generalized counseling
  • Allows comparing whether CBT and generalized counseling result in lower anxiety scores than the control condition, and whether one treatment is more effective than the other.
• Repeated Measures ANOVA
  • Examines differences in conditions experienced by the same participants over time.
  • Using one group of participants, measure them at multiple time points.
  • Example:
    • Time 1: Baseline
    • Time 2: Three weeks into treatment
    • Time 3: Six weeks into treatment
  • Shows whether participants' average scores on a dependent variable (like anxiety symptoms) change across time.

Mixed Factorial ANOVA Design

• Combines both approaches by including both a between-groups factor and a repeated measures factor.
• Typical design:
  • Separate participants into different treatment groups
  • Measure each group at multiple time points
• Example: Three groups (control, CBT, generalized counseling) all measured at baseline, three weeks, and six weeks.
• Allows comparison of group differences and how groups change over time, and whether there are differences in the patterns of change.

Advantages of Mixed Factorial Design

• Enables more sophisticated comparisons than simple between-groups or repeated measures designs.
• Determine whether:
  • Control group symptoms decrease simply due to time passing
  • CBT group symptoms decrease to a greater degree than control
  • Generalized counseling is as effective as CBT or falls somewhere between CBT and control
• Allows examination of changes in detail by comparing both groups and how they differ across time, leading to more nuanced research questions and conclusions.

Main Effects in Mixed Factorial ANOVA

• Examines each independent variable separately, ignoring the other variable.

Main Effect of Between-Groups Variable

• Calculate an average score for each group across all time points.
• Gives you a single score per group that ignores time differences.
• Example: If Group 1 (control) has a higher average than Group 2 (CBT), this suggests control participants have higher anxiety overall than CBT participants.

Main Effect of Repeated Measures Variable

• Calculate average scores for each time point across all groups.
• Shows whether there are overall changes across time, regardless of group membership.
• Example: Little change from Time 1 to Time 2, but a significant drop at Time 3, suggesting that change occurs after six weeks of treatment for everyone.

Interaction Effects

• The most valuable aspect of mixed factorial ANOVA is examining whether the two factors interact.
• An interaction occurs when the effect of one independent variable depends on the level of the other independent variable.

Types of Interaction Analysis

1. Changes Across Time for Each Group
   • Examines whether each group shows different patterns of change over time:
     • Group 1 (control): Do anxiety symptoms change from baseline to 3 weeks to 6 weeks?
     • Group 2 (CBT): Do anxiety symptoms change across the same time points?
     • Group 3 (counseling): Do anxiety symptoms change across the same time points?
   • Example: The control group shows no change, CBT shows a significant decrease, and counseling shows a moderate decrease.
2. Group Differences at Each Time Point
   • Compares groups at each specific time point:
     • Time 1 (baseline): Are groups different before treatment begins?
     • Time 2 (3 weeks): Are groups different after initial treatment?
     • Time 3 (6 weeks): Are groups different after extended treatment?
   • Baseline comparisons are particularly important in randomized controlled trials to ensure groups start equivalent.

Choosing Analysis Approach

• Driven by your research hypotheses and questions.
• Mixed factorial ANOVA provides the flexibility to test either type of interaction.
• Key advantage: Ability to combine between-groups and repeated measures IVs for more sophisticated comparisons.

When Would You Use a Mixed Factorial ANOVA?

• To test questions such as:
  • Whether a repeated-measures intervention leads to different results for different groups of people.
    • Example: Is there a significant reduction in anxiety symptoms after 3-weeks and 6-weeks of CBT, compared to baseline, and is the degree of change the same for participants with and without a diagnosis of Generalised Anxiety Disorder (GAD)?
    • This would be a 3(time point: baseline, 3 -weeks, 6-weeks) X 2(Group: GAD, no GAD) Mixed Factorial design.
  • Does a dependent variable change across time in the same way for people in different experimental conditions?
    • Example: Do anxiety symptoms change between baseline and a 2-weeks follow up test for people in a control group and for people receiving CBT?
    • This would be a 2(time point: baseline, 2-weeks) X 2(Treatment: Control, CBT) Mixed Factorial design.

Types of Data Suitable for Mixed Factorial ANOVA

• The DV must be continuous.
• There must be at least one between-groups IV that is categorical.
• There must be at least repeated-measures IV that includes two or more time- points.

Assumptions of Mixed Factorial ANOVA

Normality of Residuals

• Factorial models assume that the residuals of the model are normally distributed.
• Evaluate by visually evaluating a normal Q-Q plot.
• ANOVA is robust to violations of normality, particularly when sample sizes are large.
• Severe departures from normality can affect the validity of the ANOVA results.
• Consider transforming individual variables or using an alternative statistical test that does not assume normality ,if you encounter this.

Homogeneity of Variance

• Key assumption for mixed factorial ANOVA.
• Refers that the variances of the different groups are approximately equal.
• For mixed factorial designs, check this at each level of the repeated measure variable meaning checking if the variance is equal at each time point in the study.
• A Levene's test result is obtained for each time point.

Interpreting Levene's test:

• If p-value is less than (<) .05, the variance between groups is significantly different, violating the assumption of homogeneity.
• If the p-vale is greater than (>) .05, the variance between groups is not significantly different, satisfying the assumption of homogeneity.

Sphericity

• Refers to the assumption that the variability in differences between time-points is equal.
• If there are 3 times points in our study, we want to know that the variability in differences between time 1 and time 2, time 1 and time 3, and time 2 and time 3 is approximately equal.
• Similar to homogeneity of variance, but refers to variance in differences scores between time points, rather than the variance in scores for different groups.
• Mauchly's test is used to check the assumption of sphericity.
  • p-values less than < .05, indicates that the variance in difference scores is significantly different, and the assumption of sphericity is violated.
  • p-values greater than > .05, indicates that the variance in difference scores are not significantly different, and the assumption of sphericity is satisfied.
• If the assumption is violated, apply a correction to the ANOVA omnibus test results.

Corrections for Violation of Sphericity

• Jamovi provides Greenhouse-Geisser correction and a Huynh-Feldt correction.
• To decide which one to use, look at the epsilon ({varepsilon}) value for the Greenhouse-Geisser in the test of sphericity table.
  • If ({varepsilon}) < .75, choose Greenhouse-Geisser correction.
  • If ({varepsilon}) > .75, use the Huynh-Feldt correction.
• The assumption of sphericity is always met if your repeated measures IV only has two levels.
• If there are three or more levels for our repeated measures IV, we need to check the output for this test.

Interpreting the Output and Reporting the Results

Omnibus tests

• The omnibus test results are divided into two different tables.
  • The first provides the omnibus test (F tests) for the main effect of the repeated measures factor, and the interaction between the repeated measured and between groups factors.
  • The second provides the omnibus test results for the between groups factor.
• Tables provide the following statistics for the main effects and interaction effect
  • ANOVA, the degrees of freedom for the effect (df1) and the degrees of freedom for the residuals (df2) are both reported.
  • The value and significance of the F statistic.
    • If the value is significant, it indicates there is a difference somewhere among the groups.
  • The effect size for the main effects and interaction is provided in the form of partial eta square (\eta^2_p), which tells us how much variance in the DV the effect accounts for.

Within subjects effects table

• Provides results for the main effect of the repeated measures IV (within subjects is another term for repeated measures), as well as the interaction between the between groups and repeated measures IV.
  • Provides the statistics for the main effect of the repeated measure IV time.
    • The main effect of time is significant and accounts for 49% of variance in satisfaction ratings, F(2, 20) = 9.67, p = .001, \eta^2_p = .49.
  • Provides the statistics for the interaction between the repeated measure and between-groups IVs.
    • The interaction between time and participant group is significant and accounts for 72% of variance in satisfaction ratings, F(2, 20) = 26.26, p < .001, \eta^2_p = .72.

Between subjects effects table:

• Provides the results for the main effect of the between groups IV (between subjects is another term for between groups), as well as the interaction between the between groups and the repeated measures IV.
  • Provides the statistics for the main effect of the between-groups IV.
    • The main effect of group is significant and accounts for 38% of variance in satisfaction ratings, F(1, 10) = 6.15, p = .033, \eta^2_p = .38.

Simple effects

• If a significant interaction is found, follow this up with pairwise comparisons to understand the interaction.

Post hoc tests

• Provides the statistics for the pairwise comparisons we need to explore the interaction effects, in the form of a t-tests.
• A correction should be used to adjust for the inflated Type 1 errors (e.g., Tukey correction).
  • Table will provide every possible group comparison. Not all of these will be relevant to your research question, and so you should interpret/report only those that are.
• The t-test results indicating if the difference between groups is significant.
  • Report as t(df) = X.XX, p = .XXX.
• Unlike between-subjects factorial ANOVA, we are not provided with Cohen's d for a mixed factorial ANOVA, so be sure to report the descriptive statistics (Ms, SDs) so readers can get a sense of the magnitude of each difference.

Reporting Observations across Time Points and Groups

• For the group of customers, satisfaction ratings at Time 2 were significantly higher than ratings at Time 1, t(10) = -5.242, p = .003. Satisfaction at Time 3 was significantly lower than Time 2, t(10) = 5.22, p < .004, but was not any different to Time 1 t(10) = 0.99, p = .910
• For the group of employees, satisfaction ratings at Time 2 were significantly lower than ratings at Time 1, t(10) = 7.05, p < .001. Satisfaction at Time 3 was not different to Time 2, t(10) = -0.84, p = .954, but was significantly lower than Time 1 t(10) = 4.37, p = .013

Simple effects Plot

• Provides a visual illustration of the simple effects.
• For customers, satisfaction levels increase from Time 1 to Time 2, and then decrease again at Time 3.
• For employees, satisfaction levels decrease from Time 1 to 2 and then remain low at Time 3.

Summary

• Both time and group had significant main effects on satisfaction, however more importantly, there is a significant interaction between time and group. This means the changes in satisfaction ratings across time that occurred was different employees than it was for customers.
• The simple effects found that for customers of the retail company, their satisfaction levels significantly increase at Time 2 compared to Time 1, and then decreased back to baseline levels at Time 3. This means customers were much more satisfied with the company after it introduced a "customer is always right policy", and their satisfaction levels significantly dropped after this policy was abolished.
• A different pattern was observed for employees of the company. For these participants, satisfaction dropped significantly after the introduction of the new policy (Time 1 compared to Time 2), and then remained low even after the policy was abolished (Time 3 was not different to Time 2, and both Time 2 and 3 were significantly lower than Time 1).

Descriptive statistics

• When reporting the results of any kind of ANOVA, you should report the means and standard deviations for each group at each level of the repeated measures IV.
• Can obtain these statistics and plot using the Exploration > Descriptives function in Jamovi.
• You should always report descriptive statistics, however it's especially important in situations like this in which we do not have effect sizes for each of the simple-effects pairwise comparisons.
• Provides the Ms and SDs gives the reader a general sense of the magnitude of the differences reported

Reporting results

General guidelines for reporting results of a mixed factorial ANOVA

• Report the results of the omnibus tests for both the main effects and interaction. The statistics are reported in the format of: F(df1, df2) = X.XX, p = .XXX e.g., F(1, 193) = 20.31, p <.001.
• If the interaction is significant, report the simple effects and specify if a correction was applied to them (e.g., Bonferonni, Tukey). Report each of the key comparisons that relate to your hypotheses. Include descriptive statistics (M, SD) for each group, if they are significantly different or not and the direction of the difference. E.g., At Time 1, Group 1 (M = X.XX, SD = X.XX) had significantly higher scores than Group 2 (M = X.XX, SD = X.XX), t(X) = X.XX, p =.XXX.
• If the interaction is not significant, do not run simple effects. Instead, report the results of post -hoc comparisons for each main effect (describe if any corrections were made, provide descriptive stats, t-test results, and effect size for each post-hoc comparison). We generally only report these post-hoc tests if the interaction is not significant.

Mixed Factorial ANOVA in Jamovi - Study Notes

Study Context

• The analysis examines customer satisfaction changes in a retail company across three time points related to a "customer is always right" policy.
• Time 1 represents baseline, Time 2 follows policy introduction, and Time 3 occurs after policy abolishment.
• Two groups were studied: customers and employees of the retail company.

Variables Structure

• The design includes a repeated measures independent variable (time) with three levels and a between-groups factor (group) with two levels.
• Time serves as the within-subjects factor while group represents the between-subjects factor.
• This creates a mixed factorial design combining both repeated measures and independent groups elements.

Setting Up the Analysis in Jamovi

• Navigate to Analyses tab, select ANOVA, then choose repeated measures ANOVA option.
• Rename the repeated measures factor from "RM factor 1" to something meaningful like "time" and label the levels appropriately (Time 1, Time 2, Time 3).
• Drag the corresponding data variables into the repeated measures cells boxes.
• Move the between-groups variable into the between subjects factor box to complete the model specification.

Output Configuration Options

• Under effect size, select partial eta squared to determine the proportion of variance explained by each effect.
• Enable assumption checks including sphericity tests, homogeneity tests, and QQ plots.
• If sphericity is violated, corrections can be applied using Greenhouse-Geisser or Huynh-Feldt options.
• Configure post-hoc tests for main effects and simple effects analysis when interactions are significant.

Assumption Testing Results

• Mauchly's test of sphericity examines whether variability in difference scores between repeated measures is consistent.
  • A non-significant result (p = .563) indicates the sphericity assumption is satisfied.
• Levene's test checks homogeneity of variance between groups at each time point, with non -significant results indicating equal variances.
• The QQ plot of residuals shows normal distribution when points align closely with the predicted line, confirming multivariate normality.

Main ANOVA Results

• The analysis produces two tables: one for repeated measures effects and interactions, another for between-groups main effects.
  • The main effect of time was significant (F = 9.76, p = 0.001, partial {eta}^2 = 49\%), indicating satisfaction changed significantly across time points.
  • The interaction between time and group was also significant (F = 26.26, p < 0.01, partial {eta}^2 = 72\%), suggesting different change patterns for customers versus employees.
  • The main effect of group was significant (F = 6.15, p = 0.033, partial {eta}^2 = 38\%), showing overall differences between customer and employee satisfaction levels.

Simple Effects Analysis for Customers

• Post-hoc comparisons reveal specific patterns for customer satisfaction changes.
  • From Time 1 to Time 2, satisfaction increased significantly by 3.33 points (t = -5.42, p = 0.03), indicating the policy introduction boosted customer satisfaction.
  • From Time 2 to Time 3, satisfaction decreased significantly by 4.17 points (t = 5.22, p = 0.04), showing satisfaction dropped when the policy was removed.
  • Comparing Time 1 to Time 3 showed no significant difference (t = 0.99, p = 0.910), indicating satisfaction returned to baseline levels after policy removal.

Simple Effects Analysis for Employees

• Employee satisfaction showed a contrasting pattern.
  • From Time 1 to Time 2, satisfaction decreased significantly by 4.33 points (t = 7.05, p < 0.01), indicating the policy introduction negatively affected employee satisfaction.
  • From Time 2 to Time 3, there was no significant change (t = -0.84, p = 0.954), meaning satisfaction remained low after policy removal.
  • Comparing Time 1 to Time 3 showed significantly lower satisfaction at Time 3 by 3.76 points (t = 4.37, p = 0.013), indicating lasting negative effects on employee satisfaction.

Visual Interpretation

• The interaction plot displays customer satisfaction starting around 3.5, increasing to approximately 7 at Time 2, then returning to about 3.5 at Time 3.
• Employee satisfaction begins around 6, drops dramatically to below 2 at Time 2, and remains low at Time 3.
• This visualization clearly shows the opposing effects of the policy on the two groups.

Practical Implications

• The policy created a temporary benefit for customers but caused lasting damage to employee satisfaction.
• While customer satisfaction returned to baseline after policy removal, employee satisfaction remained significantly lower than initial levels.
• This suggests the policy undermined employee faith in the company with persistent negative effects.
• The findings recommend against implementing such policies due to their detrimental impact on staff satisfaction and potential long- term consequences for company performance.

Reporting Requirements

• Complete results reporting requires means and standard deviations for each group at each time point, obtainable through the descriptives function in Jamovi.
• These descriptive statistics correspond to each point on the interaction plot and provide essential information for comprehensive result presentation.
• The analysis should include assumption test results, main effects, interaction effects, and relevant simple effects comparisons with appropriate effect sizes and significance levels.

Assessment Update

• End of semester test:
  • Opens 9am June 13th, close 9am June 14th
  • Content cover: Weeks 1-12
  • Format:
    • 60 MCQs (covering all weeks)
    • 1 SAQ covering Week 1-4 (research methods)
    • 2 SAQs covering Week 5-12 (output interpretation)
  • Time limit: 2 hours + 10 mins reading time
  • Weighting: 40%

Tips for the Test

• Open-book test, but study!
• Be familiar with the content.
• There isn't enough time to look up everything.
• Responses to SAQs must be in your own words: Do not cut and paste from outside sources.
• Practice good time management
  • Set a clock to alert you 15 minutes before the deadline
  • If you get stuck on a question, move on and you can come back later
• Check computer and internet access - if needed, book a computer or work-space on campus, in a public library, or using a friend/relatives wifi.
• Turn off all distractions

If you have IT problems on the day:

• Log back into the system - Your test time accounts for these types of minor problems.
• If you can't get back into the test, take a screen shot, photo, or video of your screen as evidence and notify your unit coordinator (Natalia Albein-Urios) right away by email.
• IT support will only be available until 4pm on Friday.
• Teaching staff will only be available until 5pm on Friday.

ANOVA Overview

• Analysis of Variance (ANOVA) are a family of analyses used to test differences between three or more means
  • Between-groups ANOVAS = test differences in means of different groups
  • Repeated-measures ANOVAS = tests differences between means scores from same group of participants measured at multiple time-points.

Mixed Factorial ANOVA

• Factorial ANOVAs include more than one IV
  • Mixed factorial ANOVA includes at least one…
    • Between Groups IV
    • Repeated Measures IV
• Allows us to evaluate:
  • Main effects - the effect of each IV on its own
  • Interactions - if the effect of one IV changes at different levels of the other IV

Review

An experiment compared the memory ability of science students and arts students. The participants were all tested under three conditions in which they had to memorise and recall different types of words (complex English words, simple English words, Latin words). This design would be described as:

• C. 2 x 3 mixed factorial design

What is a main effect in mixed factorial ANOVA?

• a. The effect of one IV ignoring the other IVs

What does it mean when there is an interaction in mixed factorial ANOVA?

• d. The effect of one IV changes over levels of another IV

Sphericity

Is the assumption of sphericity satisfied?

• No, the assumption of sphericity is not satisfied.

Would you use a Greenhouse-Geisser or Huynh-Feldt correction to address this violation?

• If GG \epsilon \le.75 \rightarrow use GG correction
• If GG \epsilon > .75 \rightarrow use HF correction
• Huynh-Feldt correction.