Experimental Designs with Multiple Independent Variables: Factorial Analysis, Interactions, and Real-World Applications

Single-Factor Experimental Designs versus Factorial Designs

  • Single-Factor Experimental Designs

    • Evaluate the relationship between a single independent variable (IV) and a dependent variable (DV) to test for simple differences across experimental conditions or interventions.
    • Okonofua et al. (2016) Study 3: Evaluated school disciplinary approaches by testing a punishment-oriented, zero-tolerance mindset against an empathic, restorative discipline approach among middle school math teachers.
      • Independent Variable: Teacher training condition (2 levels: Empathy mindset intervention vs. Control training).
      • Intervention details: The empathy mindset module reminded teachers that adolescent social and biological changes cause insecurity, and that punishment causes stress, overreaction, and school disengagement; it prompted teachers to reflect on students' desire to feel heard and respected. The control module focused on classroom technology applications.
      • Dependent Variable: Percentage of students suspended over the subsequent academic year.
      • Results: Students of teachers in the empathy mindset group had a 4.6%4.6\% suspension rate compared to 9.8%9.8\% in the control group (evaluated as a simple difference: empathy minus control).
    • Carrier & Pashler (1992): Evaluated the memory benefits of retrieval practice compared to passive study.
      • Independent Variable: Study condition (manipulated as a within-groups variable with 2 levels: Pure study vs. Test trial / retrieval practice).
      • Materials: 40 English-Yupik vocabulary word pairs from Sivuqaq (St. Lawrence Island) inhabitants in northwestern Alaska (e.g., AGLUK - JAW, MESUQ - JUICE).
      • Procedure: In the pure study condition, a computer displayed the complete word pair for 10 seconds10\,\text{seconds}. In the test trial condition, the computer displayed the Yupik word alone for 5 seconds5\,\text{seconds}, followed by the full pair for 5 seconds5\,\text{seconds}. Memory was tested 1 day later.
      • Dependent Variable: Memory accuracy (number of words recalled on Day 2).
      • Results: Participants recalled an average of 3.93.9 words in the pure study condition and 4.64.6 words in the retrieval practice condition (evaluated as a simple difference: retrieval practice minus pure study).

  • Introduction to Factorial Designs

    • Definition: An experimental design in which two or more independent variables (also termed factors) are crossed, creating conditions (cells) that represent every possible combination of the levels of each independent variable.
    • Design Notation: Factorial designs are written as level1×level2×⋯×leveln\text{level}_{1} \times \text{level}_{2} \times \dots \times \text{level}_{n}.
      • The number of numbers indicates the total number of independent variables.
      • The value of each individual number indicates the number of levels for that specific independent variable.
      • Multiplying the numbers together calculates the total number of experimental conditions or cells in the study.
      • Example: A 2×22 \times 2 factorial design crosses two independent variables with two levels each, producing 2×2=42 \times 2 = 4 unique cells.

Understanding Interaction Effects

  • Core Concepts of Interactions

    • Definition: An interaction effect (or interaction) occurs when the effect of one independent variable on a dependent variable depends on the level of another independent variable.
    • Mathematical Definition: An interaction represents a "difference in differences." It tests whether the magnitude or direction of the difference between levels of one IV changes across levels of a second IV.

  • Intuitive Types of Interactions

    • Crossover Interaction
      • Graphical Representation: Visible on a line graph as lines that cross each other, having slopes with opposite signs (one positive, one negative).
      • Linguistic Phrasing: Described using the phrase "it depends."
      • Example (Food Temperature Preference): Evaluates food temperature preference (2×22 \times 2 design: Food Type [Ice Cream vs. Pancakes] ×\times Temperature [Cold vs. Hot]; DV = Liking rating).
        • People prefer ice cream cold rather than hot (cold minus hot equals a positive value).
        • People prefer pancakes hot rather than cold (cold minus hot equals a negative value).
        • Conclusion: How much a person likes a food temperature depends on which food is being served.
    • Spreading Interaction
      • Graphical Representation: Visible on a line graph as nonparallel lines that diverge or spread apart without crossing.
      • Linguistic Phrasing: Described using the phrase "especially when."
      • Example (Sandwich Ingredients): Evaluates sandwich deliciousness (2×22 \times 2 design: Bacon [No Bacon vs. Bacon] ×\times Avocado [No Avocado vs. Avocado]; DV = Rated deliciousness).
        • When there is no bacon, adding avocado leads to a small increase in deliciousness.
        • When bacon is present, adding avocado leads to a substantial increase in deliciousness.
        • Conclusion: Sandwiches are more delicious with bacon or avocado alone, but taste especially wonderful when both ingredients are combined.

  • Real-World and Theoretical Examples of Interactions

    • Leadership Style Evaluation (Williams & Tiedens, 2016): Evaluated public liking of leaders based on gender and behavior (2×22 \times 2 design: Leader Gender [Male vs. Female] ×\times Leader Behavior [Dominant vs. Non-dominant]). Liking depends on leader behavior: people tend to like male leaders who act dominant, but do not like female leaders who display dominant behavior.
    • Daycare and Child Development (Burchinal et al., 2010): Evaluated social and intellectual development of children based on care environment. The developmental impact of daycare depends on care quality: high-quality daycare provides social and intellectual development benefits compared to parental care alone, whereas poor-quality daycare fails to provide these benefits.

Manipulated Variables, Participant Variables, and Testing Limits

  • Manipulated Variables versus Participant Variables

    • Manipulated Variables: Independent variables assigned directly by researchers through experimental control (e.g., training modules, study methods, stimulus timing).
    • Participant Variables: Measured variables whose levels are selected based on inherent participant characteristics (e.g., age, gender, ethnicity, self-esteem). Although participant variables are measured rather than manipulated, they are analyzed as independent variables in factorial designs.

  • Testing Limits and External Validity

    • Combining manipulated and participant variables allows researchers to test the external validity and generalizability of a causal relationship across different populations or situations.
    • Okonofua et al. (2022): Evaluated whether the empathy mindset training generalized across student racial and ethnic categories (2×22 \times 2 design: Teacher Training [Empathy vs. Control] ×\times Student Ethnicity [Black or Hispanic vs. Not Black or Hispanic]; sample of 66 middle school math teachers and 5,822 students).
      • Empathy Mindset / Not Black or Hispanic: 14.9%14.9\% suspension rate.
      • Control Training / Not Black or Hispanic: 15.9%15.9\% suspension rate.
      • Empathy Mindset / Black or Hispanic: 20.8%20.8\% suspension rate.
      • Control Training / Black or Hispanic: 26.5%26.5\% suspension rate.
      • Conclusion: The empathy intervention reduced suspensions for all student groups, but showed a much stronger effect for Black and Hispanic students. Student ethnicity moderated the intervention's impact.
    • Batman Effect Study (White et al., 2017): Evaluated task persistence on a boring computer task across age groups using perspective-taking interventions (2×22 \times 2 design: Age [4-year-olds vs. 6-year-olds] ×\times Perspective Condition ["Hardworking hero" Batman perspective vs. Third-person self-reflection perspective]).
      • Procedure: Children in the Batman condition asked themselves, "Is Batman working hard?" Children in the third-person condition asked themselves, "Is [child's name] working hard?"
      • 4-year-olds persistence: Batman perspective = 32.5%32.5\%, Third-person = 26.5%26.5\% (Difference = 6.0%6.0\%).
      • 6-year-olds persistence: Batman perspective = 57.0%57.0\%, Third-person = 44.0%44.0\% (Difference = 13.0%13.0\%).
      • Conclusion: Although 6-year-olds persisted longer overall, the performance benefit of adopting the Batman perspective was comparable across both age groups. The absence of a statistically significant interaction confirms that the "Batman effect" generalizes across these childhood age levels.

  • Interactions as Moderators

    • In factorial terminology, a moderator is an independent variable that modifies the direction or strength of the relationship between another independent variable and a dependent variable.
    • Every moderator effect constitutes an interaction effect, indicating that the impact of one IV depends on the level of the moderating IV.

Analyzing and Interpreting Factorial Results (2×22 \times 2 Designs)

  • Main Effects and Marginal Means

    • Main Effect: The overall effect of a single independent variable on the dependent variable, averaging across all levels of the other independent variable(s). In a 2×22 \times 2 design, there are two main effects to evaluate.
    • Marginal Means: The arithmetic means for each individual level of an independent variable, calculated by collapsing across the levels of the second independent variable.
      • If sample sizes in each cell are equal, marginal means are computed as simple arithmetic averages.
      • If cell sample sizes are unequal, marginal means are computed using weighted averages that weight larger cell sizes more heavily.
    • Roediger & Karpicke (2006) Main Effects Analysis (2×22 \times 2 Memory Study):
      • Data Matrix (Percentage of ideas recalled):
        • Pure Study (SSSS\text{SSSS}): 5 min5\,\text{min} delay = 83%83\%, 1 week1\,\text{week} delay = 40%40\%.
        • Retrieval Practice (STTT\text{STTT}): 5 min5\,\text{min} delay = 71%71\%, 1 week1\,\text{week} delay = 61%61\%.
      • Learning Condition Marginal Means: Pure Study (SSSS\text{SSSS}) = 61.5%61.5\% (average of 83%83\% and 40%40\%); Retrieval Practice (STTT\text{STTT}) = 66%66\% (average of 71%71\% and 61%61\%).
      • Final Test Delay Marginal Means: 5 min5\,\text{min} delay = 77%77\% (average of 83%83\% and 71%71\%); 1 week1\,\text{week} delay = 50.5%50.5\% (average of 40%40\% and 61%61\%).
      • Statistical Findings: The main effect of Final Test Delay (77%77\% vs. 50.5%50.5\%) was statistically significant (its 95%95\% confidence interval did not include zero; F(1,174)=122.53F(1, 174) = 122.53). The main effect of Learning Condition (61.5%61.5\% vs. 66%66\%) was not statistically significant (its 95%95\% confidence interval included zero; F(2,174)=2.32,η2=.03F(2, 174) = 2.32, \eta^2 = .03).
    • Okonofua et al. (2022) Main Effects Analysis:
      • Student Ethnicity Marginal Means: Not Black/Hispanic = 15.4%15.4\%; Black/Hispanic = 23.7%23.7\% (Statistically significant main effect).
      • Teacher Training Marginal Means: Empathy mindset = 17.3%17.3\%; Control = 20.4%20.4\% (Statistically significant main effect).

  • Calculating and Estimating Interaction Effects

    • Table Method (Row Difference Analysis):
      1. Compute the difference between levels of the second IV at the first level of the first IV.
      2. Compute the difference between levels of the second IV at the second level of the first IV.
      3. Subtract one difference score from the other to calculate the difference in differences.
      • Roediger & Karpicke (2006) Row Calculation: Pure study difference = 83−40=4383 - 40 = 43; Retrieval practice difference = 71−61=1071 - 61 = 10. Difference in differences = 43−10=3343 - 10 = 33 points (F(2,174)=18.48,m2=.18F(2, 174) = 18.48, m^2 = .18; statistically significant interaction).
    • Table Method (Column Difference Analysis):
      1. Compute the difference between levels of the first IV at the first level of the second IV.
      2. Compute the difference between levels of the first IV at the second level of the second IV.
      • Roediger & Karpicke (2006) Column Calculation: 5 min5\,\text{min} delay difference = 83−71=1283 - 71 = 12; 1 week1\,\text{week} delay difference = 40−61=−2140 - 61 = -21. Difference in differences = 12−(−21)=3312 - (-21) = 33 points.
    • Batman Study Table Calculation: 4-year-olds difference = 32.5−26.5=6.032.5 - 26.5 = 6.0; 6-year-olds difference = 57.0−44.0=13.057.0 - 44.0 = 13.0. Inferential statistical testing confirmed the difference between 6.06.0 and 13.013.0 was not statistically significant, establishing no interaction.
    • Graphical Detection: Plotted as a line graph, nonparallel lines or diverging slopes indicate the presence of an interaction effect, whereas parallel lines indicate the absence of an interaction effect.

  • Describing Interactions in Words

    • Simple Effects Approach: Focus on the first level of the independent variable on the x-axis, describe the change across levels of the second independent variable, then move to the next level on the x-axis and describe the change.
      • Example: "When the memory test occurs after 5 minutes5\,\text{minutes}, the pure study group had better memory than the retrieval practice group; but after 1 week1\,\text{week}, retrieval practice is better than pure study."
    • Key Phrases Approach: Frame the finding using specific conditional phrasing such as "especially for", "only when", or "depends on."
      • Example: "The empathy mindset intervention reduced suspension rates, especially for students who are Black or Hispanic."
      • Example: "The effectiveness of the learning condition depends on whether people are tested right away or 1 week1\,\text{week} later."

  • Hierarchy of Effects

    • When an interaction effect and main effects are present in the same dataset, the interaction effect is almost always the most important finding.
    • A main effect averages across conditions and can obscure or conceal the true pattern of results identified by a significant interaction.

Advanced Variations in Factorial Designs

  • Participant Assignment Variations

    • Independent-Groups (Between-Subjects) Factorial Design: Both independent variables are manipulated using independent groups (different participants serve in each cell of the design).
      • Okonofua et al. (2022): Independent teachers were assigned to training conditions, and independent student groups represented ethnic classifications.
      • Roediger & Karpicke (2006): Required four distinct participant groups. Assigning 3030 participants per cell in a 2×22 \times 2 between-subjects design required a total sample size of 30×4=12030 \times 4 = 120 participants.
    • Within-Groups (Repeated-Measures) Factorial Design: Both independent variables are manipulated as within-groups (the exact same group of participants experiences all cells in the design).
      • Efficiency: Requiring 1212 observations per cell in a 2×22 \times 2 within-groups design requires only 1212 total participants (compared to 4848 in an independent-groups design).
      • Requirement: Demands experimental counterbalancing of conditions to eliminate order effects and protect internal validity.
    • Mixed Factorial Design: One independent variable is manipulated as independent-groups, while another independent variable is manipulated as within-groups.
      • Roediger & Karpicke (2006, Study 1): Learning condition was manipulated within-groups (participants studied one text passage via pure study and another passage via retrieval practice), while final test delay was manipulated between-groups (5 min5\,\text{min} vs. 1 week1\,\text{week} delay). Assigning 3030 participants per cell required 6060 total participants (3030 in the 5 min5\,\text{min} delay condition and 3030 in the 1 week1\,\text{week} delay condition, each tested across both learning conditions).

  • Increasing Levels of an Independent Variable

    • 3×23 \times 2 Factorial Design: Combines one independent variable with 3 levels and a second independent variable with 2 levels, creating 3×2=63 \times 2 = 6 distinct cells.
      • Roediger & Karpicke (2006) Full Study: Crossed Learning Condition (SSSS\text{SSSS} [repeated study], SSST\text{SSST} [single test], STTT\text{STTT} [repeated test]) with Final Test Delay (5 min5\,\text{min}, 1 week1\,\text{week}). Assigning 3030 subjects per cell yielded 180180 total subjects (9090 tested after 5 min5\,\text{min}, 9090 tested after 1 week1\,\text{week}).
      • Results Matrix (% of ideas recalled):
        • SSSS\text{SSSS}: 5 min=83%5\,\text{min} = 83\%, 1 week=40%1\,\text{week} = 40\%
        • SSST\text{SSST}: 5 min=77%5\,\text{min} = 77\%, 1 week=55%1\,\text{week} = 55\%
        • STTT\text{STTT}: 5 min=71%5\,\text{min} = 71\%, 1 week=61%1\,\text{week} = 61\%
    • 3×43 \times 4 Factorial Design: Combines one independent variable with 3 levels and a second independent variable with 4 levels, creating 3×4=123 \times 4 = 12 distinct cells.
      • Illustration (Learning Condition [SSSS\text{SSSS}, SSST\text{SSST}, STTT\text{STTT}] ×\times Test Delay [5 min5\,\text{min}, 2 days2\,\text{days}, 1 week1\,\text{week}, 1 year1\,\text{year}]):
        • 5 min5\,\text{min} delay: SSSS=83%\text{SSSS} = 83\%, SSST=77%\text{SSST} = 77\%, STTT=71%\text{STTT} = 71\%
        • 2 days2\,\text{days} delay: SSSS=52%\text{SSSS} = 52\%, SSST=60%\text{SSST} = 60\%, STTT=67%\text{STTT} = 67\%
        • 1 week1\,\text{week} delay: SSSS=40%\text{SSSS} = 40\%, SSST=55%\text{SSST} = 55\%, STTT=61%\text{STTT} = 61\%
        • 1 year1\,\text{year} delay: SSSS=15%\text{SSSS} = 15\%, SSST=35%\text{SSST} = 35\%, STTT=40%\text{STTT} = 40\%

  • Increasing the Number of Independent Variables (Three-Way Designs)

    • 2×2×22 \times 2 \times 2 Factorial Design: Incorporates three independent variables, each with two levels, creating 2×2×2=82 \times 2 \times 2 = 8 experimental conditions/cells.
    • Okonofua Extension Example: Teacher Training (Empathy vs. Control) ×\times Student Ethnicity (Black/Hispanic vs. Not Black/Hispanic) ×\times Student Gender (Boys vs. Girls); DV = Student suspension rate.
    • Data Structure:
      • Boys Cells: Empathy/Not Black = 14%14\%, Empathy/Black = 20%20\%, Control/Not Black = 16%16\%, Control/Black = 28%28\%
      • Girls Cells: Empathy/Not Black = 1%1\%, Empathy/Black = 4%4\%, Control/Not Black = 2%2\%, Control/Black = 6%6\%
    • Main Effects in a Three-Way Design (Tests 3 simple overall differences, averaging across the other two variables):
      • Main Effect of Teacher Training: Empathy Mindset = 9.75%9.75\% (average of 14%14\%, 20%20\%, 1%1\%, 4%4\%); Control = 13%13\% (average of 16%16\%, 28%28\%, 2%2\%, 6%6\%).
      • Main Effect of Student Ethnicity: Not Black/Hispanic = 8.25%8.25\% (average of 14%14\%, 16%16\%, 1%1\%, 2%2\%); Black/Hispanic = 14.5%14.5\% (average of 20%20\%, 28%28\%, 4%4\%, 6%6\%).
      • Main Effect of Student Gender: Boys = 19.5%19.5\% (average of 14%14\%, 16%16\%, 20%20\%, 28%28\%); Girls = 3.25%3.25\% (average of 1%1\%, 2%2\%, 4%4\%, 6%6\%).
    • Two-Way Interactions in a Three-Way Design (Evaluates 3 distinct two-way interactions):
      1. Training Condition ×\times Student Gender (averaging over student ethnicity): Empathy Boys = 17%17\%, Empathy Girls = 2.5%2.5\%; Control Boys = 22%22\%, Control Girls = 4%4\%.
      2. Training Condition ×\times Student Ethnicity (averaging over student gender): Empathy Not Black = 7.5%7.5\%, Empathy Black = 12%12\%; Control Not Black = 9%9\%, Control Black = 17%17\%.
      3. Student Gender ×\times Student Ethnicity (averaging over training condition): Boys Not Black = 15%15\%, Boys Black = 24%24\%; Girls Not Black = 1.5%1.5\%, Girls Black = 5%5\%.
    • Three-Way Interaction Definition:
      • Determines whether the two-way interaction between two independent variables depends on the level of a third independent variable ("the difference in differences is different").
      • A three-way interaction is present if the pattern of a two-way interaction on one side of a graph differs from the pattern of the two-way interaction on the other side of the graph (e.g., a strong two-way interaction exists for boys, but no two-way interaction exists for girls).
      • If the exact same two-way interaction pattern is present at both levels of the third independent variable, there is no three-way interaction.

Identifying Factorial Designs in Scientific Literature and Popular Media

  • Empirical Journal Articles

    • Method Section: Authors explicitly describe design parameters using standard notation (e.g., "a 3×23 \times 2 between-subjects design", "a 2×2×22 \times 2 \times 2 factorial design") and specify whether variables are between-groups or repeated-measures.
    • Results Section: Reports inferential statistics evaluating main effects and interactions, specifying FF-statistics, degrees of freedom, pp-values (p<0.05p < 0.05), effect sizes (η2\eta^2, m2m^2), and 95%95\% confidence intervals (95%95\% CI).
    • Sample Journal Description (Roediger & Karpicke, 2006, p. 253): "The results… were submitted to a 2×32 \times 3 ANOVA, with retention interval (5 min5\,\text{min} or 1 week1\,\text{week}) and learning condition (SSSS\text{SSSS}, SSST\text{SSST}, or STTT\text{STTT}) as independent variables. This analysis revealed a main effect of retention interval, F(1,174)=122.53,2.41F(1, 174) = 122.53, 2.41, indicating that forgetting occurred. The effect of learning condition was only marginally significant, F(2,174)=2.32,n2=.03F(2, 174) = 2.32, n^2 = .03. However, these effects were qualified by a significant Learning Condition ×\times Retention Interval interaction, F(2,174)=18.48,m2=.18F(2, 174) = 18.48, m^2 = .18."

  • Popular Media Reports

    • Journalists typically omit standard mathematical notation (such as "2×22 \times 2 factorial design") to simplify reading, but leave structural linguistic clues.
    • Linguistic Indicators: Look for conditional phrases like "it depends" or "only when", or discussions incorporating participant variables alongside manipulated variables.
    • Media Example 1 (Cherry, 2019): Reported on music and memory (2×22 \times 2 design: Music Type [Neutral vs. Positive] ×\times Musical Training [Musically Trained vs. Musically Naive]; DV = Performance on learning tests). Used "it depends" to describe the interaction: musically trained students performed better with neutral music, whereas musically naive students learned better with positive music.
    • Media Example 2 (Andics et al., 2016; Howard, 2016 CNN report): Reported on a 2×22 \times 2 within-groups MRI study evaluating canine brain processing of speech. Dogs wore dog-sized headphones in an MRI machine.
      • IV 1 (Lexical Content): Praise words ("well done", "good boy", "clever") vs. Neutral/meaningless words ("even if", "although", "however").
      • IV 2 (Intonation): Praising intonation vs. Neutral intonation.
      • DV: Activity in the brain's reward center.
      • Reporting: Used "only when" to describe the interaction effect, noting that the brain's reward center lit up "like a Christmas tree" only when a praise word was spoken to the dogs in a praising tone of voice.
    • Media Example 3 (Stein, 2009): Reported on Joanne Wood et al. (University of Waterloo) study on self-esteem (3232 male, 3636 female psychology students; DV = Mood).
      • Participant IV: Self-esteem (High vs. Low).
      • Manipulated IV: Repeating positive self-statements ("I am a lovable person").
      • Reporting: Noted that repeating the phrase failed to improve mood for participants with low self-esteem—they actually felt worse—and "the gap between those with high and low self-esteem widened" (signaling a spreading interaction).
    • Media Example 4 (Manis, 2022 in PsyPost): Reported on Okonofua's empathy mindset intervention, noting: "Results showed that the empathic-mindset intervention did reduce suspension rates and mitigated racial disparities in suspension. The reduction rate was greatest for Black and Hispanic students."