Within-Participants Designs: Advantages vs. Disadvantages
Key advantage: estimate and remove variance due to individual differences.
Results in a smaller error term in the F-test (smaller SS{error}, smaller MS{error}, larger F-value).
More powerful; more likely to find a significant effect if it exists.
Fewer participants needed to detect the same sized effect compared to between-participants design.
Disadvantage: experiencing one condition may affect scores in other conditions.
Differences may emerge due to order effects rather than real effects of IV. Types include:
Practice effects: Performance improves in later conditions.
Fatigue effects: Performance deteriorates in later conditions.
Sensitization effects: Greater sensitivity/reactivity in later conditions.
Carryover effects: Previous conditions set a standard for later conditions.
Counterbalancing: Randomly allocate participants to complete conditions in different sequences to deal with order effects.
Not always possible (e.g., pre-post intervention studies).
Mixed Factorial Designs
Contain at least one between-participants (BP) factor and one within-participants (WP) factor.
BP factor: different people in each condition.
WP factor: same people in each condition.
Focus on two-way mixed designs: one fixed BP factor, one fixed WP factor.
A fully within-participants design can technically be considered to have elements of a “mixed” design. The participant factor is just a random effect that we want to account for.
However a study design is only considered a “mixed design” if it contains both and factor(s).
Mixed Designs
The BP factor is independent within levels of the fixed between-participants factor.
Interaction with the fixed within-participants factor is independent within levels of the BP factor (each participant only participates in one condition of the BP factor).
The WP factor is correlated within levels of the fixed between-participants factor.
Interaction with the fixed within-participants factor is correlated with the BP factor because each participant participates in every condition of the WP factor.
The interaction is nested within levels of the fixed between-participants factor. We can get all possible crossed combinations of the participant factor and the WP factor but not the BP factor.
Two-Way Mixed ANOVA
Involves three omnibus tests:
Main effect of BP factor
Main effect of WP factor
WP x BP interaction
Main effect of BP factor: Compares marginal DV means of the two between-participants conditions, averaging over the within-participants factor.
Main effect of WP factor: Compares marginal DV means of the three within-participants conditions, averaging over the between-participants factor.
Interaction: Compares the cell DV means of one factor at each level of the other factor to see whether the effect of that factor changes at each level of the other factor.
Two-way mixed ANOVA partitions variance in the DV into: variance attributable to the different conditions/levels of the between-participants factor, within-participants factor, systematic variation in DV resulting from changes in the effect of the WP factor across levels of the BP factor, and Variation in DV attributable to individual differences.
*
Sum of Squares
Total sum of squares: SS_{total}. How much does each individual score/observation vary from the grand mean?
BP factor sum of squares: SS_{BPF}. How much does each BP condition mean vary from the grand mean?
WP factor sum of squares: SS_{WPF}. How much does each WP condition mean vary from the grand mean?
WP factor x BP factor interaction sum of squares: SS_{WPF\times BPF}. How much more does each cell mean vary from the grand mean after accounting for the additive effects of the WP factor and BP factor?
Participants within BP groups sum of squares: SS_{participants}. How much does each participant’s mean vary from the mean of their own BP condition?
WP factor x participant interaction sum of squares: SS_{WPF\times participants}. How much more does each individual score vary from the grand mean after accounting for all the effects above?
Degrees of Freedom
Total degrees of freedom: df_{total}. N − 1 = bwn − 1
BP factor degrees of freedom: df_{BPF}. b − 1
WP factor degrees of freedom: df_{WPF}. w − 1
WP factor x BP factor interaction degrees of freedom: df{WPF \times BPF}. df{WPF} × df_{BPF} = (w − 1)(b − 1)
Participants within BP groups degrees of freedom: df_{participants}. b × (n − 1)
WP factor x participant interaction degrees of freedom: df{WPF \times participants}. df{WPF} × df_{participants} = w − 1 × b × (n − 1)
Where: b = number of levels of BP factor, w = number of levels of WP factor, n = number of participants per BP condition, N = total number of observations.
Calculating F-Ratios
Step 1: Calculate SS and df required to determine:
Variance due to main effect of BP factor: MS_{BPF}
Variance due to main effect of WP factor: MS_{WPF}
Variance due to WP factor x BP factor interaction: MS_{WPF\times BPF}
Variance due to participant individual differences within BP groups: (MS_{participants})
Variance due to WP factor x participant interaction: (MS_{WPF\times participants})
Step 2: Calculate F-ratios:
F = \frac{MS{BPF}}{MS{participants}}
F = \frac{MS{WPF}}{MS{WPF\times participant}}
F = \frac{MS{WPF\times BPF}}{MS{WPF\times participant}}
Note that the error term for the Main effect of the BP factor is Participants within groups whereas the error term for both the Main effect of the WP factor and WPF x BPF Interaction is the Time x Participant interaction
Advantages of Mixed Designs
Combining strengths and offsetting limitations of between-participants and within-participants designs.
Testing interactions between WP and BP factors.
Determine whether WP effects differ across BP groups.
Mirror real-world scenarios.
Choosing the most appropriate way to test each factor.
Appropriate factors to test between-participants:
participant characteristics
intervention conditions
Appropriate factors to test within-participants:
stimulus/task characteristics
time effects
pre-post effects
Calculating and removing error variance due to individual differences.
Fewer participants needed compared to fully between-participants designs.
Including a between-participants factor can help illuminate the potential influence of order/sequencing effects when counterbalancing is not possible. Randomly assign people to a control condition to examine “natural” variations in DV due to potential order effects to be confident that any WP effects seen are valid.