Accounting for Individual Differences

Individual Differences and Experimental Research

• Individual differences present challenges to experimental researchers.
• ANOVA is used when the IV has 3 or more levels (groups).
• Multiple comparisons increase the chance of Type I error, so the alpha level is adjusted using:
  • Modified Bonferroni Correction (manual).
  • SPSS post-hoc tests (automatic).
• Contrast tests allow comparing multiple groups to multiple groups.
• Between subjects ANOVA tests compare differences between within-groups and between-groups variance.

Sources of Error

• Main sources of error:
  • Individual differences.
  • Errors in control (justified or unforeseen).
  • Errors in measurement:
    • Systematic (e.g., wrong settings).
    • Random (e.g., unreliable instrument).
• Errors in control and measurement should be minor due to good research design.
• Individual differences are the main source of error and can interfere with analyses when they exist between groups.

Individual Differences

• Examples:
  • Age, gender, sexual identity.
  • Personality, psychological constructs.
• Reducing individual differences:
  • Select a more homogenous group of subjects (not always easy or desirable).
  • Standardizing procedures.
  • Training subjects sufficiently.
• These strategies are relatively ineffective compared to other approaches.

Minimizing Individual Differences

• Experimental designs differ in how they deal with individual differences:
  • Between subjects (independent groups, between subjects ANOVA).
  • ANCOVA - ANalysis of COVAriance (for between subjects designs).
  • Within subjects (no independent groups, repeated measures ANOVA).

Repeated Measures ANOVA

Repeated Measures Designs

• Key feature: how they deal with individual differences.
  • Between subjects (independent groups, between subjects ANOVA).
  • ANCOVA - ANalysis of COVAriance (for between subjects designs).
  • Within subjects (no independent groups, repeated measures ANOVA).
• Effectively eliminate group non-equivalence.
• Individual differences are managed because all individuals are together in the same group.
• Time-related threats may be introduced.

Example: Meditation and Headache Duration

• Fake data for a study on the impact of meditation on headache duration per week.
• Analyzed from two design perspectives:
  • Between subjects using an independent groups ANOVA.
  • Within subjects using a repeated measures ANOVA.
• The purpose is to explain where individual differences are in each design and what can be done to manage their influence on our findings.

Between Subjects Data

• DV: headache duration (hours/week).
• Participants meditated regularly for 1, 2, 3, 4, or 5 weeks.

• IV: Week
• The starting point is the differences between these means.
• F = \frac{Between \, Groups \, Variance}{Within \, Groups \, Variance}
• F = \frac{Effect + Individual \, Differences}{Individual \, Differences}

Independent Groups ANOVA Output

• A between subjects design must accept individual differences being part of both between and within groups variance.
• Sum of Squares (SS) represent the amount of variance of each type, between and within groups.
• Between groups SS = 1291.440, Within Groups SS = 451.200.

Individual Differences in Between Groups Design

• Despite having to accept some level of individual differences influence, there are still issues and solutions.
  • Issue: Group non-equivalence due to individual differences contributes to Between Groups Variance.
    • Could mean any IV effect we observe is actually due to individual differences (Type I Error).
    • Solution: Randomisation to distribute individual differences, but groups may still differ on average.
  • Issue: Inflated Within Groups Variance due to individual differences being more extreme in some/across groups.
    • Could violate assumptions, could lead to non-significance when IV actually had impact (Type II Error).
    • Solution: Attempt to exert more experimental control, standardise processes etc.
    • Unlikely to completely account for individual differences.

Removing Individual Differences

• From Between Groups Variance?
  • Not completely…
• From Within Groups Variance?
  • Not completely…
• Completely…?
  • Repeated Measures design removes individual differences by putting all participants in all conditions.
  • No different groups, no chance for individual differences to be a problem… or is there?

Change of Terminology

• Independent Groups ANOVA => Between Groups, Within Groups
• Repeated Measures ANOVA => Between Levels, Within Levels

Repeated Measures Data

• Measure headache duration at weekly intervals while engaging in regular meditation.
• Individual differences cannot explain the differences between levels (weeks) because each week has the same people!

Between Levels Variance

• Could be due to:
  • Errors in control
  • Errors in measurement
  • Effects of IV/Treatment
  • But not individual differences.

Within Levels Variance

• Due to:
  • Errors in control.
  • Errors in measurement
  • Individual differences.
  • But not the IV/Treatment.
• Participants may vary in reaction to IV/treatment within a level, but this is an individual difference!
• Having the same participants in all conditions removes individual differences from between levels but not from within levels variance. This is a problem.

Balance

• Between Groups ANOVA calculation:
  • F = \frac{Between \, Groups \, Variance}{Within \, Groups \, Variance} = \frac{Effect + Individual \, Differences}{Individual \, Differences}
  • The equation is balanced as individual differences (and error) exist on each side, allowing us to obverse the effects of the IV.
• Repeated Measures ANOVA calculation:
  • F = \frac{Between \, Levels \, Variance}{Within \, Levels \, Variance} = \frac{Effect}{Individual \, Differences}
  • The equation is not balanced as individual differences exist only on the Within Levels side, interfering with our ability to obverse the effects of the IV.
• To restore balance we must somehow remove individuals differences from Within Levels, which we do statistically.

Repeated Measures ANOVA Output

• The Repeated Measures ANOVA output has a table with the stats for the Within Levels individual differences.
• SPSS calculates this and accounts for it during the ANOVA calculation, meaning the individual differences are removed… our equation is balanced again!

Assumptions of Repeated Measures ANOVA

• Independence.
• Scale data.
• Normality.
• Homogeneity of Variance.
• Sphericity.

Sphericity

• The variance of the difference scores between any two conditions is the same as the variance of the difference scores between any two other conditions.
• Any two conditions should have similar variance in difference scores to any other combination of two conditions.

• Difference scores: Cond 1 – Cond 2

• Difference scores: Cond 2 – Cond 3

• Difference scores: Cond 1 – Cond 3

Test for Sphericity

• If Mauchly’s W is statistically significant (p < .05) then the data departs from sphericity.
• Like most assumption tests, understanding the output that indicates meeting or violating the assumption is the most important aspect.

Interpreting the Overall Repeated Measures ANOVA

• Read the Sphericity Assumed line if the assumptions of Sphericity is met.
• If Sphericity is violated, generally look at Huynh-Feldt line for violation-adjusted results.

Follow-Ups and Effect Sizes

• Post-hoc and planned comparisons available.
• Eta squared as a measure of variance in DV explained by IV.
• Overall ANOVA effect size.

Evaluation of Repeated Measures Designs

• Between Levels:
  • Effectively removed individual differences.
  • Introduces new confounds (order effects).
  • NOTE: Counterbalancing may not be sufficient if order effects are asymmetrical (like our weeks of mediation).
• Within Levels:
  • Can statistically removes largest source of error variance (individual differences).
  • Effectively makes the design more sensitive to detecting the effects of the IV (more powerful and therefore fewer participants are required).

Evaluation of Repeated Measures Designs

• Assumptions
  • Assumptions are more complex.
  • Complex assumptions are hard to meet. The risk of a Type 1 error is raised.
  • Epsilon adjustments can be punishing.
  • Implications for statistical power (significance more difficult to observe after adjustment).

One-way ANCOVA

One-Way Analysis of Covariance (ANCOVA)

• We are moving back to independent groups for this analysis (not every situation can use repeated measures).
• Experimentalists are interested in the “typical person”. To them individual differences are the biggest source of error variance (they are annoying!).
• Our objective is to remove individual differences from our analysis to better observe the impact of the IV.

The F-Ratio (Between Subjects Design)

• F = \frac{Between \, Groups \, Variance}{Within \, Groups \, Variance}
• Between Groups Variance: The means differ across conditions.
• Within Groups Variance: Individual scores differ within each condition.
• To meet our objective, we must address individual differences in both these sources of variance.

Addressing Individual Differences

• Independent Groups (Between Subjects) Design
  • Random allocation.
  • Match groups?
    • Potentially introduces bias.
• Repeated Measures (Within Subjects) Design
  • Same participants in each condition.
    • Introduces confound: order effects.
• Between Groups/Levels Variance.
  • Concerned about systematic bias.

Addressing Individual Differences

• Independent Groups (Between Subjects) Design
  • Standardise procedures.
  • Give practice.
  • Lots of error can still remain.
• Repeated Measures (Within Subjects) Design
  • Individual differences are subtracted statistically.
    • A potentially more powerful design.
• Within Groups/Levels Variance.
  • Concerned about error variance. The treatment becomes difficult to detect.

Removing Individual Differences in Between Subjects Designs

• We make use of a technique that is used by those seeking to explain individual differences: partial correlation.
• Partialling through regression: The use of correlations between variables to control for the influence of a covariate.
• Covariate: A variable (usually an individual difference) other than the IV that is known to be correlated with the DV.
• If the covariate isn’t correlated with the DV, it is unlikely to impact your observation of IV and DV interaction.
• Effectively, the covariate could provide a rival explanation to your IV effecting DV explanation.
• Adding a covariate also makes it harder to observe significance, so we want to be sure it is correlated.
• Measurement of the covariate should happen before the experiment, as you don’t want the IV to impact the covariate in any way.
• You don’t need to do any correlating yourself, it is all done automatically through SPSS.

ANCOVA

• ANCOVA ‘controls’ for the influence of the covariate.
• The influence of the covariate is controlled by holding the covariate ‘constant’.
• Holding the covariate constant means that in the analysis the impact of the covariate is equalised across participants.
• All participants are ‘the same’ in terms of the covariate. The covariates influence has been accounted for.
• One way to explain what the ANCOVA is doing could be: “If everyone had the same covariate score, then the IV impacts the DV in this way.”
• ANCOVA does not account for all individual differences, just the ones we measure. These are usually the largest known influences or nuisances in a given situation (larger correlation between covariate and DV).
• Through measurement and partial correlation we are controlling variables that cannot be controlled experimentally!

Example: Teaching Methods and Spelling Ability

• Fake data for a study looking at the impact of teaching methods on spelling ability.
• We will look at this data from two analysis perspectives:
  • Using an independent groups ANOVA.
  • Using an independent groups ANCOVA.
• The purpose of this is observe the impact that removing the individual differences can have in a between subjects design.

Example of Independent Groups ANOVA

• DV: Spelling Ability

• IV: Teaching Method

Output of Independent Groups ANOVA

• A non-significant result, could explained by:
  1. The manipulation/IV is ineffective (teaching method is ineffective).
  2. There was not enough statistical power to detect significance (sample not large enough to be confident in this effect size).
  3. Individual differences are a rival explanation which we did not control for!

Introducing a Covariate

• One potential rival explanation for student’s individual differences in spelling ability AND reaction to teaching methods could be their “Verbal IQ”.
• We can safely assume Verbal IQ is correlated with our DV of spelling ability.
• Let’s make Verbal IQ our covariate…
• This allows us to see if our IV (teaching method) impacts on our DV (spelling ability) when Verbal IQ is controlled for – statistically removed from the picture!
• We need to have measured Verbal IQ before any teaching has taken place – teaching could increase Verbal IQ*!
• at least temporarily

Example of Independent Groups ANCOVA

• DV: Spelling Ability
• COVARIATE: Verbal IQ

• IV: Teaching Method

Variability in Spelling Scores for Method One

• The within groups variability is calculated from the sum of the squared deviations of each score from the mean.

Each Participant has a score on the DV as well as Covariate

• Regression line (line of best fit)

Our ANCOVA Output

• Now the impact of our teaching method IV is statistically significant, F(2, 20) = 4.96, p = .018.
• When controlling for verbal IQ, our different teaching methods do have an impact on spelling ability!
• Put another way, our ANOVA non-significant result was due to the ‘noise’ the individual differences in verbal IQ put into our data.
• Verbal IQ is significant too, but we expected that due to their almost certain correlation.

Follow Up Tests and Effect Sizes

• ANCOVA is still ambiguous, so we would still need to do follow-up tests.
• Post-hoc and planned comparisons (contrasts) still viable.
• Eta squared remains the overall effect size for the ANCOVA and is calculated like this:
  • \eta^2 = \frac{SS{for \, METHOD}}{SS{Corrected \, Total}}
  • Sums of Squares (SS) from ANCOVA table.

Evaluating ANCOVA

• ANCOVA attempts to manage individual differences impacting on our observations/analysis.
  • Between Groups – Equalises groups on covariate (all participants are ‘the same’ in covariate terms).
  • Within Groups – Reduces ‘noise’ by removing covariate differences.
• Non-covariate individual differences remain, which we hope randomisation somewhat evenly distributes.
• Overall, a more powerful test than ANOVA.
• Requires knowledge and measurement of covariate prior to data collection.
• More difficult to observe significance due to inclusion of a covariate.
• Assumptions are more complex and difficult to meet.