Comprehensive Study Notes on ANOVA Effect Sizes, Statistical Power, and Follow-Up Post Hoc Analyses

Foundations of Effect Size in ANOVA

  • Definition and core purpose:

    • Effect size quantifies the magnitude of an experimental effect or observed difference between groups.
    • While an analysis of variance (ANOVA\text{ANOVA}) determines whether a statistically significant difference exists among group means, it does not communicate the scale of that difference. Effect size indicates whether an identified difference is practically small or large.
    • Statistical analysis fundamentally centers on variability, utilizing variability in one variable to predict or account for variability in another variable.
  • Universal reporting and interpretation standard:

    • In a results section, effect size is reported as a decimal value.
    • To interpret an effect size substantively, shift the decimal point two places to the right to convert it into a percentage.
    • Standard interpretive template for any effect size: "[Percentage] of the variability in the dependent variable was accounted for by the variability in the independent variable (or condition)."

Measures of Effect Size: Eta Squared, Omega Squared, and Partial Eta Squared

  • Eta Squared (η2\eta^2):

    • Biased statistic: Overestimates the proportion of variance explained in the broader population.
    • Application domain: Acceptable when research interest is strictly restricted to the specific sample or group evaluated (e.g., assessing differences solely within a single defined classroom without generalizing to external populations).
    • Mathematical definition: Sum of squares treatment divided by sum of squares total:     η2=SStreatmentSStotal\eta^2 = \frac{SS_{\text{treatment}}}{SS_{\text{total}}}
    • Level of Processing and Word Recall experiment:
    • Design: Independent variable has five processing conditions/levels: counting, rhyming, adjective, imagery, and intentional processing. The dependent variable is the number of words recalled.
    • Calculated value: η2=.447\eta^2 = .447.
    • Results section reporting: η2=.447\eta^2 = .447.
    • Interpretive statement: "44.7% of the variability in recall was accounted for by the level of processing condition (the independent variable)."
  • Omega Squared (ω2\omega^2):

    • Unbiased statistic: Adjusts for sample bias and provides a superior, accurate estimate for inferential statistics and population generalization.
    • Mathematical formulation:     ω2=SStreatment−(j−1)MSerrorSStotal+MSerror\omega^2 = \frac{SS_{\text{treatment}} - (j - 1)MS_{\text{error}}}{SS_{\text{total}} + MS_{\text{error}}}     where jj represents the number of groups or experimental conditions.
    • Calculation in the Level of Processing experiment:
    • Number of conditions: j=5j = 5.
    • Degrees of freedom correction: j−1=5−1=4j - 1 = 5 - 1 = 4.
    • Mean square error: MSerror=9.67MS_{\text{error}} = 9.67.
    • Resulting value: ω2=.393\omega^2 = .393.
    • Interpretive statement: "39.3% of the variability in recall scores was accounted for by the level of processing."
    • Practical significance of the difference between η2\eta^2 and ω2\omega^2:
    • In psychology, an effect accounting for less than 1%1\% (.01.01) of total variability is conventionally considered negligible or worthless.
    • The difference of over 5%5\% between η2=.447\eta^2 = .447 (44.7%44.7\%) and ω2=.393\omega^2 = .393 (39.3%39.3\%) is substantial, demonstrating how markedly η2\eta^2 can inflate effect estimates.
  • Partial Eta Squared (ηp2\eta_p^2):

    • Characteristics: Typically biased, but functions as an unbiased estimate under specific multi-factor configurations.
    • Single independent variable design: In a one-way design with only one independent variable, ηp2\eta_p^2 is mathematically identical to regular η2\eta^2 and remains biased.
    • Multiple independent variables: In factorial designs containing two or more independent variables, ηp2\eta_p^2 partials out other factors, behaving differently from regular η2\eta^2.
    • Software convention: In SPSS, the One-Way ANOVA command outputs Omega Squared fixed effect (ω2\omega^2), whereas the General Linear Model Univariate command outputs Partial Eta Squared (ηp2\eta_p^2).

Statistical Power and Sample Size Determination

  • Theoretical Framework of Statistical Power:

    • Null hypothesis testing decisions:
    • Type I Error (α\alpha): The probability of rejecting the null hypothesis when the null hypothesis is true (conventionally set at α=.05\alpha = .05 in psychology).
    • Type II Error (β\beta): The probability of failing to reject the null hypothesis when the null hypothesis is false.
    • Statistical Power (1−β1 - \beta): The probability of correctly rejecting the null hypothesis when it is false.
    • Operational definition: The ability of a statistical test to detect an effect of the independent variable if that effect is genuinely present.
  • Power Standards and Diagnostics:

    • Conventional threshold: The discipline standard in psychology requires a minimum power of .80.80 (80%80\%).
    • Evaluation: If an experimental analysis reports observed power equal to .67.67, the design is underpowered, meaning there was an inadequate probability of detecting a real effect. An observed power of .80.80 or above reflects sufficient statistical power.
  • Sample Size Planning via G*Power:

    • G*Power is a free, downloadable software program used to calculate the required sample size (NN) necessary to detect an effect of a specified magnitude prior to data collection.
    • Input parameters: Alpha level (α=.05\alpha = .05), target power level (.80.80), and anticipated effect size.
  • Subdisciplinary Sample Size Requirements per Cell:

    • Perception and Neuroscience: Typically investigate strong, biologically constrained phenomena; standard requirements are 1212 to 2020 participants per cell.
    • A two-group tt-test requires 12×2=2412 \times 2 = 24 participants total, or 20×2=4020 \times 2 = 40 participants total.
    • Social Psychology: Evaluates subtler behavioral differences with smaller effect sizes; standard conventions require 4040 to 5050 participants per cell, or 100+100+ participants per study.
    • General Rule of Thumb: A minimum of 4040 participants is recommended for each condition, or 4040 participants for each independent/dependent variable in multivariate models such as multiple regression.

Logic of Follow-Up Tests and Multiple Comparisons

  • The Omnibus Test Prerequisite:

    • Follow-up tests are conducted exclusively after an omnibus ANOVA\text{ANOVA} demonstrates statistical significance (p<.05p < .05).
    • An omnibus ANOVA\text{ANOVA} only indicates that a difference exists somewhere among the group means; follow-up tests are required to locate precisely which groups differ.
    • If an omnibus ANOVA\text{ANOVA} is non-significant (p≥.05p \ge .05), no differences exist, making it invalid to run follow-up tests.
  • Error Inflation Problem:

    • Running multiple pairwise tt-tests creates cumulative, inflated Type I error across the family of comparisons.
    • The core strength of the omnibus ANOVA\text{ANOVA} is maintaining the familywise Type I error rate (α\alpha) strictly at .05.05 across all simultaneous comparisons.
  • Bonferroni Test for Planned Comparisons:

    • Mechanism: Spreads the overall alpha budget across the exact number of planned comparisons by dividing the familywise alpha (.05.05) by the total number of comparisons (cc):     αadjusted=αc\alpha_{\text{adjusted}} = \frac{\alpha}{c}
    • Example: For 1010 pairwise comparisons:     αnew=.0510=.005\alpha_{\text{new}} = \frac{.05}{10} = .005     A comparison must yield a pp-value strictly less than .005.005 to reject the null hypothesis for that pairing.
    • Ideal usage: Appropriate when a researcher has a small number of planned (a priori) comparisons of theoretical interest, avoiding irrelevant combinations (e.g., ignoring comparisons between a placebo in one condition and an unrelated third condition in a factorial design).
    • Limitation: If the number of comparisons is too high, the required threshold becomes excessively stringent (e.g., p<.00005p < .00005), rendering true effects almost impossible to detect.
  • Comparison Categories:

    • A Priori Comparisons: Planned prior to data collection based on specific theoretical hypotheses; utilize a smaller comparison pool to conserve alpha.
    • Post Hoc Comparisons: Unplanned tests chosen after inspecting data; assume that all possible pairwise comparisons are being executed and integrate mathematical penalties to hold the overall error rate constant.

Post Hoc Test Selection: Tukey's HSD vs. Games-Howell

  • Pairwise Comparisons:

    • Testing differences between two specific group means evaluated in isolated pairs.
  • Tukey's Honestly Significant Difference (Tukey's HSD):

    • Most common post hoc test in behavioral sciences; historically favored for its ease of hand calculation prior to computers.
    • Selection condition: Selected when the assumption of homogeneity of variance is MET (indicated by a non-significant Levene's test, p>.05p > .05).
  • Games-Howell Test:

    • Non-parametric adjustment post hoc test.
    • Selection condition: Selected when the assumption of homogeneity of variance is VIOLATED (indicated by a statistically significant Levene's test, p≤.05p \le .05).
    • Adjusts degrees of freedom and standard error estimates internally to correct for unequal variances, functioning analogously to the "equal variances not assumed" correction in an independent-samples tt-test.
  • SPSS Best Practice Execution:

    • In SPSS, check both the Tukey and Games-Howell options concurrently during dialog setup.
    • Once output is generated, inspect Levene's test:
    • If Levene's test is non-significant (p>.05p > .05), evaluate and report the Tukey test results.
    • If Levene's test is significant (p≤.05p \le .05), evaluate and report the Games-Howell results.

Complex Comparisons and Linear Contrasts

  • Complex Comparisons:

    • Comparisons where two or more group means are mathematically combined and evaluated against another single group or set of combined groups.
  • Linear Contrasts:

    • Method for computing complex comparisons by calculating a weighted average of specified group means based on their relative sample sizes and comparing them against target conditions.
    • Five-condition drug trial illustration:
    • Experimental conditions: Low Dose, Medium Dose, High Dose, No Drug, and Placebo.
    • Contrast 1 (Drug vs. No Drug): Combines Low, Medium, and High dosage groups together via weighted averaging and contrasts them against the No Drug condition to assess whether the chemical agent works overall.
    • Contrast 2 (Drug vs. Placebo): Combines Low, Medium, and High dosage groups together and contrasts them against the Placebo condition to assess the therapeutic effect above and beyond psychological expectancy.
    • Contrast 3 (Placebo vs. No Drug): Contrasts Placebo directly against No Drug to evaluate the isolated effect of participant expectancy.

Experimental Case Study: Maternal Retrieval Time in Rats

  • Experimental Design and Hypothesis:

    • Research question: Does the chronological age of a rat pup affect the latency of maternal retrieval back to the nest?
    • Paradigm: A researcher removes a baby rat (rat pup) from its nest, places it at a standardized, fixed distance within the cage, and records the time in seconds required for the mother rat to grasp the pup by the scruff of the neck and carry it back into the nest.
    • Independent Variable: Age of the rat pup with three levels:
    • 5 days old5\text{ days old}
    • 20 days old20\text{ days old}
    • 35 days old35\text{ days old}
    • Dependent Variable: Retrieval time measured in seconds.
  • SPSS Variable Setup and Data Structure:

    • Variable View setup:
    • Independent variable: Age (Numeric).
    • Values assigned: 1 = 5 day old5\text{ day old}, 2 = 20 day old20\text{ day old}, 3 = 35 day old35\text{ day old}. (Alternative valid numerical codings include 0, 1, 2 or 5, 20, 35).
    • Dependent variable: Time (Numeric; time in seconds).
    • Sample size allocation: N=18N = 18 total rat pups (n=6n = 6 pups per age condition):
    • Rows 1 through 6: Group 1 (5 days old5\text{ days old}, values: 15, 10, 25, 15, 20, 18 seconds; initial sequence listed 15, 15, 20, 25, 23, 20).
    • Rows 7 through 12: Group 2 (20 days old20\text{ days old}, values: 30, 15, 20, 25, 23, 20 seconds).
    • Rows 13 through 18: Group 3 (35 days old35\text{ days old}, values: 40, 35, 50, 43, 45, 45 seconds; alternate listed 40, 35, 50, 43, 45, 40).
  • Descriptive Statistics and Diagnostic Verification:

    • Group means:
    • 5 days old5\text{ days old}: Mean (MM) = 17 seconds17\text{ seconds}.
    • 20 days old20\text{ days old}: Mean (MM) = 22 seconds22\text{ seconds}.
    • 35 days old35\text{ days old}: Mean (MM) = 42 seconds42\text{ seconds}.
    • Standard deviations: Identical across all three groups.
    • Levene's Test for Homogeneity of Variances (based on Mean):
    • Significance value: p=1.000p = 1.000.
    • Diagnostic interpretation: Because p=1.000>.05p = 1.000 > .05, the assumption of homogeneity of variance is met with complete certainty. The researcher formally retains Tukey's HSD for post hoc interpretation.
  • Omnibus ANOVA Output:

    • Between-groups degrees of freedom: dfbetween=2df_{\text{between}} = 2.
    • Within-groups (error) degrees of freedom: dfwithin=15df_{\text{within}} = 15.
    • Test statistic: F(2,15)=40.127F(2, 15) = 40.127.
    • Significance: p<.001p < .001.
    • Conclusion: Reject the null hypothesis; there is a statistically significant omnibus difference in retrieval times among the three age groups.
  • Tukey's HSD Post Hoc Results:

    • The SPSS output displays four significant comparison rows (p<.05p < .05) because each significant contrast is printed bidirectionally as a duplicate:
    • 5 days5\text{ days} vs. 35 days35\text{ days} is identical to 35 days35\text{ days} vs. 5 days5\text{ days} (p<.05p < .05).
    • 20 days20\text{ days} vs. 35 days35\text{ days} is identical to 35 days35\text{ days} vs. 20 days20\text{ days} (p<.05p < .05).
    • The comparison between 5 days5\text{ days} and 20 days20\text{ days} is not statistically significant (p≥.05p \ge .05).
    • Substantive conclusion: Mother rats retrieved 35-day-old35\text{-day-old} rat pups significantly slower (taking longer in seconds) than both 5-day-old5\text{-day-old} rat pups and 20-day-old20\text{-day-old} rat pups.

Step-by-Step SPSS Procedures and APA Results Reporting

  • Procedure 1: One-Way ANOVA Command (Compare Means):

    • Menu path: Analyze > Compare Means > One-Way ANOVA...
    • Dialogue entries:
    • Move Time into Dependent List.
    • Move Age into Factor.
    • Click Post Hoc... > Check both Tukey and Games-Howell > Click Continue.
    • Click Options... > Check Descriptive, Fixed and random effects, and Homogeneity of variance test > Click Continue.
    • Click OK.
    • Effect size reported: Omega Squared fixed effect (ω2=.813\omega^2 = .813).
    • Results section write-up:     "A one-way analysis of variance was conducted with age of rat as the independent variable and time of retrieval in seconds as the dependent variable. Levene's test indicated that we met the assumption of homogeneity of variance (p=1.000p = 1.000). There was a significant difference among group means, F(2,15)=40.127F(2, 15) = 40.127, p<.001p < .001, ω2=.813\omega^2 = .813. Tukey post hoc tests indicated that the 35-day-old rats (M=42M = 42) were retrieved more slowly than 5-day-old rats (M=17M = 17) or 20-day-old rats (M=22M = 22)."
  • Procedure 2: General Linear Model Command (Univariate):

    • Menu path: Analyze > General Linear Model > Univariate...
    • Dialogue entries:
    • Move Time into Dependent Variable.
    • Move Age into Fixed Factor(s).
    • Click Post Hoc... > Move Age to Post Hoc Tests for box > Check both Tukey and Games-Howell > Click Continue.
    • Click Options... > Check Homogeneity tests > Click Continue.
    • Click OK.
    • Output reading: Locate the row for the independent variable (Age) and the row for Error in the Tests of Between-Subjects Effects table:
    • Degrees of freedom: 22 and 1515.
    • FF-ratio: 40.12740.127.
    • Significance: p<.001p < .001.
    • Effect size: Partial Eta Squared (ηp2=.843\eta_p^2 = .843).
    • Statistical power: Observed Power = 1.0001.000.
    • Results section write-up:     "A one-way analysis of variance was conducted with age of rat as the independent variable and time of retrieval in seconds as the dependent variable. Levene's test indicated that we met the assumption of homogeneity of variance (p=1.000p = 1.000). There was a significant difference among group means, F(2,15)=40.127F(2, 15) = 40.127, p<.001p < .001, ηp2=.843\eta_p^2 = .843, observed power =1.000= 1.000. Tukey post hoc tests indicated that the 35-day-old rats (M=42M = 42) were retrieved more slowly than 5-day-old rats (M=17M = 17) or 20-day-old rats (M=22M = 22)."
  • Comparison of Reporting Formats:

    • The statistical decisions, degrees of freedom, FF-ratio, pp-values, means, standard deviations, and post hoc pairwise comparisons are identical across both procedures.
    • The only reporting divergences are:
    • The One-Way ANOVA command yields and reports Omega Squared fixed effect (ω2=.813\omega^2 = .813) without observed power.
    • The Univariate command yields and reports Partial Eta Squared (ηp2=.843\eta_p^2 = .843) alongside Observed Power (1.0001.000).

Extension to Advanced Factorial and Multivariate Designs

  • Structural Continuity across Complex Models:
    • Factorial Designs: Introduce additional independent variables, generating multiple main effects and interaction terms.
    • Multivariate ANOVA (MANOVA\text{MANOVA}): Introduces multiple dependent variables simultaneously.
    • Procedural invariance: The conceptual and procedural workflow remains identical regardless of design complexity:
    • Run the omnibus test.
    • Decompose statistically significant interaction effects into smaller simple-effects ANOVAs.
    • Verify Levene's test of homogeneity of variance.
    • Execute Tukey's HSD or Games-Howell follow-up tests to pinpoint pairwise differences.
    • Interpret effect sizes by stating the percentage of dependent variable variance explained by the factor.

Questions & Discussion

  • Why select both Tukey and Games-Howell in SPSS options prior to knowing the outcome of Levene's test?

    • Researchers check both boxes simultaneously so that SPSS generates both tests in a single run. Once output is generated, the researcher consults Levene's test: if p>.05p > .05, the Tukey output is used; if p≤.05p \le .05, the Games-Howell output is used. This eliminates the need to re-run the analysis after checking assumptions.
  • Why does the Tukey post hoc table indicate four significant lines if there are only three groups?

    • SPSS prints every pairwise comparison bidirectionally. A comparison between Group 1 and Group 3 is duplicated as Group 3 versus Group 1. Thus, four significant rows in the output reflect exactly two distinct significant comparisons (3535 vs. 55 days, and 3535 vs. 2020 days).
  • Does the numeric coding used for the independent variable in SPSS affect the ANOVA output?

    • No. Coding the groups as 1, 2, 3, 0, 1, 2, or the actual day values 5, 20, 35 yields identical sums of squares, degrees of freedom, FF-ratios, and significance values in an ANOVA, provided the groupings are specified consistently.
  • Course Schedule Notice:

    • Review of Chapter 16 is scheduled for the upcoming Tuesday session following the Fall break.