Comprehensive Study Notes on ANOVA Effect Sizes, Statistical Power, and Follow-Up Post Hoc Analyses
Foundations of Effect Size in ANOVA
Definition and core purpose:
- Effect size quantifies the magnitude of an experimental effect or observed difference between groups.
- While an analysis of variance () determines whether a statistically significant difference exists among group means, it does not communicate the scale of that difference. Effect size indicates whether an identified difference is practically small or large.
- Statistical analysis fundamentally centers on variability, utilizing variability in one variable to predict or account for variability in another variable.
Universal reporting and interpretation standard:
- In a results section, effect size is reported as a decimal value.
- To interpret an effect size substantively, shift the decimal point two places to the right to convert it into a percentage.
- Standard interpretive template for any effect size: "[Percentage] of the variability in the dependent variable was accounted for by the variability in the independent variable (or condition)."
Measures of Effect Size: Eta Squared, Omega Squared, and Partial Eta Squared
Eta Squared ():
- Biased statistic: Overestimates the proportion of variance explained in the broader population.
- Application domain: Acceptable when research interest is strictly restricted to the specific sample or group evaluated (e.g., assessing differences solely within a single defined classroom without generalizing to external populations).
- Mathematical definition: Sum of squares treatment divided by sum of squares total:
- Level of Processing and Word Recall experiment:
- Design: Independent variable has five processing conditions/levels: counting, rhyming, adjective, imagery, and intentional processing. The dependent variable is the number of words recalled.
- Calculated value: .
- Results section reporting: .
- Interpretive statement: "44.7% of the variability in recall was accounted for by the level of processing condition (the independent variable)."
Omega Squared ():
- Unbiased statistic: Adjusts for sample bias and provides a superior, accurate estimate for inferential statistics and population generalization.
- Mathematical formulation: where represents the number of groups or experimental conditions.
- Calculation in the Level of Processing experiment:
- Number of conditions: .
- Degrees of freedom correction: .
- Mean square error: .
- Resulting value: .
- Interpretive statement: "39.3% of the variability in recall scores was accounted for by the level of processing."
- Practical significance of the difference between and :
- In psychology, an effect accounting for less than () of total variability is conventionally considered negligible or worthless.
- The difference of over between () and () is substantial, demonstrating how markedly can inflate effect estimates.
Partial Eta Squared ():
- Characteristics: Typically biased, but functions as an unbiased estimate under specific multi-factor configurations.
- Single independent variable design: In a one-way design with only one independent variable, is mathematically identical to regular and remains biased.
- Multiple independent variables: In factorial designs containing two or more independent variables, partials out other factors, behaving differently from regular .
- Software convention: In SPSS, the
One-Way ANOVAcommand outputs Omega Squared fixed effect (), whereas the General Linear ModelUnivariatecommand outputs Partial Eta Squared ().
Statistical Power and Sample Size Determination
Theoretical Framework of Statistical Power:
- Null hypothesis testing decisions:
- Type I Error (): The probability of rejecting the null hypothesis when the null hypothesis is true (conventionally set at in psychology).
- Type II Error (): The probability of failing to reject the null hypothesis when the null hypothesis is false.
- Statistical Power (): The probability of correctly rejecting the null hypothesis when it is false.
- Operational definition: The ability of a statistical test to detect an effect of the independent variable if that effect is genuinely present.
Power Standards and Diagnostics:
- Conventional threshold: The discipline standard in psychology requires a minimum power of ().
- Evaluation: If an experimental analysis reports observed power equal to , the design is underpowered, meaning there was an inadequate probability of detecting a real effect. An observed power of or above reflects sufficient statistical power.
Sample Size Planning via G*Power:
- G*Power is a free, downloadable software program used to calculate the required sample size () necessary to detect an effect of a specified magnitude prior to data collection.
- Input parameters: Alpha level (), target power level (), and anticipated effect size.
Subdisciplinary Sample Size Requirements per Cell:
- Perception and Neuroscience: Typically investigate strong, biologically constrained phenomena; standard requirements are to participants per cell.
- A two-group -test requires participants total, or participants total.
- Social Psychology: Evaluates subtler behavioral differences with smaller effect sizes; standard conventions require to participants per cell, or participants per study.
- General Rule of Thumb: A minimum of participants is recommended for each condition, or participants for each independent/dependent variable in multivariate models such as multiple regression.
Logic of Follow-Up Tests and Multiple Comparisons
The Omnibus Test Prerequisite:
- Follow-up tests are conducted exclusively after an omnibus demonstrates statistical significance ().
- An omnibus only indicates that a difference exists somewhere among the group means; follow-up tests are required to locate precisely which groups differ.
- If an omnibus is non-significant (), no differences exist, making it invalid to run follow-up tests.
Error Inflation Problem:
- Running multiple pairwise -tests creates cumulative, inflated Type I error across the family of comparisons.
- The core strength of the omnibus is maintaining the familywise Type I error rate () strictly at across all simultaneous comparisons.
Bonferroni Test for Planned Comparisons:
- Mechanism: Spreads the overall alpha budget across the exact number of planned comparisons by dividing the familywise alpha () by the total number of comparisons ():
- Example: For pairwise comparisons: A comparison must yield a -value strictly less than to reject the null hypothesis for that pairing.
- Ideal usage: Appropriate when a researcher has a small number of planned (a priori) comparisons of theoretical interest, avoiding irrelevant combinations (e.g., ignoring comparisons between a placebo in one condition and an unrelated third condition in a factorial design).
- Limitation: If the number of comparisons is too high, the required threshold becomes excessively stringent (e.g., ), rendering true effects almost impossible to detect.
Comparison Categories:
- A Priori Comparisons: Planned prior to data collection based on specific theoretical hypotheses; utilize a smaller comparison pool to conserve alpha.
- Post Hoc Comparisons: Unplanned tests chosen after inspecting data; assume that all possible pairwise comparisons are being executed and integrate mathematical penalties to hold the overall error rate constant.
Post Hoc Test Selection: Tukey's HSD vs. Games-Howell
Pairwise Comparisons:
- Testing differences between two specific group means evaluated in isolated pairs.
Tukey's Honestly Significant Difference (Tukey's HSD):
- Most common post hoc test in behavioral sciences; historically favored for its ease of hand calculation prior to computers.
- Selection condition: Selected when the assumption of homogeneity of variance is MET (indicated by a non-significant Levene's test, ).
Games-Howell Test:
- Non-parametric adjustment post hoc test.
- Selection condition: Selected when the assumption of homogeneity of variance is VIOLATED (indicated by a statistically significant Levene's test, ).
- Adjusts degrees of freedom and standard error estimates internally to correct for unequal variances, functioning analogously to the "equal variances not assumed" correction in an independent-samples -test.
SPSS Best Practice Execution:
- In SPSS, check both the Tukey and Games-Howell options concurrently during dialog setup.
- Once output is generated, inspect Levene's test:
- If Levene's test is non-significant (), evaluate and report the Tukey test results.
- If Levene's test is significant (), evaluate and report the Games-Howell results.
Complex Comparisons and Linear Contrasts
Complex Comparisons:
- Comparisons where two or more group means are mathematically combined and evaluated against another single group or set of combined groups.
Linear Contrasts:
- Method for computing complex comparisons by calculating a weighted average of specified group means based on their relative sample sizes and comparing them against target conditions.
- Five-condition drug trial illustration:
- Experimental conditions: Low Dose, Medium Dose, High Dose, No Drug, and Placebo.
- Contrast 1 (Drug vs. No Drug): Combines Low, Medium, and High dosage groups together via weighted averaging and contrasts them against the No Drug condition to assess whether the chemical agent works overall.
- Contrast 2 (Drug vs. Placebo): Combines Low, Medium, and High dosage groups together and contrasts them against the Placebo condition to assess the therapeutic effect above and beyond psychological expectancy.
- Contrast 3 (Placebo vs. No Drug): Contrasts Placebo directly against No Drug to evaluate the isolated effect of participant expectancy.
Experimental Case Study: Maternal Retrieval Time in Rats
Experimental Design and Hypothesis:
- Research question: Does the chronological age of a rat pup affect the latency of maternal retrieval back to the nest?
- Paradigm: A researcher removes a baby rat (rat pup) from its nest, places it at a standardized, fixed distance within the cage, and records the time in seconds required for the mother rat to grasp the pup by the scruff of the neck and carry it back into the nest.
- Independent Variable: Age of the rat pup with three levels:
- Dependent Variable: Retrieval time measured in seconds.
SPSS Variable Setup and Data Structure:
- Variable View setup:
- Independent variable:
Age(Numeric). - Values assigned:
1= ,2= ,3= . (Alternative valid numerical codings include0, 1, 2or5, 20, 35). - Dependent variable:
Time(Numeric; time in seconds). - Sample size allocation: total rat pups ( pups per age condition):
- Rows 1 through 6: Group 1 (, values:
15, 10, 25, 15, 20, 18seconds; initial sequence listed15, 15, 20, 25, 23, 20). - Rows 7 through 12: Group 2 (, values:
30, 15, 20, 25, 23, 20seconds). - Rows 13 through 18: Group 3 (, values:
40, 35, 50, 43, 45, 45seconds; alternate listed40, 35, 50, 43, 45, 40).
Descriptive Statistics and Diagnostic Verification:
- Group means:
- : Mean () = .
- : Mean () = .
- : Mean () = .
- Standard deviations: Identical across all three groups.
- Levene's Test for Homogeneity of Variances (based on Mean):
- Significance value: .
- Diagnostic interpretation: Because , the assumption of homogeneity of variance is met with complete certainty. The researcher formally retains Tukey's HSD for post hoc interpretation.
Omnibus ANOVA Output:
- Between-groups degrees of freedom: .
- Within-groups (error) degrees of freedom: .
- Test statistic: .
- Significance: .
- Conclusion: Reject the null hypothesis; there is a statistically significant omnibus difference in retrieval times among the three age groups.
Tukey's HSD Post Hoc Results:
- The SPSS output displays four significant comparison rows () because each significant contrast is printed bidirectionally as a duplicate:
- vs. is identical to vs. ().
- vs. is identical to vs. ().
- The comparison between and is not statistically significant ().
- Substantive conclusion: Mother rats retrieved rat pups significantly slower (taking longer in seconds) than both rat pups and rat pups.
Step-by-Step SPSS Procedures and APA Results Reporting
Procedure 1: One-Way ANOVA Command (
Compare Means):- Menu path:
Analyze>Compare Means>One-Way ANOVA... - Dialogue entries:
- Move
TimeintoDependent List. - Move
AgeintoFactor. - Click
Post Hoc...> Check bothTukeyandGames-Howell> ClickContinue. - Click
Options...> CheckDescriptive,Fixed and random effects, andHomogeneity of variance test> ClickContinue. - Click
OK. - Effect size reported: Omega Squared fixed effect ().
- Results section write-up: "A one-way analysis of variance was conducted with age of rat as the independent variable and time of retrieval in seconds as the dependent variable. Levene's test indicated that we met the assumption of homogeneity of variance (). There was a significant difference among group means, , , . Tukey post hoc tests indicated that the 35-day-old rats () were retrieved more slowly than 5-day-old rats () or 20-day-old rats ()."
- Menu path:
Procedure 2: General Linear Model Command (
Univariate):- Menu path:
Analyze>General Linear Model>Univariate... - Dialogue entries:
- Move
TimeintoDependent Variable. - Move
AgeintoFixed Factor(s). - Click
Post Hoc...> MoveAgetoPost Hoc Tests forbox > Check bothTukeyandGames-Howell> ClickContinue. - Click
Options...> CheckHomogeneity tests> ClickContinue. - Click
OK. - Output reading: Locate the row for the independent variable (
Age) and the row forErrorin the Tests of Between-Subjects Effects table: - Degrees of freedom: and .
- -ratio: .
- Significance: .
- Effect size: Partial Eta Squared ().
- Statistical power: Observed Power = .
- Results section write-up: "A one-way analysis of variance was conducted with age of rat as the independent variable and time of retrieval in seconds as the dependent variable. Levene's test indicated that we met the assumption of homogeneity of variance (). There was a significant difference among group means, , , , observed power . Tukey post hoc tests indicated that the 35-day-old rats () were retrieved more slowly than 5-day-old rats () or 20-day-old rats ()."
- Menu path:
Comparison of Reporting Formats:
- The statistical decisions, degrees of freedom, -ratio, -values, means, standard deviations, and post hoc pairwise comparisons are identical across both procedures.
- The only reporting divergences are:
- The
One-Way ANOVAcommand yields and reports Omega Squared fixed effect () without observed power. - The
Univariatecommand yields and reports Partial Eta Squared () alongside Observed Power ().
Extension to Advanced Factorial and Multivariate Designs
- Structural Continuity across Complex Models:
- Factorial Designs: Introduce additional independent variables, generating multiple main effects and interaction terms.
- Multivariate ANOVA (): Introduces multiple dependent variables simultaneously.
- Procedural invariance: The conceptual and procedural workflow remains identical regardless of design complexity:
- Run the omnibus test.
- Decompose statistically significant interaction effects into smaller simple-effects ANOVAs.
- Verify Levene's test of homogeneity of variance.
- Execute Tukey's HSD or Games-Howell follow-up tests to pinpoint pairwise differences.
- Interpret effect sizes by stating the percentage of dependent variable variance explained by the factor.
Questions & Discussion
Why select both Tukey and Games-Howell in SPSS options prior to knowing the outcome of Levene's test?
- Researchers check both boxes simultaneously so that SPSS generates both tests in a single run. Once output is generated, the researcher consults Levene's test: if , the Tukey output is used; if , the Games-Howell output is used. This eliminates the need to re-run the analysis after checking assumptions.
Why does the Tukey post hoc table indicate four significant lines if there are only three groups?
- SPSS prints every pairwise comparison bidirectionally. A comparison between Group 1 and Group 3 is duplicated as Group 3 versus Group 1. Thus, four significant rows in the output reflect exactly two distinct significant comparisons ( vs. days, and vs. days).
Does the numeric coding used for the independent variable in SPSS affect the ANOVA output?
- No. Coding the groups as
1, 2, 3,0, 1, 2, or the actual day values5, 20, 35yields identical sums of squares, degrees of freedom, -ratios, and significance values in an ANOVA, provided the groupings are specified consistently.
- No. Coding the groups as
Course Schedule Notice:
- Review of Chapter 16 is scheduled for the upcoming Tuesday session following the Fall break.