One-Way ANOVA: Comprehensive Study Notes

Introduction to One-Way ANOVA

• ANOVA stands for Analysis of Variance: a statistical technique to determine if two or more groups are statistically different on a single dependent variable.
• Core idea: test the null hypothesis that all group means are equal against the alternative that at least one group mean is different.
• Narratives from the lecture emphasize: separating signal from noise in human data; variability is inevitable in psychological data; ANOVA helps determine if observed differences exceed random variation.
• Typical lab context: compare performance across multiple experimental conditions (e.g., control, deadline with short reading time, and instruction about what to look for in fake headlines).
• Analogy used: signal in noise — is there a real effect (signal) in the data beyond random fluctuation (noise)?

Key Concepts and Definitions

• Populations vs samples:
  • Population: the entire group we want to learn about (e.g., all students’ performance under different study conditions).
  • Sample: a subset we actually measure (e.g., a group of participants in each experimental condition).
  • ANOVA uses sample data to make inferences about the population(s) from which the samples were drawn.
• One-way ANOVA setup:
  • One independent variable (IV) that is categorical with two or more levels (groups).
  • One dependent variable (DV) that is continuous (e.g., test scores, accuracy).
  • No inherent ordering of groups (i.e., levels are categories, not a numeric continuum).
• Variability components in ANOVA:
  • Between-group variability (SS_between): variation due to the effect of the IV across group means.
  • Within-group variability (SS_within): variation due to randomness and other factors within each group.
  • Total variability (SS_total): overall variability in the data.
  • Relationship: SStotal = SSbetween + SS_within.
• F-statistic (F ratio):
  • $F = \frac{MS<em>{Between}}{MS</em>{Within}} = \frac{SS<em>{Between} / df</em>{Between}}{SS<em>{Within} / df</em>{Within}}$
  • If group means are truly equal, F tends to be small (near 1); if there are differences, F tends to be larger.
• Degrees of freedom (df):
  • Total df: $df_{Total} = N - 1$ where N is total number of observations.
  • Between-group df: $df_{Between} = k - 1$ where k is the number of groups.
  • Within-group df: $df_{Within} = N - k$
• p-value: the probability, under the null hypothesis, of observing an F as extreme as (or more extreme than) the one computed from the data. A small p-value suggests the null hypothesis is unlikely.
• Hypotheses:
  • Null hypothesis H0: All population means are equal, i.e., (\mu1 = \mu2 = … = \mu_k).
  • Alternative hypothesis Ha: At least one group mean differs from the others.
• Relationship to the t-test:
  • With two groups, ANOVA and a t-test give equivalent conclusions; in fact, for two groups, $F = t^2$ with df1 = 1 and df2 = N - 2.
  • ANOVA generalizes to three or more groups where a t-test is not appropriate for multiple comparisons without correction.
• Assumptions of ANOVA:
  • Independence of observations: each observation is independent of others.
  • Normality: DV is approximately normally distributed within each group. Robustness improves with larger samples.
  • Homogeneity of variances: groups have roughly equal variances (homoscedasticity).
• What ANOVA does and does not tell you:
  • Tells you whether there is a statistically significant difference somewhere among the group means.
  • Does not identify which specific groups differ; requires post hoc tests if significant.
• Post hoc tests:
  • Used after a significant ANOVA to compare all pairs of group means (e.g., deadline vs. instruction, deadline vs. control, instruction vs. control).
  • Common methods: Tukey's HSD (honest significant difference), Bonferroni correction, among others.
• Effect size in ANOVA:
  • Eta squared ($\eta^2$): the proportion of total variance explained by the IV.
  • $\eta^2 = \frac{SS<em>{Between}}{SS</em>{Total}}$
  • Partial eta squared is another common measure when multiple effects are considered in other designs.
• Type of sums of squares:
  • Type III sums of squares are the default in many software packages and are used to partition variance in a way that handles unbalanced designs. Other types (Type I, II) exist and can yield different results in some designs.
• Power and sample size:
  • Statistical power: probability of detecting an effect if one actually exists.
  • Power analysis helps determine needed sample size; larger samples yield higher power.
• Common questions and practical considerations:
  • Always start with graphing data to see whether groups look different before relying on F and p-values.
  • The lecture emphasizes interpreting in context: are means and variances meaningfully different, not just statistically different?
  • Sampling biases and confounds can mimic effects; ensure random assignment and representative sampling where possible.

Data Structure and How to Calculate ANOVA by Hand

• Data layout:
  • One DV measured for each participant.
  • One IV with k levels (groups).
  • Each group j has nj observations with mean (\bar{X}{.j}).
• Grand mean:
  • $\bar{X}<em>{..} = \frac{1}{N} \sum</em>{i=1}^{N} X_i$
• Sum of Squares:
  • Total: $SS<em>{Total} = \sum</em>{i=1}^{N} (X<em>i - \bar{X}</em>{..})^2$
  • Between: $SS<em>{Between} = \sum</em>{j=1}^{k} n<em>j(\bar{X}</em>{.j} - \bar{X}_{..})^2$
  • Within: $SS<em>{Within} = \sum</em>{j=1}^{k} \sum<em>{i=1}^{n</em>j} (X<em>{ij} - \bar{X}</em>{.j})^2$
• Mean Squares:
  • Between: $MS<em>{Between} = \frac{SS</em>{Between}}{df<em>{Between}} = \frac{SS</em>{Between}}{k-1}$
  • Within: $MS<em>{Within} = \frac{SS</em>{Within}}{df<em>{Within}} = \frac{SS</em>{Within}}{N - k}$
• F-statistic:
  • $F = \frac{MS<em>{Between}}{MS</em>{Within}}$
• Example interpretation:
  • If SS Between is small relative to SS Within (no real differences), F will be close to 1 and p will be large.
  • If SS Between is large relative to SS Within (differences among group means), F will be large and p will be small, leading to rejection of H0.
• Degrees of freedom recap:
  • df Total = N − 1
  • df Between = k − 1
  • df Within = N − k
• Special note about big-sample vs small-sample behavior:
  • With large samples, ANOVA can be robust to some violations of normality and equal variances, but severe violations can bias results.

Interpreting p-values and Making Inferences

• p-value interpretation:
  • If p < 0.05 (alpha level), reject H0: there is evidence that not all group means are equal.
  • If p ≥ 0.05, fail to reject H0: not enough evidence that there is a difference among the groups.
  • A p-value does not indicate the size or importance of the effect; it only reflects whether the observed data are unlikely under H0.
• Alpha level and error rates:
  • Alpha (often 0.05) is the probability of a Type I error (false positive) when the null is true.
  • Do not confuse p-values with effect sizes or practical significance.
• Type I and Type II errors:
  • Type I: rejecting H0 when it is true (false positive).
  • Type II: failing to reject H0 when an actual effect exists (false negative).
  • Statistical power (1 − Type II error) increases with larger sample size, larger effect sizes, and lower variability.
• When to use post hoc tests:
  • Use when ANOVA is significant to determine which specific group pairs differ.
  • Choose method (e.g., Tukey HSD or Bonferroni) based on study design and number of comparisons.

Assumptions in Detail and Diagnostics

• Independence of observations:
  • Each observation should be independent; violation can inflate Type I or Type II error rates.
  • Achieved via proper random assignment and avoiding clustering (e.g., participants influencing each other).
• Normality:
  • DV should be approximately normally distributed within each group.
  • More critical for small samples; with large samples, ANOVA is fairly robust to violations.
• Homogeneity of variances (homoscedasticity):
  • Variances across groups should be roughly equal.
  • Violations can distort F and p-values; use Levene’s test or Bartlett’s test to assess.
  • If violated, alternatives include Welch ANOVA or nonparametric tests (e.g., Kruskal-Wallis).
• Reporting table in ANOVA output:
  • A typical ANOVA table reports: source (Condition), df, SS, MS, F, p-value, and often effect size (e.g., eta-squared).
  • Software note: Type III sums of squares are common; eta-squared is commonly reported as a measure of effect size.

Practical Examples and Scenarios

• Example 1: Fake news headline task with three conditions (control, deadline, instructions).
  • DV: accuracy in identifying real vs. fake headlines.
  • IV: condition with 3 levels (categorical independent variable).
  • ANOVA asks: is there a statistically significant difference in accuracy across these three conditions?
• Example 2: Music study (silent, lo-fi beats, classical) and a dependent variable like final-test performance.
  • Demonstrates a one-way ANOVA with a single IV and three levels.
• Example 3: Anxiety after different treatments (cognitive behavioral therapy, medication, both).
  • Shows how understanding variability within and between groups informs whether therapy has a detectable effect.
• N-way design caveat:
  • If there were two independent variables (e.g., instruction vs. no instruction) crossed with deadline length (short vs. long), this would require examining main effects and interaction effects; such designs move beyond one-way ANOVA into factorial ANOVA.

Advanced Topics Mentioned in the Lecture

• Post-hoc versus a priori planned contrasts:
  • Post hoc tests (e.g., Tukey, Bonferroni) compare all pairs after a significant ANOVA.
  • In some designs, planned contrasts may be specified in advance to test specific hypotheses without a full post hoc correction.
• Effect sizes and interpretation:
  • Eta squared provides a sense of the magnitude of the effect (what proportion of total variance is explained by the IV).
  • Larger eta-squared indicates a stronger effect; however, interpret within the context of the field and measurement reliability.
• Do-calculus and mediation (brief aside):
  • Mediation analysis deals with whether a third variable explains the relationship between two others.
  • The do-operator (Judea Pearl) is a formal tool to reason about causality by intervening on a variable and observing the effect on another.
  • Correlation does not imply causation; do-calculus provides a formal framework for causal inference under certain assumptions.

Key Formulas to Remember (LaTeX)

• F-statistic: $F = \frac{MS<em>{Between}}{MS</em>{Within}} = \frac{SS<em>{Between} / df</em>{Between}}{SS<em>{Within} / df</em>{Within}}$
• Degrees of freedom:
  • $df_{Between} = k - 1$
  • $df_{Within} = N - k$
  • $df_{Total} = N - 1$
• Sums of Squares:
  • Total: $SS<em>{Total} = \sum</em>{i=1}^{N} (X<em>i - \bar{X}</em>{..})^2$
  • Between: $SS<em>{Between} = \sum</em>{j=1}^{k} n<em>j (\bar{X}</em>{.j} - \bar{X}_{..})^2$
  • Within: $SS<em>{Within} = \sum</em>{j=1}^{k} \sum<em>{i=1}^{n</em>j} (X<em>{ij} - \bar{X}</em>{.j})^2$
• Means:
  • $MS<em>{Between} = \frac{SS</em>{Between}}{df_{Between}}$
  • $MS<em>{Within} = \frac{SS</em>{Within}}{df_{Within}}$
• Effect size (Eta squared): $\eta^2 = \frac{SS<em>{Between}}{SS</em>{Total}}$
• Relationship to t-test (two groups): $F = t^2, \quad df1 = 1, \quad df2 = N - 2$
• Type III sums of squares (brief note): Common default in software; used to partition variance when dealing with unbalanced designs.

Quick Guidelines for Exam and Reporting

• When you have two or more groups, consider one-way ANOVA as the primary test for differences in a single DV across those groups.
• If ANOVA is significant, report: F-statistic, degrees of freedom, p-value, and effect size (e.g., $\eta^2$).
• Check assumptions: independence, normality, and homogeneity of variances; mention what you would do if an assumption is violated (e.g., use Welch ANOVA or a nonparametric alternative).
• Use post hoc tests to determine which specific groups differ after a significant ANOVA.
• Interpret in context: statistical significance does not always imply practical significance; consider effect sizes and the study design.

Summary Takeaways

• One-Way ANOVA tests whether multiple group means differ on a single DV by comparing between-group variance to within-group variance via the F statistic.
• A significant F (p < 0.05 typically) indicates at least one group mean differs, but not which ones; follow-up post hoc tests are needed.
• Assumptions matter: independence, normality, and equal variances; robustness improves with large sample sizes.
• Always visualize data first and interpret results in light of theory, measurement reliability, and real-world relevance.
• The framework generalizes to more groups; with two groups it dovetails with the t-test; with more groups, ANOVA is the standard approach.