Assumptions of ANOVA, Effect Size, and Power

Several assumptions underlie the use of ANOVA. Some mirror those in regression, while others are unique.

The scores within each group should follow a normal distribution.
There is a question of whether data needs to actually be normally distributed or conceptually should be.
If data is expected to be skewed in one condition relative to others, it can be problematic as ANOVA assumes normal distributions across groups.
Issue is more pronounced with unequal sample sizes because skewness can distort group statistics.
ANOVA functions optimally with roughly equal sample sizes across groups.
Skewed distribution + unequal sample sizes = test instability.
This instability can lead to failure to find significance or, conversely, false positives.
Non-normality can be problematic when doing pairwise comparisons using directional tests (expecting one group to be greater than another).
Skew can bias statistics, which might not be accounted for by the omnibus error term.
Outliers can disproportionately affect means, potentially causing instability or false positives.

Another assumption of ANOVA is homogeneity of variance: variances should be roughly equal across groups.
Variances don't have to be exactly the same, but they should be similar.
Sensitive to imbalanced cells
If sample sizes vary greatly across groups and the variances differ, it affects comparisons and the omnibus F test.

Observations should be independent.
This is primarily a design issue.
Between-subjects design: Different people should be in different conditions.
Observations can be dependent if there is contamination within a study where participants in one condition somehow inform people in another group how they should behave during the study, or what they learn from someone's experience in a different group influences their behavior when they're in a different condition of the study.
Lack of independence means the tested differences may not be what the researchers think they are testing.
Good experimental design (random assignment, eliminating confounds, reducing experimenter effects) helps ensure independence.

Levene's test of homogeneity of variances can be used.
The goal is not to have a statistically significant Levene's test.
A p-value for Levene's test should be greater than 0.05.
If Levene's test is significant (p < 0.05), the homogeneity of variance assumption is violated.
When Levene's test is significant, check robust tests such as Welch or Brown-Forsythe tests.
These tests adjust degrees of freedom, affecting the test statistics and reducing the F ratios.
Evaluate the statistical significance of the robust tests if Levene's test is significant.

It's important to test assumptions, but it's something that is often not done.
If the omnibus F is significant, but Levene's test is also significant, and robust tests are not significant, be cautious when interpreting results.
Robust tests correct for potential bias from heterogeneous variances.
If results are consistent across all tests, you can be more confident in their validity.

ANOVA is primarily a significance test.
Statistical significance doesn't equate to practical significance.
A p-value of less than 0.05 does not automatically make a result meaningful.
Effect size indicates the magnitude of the effect.
Along with statistical findings, present information about the magnitude of the effect
In ANOVA, effect size measures give an idea of the magnitude of an effect.

The most common measure of effect size in ANOVA is $\eta^2$ (eta squared).
$\eta^2$ is the ratio of variance accounted for by the effect, relative to the total variation in the outcome.
It is a ratio of sums of squares: sums of squares for the model divided by sums of squares total.
$\eta^2 = \frac{SS<em>{model}}{SS</em>{total}}$
In a one-way ANOVA, $\eta^2$ is equal to $R^2$ for the omnibus test.
$\eta^2$ can also be calculated for planned comparisons by dividing the sums of squares for the comparison by the total sums of squares.
It represents the percentage of total variation accounted for by the omnibus test or a specific mean comparison.
$\eta^2$ tends to overestimate the effect size in the population.
Cohen's d is another common effect size measure.

Jacob Cohen provided guidelines for evaluating $\eta^2$ , but emphasized they are somewhat arbitrary.
Small effect: $\eta^2$ around 0.01 (1% of the variance).
Medium effect: $\eta^2$ around 0.06.
Large effect: $\eta^2$ around 0.15.
The importance of a result lies in its practical significance.
A large effect found in a controlled lab setting may not be relevant in real-world settings.
Ultimately guidelines are arbitrary and depend on real-world context.

Effect size can be used to predict the plausibility of a research design via power analysis.
Power analysis determines the number of participants needed to increase a study's statistical power.
Power is the inverse of type II error (failing to detect a real effect).
A powerful significance test reduces the likelihood of type II error.
Knowing the likely effect size in advance helps determine the number of subjects needed to achieve sufficient power.
Power analysis is increasingly required by master's committees and grant applications.
It demonstrates whether the research design is likely to yield statistically meaningful results.
A detailed discussion of power analysis is beyond the scope of this course; this course introduces the concept of power and talks about a measure of effect size as a starting point.