Week 2: Factorial ANOVA

What are Questionable Research Practices?

P-hacking, HARKing, Selective reporting and fraud.

What are examples of P-hacking?

- Data peeking = collecting more data if results are not significant yet.

- Stopping data collection early if results are already significant.

- Change exclusion/inclusion criteria depending on the results.

- Only reporting conditions/independent variables that worked.

- Only reporting outcome/dependent variables that worked.

- Include or remove covariates depending on the results.

- Switch to a one-sided test when the study design was two-sided.

- Rounding down the p-value (.054 → .05)

What is HARKing?

Hypothesizing after results are known. Presenting an unexpected result as if you had predicted it from the start.

How does the process of preregistration work?

- Develop idea → design study → write report → publish report

- After designing the study, you have to preregister: hypotheses, planned method, sampling plan, planned analyses and inference criteria.

- This does not prevent publication bias.

What are registered reports and what are its advantages?

Peer reviewed articles which journals have decided to publish before results are known. This prevents publication bias, QRP’s, file-drawering and reviewer bias.

What is a factorial ANOVA?

An experimental design with more than one factor of interest. It has a categorical independent variable and a continuous dependent variable. It assesses differences in conditions between different groups by looking at the group means.

What’s the difference between a 2x2 ANOVA and a 2x2x2 ANOVA?

- A 2x2 ANOVA has two conditions and two levels of a factor.

- A 2x2x2 ANOVA has two conditions and two factors with two levels.

- Another example, a 3x2x2 ANOVA has three conditions and two factors.

Consider a design with the factors Gender (Male, Female, Other), SES (low, middle, high), and Religion (Catholic, Protestant, Other religion, None) what is the name of this design?

3x3x4 ANOVA.

What’s the difference between a one-way ANOVA and a factorial ANOVA?

The factorial ANOVA has more than one predictor variable, the simple ANOVA only one.

Why would you prefer an ANOVA to multiple t-tests?

Multiple t-tests cause an inflation in type 1 error rate, an ANOVA compares all group means at once.

What are the hypotheses for an ANOVA?

The null hypotheses states “not all means are equal”, the informative hypothesis states “there is atleast one difference between the means” —> not very informative.

What do you analyze in an ANOVA?

Between-group variation and within-group variation.

What are the assumptions of an ANOVA?

- Random sample

- Independent observations: people in different groups don’t influence each other

- Dependent variable is at least interval scale and independent are nominal

- Dependent variable is normally distributed in each group

- Homogeneity of variances: roughly equal variance within-group

- No outliers

- Normality of residuals

What are main effects and how do you determine them?

- Main effect of condition means that there is an overall difference in averages between the control and the experimental group. You use the interaction graph to assess the grand mean of both groups and compare them (assuming the study has equal numbers in both groups). You eyeball by finding the middle value between the beginning of two lines and end of both lines. If there is a difference in these two marginal means, this means there is a main effect of condition.

- Main effect of factor means a difference between two groups. We assess this by looking at the grand mean of each line and comparing them. There is a main effect when these values differ when lining them up to the y-axis.

What is an interaction effect and how do you determine one?

- This means that the effect of one factor depends on the value of the other factor. You can ask = is the effect of A the same for all levels of B? There is an interaction when the lines do not run parallel to each other.

- In ANOVA look at the F-test statistic and p-value.

What do you do when you find a significant interaction effect in ANOVA?

Follow-up with simple main effects analysis to determine which variables differ from each other.

How do you check for homogeneity of variances?

- Check in descriptives if the standard deviations are about equal in all groups.

- Levene’s test, when this test is not significant this means we do not reject the null hypothesis of equal variances. The assumption is met.

What could cause a significant Levene’s test when the SD’s are similar?

You have large sample sizes per group, so small effects become significant. Larger N means more power. You should not worry about small violations.

How do you check for normality of residuals assumption?

- Statistical tests such as Shapiro-Wilks or Kolmogorov-Smirnov.

- Plot the residuals and visually inspect them with Q-Q plot of residuals. Plots the theoretical quantiles against the observed quantiles. The observed residuals should be on the line.

Which correction should you use under post-hoc tests and what does the table tell us?

- Bonferonni correction otherwise increase in type 1 error. It divides the alpha level with the number of tests. Or you multiply the p-value with the amount of tests. Decreases power.

- Shows us the pairwise comparisons of all levels of a factor and the Bayes factor which compares the hypothesis.

What is the partial eta-squared and what are its rules?

Small effect .01, medium effect .06, large effect .14. This measures the proportion of the total variance of the independent variables on the dependent variable.

How do you calculate the experiment wise error rate a_ew?

- 1 - (1 - α_pc)^c, where αpc is the chosen significance level per comparison (usually 0.05), and c is the number of comparisons.

What is your type 1 and 2 error risk when conducting 3 tests with alpha level of 5%

14% chance for type 1 and 86% chance for type 2.

What is the advantage and disadvantage of an omnibus test?

Better control over your type 1 error but you don’t know where the difference is when your null hypothesis is rejected.

What are pairwise comparisons and simple main effects used for?

- After a significant main effect use pairwise comparisons. This has more power than a Bonferonni correction.

- After a significant interaction effect use simple main effects. Simple refers to restricting the test to considering one level of the other factor, different to main effects which test if there is an overall effect. Perform only when interaction is significant.

What are planned comparisons and how are they tested?

- Used when we aren’t interested in all possible comparisons, but specific ones formulated before seeing the results.

- Tested using contrasts: linear combinations of the different levels of a factor.

What are simple contrasts?

Contrast certain means pairwise while selecting a reference category all other groups are compared against. There are less tests performed with simple contrasts than with pairwise  6-group ANOVA with 15 pairwise becomes only 5 simple contrasts.

What are repeated contrasts?

Each group is contrasted against the next group. Makes sense when the grouping variable is ordinal (such as increasing dose of medicine). At what level does a difference appear? Group 1 vs group 2 and group 2 vs group 3 separately.

What is the difference between one-sided and two-sided testing?

- In NHST one-sided testing is more specific and more powerful. Here only positive or negative values would be in line with HA.

- In two-sided any value except 0 would be in line with the alternative hypothesis.

What is the use of an informative hypothesis and how is it tested?

Allows us to make more specific conclusions. Bayesian method use Bain module to assess this, you state the hypothesis of interest and a competitor such as the unconstrained (leaves open all options), the null, the complement or a competing theory. Check the Bayes Factor Matrix.

How do you choose the best model?

- One with the highest BF, if this is above 10 it receives strong support.

- The one with the highest PMP.

What is the error probability?

The sum of the PMPs for the other hypotheses you didn’t choose, also called conditional error probabilities.

What are polynomial contrasts?

- It tells us the pattern/trend in data.

- Linear means we are testing if there is a linear in/de-crease in data (do all means go up/down).

- A quadratic contrast means a u-shape or inverted u-shape, so if group 1 is higher than group 2 and group 3 is higher than group 2.

- Quadratic can give us a significant result if the repeated contrast doesn’t give us a significant result due to the higher power