lecture 9- alpha error accumulation

What are decision errors?

• there is a test decision at end of hypothesis test

  • P<.05 - significant, test decision for H1, non-zero effect

  • P>.05 - non-significant, test decision for H0, effect may be zero

• This decision may be wrong

• type I / alpha errors are controlled in the hypothesis test

  • Comparing p to 0.05. 5% probability that a test decision for H1 even though H0 is true.

• Type II / beta errors are reduced by high statistical power (sample size)

  • Stat power= probability of detecting an existing effect

  • Type II - 1-power (prob of not detecting an effect if it exists)

What is the problem of multiple testing?

• researchers often conduct more than one hypothesis test

• Can be legit or because they are fishing

• What is the probability they make an alpha error across multiple tests?

What is a composite hypotheses?

• an OR hypothesis

• E.g. H1: E ≠ S OR E ≠ W OR E ≠ NI OR S ≠ W OR S ≠ NI OR W ≠ NI

• if at least one of the statements is true, then H1 is true

• Each statement is tested with one t-test

What is the probability of making a correct decision for H0 in multiple testing?

E.g. If there is truly no difference in knowledge between countries:

• probability of NOT making an alpha error in a single test: 1-a

• Probability of NOT making alpha error in any test: 1 - ɑ) ・(1 - ɑ) ・(1 - ɑ) ・(1 - ɑ) ・(1 - ɑ) ・(1 - ɑ)

= (1 - ɑ)6

• for a = 0.05, this is 73.5%

• So 26.5% chance of making at least one error

• This is alpha error accumulation.

What is alpha error accumulation?

• increase in probability of making a type 1 error when conducting multiple statistical tests

What is family-wise error rate?

• probability of making at least one type I error across multiple tests

What is comparison-wise error rate?

• probability of making a type I error in a single test

What is the Sidak correction method?

• an alpha error correction method- lowering the alpha level in each test to keep family-wise error rate small

• Probability of making at least one type I error: 1 - (1 - ɑ)N

• Simply invert the function to calculate the FWER. α= 1-√(N&1-FWER)

What is the Bonferroni correction method?

• α=  FWER/N

• Bonferroni always corrects a bit too much but is much easier and used more often.

When should you correct?

• the goal of a hypothesis test is to provide a way of deciding about a research hypothesis

• This decision is necessarily temporary / intermediate and part of a bigger research program where theories can change over time

• Only correcting at the level of an individual study is enough - usually test one research hypotheses

• If they are composite hypotheses that require multiple tests, these are a family of tests.

• Correct for families, not all tests.

• Use alpha error correction methods for OR hypothesis but not for AND hypothesis

Are all families equal?

• correction methods need to be applied for OR hypotheses where discovery is claimed if at least 1 is significant

• But not all composites are OR hypotheses e.g. reading a book before bed is better for sleep quality than a podcast, which is better than scrolling on your phone.

• OR hypotheses: single type I error in any test- wrong decision for H1- grows more likely with many tests.

• AND hypotheses: all hypothesis tests need to make type 1 errors for a wrong decision for H1- grows less likely with many tests.