Multiple Testing

Define Multiple Testing

Multiple testing is about accounting for the number of hypotheses tested when interpreting statistical evidence

What is p hacking

The misuse of data analysis to force a study's results to look statistically significant (p < 0.05) even when no real underlying effect exists

What is FWER

The family-wise error rate is defined as the probability of any false positive in a family of tests

What is the process we follow in multiple testing

ANOVA —> Post Hoc

Methods for Padj

Disadvantage of bonferroni

It is conservative (cant be applied when the n number of tests is large)

Low power

Tukey is more powerful than Bonferroni for pairwise comparisons and should be preffered. True or false

True

Which 2 tests are specific to an ANOVA design

Tukey and Dunnett

Multiple testing and multiple comparisons- which one is overlapping and which one is not

overlapping- multiple comparisons

non overlapping- Multiple testing

Define FDR

The false dicovery rate (FDR) is the proportion of false positives among all positives (detected associations) in a family of tests

FDR when is it useful?

FDR controls the fraction of false positives in this smaller set

False discovery rate controlled by the

Benjamini-Hochberg procedure

Where does the subset property hold

FWER (Bonferroni): Yes, the subset property holds

FDR (Benjamini-Hochberg): No, the subset property does not hold

A researcher investigates the expression of 500 genes in cancer patients compared with healthy controls. All 500 null hypotheses are actually true.

a) How many p-values would you expect to be below 0.05 purely by chance? (1 mark)

b) Assuming all tests are independent, calculate the probability of obtaining at least ONE false positive among the 500 tests. (1 mark)

c) Which correction would you use to control the family-wise error rate at 5%? (1 mark)

A researcher investigates whether 10 different chemotherapy drugs have an effect on tumour volume. They perform 10 independent hypothesis tests.

The following unadjusted p-values are obtained:

Drug	Unadjusted p-value
A	0.001
B	0.003
C	0.008
D	0.012
E	0.025
F	0.031
G	0.041
H	0.050
I	0.072
J	0.091

a) Calculate the Bonferroni-adjusted significance threshold using α = 0.05. (1 mark)

b) Which drugs remain statistically significant after Bonferroni correction? (1 mark)

c) Explain why Bonferroni correction is considered conservative. (1 mark)

A researcher investigates 5,000 proteins to identify potential biomarkers for ovarian cancer.

They apply two different corrections:

• Method A: Bonferroni correction.

• Method B: Benjamini-Hochberg correction.

a) Which method controls the family-wise error rate, and which controls the false discovery rate? (1 mark)

b) The researcher is conducting exploratory research and wants to identify as many promising proteins as possible for future validation. Which method would you recommend? Explain your answer. (1 mark)

c) Which method is generally more conservative, and why? (1 mark)

A researcher investigates the differential expression of 10,000 genes between cancer patients and healthy controls.

After applying Benjamini-Hochberg correction to all 10,000 genes, the researcher selects 50 significant genes related to the immune system.

The researcher then claims that the false discovery rate among these 50 genes is guaranteed to be 5%.

a) Is this conclusion correct? Explain why or why not. (1 mark)

b) Would the same conclusion apply if the researcher had used Bonferroni correction instead? Explain. (1 mark)

c) Name the statistical property that explains this difference. (1 mark)

A researcher investigates 20 proteins to determine whether their expression differs between tumour and healthy tissue.

The p-values are stored in an R vector called p.values.

a) Write the R code to apply Bonferroni correction and store the adjusted p-values in an object called p_bonf. (1 mark)

b) Write the R code to apply Benjamini-Hochberg correction and store the adjusted p-values in an object called p_bh. (1 mark)

c) Write R code to count how many proteins remain statistically significant after each correction, using α = 0.05. (1 mark)