P-values & Statistical Significance

p-values

a test of “statistical significance” using a p-value is the most common statistical test to be found in the published research literature

-we will focus on understanding p-values and their widespread misuse in Null Hypothesis Significance Testing (NHST)

Null Hypothesis Statistical Testing (NHST)

NHST as reported in most published papers is a problematic combination of R.A. Fisher’s significance test and the hypothesis testing approach described by Neyman & Pearson

-they hybrid nature of NHST is at the root of its problems

too many “positive” results with P<0.05

vast majority of published studies report positive results; >90% in some disciplines

-this would be very unlikely if there was genuine uncertainty about the hypotheses tested

-Many published results are false

the results of most studies cannot be replicated

why is there too many positive results with P?

P<0.05 is not a very difficult threshold to cross

-the more tests one runs, the more likely statistically significant results are to emerge from the data analysis

-there are methods to adjust p-values when multiple tests are run (these adjustments are rarely used in public health)

multiple comparisons

can produce associations simply by chance

-must control the alpha at 0.05

family wise error rate

probability of obtaining at least one false positive in a family of hypothesis tests

data dredging (selective outcome reporting bias)

there are two basic ways of getting a “statistically significant” result from a data set:

-Hypothesizing After the Results are Known (HARKing)

-P-hacking

HARKing

-obtain a lot of data with lots of variables

-conduct lots of data analysis making comparisons between subjects and variables using p-values

-identify an association with a p-value below 0.05

-make up an explanation about this association

-write a paper that presents this after the explanation as a pre-specified hypothesis that your study was always designed to test

P-hacking

1. start with a vague hypothesis (Program x reduces adolescent drug use)

2. collect lots of data about drug use (daily, weekly, lifetime)

3. run a data analysis examining the effects of participating in Program X on drug use

4. if the main effect isn’t statistically significant, keep running analyses or try different combinations (only alcohol, cigarettes and alcohol, etc.)

5. once the p-value below 0.05 is reached, write your paper

6. do not report any analyses conducted that produced p-values above 0.05

practical significance

tells us nothing about the clinical significance of a result

-used to study whether results of a study are meaningful in the real world

unfalsifiable hypothesis

cannot be proven false because it is impossible to test conclusively