Statistical significance and QRPs

Null Hypothesis Significance Testing (NHST)

A statistical method that evaluates whether observed data differ significantly from a null hypothesis, which assumes no effect or relationship.

p-value

The probability of observing results as extreme as the ones obtained, assuming the null hypothesis is true.

Type 1 Error (False Positive)

Incorrectly rejecting a true null hypothesis, concluding an effect exists when it does not.

Type 2 Error (False Negative)

Failing to reject a false null hypothesis, concluding no effect exists when one does.

Familywise Error

The increased probability of making at least one Type 1 error across multiple statistical tests.

Questionable Research Practices (QRPs)

Research behaviors that compromise the integrity of findings, such as selective reporting, cherry-picking data, or p-hacking.

p-Hacking

Manipulating data or analyses (e.g., repeatedly testing or stopping data collection early) to achieve a statistically significant p-value.

HARKing (Hypothesizing After Results are Known)

Formulating hypotheses after analyzing the data and presenting them as if they were pre-planned.

Sharpshooter Fallacy

Drawing conclusions by selectively focusing on patterns that fit a hypothesis, ignoring data that doesn’t.

File Drawer Problem

The tendency for studies with null or insignificant results to remain unpublished, skewing the literature.

Publication Bias

The preference for publishing studies with significant or positive results, leading to an unbalanced representation in the literature.

Pre-registration

The practice of publicly documenting study methods and hypotheses before data collection begins to prevent biases.

Registered Reports

A publication format where study plans are peer-reviewed and approved before data collection, ensuring transparency and methodological rigor.