tutorial - reliability

What is scientific reliability about?

Repeatability and reproducibility

If a scientific observation is reliable, it is repeatable and reproducible.

Which of these is scientific reliability NOT the same as?

Accuracy/truth, trustworthiness, objectivity, or validity

Reliability is distinct from accuracy/truth, trustworthiness/credibility, objectivity, and validity.

What was one key reason behind the replication crisis?

A lack of reliability, repeatability, and replicability of published studies

The replication crisis stemmed largely from findings that couldn't be reliably repeated or reproduced.

In the Reproducibility Project, 97% of the 100 original studies showed a statistically significant effect. What percentage showed significant findings upon replication?

36%

Only 36 of the 100 replicated studies found significant findings, despite 97% originally showing significance.

What does inter-rater reliability measure?

The extent to which two or more independent observers/raters agree when scoring the same thing

High inter-rater reliability means different raters would land on roughly the same result.

What does test-retest reliability measure?

Whether a person gets a similar score on the same test given on two separate occasions

High test-retest reliability means the measure is picking up something stable, not just day-to-day noise.

What does internal reliability refer to?

Whether items in the same questionnaire agree with each other

High internal consistency means people who score higher on some items tend to score higher on other related items too.