Social Psych 2 Lecture 12 The Reproducibility Crisis and Credibility Revolution

attention economy of scientific journals

scientists compete for work to be published so they want unique data/conclusions

increasing scientific publications each year = increasing strain on scientists to read and review = decreasing reliability of review = magnifying the attention economy for scientists other compete to have work noticed and have an impact

response to reproducibility crisis

credibility revolution and open science movement: replication of published psych studies to ensure they have significant results

vulnerabilities: p-hacking (selectively running tests until 0.05 appears), low statistical power (probability hat the study can correctly detect true difference), publication bias to positive results

why is there reproducibility crisis

p-hacking: selectively running tests until reach p<0.05

low statistical power: does not correctly detect true difference

publication bias to positive results because of attention economy

reproducibility

many published research findings need to be reproduced by independent researchers using the original study’s existing data and computer code/running analysis

verifies data integrity (checks if the tests are done honestly/accurately)

combats bias and fraud (transparency)

supports open science

reproducible workflow: scripted analysis with code so traceable every step and anyone can rerun it on public repository which allows for in-depth scrutiny, re-analysis, equity among researchers

replicability

running a braw-new study with fresh data to see if you get the same overall finding

fighting reproducibility crisis

preregistration: research lock in hypothesis, sample sizes, and analysis on platforms before collecting data

registered reports: journals review and accept study design before data collection

preregistration

researchers lock in hypothesis, sample sizes, and analysis on platforms like Open Science before collecting data

registered reports

journal review and accept study designs before collection

questionable research practices (QRP)

p-hacking (testing until p-value reached) or HARKing (hypothesizing after the results are known)

Limbo (bystanders), overselling, post-hoc storytelling, p-value fishing, creative outliers, plagiarism, non-publication, partial publication, inventing data (LaCour data fraud)

injunctive/prescriptive norm

what SHOULD happen

science SHOULD be reproducible = ideal

descriptive norm

what ACTUALLY happens

many published findings do not replicate

how can study wrong

false positive: results say there is an effect but there isn’t (arose from chance and questionable research practices (p-hacking/HARKing) and low statistical power)

false negative: results say there is no effect but there is (arose from low statistical power)

p-hacking

running tests until statistical significance is reached - so many times that chance alone resulted in positive effect

excluding or including outliers, changing probability test, running tests separately for different traits

reproducible workflow:

scripted analysis with code so traceable every step and anyone can rerun it on public repository which allows for in-depth scrutiny, re-analysis, equity among researchers

scientific sketicism

not cynical.

blinding

unaware of information that can bias them

preventing placebo effect, experimenter bias, p-hacking