Advanced Research Methods: Reliability, Reproducibility, and Open Science in Psychology

Recapping the ‘Crisis’ and Moving Forward

• The lecture begins with a recap of the situation in psychology and related sciences regarding questionable research practices (QRPs) that were prevalent until about ten years ago.
• While the field is not yet perfectly executing research, there has been a significant realization that these practices needed to be changed.
• Improvements are actively being made, and recent developments focus on structured methods to enhance the reliability of scientific findings.

Questions & Discussion: Comparing Study Results

• The speaker presents a side-by-side comparison of two independent studies to gauge which result should be more surprising.
• Scenario 1: A well-powered study finds that sleep deprivation impairs attention.
• Scenario 2: A small study finds that a $10\text{ minute}$ app permanently raises IQ by $15$ points.
• Audience Observation: Most students identify the second study as more surprising.
• Trustworthiness Factors for Scenario 1:
  • It is "well powered," meaning the statistical design and number of participants are sound.
  • It is intuitive because it aligns with prior knowledge and existing evidence regarding the relationship between sleep and attention.
• Red Flags for Scenario 2:
  • It is a small study, making the results potentially less reliable.
  • The claim is highly specific yet lacks generalizability.
  • The IQ Metric: IQ tests are centered on an average of $100$ for the population. If the population gets smarter, the tests are made harder to maintain this mean. The standard deviation is typically $15$ points. Therefore, a $15\text{-point}$ increase represents a Cohen’s d of $1$, or one entire standard deviation. Moving a population up by an entire standard deviation permanently via a $10\text{ minute}$ app is highly unlikely.
  • Compared to proven methods of raising IQ—such as better nutrition in malnourished countries or long-term education—a short-duration app lacks external plausibility.

Evaluating Evidence: Prior Plausibility vs. Statistical Significance

• Prior Plausibility: This refers to the likelihood a hypothesis is true based on previous evidence and theoretical knowledge before a new study is conducted.
• The P-value Limitation: Standard statistical tests (e.g., t-tests, ANOVA, f-tests) result in a p-value that does not take prior plausibility into account. A p-value of $0.05$ is the same regardless of whether the hypothesis is far-fetched or robustly supported by history.
• Scientific Bar for Evidence: If a result goes "against the grain" or is highly surprising, the bar for evidence should be significantly higher. Relying on literature and prior theoretical knowledge is the first step in factoring in plausibility.

The Three Tiers of Repeatability: Reproducibility, Robustness, and Replicability

To evaluate the body of research, scientists focus on three distinct themes often referred to as a "three-tier system":

• Reproducibility (The Low Bar):
  • Definition: Taking the data from the original article and running the exact same analysis code to see if the results match.
  • Requirement: Access to original data and the authors' code.
• Robustness (The Middle Bar):
  • Definition: Using the same original data but running alternative analyses (ideally improvements on the original model) to see if the conclusions remains similar.
  • Requirement: Reaching the same interpretation even if the decimal points do not match exactly.
• Replicability (The High Bar):
  • Definition: Conducting the study again by collecting entirely new data to see if the original claim is supported.
  • Requirement: New participants and similar evidential outcomes.

Current State of the Field: Recent Data on Reliability

Historical data from around $11$ years ago suggested that replicability in psychology was around $33\%$ to $36\%$. Recently published data (April 1st) provides updated numbers across the three tiers:

• Data on Reproducibility:
  • Precisely Reproducible: $53.6\%$.
  • Approximately Reproducible: Nearly $75\%$.
  • Shared data and code policies correlate with better reproducibility. Currently, fields like political science and economics have higher sharing rates than psychology, which sits in the middle.
  • Reasons for non-reproducibility:
  • "Background" actions not documented in code (e.g., manual outlier removal).
  • Missing steps between raw data and final analysis.
  • Reluctance to share data due to fear of errors being discovered or participant privacy concerns.
  • Technical failures: Code simply failing to run across different systems.
• Data on Robustness:
  • Only $34\%$ of re-analyses landed very close to the original effect size.
  • A higher percentage reached the same general conclusion, but some found no effect or even the opposite effect.
  • The general trend shows that re-analysis effect sizes are smaller than the original reports.
• Data on Replicability:
  • Effect sizes typically shrink upon replication.
  • In a recent large-scale assessment, the original median correlation ($r$) was $0.25$, which decreased to $0.1$ in replications.

Identifying Red Flags in Scientific Literature

When consuming research, specific "warning signs" invite caution regarding the validity of claims:

• Small samples paired with large claims: (e.g., small $N$ claiming permanent IQ changes).
• Multiple outcomes: Measuring many variables but only reporting the ones that achieved significance.
• No clear primary hypothesis: The lack of a defined goal before the study began.
• Closed Materials: No sharing of data, code, or specific materials like questionnaires.
• P-values near the threshold: A consistent pattern of p-values just below $0.05$.
• Causal Language from Correlational Data: Asserting X "causes" Y when the study design only shows a relationship.
• Lack of context: No prior literature or direct replications to support findings.

Best Practices for Trustworthy Science

• Pre-registration:
  • Researchers document their study plan (participants, outlier exclusion, statistical models) on a site like osf.io before collecting data.
  • This prevents being influenced by the data results during analysis (undisclosed flexibility).
  • Confirmatory vs. Exploratory: Pre-registration allows readers to distinguish between planned tests used to confirm hypotheses and exploratory tests conducted after seeing the data.
• Open Materials and Data:
  • Sharing questionnaires and code makes the process transparent.
  • Synthetic Datasets: When the original data is too sensitive to share (e.g., rare clinical conditions where participants could be identified), researchers use algorithms to systematically modify values. This preserves the correlations and statistical patterns while ensuring individual participants cannot be traced.
• Registered Reports:
  • This is a more advanced version of pre-registration.
  • The study plan is sent to a journal for peer review before data is collected.
  • If the design is sound, the journal gives an "in-principle acceptance," agreeing to publish the results regardless of whether they are significant or support the hypothesis.
  • This shifts the incentive from finding "sexy" results to conducting high-quality, rigorous science.

Conclusion: The Goal of Higher Standards

• The objective is not to be cynical about psychology but to move toward a cumulative science where results can build upon each other.
• Transparency vs. Bold Claims: Scientific claims should move from absolute statements (e.g., "X proves Y") to measured statements (e.g., "In this design with this sample, we observed evidence consistent with X predicting Y").
• Trustworthy science is built on transparency, transparency regarding tests run, data selection, and clear labeling of exploratory findings.