Replication Crisis in Psychology (1)

The Replication Crisis in Psychology and the Open Science Movement

Overview of the Crisis

• The replication crisis refers to recurring challenges in replicating psychological studies, raising questions about the reliability of scientific findings.

• The Open Science Movement aims to increase transparency and accessibility of research processes and findings in psychology.

Key Events in the Crisis

2011: Significant Papers and Claims

• Daryl Bem's ESP Study: Published a paper in the Journal of Personality and Social Psychology (JPSP) claiming evidence for extrasensory perception (ESP).

  • The significant effects were criticized for not correcting the familywise error rate, leading to excessive false positives (Type I errors).

Data Fabrication and Faux Findings

• Diederik Stapel Case: Discovered in 2011 for fabricating data.

• Overlapping issues in scientific methods resulted in major findings failing replication, including:

  • Ego depletion

  • Embodied cognition

  • Power posing

  • Behavioral priming

Open Science Collaboration (2015)

• Attempted replicating 100 studies from top psychology journals:

  • Only 39% of original studies were successfully replicated.

  • 25% replication success rate in social psychology versus 50% in cognitive psychology.

  • Effect sizes were frequently overestimated in original studies.

Challenges to Replicability

• Sampling Variability: Results can vary significantly due to differing sample characteristics.

• Hidden Moderators: Variables like cultural context and demographics affect outcomes.

• Low Statistical Power: Contributes to difficulties in replicating findings, with false positives potential in original studies and false negatives in replication attempts.

Publication Bias in Academia

Bias Towards Significant Findings

• Academic journals preferentially publish:

  1. Significant findings over non-significant ones.

  2. Novel findings with surprising outcomes,

• Non-significant findings often end up in a researcher’s "file drawer" and never see publication.

Academic Incentives

• Academia encourages the pursuit of "flashy" results to secure promotions, jobs, and media coverage.

• This incentivization leads to questionable research practices and increases the prevalence of false positives in published literature.

Error Types and Consequences

False Positives and False Negatives

• Type I Errors: Incorrectly claiming an effect exists when it doesn't (e.g., p < .05).

• Type II Errors: Failing to recognize an effect that actually exists.

• A study by Simmons, Nelson, & Simonsohn (2011) illustrated how easily false positives can be obtained through statistical manipulation.

Implications of False Findings

• Health research can wrongly indicate effectiveness of treatments, leading to wasted resources and erosion of credibility in psychological science.

• Increasing false positives creates skepticism around published research.

Methodological Challenges in Research

Questionable Research Practices

• Researchers may engage in practices that go against scientific integrity, motivated by the need to validate results statistically.

• Decisions in study conduct, from sample size to handling outliers, impact the chance of achieving significant results.

Data Collection Techniques

• Commonly, researchers collect data, analyze, and if results are not significant, continue data collection until achieving p < .05。

• Frequent testing to reach significance increases the likelihood of false positives.

Solutions to Reduce False Positives

Proposed Changes in Research Practices

• Prespecified Termination Rules: Researchers should determine data collection rules beforehand.

• Minimum Observations: Recommends at least 20 observations per cell in experimental designs.

• Disclosure Requirements: Full disclosure of all variables used, including details of failed manipulations and changes in analyses.

Importance of Open Science

• Open practices can help eliminate selective reporting and increase transparency, improving the credibility of psychological research and findings.

The Role of Hypothesis Development

Bad vs. Good Hypotheses

• Bad Hypothesis Example: Non-directional claims that lack clarity.

• Good Hypothesis Example: Directional, stating clear expectations regarding performance discrepancies (e.g., cultural comparisons) with process models included.

Preregistration and Transparency

• Researchers encouraged to preregister hypotheses and methodologies to ensure rigorous examinations.

• The use of public repositories for findings helps mitigate the file drawer problem.

Advanced Statistical Methods

Importance of Effect Size

• Cohen's guidelines on effect sizes indicate necessary sample sizes for detecting small effects.

• Use tools like G*Power to determine statistical power.

Exploratory vs. Confirmatory Research

• Distinction made between exploratory (theoretical) work and confirmatory studies; both serve important roles but should be labeled correctly to avoid misrepresentation.

Behavioral Priming as a Case Study

Mechanisms and Outcomes

• Behavioral (or social) priming influence outcomes without conscious awareness, illustrated by the elderly walking study.

• It was replicated only under certain expectations.

Research Design Considerations

• Substantial study designs across various subjects, indicating that significant effects require robust methodologies. Attention needed to verify findings across diverse populations.

Final Thoughts on Science and Innovation

• Emphasis on the equivalence of replication and innovation in the scientific process.

• Psychological research is evolving, embracing a more open scientific culture, supported by social media for broader dissemination and engagement.