Recommendations for Increasing Replicability in Psychology - Asendorpf et al. (2013)

Recommendations for Increasing Replicability in Psychology

Abstract

  • The replicability of findings is crucial in empirical science.

  • This article seeks to provide concrete recommendations for improving replicability in psychological research.

  • It emphasizes the systemic challenges that must be addressed across various levels of practice, evaluation, and reward.

Key Terms

  • Replicability: The ability to obtain consistent results using the same methodology.

  • Confirmation Bias: The tendency to favor information that confirms one’s preconceptions.

  • Publication Bias: The tendency to publish studies with positive results more often than those with negative or inconclusive results.

  • Generalizability: The extent to which findings can be applied to settings other than the one in which the research was conducted.

  • Research Transparency: Openness about research methods and data.

Preamble

  • The article recommends sensible improvements for future research practices without focusing on past failures.

  • The proposals aim to enhance documentation, publication, evaluation, and funding practices in research.

  • Recognizes the pluralistic nature of science and suggests these recommendations serve as evaluative ideals rather than strict rules.

Context of the Current Replicability Debate

  • In recent years, questions regarding the replicability of psychological research have surfaced (Ioannidis, 2005; Lehrer, 2010; Yong, 2012).

  • Existing debates often focus on misconduct rather than systemic factors contributing to irreproducibility.

  • Issues include a lack of data transparency, a focus on eye-catching publications, minimal incentives for null results, and validation of contradictory findings within studies.

  • Psychological research leans towards confirming existing hypotheses, with Fanelli (2010) noting rates of confirmed hypotheses from 70% to 92%.

  • A global poll revealed psychologists estimated a mean replication rate of just 53% (Fuchs et al., 2012).

Psychological Research Practices and Confirmation Bias

  • Excessive flexibility in data collection and analysis contributes to publication bias.

  • Poll results indicate that 61% of psychologists decided to collect more data after reviewing initial outcomes, while 39% ceased data collection after finding significant results.

  • Multiple methods are used, often leading to selective reporting of significant results (Simmons et al., 2011).

Concerns About Statistical Null-Hypothesis Testing

  • The validity of the null-hypothesis testing approach has been questioned following implausible findings (e.g. Bem, 2011).

  • Critics suggest Bayesian statistics as an alternative (Wagenmakers et al., 2011) and advocate treating stimuli as random factors to assess variability in results (Judd et al., 2012).

Definitions of Key Concepts

Data Reproducibility

  • Refers to obtaining the same results from the same data set using identical methodology. Requirements include:

    • Raw data

    • Code book

    • Knowledge of the analysis performed

Replicability

  • Achieving the same findings in different random samples from a multidimensional space partaking in significant research facets:

    • Individuals

    • Situations

    • Operationalizations

    • Time points

  • Replications must show minimal differences attributable to unsystematic and sampling errors.

Generalizability

  • Demonstrates that findings are not dependent on any unmeasured variables exhibiting systematic effects.

    • Generalizability requires replicability but applies to extended conditions under which effects apply.

  • The relationship between these concepts is hierarchical: Reproducibility is necessary but insufficient for replicability; replicability is necessary but insufficient for generalizability.

Recommendations for Study Design and Data Analysis

Increasing Sample Size

  • Focus on increasing sample sizes to enhance power, reduce error, and improve replicability.

  • Typical median sample sizes in psychology are around 40, contributing to low power averages of approximately .35 (Bakker et al., 2012).

  • A recommendation for authors, editors, reviewers, and readers is to push for larger samples in research.

Increase Measurement Reliability

  • Higher reliability in measurement reduces error variance and improves replicability.

Enhance Study Design Sensitivity

  • Control methodological sources of error rigorously and maintain systematic identification of errors through clear instructions and study conditions.

Improve Statistical Analysis

  • Use appropriate statistical tests and treat samples as random factors to enhance analysis of variance in data.

Limit Multiple Underpowered Studies

  • Many underpowered studies may yield false results, leading to misinterpretations of consistency across findings.

  • Researchers must prioritize high-powered, singular studies over numerous low-powered ones.

Correct for Multiple Testing

  • Implement better error control methods than Bonferroni corrections, such as random permutation tests or false discovery rate procedures.

Establish Effective Replicability Checks

  • Utilize improved measures to evaluate if results can be quantitatively replicated. Recommendations include evaluating confidence intervals and running meta-analyses on effect sizes.

Recommendations for the Publication Process

For Authors

  • Take responsibility for transparency and assessment of replicability in research publications by addressing:

    1. Contribution to increasing research transparency.

    2. Acceleration of scientific progress.

Increasing Research Transparency

  • Provide comprehensive literature reviews, including previous studies’ replication status, sample size justifications, preregister predictions, and publish materials and data openly.

For Reviewers and Editors

  • Promote good practices by encouraging tolerance for non-confirmatory results, allowing discussions about papers, and reducing stringent regulations against publishing null results.

Recommendations for Teaching Research Methods

Goals for Instructors

  • Shift focus in education towards:

    1. Teaching rigorous methodology.

    2. Promoting critical thinking and the refutation of hypotheses.

    3. Establishing a culture prioritizing accurate research over publishability.

  • Encouraging students to conduct replication studies and evaluate evidence critically.

Recommendations for Institutional Incentives

Shift Focus to Quality Over Quantity

  • Emphasize quality in research over the quantity of publications in reviews, promotions, and grants.

Support Research Practices Through Funding

  • Mandate replication studies as a requirement for funded research; emphasize quality-based assessment for funding decisions.

Revise Tenure Standards

  • Change tenure regulations to value replicability and promote the publication of null results.

Alter Informal Guild Incentives

  • Foster an academic culture that accepts non-replicable findings as part of improving scientific progress.

Implementation Considerations

Infrastructure Needs

  • Ensure systems exist that facilitate sharing of research materials and findings, emphasizing ease of access and cooperation among researchers.

  • Highlight the Open Science Framework as an emerging solution for enhancing transparency in psychological research.

Conclusion

  • Reliability in findings (replicability) is foundational to the validity of conclusions in science, including psychology.

  • Multiple proposals exist to improve replicability, requiring both technical changes and shifts in academic culture toward transparency and rigorous methodology.

References (Partial List)

  • Bakker, M., Van Dijk, A., & Wicherts, J. M. (2012).

  • Cohen, J. (1988).

  • Fanelli, D. (2010).

  • John, L. K., Loewenstein, G., & Prelec, D. (2012).

  • Nosek, B. A., Spies, J. R., & Motyl, M. (2012).

  • Wachtel, P. L. (1980).