Study Notes on Replication, Transparency, and Research Practices

Replication and Transparency

• Replication
  • Definition: Will the same results be observed if the study is conducted again?
  • Importance:
  • Crucial for statistical validity.
  • A single study holds significance only if it can be replicated.
  • Scientific progress relies on replication of studies.

Types of Replication

1. Direct Replication

   • Researchers repeat the original study as closely as possible.

2. Conceptual Replication

   • Explores the same research question but uses different procedures.

3. Replication-plus-Extension

   • Replicate original study and add variables to test additional questions.

Conceptual Replication

• Relation between variables at different levels:
  • Abstract Level: Independent variable - Thinking about alcohol, Dependent variable - Aggressive thoughts.
  • Operational Level:
  • Alcohol vs. Plant photos
  • Alcohol ads vs. Control ads
  • Reaction time to aggression words
  • Rating Donald's behavior as "hostile"

Replication Crisis

• Overview:

  • Highlighted issues in psychological science regarding reproducibility.
  • Example: 97% of studies found significant results, but only 36% of replications were significant (Open Science Collaboration, 2015).

• Key Articles:

  • False-positive psychology due to flexibility in data collection and analysis; suggests reluctant dissemination of findings aligns more with bias than with accuracy.

Systematic Reviews and Meta-analyses

• Systematic Review:

  • A methodical process to find studies fitting specific criteria for answering research questions.
  • Reduces bias.

• Meta-analysis:

  • A statistical analysis providing a quantitative summary of findings from multiple studies.

Why Might Studies Not Replicate?

• Issues with replication attempts (irrelevant operationalization in new settings).
• Problems in original studies (like a small sample size).
• Publication practices favoring significant results.

Researcher Degrees of Freedom

• Refers to the flexibility researchers have in study design, data analysis, and reporting outcomes.
• This flexibility can lead to questionable research practices (QRPs).

Questionable Research Practices (QRPs)

1. HARKing: Formulating hypotheses after results are known.
2. p-Hacking: Adjusting the analysis or data until significant results (p < .05) are achieved.
3. Selective Reporting: Only reporting significant findings.
4. Underreporting Null Results: Ignoring findings that do not support hypotheses.

Consequences of QRPs

• Inflated evidence for hypotheses, unreliable findings that do not replicate, and overall inflation of p-values and effect sizes.

Solutions to QRPs

• Transparency: Open data sharing for verification of results.
• Pre-registration of studies and analysis plans before conducting research.
• Implementing larger sample sizes to improve study reliability.

Open Science Movement

• Advocates for free sharing of data and materials to encourage collaboration and verification.

• Core Principles:

  • Transparency
  • Accessibility
  • Inclusivity

• Resources:

  • Open Science Framework (https://osf.io/) for pre-registration and data sharing.

Generalizability in Research

• Generalization Mode: Emphasis on ecological validity and representative sampling from the population.
• Theory-testing Mode: Focused on internal validity, testing specific theories, often involving WEIRD (Western, Educated, Industrialized, Rich & Democratic) samples.

Cultural Psychology

• Examines how cultural contexts shape individuals' thoughts, feelings, and behaviors.
• Challenges traditional approaches in psychology by emphasizing diverse backgrounds.

Exam Preparation

• Final Exam Details:

  • Date: April 14, 2-4 PM
  • Format: 80 multiple-choice questions covering cumulative material.
  • Materials to bring: Pencil, eraser, Carleton ID.

• Topics to Review:

  • Scientific methods and norms, types of research, reliability, validity, QRPs, open science, and data generalization practices.