PSYC 217 Lecture 9 - Replication Crisis, Open Science, and Qualitative Methods Wrap-up

Beyond Single Studies: Meta-Analysis and Null Results

• Meta-analysis: This is a technique that allows researchers to combine results from multiple studies to get a better understanding of what the actual effect size might be across a body of literature.
• The Problem of Publishing Null Studies: There is a bias in scientific publishing where significant results are more likely to be published than non-significant ones. For example, in a series of studies where Result 1 is significant, Result 2 is non-significant, and Result 3 is significant, the non-significant result may often be omitted or remain unpublished, skewing the overall perception of a phenomenon.

Learning Objectives

• By the end of this lecture, students should be able to:
  • Understand the "replication crisis" in psychology.
  • Explain different new practices implemented in response to the replication crisis.
  • Differentiate between different ways of performing qualitative research.
  • Differentiate between positivist and interpretivist approaches to research.
  • Describe the research process for qualitative research.
  • Describe specific qualitative methods: narrative research, grounded theory, and phenomenology.
  • Describe mixed methods research and the different ways it can be implemented.

The "Replication Crisis" and Questionable Research Practices

• The Replication Crisis: A phenomenon in psychology where many results from previous studies cannot be replicated by either the original researchers or independent researchers.
• Causes of the Crisis:
  • Questionable Research Practices (QRPs): Includes actions that inflate the likelihood of finding significant results.
  • p-hacking: Manipulating data or statistical analyses until a non-significant result becomes significant ($p < 0.05$).
• Questionable Studies and Examples:
  • Daryl Bem: Published a controversial study in 2011 claiming evidence for extrasensory perception (ESP).
  • The Macbeth Effect: A phenomenon where being exposed to a moral transgression creates a need to wash one's hands or engage in physical cleansing.

New Directions: Open Science and Transparency

• Open Science: A movement to encourage research data and materials to be accessible to everyone. It encourages transparency, community, and collaboration, allowing for better scrutiny and quality assessment of research.
• Preprints: These are unpublished versions of research papers made available online before formal publication. They have no pay-wall and are publicly accessible, though a major drawback is that they are not yet peer-reviewed.
• Psychological Science’s Open Badges:
  • Open Data: Authors provide the raw data for anyone to access and analyze.
  • Open Materials: Authors provide study materials (e.g., surveys, stimuli) for others to check or use for replication.
  • Pre-registration: Authors register their hypotheses and analysis plans before running the study to prevent p-hacking and hindsight bias.
• Registered Replication Reports (RRRs): Facilitated by the Open Science Foundation (OSF), this involves a specific process:
  • Write a proposal to register the hypothesis.
  • Register materials and the planned analysis.
  • Upload all data and analyses.
  • Write a final report addressing the reproducibility of the registered hypothesis.

Collaborative Replication Projects: "Many Labs"

• Collaborative Efforts: Large-scale initiatives where multiple laboratories (Lab 1, Lab 2, Lab 3… Lab N) attempt to replicate the same original study.
• Many Labs 2 Results:
  • 28 findings were re-tested.
  • Included more than 60 samples.
  • Approximately $7000$ participants were involved.
  • Authored by 186 researchers from 36 nations.
  • Replication Rate: Only $50\%$ of the findings successfully replicated.
• Hedges' g: Used as a measure of effect size in these summaries across various labs (e.g., Pace University, University of Florida, Azusa Pacific University, Ithaca College, etc.).

Research Replication Outcomes: Successes vs. Failures

• Research that has replicated well:
  • False Memory: Findings regarding the creation of false memories.
  • Primacy and Recency Effects: The tendency to remember the first and last items in a list better than the middle items.
  • The Spacing Effect: Documented by Ebbinghaus (1885), showing that learning is more effective when study sessions are spaced out over time rather than massed together.
• Research that has not replicated well:
  • Power Poses: The idea that expansive body posture leads to hormonal changes and increased confidence.
  • The Pencil in Mouth Study: The facial feedback hypothesis suggesting that holding a pencil in the mouth to force a smile leads to increased feelings of happiness.
• Human Consequences: A failure to replicate can have significant impacts on the reputations of original researchers and the validity of established psychological theories.

Effect Sizes and Confidence Intervals

• Effect Size: A point estimate of how large an effect is based on a sample. It focuses on the magnitude of the difference between groups rather than just the p-value.
• Confidence Intervals (CIs):
  • A range of values reported to index the precision of an effect size estimate (e.g., $95\% \text{ CI } [0.14, 0.46]$).
  • It represents the chance that the interval captures the true population effect size.
  • Example intervals provided: $[0.14, 0.46]$, $[-0.20, 0.16]$, and $[0.14, 0.90]$.

Researcher Degrees of Freedom

• Subjectivity in Analysis: Statistical analyses are not purely objective; they are impacted by human decisions.
• Researcher Degrees of Freedom: The flexible decisions made after data collection, such as:
  • Which variables to control for.
  • Whether to collect more participants.
  • Which statistical models to use.
• Case Study: Soccer Data: 29 research teams were given the same data to determine if referees give more red cards to dark-skinned players. Using different statistical methods, teams found different relationships, ranging from "three times as likely" to "equally likely" (non-significant).
• Proposed Solutions:
  • Need for large sample sizes ($N$).
  • Split-half analysis: Dividing data to test findings on a second subset.
  • Distinguishing between Exploratory research (generating hypotheses) and Confirmatory research (testing pre-registered hypotheses).

Philosophical Approaches: Positivism vs. Interpretivism

• Positivism:
  • Assumes data are measurable, observable, and neutral.
  • Underlies quantitative methods.
  • Focuses on the relationship between variables ($X$ and $Y$), using operational definitions and statistical analyses to draw conclusions.
• Interpretivism:
  • Allows for data to be subjectively constructed.
  • Focuses on individual experiences and in-depth accounts.
  • Underlies qualitative methods.

Qualitative Research Process

• Steps in Qualitative Research:
  1. Collect Data: Utilizing techniques like interviews, talking circles, and archival research.
  2. Transcribe: Turning recorded data into written transcripts.
  3. Code: Identifying themes or codes within the qualitative data.
  4. Analyze: Thinking broadly about what the codes reveal regarding the research question.

Types of Qualitative Research Methods

• Narrative Research: Focuses on the stories individuals tell about their experiences.
  • Inductive Approach: A bottom-up approach where themes emerge from responses and patterns are found in the data.
  • Deductive Approach: A top-down approach where analysis begins with expected themes based on prior theories to test expectations.
• Phenomenology: Assumes there is a universality or commonality in particular experiences.
  • Generally inductive and purely descriptive.
  • Aims to make generalizations about aspects of an experience over time.
• Grounded Theory: Involves finding relationships between themes to generate and develop theory.
  • Expected relationships lead to hypotheses, which lead to more data collection for testing.
  • Data Saturation: Collecting data until no additional themes emerge.
  • This is an iterative process (Data Collection → Data Coding → Data Analysis), making it resource- and time-intensive.

Mixed Methods Research

• Definition: Combines both quantitative and qualitative research, assuming both data types are necessary to answer a research question.
• Examples of Implementation:
  • Qualitative: "Can you explain the ways you feel aggressive?" vs. Quantitative: "On a scale of $1-5$, how aggressive do you feel?"
• Sequential Forms:
  • Qualitative data can direct attention to relationships and help with theory generation, followed by quantitative data to support the theory.
  • Quantitative data can be collected first, followed by qualitative data to help clarify or explain the quantitative results.

PSYC 217 Wrap-up and Core Concepts

• Summary of Course Topics:
  • Science Foundations: Hypotheses, Falsifiability, Operationism vs. Essentialism, Variables, and Ethics.
  • Designing & Conducting Research: Measurement, Questionnaires, Experimental Design, Observation, Case Studies, Quasi-Experiments, and Factorial Designs.
  • Analysis and Interpretation: Descriptive Analyses, Correlation, Central Tendency, Variability, Probability, Inference Basics, t-ratio, Sampling Distributions, and Generalizability.
• The t-ratio Formula:     $t = \frac{\bar{X}_1 - \bar{X}_2}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}}$
  • Representation: $\text{Signal (Between-Group Difference)} / \text{Noise (Within-Group Variability)}$.

Final Conclusions in Research

• Probabilistic Nature: When research is done honestly, results are entirely probabilistic. Data either support the research hypothesis and are inconsistent with the null, or vice versa. There is always a risk of a Type 1 error.
• Philosophy of Science: Nothing is ever fully "proven" or "disproven." Scientific progress involves merely gaining confidence in theories and hypotheses over time.
• Skepticism: Maintaining a skeptical mindset is essential in evaluating scientific claims.