Lecture 8 Glossary - Replication, QRPs, Generalizability and Qualitative Research

Comprehensive vocabulary flashcards defining replication approaches, questionable research practices, open science procedures, meta-analysis, generalizability modes, and qualitative research standards.

Direct replication

Repeating the original study as closely as possible, with the same operationalizations, to see whether the original effect shows up again in new data; confirms the original finding but does not test it in a new context, and reproduces any construct- or internal-validity flaws the original had.

Conceptual replication

Testing the same research question — the same conceptual variables — but operationalizing them differently than the original study did.

Replication-plus-extension

Replicating the original effect while adding new variables or conditions to ask further questions.

Questionable research practices (QRPs)

Ways researchers can — often unintentionally — mislead about the strength of their evidence.

Underreporting null findings

A questionable research practice (QRP) where only the dependent variables or conditions that "worked" are reported, while the ones that did not are quietly omitted.

p-hacking

A questionable research practice (QRP) where many different ways of analyzing the same data are tried until a $p$-value just under $.05$ is found, without disclosing all the versions attempted.

HARKing

"Hypothesizing after the results are known"; a questionable research practice (QRP) where an unexpected, after-the-fact result is written up as though it had been predicted all along.

Open materials

A transparent research practice of publishing all conditions and measures that counters underreporting null effects.

Open data

A transparent research practice of publishing the full dataset that counters p-hacking.

Preregistration

A transparent research practice of time-stamping and publishing hypotheses and analysis plans before data collection to counter HARKing.

Open Research Practices checklist

A five-step framework comprising sufficiency (adequately powered sample size), preregistration, transparency (reporting the full research process and findings), availability (publicly accessible data/materials), and honesty (publishing corrections or retractions when mistakes are found).

Meta-analysis

Mathematically averaging the effect sizes of every study (published and unpublished) that has tested the same variables, to see what conclusion the whole body of evidence supports.

File drawer problem

The problem wherein meta-analyses built only from published studies can overestimate the true effect size because stronger, significant results are more likely to get published than null or negligible ones.

Publication bias

The tendency for significant, "positive" results to be published more often than null results, skewing the visible literature.

Theory-testing mode

Designing a study to test support for a theory, prioritizing internal validity over external validity — even if that means an artificial situation or a convenience sample.

Generalization mode

Shifting the goal to estimating whether, and how well, an effect or theory holds up in the real world for real populations — prioritizing external validity, usually through field research and probability sampling.

Universality assumption

The implicit assumption, in theory-testing mode, that if a mechanism is real it should show up in (almost) any sample — letting researchers validly test a theory on a convenience sample before testing generalizability later.

WEIRD

Western, Educated, Industrialized, Rich, and Democratic; describes the kind of unrepresentative sample that most psychology research actually draws on.

Cultural psychology

The study of how cultural context shapes thought, feeling, and behavior; a "special case" that blends theory-testing and generalization mode by deliberately testing whether findings hold across different cultural samples.

Ecological validity (mundane realism)

How similar a study's tasks and manipulations are to situations people actually encounter in everyday life; an aspect of external validity, generally higher in field research than lab research.

Experimental realism

How much a lab experiment is designed so participants experience authentic emotions, motivations, and behaviors, regardless of how artificial the surface setup looks — can be high even when ecological validity is low.

Quantitative research

Relies on numerical data and analysis to make objective frequency, association, and causal claims; minimizes the researcher's own influence and aims to generalize findings across a population, but focuses on the average rather than the individual.

Qualitative research

Aims to understand thought, feeling, and behavior by obtaining an in-depth account of participants' own perspective on their world; focuses on the individual, acknowledges the researcher's larger interpretive influence, and studies experience as situated in its natural context.

Purposeful (purposive) sampling

A qualitative sampling technique where participants are deliberately chosen because they can richly inform the research question, rather than selected to statistically represent a population.

Saturation

The point at which additional participants or data stop producing new themes or insights, indicating to a qualitative researcher when to stop collecting data instead of relying on a pre-set sample size.

Trustworthiness

Qualitative research's answer to the four validities, built from four parallel criteria: credibility, transferability, dependability, and confirmability.

Credibility

The qualitative parallel to internal validity; established through triangulation, detailed evidence, prolonged observation, and skillful interview technique.

Transferability

The qualitative parallel to external validity; established through a detailed description of the sample, setting, and results, so readers can judge whether findings apply to their own situations.

Dependability

The qualitative parallel to reliability; established through more than one researcher participating in analysis, peer debriefing, and rigorous, systematic procedures.

Confirmability

The qualitative parallel to objectivity; established through a detailed audit trail of procedures and interpretations, plus member checking (asking participants whether the researcher's interpretations make sense).