Lecture 16: Replicability and Generalizability

replication

if you did the same study again, would you get the same results?

• important for research integrity

• to be important, must be replicable

direct replication

repeating the methods of a study as closely as possible to see if the same result occurs with a different sample

• primary goal → to make sure the effect was not specific to those participants, that lab, those researchers, etc

• secondary goal → to make sure the researchers did not engage in questionable practices, or even outright fraud

conceptual replication

testing the same research question with different methods

• goal → to make sure effect was not operationalization-specific

• same conceptual variables, different operationalization definitions

  • could use different manipulation of the IV and/or a different measure of the DV

replication-plus-extension

repeating the original methods and adding something new

• goal → to determine “boundaries” of the effect

• additional sample(s), condition(s), and/or measure(s)

meta-analysis

a mathematical compilation of studies that all tested the same effect

• includes direct replications, conceptual replications, and replications-plus-extensions

• tests overall effect and moderators of the effect

file drawer problem

meta-analyses tend to overestimate effects because null and contradictory effects are difficult to publish

• null effects get forgotten in file drawer

• one solution: contact colleagues to collect “failed” studies

  • another solution: statistically estimate how many null studies would have to exist to undo the effect (fail-safe N)

fail-safe N

statistically estimate how many null studies would have to exist to undo the effect

pre-registration

documenting your planned study before conducting it and making the results available regardless of the “success'“ of the study

external validity

partly addressed through conceptual replications and replication-plus-extension

ecological validity

does the situation you create in your study resemble the real world?

field studies

studies that take place in the real world instead of a contrived setting (i.e., a lab)

generalization/discovery mode

• frequency claims

• goal is to make a claim a/b a population

• OR discover a phenomenon

• real world matters

• external validity essential

theory-testing mode

• association and causal claims

• goal is to test a theory rigorously, isolate variables

• prioritize internal validity

• artificial situations may be required

• real world comes later (or first)

• external validity NOT the priority