Quasi Experiments, Small N Designs, Replicability and Generalizability

quasi independent variable

don’t actually manipulate the variable (ex. can’t manipulate how many languages someone speaks)

quasi experiments

researchers identify an independent variable but do not manipulate it or randomly assign participants to conditions

• true experiments call for manipulation of independent variable and random assignment of people to conditions

when are quasi experiments necessary

when the independent variable cannot (practically) or should not (ethically) be manipulated

nonequivalent groups design

a quasi experimental study that has at least two groups (between subjects)

repeated measures quasi experiments

• interrupted time series design

• interrupted time series with reversal

• nonequivalent control group interrupted time series

(within subjects)

interrupted time series design

a quasi experiment that measures the dependent variable repeatedly before, during, and after some event (the interruption)

• ex. working memory test before introducing a second language program, assess memory after program starts; if memory improves after program, maybe program did help

interrupted time series with reversal

an interrupted time series design in which measurements continue after things have gone back to their originals state

• ex. working memory before program, working memory during program, assess working memory after program starts and see if it returns to OG levels

increases internal validity

nonequivalent control group interrupted time series

include a control group for which the key event does not occur

• one group takes measure before, during, and after interruption

• second group don’t take the program and continue with tests

internal validity and quasi experiments

always lower in quasi experiments than in well designed true experiments

selection effects

the major potential confound in nonequivalent groups designs (participants may have huge differences between levels/groups)

matched groups

a particularly useful strategy in quasi experimental designs (but also relevant to true experiments) [make sure groups are equivalent in certain ways despite not randomly assigning]

isn’t a quasi experimental just a correlational study

quasi experiments typically improve on correlational studies by attempting to rule out certain threats to internal validity by establishing temporal precedence

small n designs

gathering a lot of info from a small sample instead of a small amount of information from a big sample

single n design

a study of a single person or animal’s experience (case study)

advantages of a small or single n design

• allows researchers to study rare people or events

• provide rich data about a narrow span of experience (depth)

• avoids problems associated with averaging across participants

disadvantages of a small or single n design

findings may not generalize (low external validity)

replicability

if you did the same study again, would you get the same results

direct replication

repeating the methods of a study as closely as possible to see if you get the same results with a different sample

direct replication primary goal

to make sure the effect was not specific to those participants, that lab, those researchers, etc.

direct replication secondary goal

to make sure the researchers did not engage in questionable practices, or even outright fraud

conceptual replication

testing the same research question with different methods

• same conceptual variables, different operational definitions (different manipulation of the IV, different measure of the DV

conceptual replication goal

to make sure the effect was not specific to a particular operational definition of the variables

replication plus extension

repeating the original methods and adding something new

• determine boundaries of effect by adding additional samples, conditions, and or measures

possible definitions of successful replication

• statistical significance: was the effect significant before? is it this time?

• effect size: what was the effect size before? is the new effect size similar?

meta-analysis

a mathematic compilation of studies that all tested the same effect

• includes direct replications, conceptual replications, and replication plus extensions

• tests overall effect and moderators of the effect

  • ex. priming —> behavior

  • potential moderators —> time between priming and behavior measure, nature of prime (words or pictures), etc.

file drawer problem

meta-analyses tend to overestimate effects because null and contradictory effects are difficult to publish

solutions to file drawer problem

• contact colleagues to collect failed studies

• preregistration

preregistration

documenting your planned study before conducting it and making results available regardless of the success of the study

replication and external validity

conceptual and replication plus extension replications partly addresses external validity (does findings extend to other samples and contexts)

ecological validity

does the situation you create in your study resemble the real world (aka mundane realism)

field studies

studies that take place in the real world instead of in a contrived setting

experimental realism

is the situation you create in your study engaging, eliciting ‘real’ emotions, motivations, and behaviors (make sure participants are engaged in situation created in lab)