Research Methods Exam 3- Chapters 13 & 14

Quasi-experiments

when experimenters do not have full experimental control and cannot randomly assign

Independent group quasi-experiments

nonequivalent control group posttest only, nonequivalent control group pretest/posttest

Within group quasi-experiments

interrupted time-series, nonequivalent groups interrupted time-series

Nonequivalent control group posttest-only design

no random assignment of treatment and control groups, only tested once after exposure to each level of the IV

Organ donation example of nonequivalent control group posttest-only design

effect of being given the option to opt-in vs opt-out of organ donation on amount of people who are organ donors, nonequivalent groups because researches didn’t randomly assign the countries to either condition and did not randomly assign people to live in each country. Posttest only because people only tested after exposure to the opt-in/out option.

Nonequivalent control group pretest/posttest design

no random assignment to treatment and control groups, tested before and after exposure to each level of the IV

Interrupted time-series design

participants measured repeatedly on a DV before during and after an event occurs (ex: suicide rates & call volumes before and during release of Logic’s 1-800 song, the release of the music video, and awards shows)

Nonequivalent control group interrupted time-series design

participants not randomly assigned to groups but measured before, during, and after an event (ex states that have triplicate laws and states that do not before, during, and after the release of Oxytocin)

Which of the four validities do we lose and which do we gain in quasi-experiments

lose internal, gain external

Selection effect threat to internal validity in quasi-experiments

only relevant to independent groups designs, kinds of participants at one level of the IV are systematically different from those at the other level (ex group that chose not to have surgery likely different than group that went through with surgery)

What can be done to mitigate selection effects in quasi-experiments

use matched group or wait-list designs

Design Confound threat to internal validity in quasi-experiments

outside variable systematically varies with the levels of the IV

Maturation Threat to internal validity in quasi-experiments

occurs in designs with a pre and posttest, change resulted due to time rather than IV

How to mitigate maturation threats in quasi-experiments

include a comparison group to see if that group also has similar changes

History Threat to internal validity in quasi-experiments

external, historical event happens to everyone at the same time as the treatment variable (ex: Logic’s 1-800 song released at the same time as PSAs about the suicide hotline)

How to mitigate history threats in internal validity

include a comparison group that also went through the same historical event to see if that group also has similar changes

Regression to the Mean threat to internal validity in quasi-experiments

only in pre/posttest designs, group is selected because of their initially very high or very low scores and is closer to the mean next time they are tested

Attrition threat to internal validity in quasi-experiments

only in pre/posttest designs, only problematic when dropout is systematic (people who were disappointed in their surgery outcomes dropped out)

Why should you do quasi-experiments?

help you discover if the results apply to real world scenarios and increases external validity

Quasi-experiments vs correlational studies

quasi-experiments involve researchers actively selecting participants, but not manipulating the IV

Why Conduct a Small-N Experiment

a large N isn’t always necessary or possible especially when conducting research on human behavior or physiology

What can be done to make a causal statement from small-N experiments

using careful experimental control and studying special cases in which it isn’t necessary to generalize

Issues of small-N designs

internal validity when there is no experimental control over IV and external validity

Three types of small-N designs

stable baseline design, multiple-baseline design, reversal design

Stable-Baseline Design

observe behavior for an extended baseline period before beginning treatment

How do stable-baseling designs establish internal validity

if behavior during baseline is stable, we can be more sure of treatment effectiveness, ruling out alternative explanations

Multiple-Baseline Design

stagger the introduction of an intervention across a variety of contexts, times, situations, or participants (ex: autistic children fearing dogs- each participant had different baseline, treatment, and generalization time periods)

Reversal designs

observe a problem behavior with and without treatment, but take the treatment away for a while to see if problem behavior returns

When is it appropriate to use a reversal design

when treatment may not cause lasting change

Types of replication

direct replication, conceptual replication, replication plus extension

Direct Replication

repeat an original study as closely as possible (including flaws)

Which of the four validities does direct replication help establish?

internal validity

Conceptual Replication

using the same research question but different procedures/operationalization

Replication Plus Extension

repeat an original study but add variables/conditions to test additional questions

Replication crisis

journals prefer to publish new research rather than replications and replications often fail

What are the possible reasons a replication fails?

problem with the replication attempt, problem with the original study, or problem with publication practices

Meta-Analysis

mathematical average of the results of all studies that have tested the same variables

How is the effects of a meta-analysis calculated

by averaging all effect sizes to find an overall effect size

File Drawer Problem

a meta-analysis could be overestimating the true size of an effect because negligable or even opposite effects are less likely to be published

How to counter the file drawer problem

request both published and unpublished articles

Questionable Research Practices

underreporting null findings, HARKing, and p-hacking

What does underreporting null findings lead people to believe

that the evidence for a theory is stronger than it truly is

HARKing

Hypothesizing After the Results are Known

p-hacking

computing statistical analysis of data in a way that manipulates the p-value (remove outliers, compute scores different ways, run different types of statistics)

Family wise error

looking around in statistical analysis will result in finding a statistically significant result

Transparent Research Practces

open data & materials, preregistration

Open Data

provide full dataset so other researchers can reproduce the statistical results or conduct new analyses

Open Materials

full set of measures and manipulations are provided so other researchers can replicate more easily

Preregistration

counteracts HARKing. publishing study’s method, hypothesis, or statistical analysis in advance of data collection

Aspects to external validity

generalizing to a wider population and generalizing to other situations

If a sample comes from a population of interest does that automatically make it generalize to that population?

No

Whom do psychologists study?

mostly convenience samples and mostly from North America

Cultural psychology

examines how cultural contexts shape the way a person thinks, feels and behaves

How does the lack of diversity in research conductors impact generalizability

it may impact the questions being asked and how the studies are designed

How important is external validity in frequency claims

they must have external validity because the accuracy of the estimate depends on the external validity.

How important is external validity in association and causal claims

may not be necessary because it is sometimes more important to prioritize internal validity and testing theories

Theory-testing mode:

research to investigate support for a theory on any sample of populations, later replications can be conducted to generalize the findings

Lab study

takes place in a standard location where design and decor is held constant across participants

Field research

takes place in the real world

Replicating a lab study in the field or vice versa would be an example of which type of replication?

conceptual replication

Experimental realism

creating a lab setting that feels comfortable and natural so that participants respond more like they would in their normal environment

Are lab or field studies more important?

it depends on the purpose of the study