research methods exam 3

independent groups design

– posttest only
– pretest/posttest

within groups design

– repeated measures
– concurrent measures

Internal Validity goal

by eliminating other explanations
• selection effects
• order effects
• design confounds

design confound

an experimenter’s mistake in
designing the IV such that a second variable
happens to vary systematically with the IV

selection effects

a threat to internal validity that
occurs when the kinds of participants in one
condition are systematically different from those in the other

order effect

exposure to one level of the IV/DV can
influence responses to a subsequent level of the
IV/DV

one-group pretest/posttest design

a paradigmatic bad experiment in which one group of participants takes a pretest, experiences an intervention/treatment,
and takes a posttest

This design is problematic because it is vulnerable to many internal validity threats.

maturation threat

a change in behavior that emerges more or less spontaneously over time

A comparison group can

reveal whether there is an effect of the IV
above and beyond any maturation effect

history threat

a “historical” or external factor that
systematically affects most members of the treatment group
around the same time as the treatment itse

regression to the mean

whenever there is a random factor,
extreme scores are likely to be followed by more moderate ones.
When a group average is unusually extreme when
measured at Time 1, it is likely to be less extreme when
measured at Time 2 (closer to group’s true mean)

Only a threat when a group is measured twice and has an extreme score at pretest.

attrition threat

when a certain kind of person
drops out of a study.
It’s a problem when attrition is systematic.
– most unruly camper had to go home

Six new threats to internal validity

maturation, history, regression, attrition, testing, and instrumentation threats

Maturation Threats: Prevention

A comparison group can reveal whether there is an effect of the IV
above and beyond any maturation effect

History Threats: Prevention

A comparison group can reveal whether there is an effect of the
IV above and beyond any effect of history

Regression Threats: prevention

comparison groups + careful inspection of the pattern of results

Attrition Threats: prevention

Remove pre-test data from participants who
dropped out

testing threat

a change in participants caused by
experiencing the DV more than once

Testing Threats: prevention

testing threat: a change in participants caused by
experiencing the DV more than once

instrumentation threat

occurs when a measuring instrument changes over time

Instrumentation Threats: prevention

– if two different tests are used, make sure they are calibrated
– use clear coding manuals to retrain coders throughout experiment
– counterbalance versions of the test
– use posttest-only desig

selection-history threat

an outside event or factor systematically
affects participants at one level of the IV

selection-attrition threat

participants in only one group experience attrition

observer bias

when a researcher’s biases, beliefs, or expectations
influence how they interpret participants’ behavior

observer effects

when a researcher’s biases, beliefs, or expectations
influence the actual behavior of the participants

Demand characteristics:

participants pick up on cues that lead them to guess the experiment’s hypothesis, which influences their behavior

double-blind study

a study in which neither the researcher nor the participant
knows what level of the IV the participant is experiencing

placebo effect

participants really do improve but only because they believe
they are receiving a valid or effective treatment

Demand characteristics:

participants pick up on cues that lead them
to guess the experiment’s hypothesis, which influences their
behavior

placebo effect:

participants really do improve but only because they believe
they are receiving a valid or effective treatment

null effect

when the IV does not appear to make a difference on the DV

weak manipulations:

the difference between levels of the IV is too small
to matter or be meaningful.

insensitive measures:

null result emerges because the operationalization
of the DV does not have enough sensitivity to detect a difference between levels of the IV

ceiling effect:

participants’ scores are squeezed together at the top end of
the DV scale

floor effect:

participants’ scores are squeezed together at the bottom end
of the DV scale

Questions too easy

ceiling effect because everyone is answering
all questions correctly

Questions too difficult:

floor effect because no one can answer a single
question correctly

manipulation check:

an extra dependent variable (DV) that a researcher inserts into their experiment to ensure that their experimental manipulation
worked

pilot study:

instead of adding a DV into the main experiment, a researcher can run a separate pilot study to test the effectiveness of the IV manipulation

measurement error

a human or instrument factor that can randomly inflate or deflate a person’s score on the dependent variable

individual differences:

differences across participants that add variability in DV scores

Individual differences: solutions

Change the design: using a within-groups design instead of an independent groups design accommodates for individual differences

Add more participants: the more people you measure, the less impact any single person will have on the group’s average

situation noise:

any kind of external distraction that could cause variability within
groups that obscures between-groups differences

power

the likelihood that a study will yield a statistically significant result if the IV really does have an effect on the DV

external validity:

do the results generalize to other people, times, or situations?

simple random sampling

The most basic form of probability sampling in which everyone in a
population is assigned a number and a random process is used to
select a subset of those numbers (like a lottery).

systematic sampling

This method is similar to simple random sampling.
Generate two random numbers (e.g., 8, 11) and use those numbers
to systematically select individuals from the population
(e.g., start with the 8th person on a list and select
every 11th person after that)

cluster sampling

Randomly select subgroups, and sample everyone within those groups.
The clusters are treated as the same (e.g., assignment to a cluster is arbitrary). Thus, clusters can be chosen randomly.

multistage sampling

Randomly select subgroups, and randomly sample within those groups.
This method is similar to cluster sampling.

stratified random sampling

Select particular demographic categories and then randomly select
people within those categories to keep the numbers proportionate to
the population.
Unlike cluster sampling and multistage sampling,
the subgroups used in stratified random sampling
are meaningful

oversampling

This method is related to stratified random sampling.
If a particular demographic category makes up a small percentage of
the population, sample more from that category to increase validity of
statistical estimates.

convenience sampling

Using a sample of people who are easy to contact and readily
available to participate.
– psychology professors recruiting psych
students to participate in studies
– collecting data on online platforms

purposive sampling

When you want to study certain kinds of people, you purposefully recruit only those kinds of participants

snowball sampling

Participants are asked to recommend other participants for the study.
– asking smokers to recruit other people in their
support group, who then recruit more, etc.
– useful for sampling rare populations

quota sampling

This method is the nonprobability version of stratified random sampling.
Identify subsets of the population, set a target number (i.e., quota) for
each category, and use nonrandom sampling methods to reach the
quotas

Interrogating external validity — sample size

Sampling method, not sample size, determines external validity

quasi-experiment

a study which is structured like a true experiment, but the researcher does not have full experimental control

small-N design

instead of gathering a little information from a large sample, obtain a lot
of information from just a few cases

Advantages of case studies

– experimental control
– manipulation
– studying special cases

Disadvantages of case studies

– internal validity
– external validity

The Tuskegee Syphilis Study
Three major ethics violations

1. The participants were not
treated respectfully
2. The participants were harmed
3. The participants were a targeted,
disadvantaged social group

The Milgram Obedience Studies
Ethical questions:

1. Was it ethical to put the teacher-
participants through such a stressful
experience?
2. Were there any lasting effects even
after the participants were
debriefed?

Principle of respect for persons

– Treat participants as autonomous agents

Principle of beneficence

– Protect participants from harm

Principle of justice

– The sample of participants should
reflect the population that will benefit from the
study

Principle of respect for persons

– Treat participants as autonomous agents
– Some groups have less autonomy and are entitled to extra
protection: children, cognitively disabled, prisoners

nstitutional Review Boards (IRBs)

• Every experiment run in a college or university
needs to get approval from the IRB beforehand.
• IRBs weigh the risks and benefits to
participants and make sure they are fairly
balanced.
• Everyone on the research team needs to
receive relevant safety training, including
student researchers.

informed consent:

explanation of the study is provided in a written format before a
person agrees to participate
• outlines potential risks and benefits
• signature required

deception

is justified but only to the extent it is necessary to achieve the goals of the study
- omission: withholding details from participants
- commission: lying to participants

debriefing

when deception is used, researchers must debrief participants at end of study
- explain why deception was used and the nature of the
deception

data fabrication

researchers invent data that fit their hypothesis

data falsification

researchers influence the results of a study selectively, e.g. by deleting observations or influencing participants to act in a particular way

plagiarism

representing the words or ideas
of others as your own (cite your sources!)
– self-plagiarism: recycling your own words
across different papers/publications

Animal Research

Using animal subjects allows us to test hypotheses
that cannot be empirically tested ethically on
humans

Institutional Animal Care and Use Committee (IACUC)

– Like an IRB, these committees need to approve any animal research
project before it can begin

Animal Care Guidelines: The Three R’s

– replacement: find alternatives to animals in research
when possible
– refinement: modify procedures to minimize distress
– reduction: use designs that require the fewest animal
subjects as possible