Research Methods Midterm 2

true experiment

subjects are randomly assigned into at least 2 conditions

randomized experiment

natural experiment

cause manipulated by “nature”, subjects are not randomly assigned by researcher

quasi-experiment

experiment wherein subjects are not randomly assigned

researcher typically (though not always) has some control over administration of the treatment

nonexperimental desings

observation of the size and direction of a relationship among variables

appropriate research questions for true experiments

• independent variable must be manipulated. ethically, that is

• examining the impact of 1 or very few causal variables

• good manipulations: realistic and understandable and people recognize it. subtly is not the best choice

experimental conditions

by manipulating the IV, one produces at least 2 levels of the IV

how do we know that the potential cause and effect are causally related?

covariation/association, temporal order, non-spuriousness

basic experimental design

researcher wants to compare groups who differ on whether they experince IV (experimental condition) or not (control condition)

experimental control

want to control all aspects of the experimental situation to isolate the effects of the treatment and eliminate alternative explanations

helps with nonspuriousness, single and double-blind studies, deception (confederates)

random assignment

groups of people in the conditions start the study fundamentally the “same”

from subject pool, you randomly assign participants to the conditions

experimental condition people probalistically “look like” ‘control condition people’

threats to internal validity

present whenever you suspect anything besides the IV is having an effect on the DV

ambiguous temporal precedence (threat to internal validity)

a threat to internal validity that occurs when it is unclear which of two variables came first, making it impossible to determine if one caused the other

selection bias (threat to internal validity)

systematic differences between people in the different states of the IV

history (threat to internal validity)

during the course of running your study, historical events may occur that confound the results

maturation (threat to internal validity)

people continually grow and change and these changes can affect the DV

long term study: subjects grow older and perhaps wiser and more experineced

short term study: subjects may get tired, sleepy, bored, hungry…

testing (threat to internal validity)

process of testing and retesting often affects people’s behavior and threfore the DV

instrumentation (threat to internal validity)

measure of the DV somehow changes during the course of the study

linked to stability reliability

mortality/attrition (threat to internal validity)

some people do not complete the entire study

statistically regression (threat to internal validity)

people selected because they score extremely on the DV1

even without treatment, low scorers will tend to improve over time (floor effects and high scorers will test to do worse over time (ceiling effects)

drift to the average score is called regression to the mean

additive and interactive effects (threat to internal validity)

threats that operate simultaneously and have an effective additively or multiplicatively

selection-maturation additive effect

selection history effect

pretest-posttest control group design

roxo roo

basic randomized design (1 treatment)

rxo ro

alternative treatment design with pretest

roxo roxo roo

solomon 4-group

roxo roo rxo ro

factorial designs

two or more independent varibles called factors, each with two or more levels

allows for testing combinations of treatments

survey experiment

randomized experiment given as a survey to a representative sample of the population

field experiment

an experiment that takes place in a natural of “real-world” setting

give up the experimental control of the lab for everyday realism

keep random assignment and other soruces of experimental control

audit studies 

a type of field study that is used to assess whether characteristics lead to discrimination in real markets

natural experiment

an experiment in which the independent variable is manipulated by “nature,” not by the experimenter

quasi experiment

experiment wherein subjects are not randomly assigned; researcher typically (though not always) has some control over administration of the treatment

participants cannot actively select themselves to be in one condition or another (though they might already be in a group that then received the treatment as selected by the researcher/policy being evaluated)

one-group posttest only Q-E design

XO

non-equivalent control group q-e design

oxo oo

time series q-e design

ooo—o x o—-oooo

x is usally an event or policy that affects large number of peopel

principles of q-e designs that help eliminate alternative explanations

identify and gather data on plausible threats to internal validity

control through good design

• avoid self-selection

• blinding administrator doing selection if possible/relevant

• measurement: multiple pretests and/or posttests

• comparison groups

coherent pattern matching

switching replication, reversed treatment, removed treatment, repeated treatment

ordering of survey questions

• intro that explains survey

• opening questions that are interesting and relatively easy to answer (generally not demographic questions)

• bulk of survey should be organized by common topics and sections should have a short, orienting introduction

• end with a thank you

order/context effects in surveys

when the ordering or surrounding context in which a question appears baises the responses

how to write questions

avoid confusion

keep the respondents perspective in mind1

10 things to avoid

1. jargon, slang, and abbreviations unless a special population is being surveyed

2. ambiguity, condusion, vaugeness

3. prestige bias and emotional language

4. double barreled questions

5. leading/loaded questions

6. asking questions that are beyond respondents’ capabilities to answer

7. false premises

8. asking about future intentions

9. double negatives

10. overlapping or unbalanced response categories

social desirability bias

respondents give the normative or socially acceptable answer to a question

tend to over report

being a good citizen, well informed/cultured, being morally upstanding, having a good family life…

tend to under report

having an illness/disabilit5y, engaging in illegal or deviant behavior, revealing their financial status…

remedies for social desirability bias

embed a sensitive response within more serious activies

develop a “warm up” to the question to evoke trust from the respondent

be mindful of the social distance between the interviewer and respondent

subjects or participants

the people who participate in experimental or q-e experimental studies (participants preferred language)

respondents or cases

the people who respond to survey questionnaires

survey design

with traditional survey design, we are no longer manipulating the introduction of the IV, but measuring it

nonexperimental design

observi8ng the size and direction of the relationship between the IV and DV

cross sectional

data are collected at one point in time (a snapshot)

longitidinal

data are collected at two or more points in time

repeated cross sectional study (trend study)

panel study

establishing nonspuriousness with control varibles

when using survey data, variables are “statistically controlled”

a third variable (or more) is held constant so that the relationship between the causal and resultant concepts’ variables can be examined without it affecting that relationship

is it possible to fully account for non-spuriousness in non-experimental survey designs?

no

self-administered questionnaires

a survey completed directly by respondents through the mail or online

main concern is to maximize response rate

interview questionnaires

a survey administered by a researcher/interviewer via telephone/zoom or face-to-face

telephone interviewer surveys

huge challange is the response rate

can have additional problems associated with using the telephone

because this is with another person, respondent can ask for clarification, interviewer can ask for clarity or ask follow-up questions to improve data

in-person surveys

advantages: response rate tends to be higher than mailed/web/telephone surveys, can be longer and more complicated with these types as well

disadvantages: expensive, safety concerns interviewer bias

interviewer effects

the mere presence of an interviewer or the interviewer’s personal characteristics may lead a respondent to answer questions in a particular way, blasting responses

interviewer error

systematic reporting mistakes/errors on the part of particular interviewers

population

all of the units/people that your study is about or speaks tosam

sample

a subset of this population that is studied

probability sampling

sample wherein random chance is used to select people from the population for the sample and each person has a probability of being selected that is known

nonprobability sampling

sample not generated by random chance

probability of selection into the sample is therefore unknown

representative sample

a sample that has the same distribution of characteristics as the population from which it was selected

external validity (generalizability)

extent to which findings are generalizable to that specific population (time and place)

inferential statistics

mathematical tool for estimating the likelihood that a statistical result derived from a probability sample is representative of the population

based in probability theory

sampling error

the difference between the characteristics of a sample and the characteristics of the population from which it was drawn

sample size

increasing sample size reduced sampling error — but there is a point of diminishing returns

population heterogenity increases sampling error

sampling frame

list of people or sampling units from which the sample is taken

simple random sample

every sampling element is selected on the basis of chance and has an equal chance of being selected

systematic random sample

sample is selected by selecting people for inclusion at a fixed interval

problems arise with periodicity in the ordering of sampling elements which results in the selection of a nonrandom sample

pros: easier than simple random sampling and produces results that are generally the same

cons: will produce biased results if elements are arranged in a periodic way that aligns with the sampling interval

stratified random sampling

sample elements are selected from groupings of peope within the population

SRS steps

1. identify variables related to the DV

2. use that variable to divide population into two or more mutually exclusive strata

3. draw probability samples (simple random, systematic, or cluster) from each strata

4. join subsamples to form overall sample

two types of stratified random samples

1. proportionate: sampling elements are selected from strata in propotionto their representation in the population

   1. ensures that the sample will represent the population as accurately as possible

2. disproportionate: when certain elements are oversampled to ensure that there are enough people from smaller strata in the data

   1. ensures enough cases to run effective stats

cluster sampling

naturally occurring, mixed aggregate of cases of the population with each element appearing in only one cluster

can generate multiple stages of sampling clusters of people

select clusters/elements in two or more stages

random selection of naturally occuring, largest cluster

…

last stage: random selection of people within the cluster

cluster sampling pros

makes these large area samples possible, thereby reducing costs

cluster sampling cons

increases sampling error because there are multiple samples taken (error at each stage introduced)