PS 170 Final

4 types of research questions

1. factual/procedural
* describes the facts of the world
2. hypothetical
* what might be in the future
3. normative
* how the world should be
4. empirical
* how the world is, how the world works

What makes a good research question

* ask WHY something happens
* focus on explaining general patterns
* often begin with a “puzzle” or intriguing outcome
* have interesting implications

theory

a formal statement, or system of statements, of these ideas

causal mechanism

provides a specific chain of steps, series of links, or other specific accounting of how or why changes in the causal variable (IV) affect the outcome variable (DV)

scope conditions/domain

the temporal and spatial domain in which theories are expected to operate:

* temporal: does the theory hold only in a particular time period?
* spatial: does the theory hold only in a particular place?

assumptions

claims or beliefs (often implicit) about how the world operates; things we take for granted about how the world operates; things that you assume in order to generate your prediction or for your causal mechanism to operate

expectation/predictions

What is the basic prediction of your theory? What is the relationship between the key variables you are interested in?

inductive theory building

1. get data
2. look for patterns
3. formulate a theory that fits those data

deductive theory building

1. make assumptions
2. deduce a prediction and causal mechanism from those assumptions
3. then get data to test the theory

hypothesis

tentative, but unconfirmed, expectation about the relationship between two or more phenomena

two parts of a hypothesis

1. the main prediction of theory - what is the relationship between IV and DV
* often a restatement of the main prediction
2. about your causal mechanism
* observations consistent with your explanation for why you expect a certain relationship between your IV and DV

deterministic laws

if X occurs then Y will occur with uncertainty

probabilistic laws

If X occurs then Y will occur with certainty

4 hurdles to establishing causation

1. Is there a credible causal mechanism connecting X to Y?
2. Can we rule out reverse causation?
3. Is there a covariation / a correlation between X and Y?
4. Have we controlled for all confounding variables?

unit of analysis

the cases or entities you study, the unit of observation

conceptualization

the meaning of a concept expressed in words that is derived from theory and/or observation; theoretical definition

operationalization

the process of identifying empirical indicators and the procedures for applying them to measure a concept

validity

the goodness of fit between operational definition and the concept it is purported to measure

reliability

the stability or consistency of an operational definition

coverage error

the error that occurs when the sampling frame does not match the target population

standard error

a statistical measure of the “average” sampling error for a particular sampling distribution, which indicates how much sample results will vary from sample to sample

margin of error

statistic that tells you how much sampling error to expect, given your sample size; defined so that 95% of the time the sampling error will be that size or smaller

confidence interval

a range (interval) within a population value is estimated to lie at a specific level of confidence

target population

the population to which the researcher would like to generalize his or her results

sampling frame

an operational definition of the population that provides the basis for drawing a sample; ordinarily consists of a list of cases

nonresponse bias

bias that occurs when nonrespondents differ systematically from respondents

convenience sampling

selection of cases that are conveniently available

snowball sampling

sampling procedure that uses a process of chain referral, whereby each contact is asked to identify additional members of the target population, who are asked to name others, and so on

purposive sampling

sampling that involves the careful and informed selection of typical cases or of cases that represent relevant dimensions of the population

theoretical sampling

sampling process used in qualitative research in which observations are selected in order to develop aspects of an emerging theory

4 survey modes

1. face-to-face
2. telephone
3. mail
4. internet

2 types of survey questions

open-ended and close-ended

cross-sectional designs

the most common survey design, in which data are gathered from a sample of respondents at essentially one point in time

longitudinal designs

survey design in which data are collected at more than one point in timere

trend studies

research question is investigated by repeated surveys of independently selected samples of the same population

panel studies

the same individuals are surveyed more than once, permitting the study of individual and group change

common problems with survey research

1. poor question design
2. inattention
3. problems with drawing the sample
4. social desirability bias

social desirability bias

the tendency to project ones self in a socially desirable way

(CATI)

computer assisted telephone interview

mixed-mode survey

a survey that uses more than one mode of data collection, either sequentially or concurrently, to sample and/or collect the data

close-ended question

survey questions that require respondents to choose responses from those provided

open-ended question

a survey question that requires respondents to answer their own words

leading question

a question in which a possible answer is suggested, or some answers are presented as more acceptable than others

order effects

the order of answer options could influence responses; research shows that people are more likely to pick the first answer option

double-barreled questions

a question in which two separate ideas are presented together as a unit

inattention (attention checks)

when respondents do not read closely or think about their answers; solution = attention checks: questions designed to gauge how long a person stays on a screen

“opt in” polling

members have sought out the poll and signed up to take the survey; usually in order to earn cash or rewards

histogram

a graphic display in which the height of a vertical bar represents the frequency or percentage of cases in each category of an interval/ratio variable

outliers

unusual or suspicious values that are far removed from the preponderance of observations for a variable

listwise deletion

a common procedure for handling missing values in multivariate analysis that excludes cases which have missing values on any of the variables in the analysis

imputation

a procedure for handling missing data in which missing values are assigned based on other information, such as the sample mean or known values of other variables

4 types of measurement

nominal, ordinal, ratio, and interval

nominal

a level of measurement in which numbers serve only to label categories of a variable

ordinal

a level of measurement in which different numbers indicate rank order of cases on a variable

interval

a level of measurement that has the qualities of the ordinal level plus equal distances (intervals) between assigned numbers

ratio

the highest level of measurement, which has the features of the other levels plus an absolute (non arbitrary) zero point

regression analysis

a statistical method for analyzing bivariate (simple regression) and multivariate (multiple regression) relationships among interval or ratio-scale variables

regression coefficients

a bivariate regression statistic indicating how much the dependent variable increases (or decreases) for every unit change in the independent variable; the slope of a regression line

statistical significance (p-value)

the likelihood that the association between variables could have arisen by chance; “p” stands for “probability” that the association occurred randomly, given the size of the effect, the variability of your date, and your sample size

fit of the model (R-squared)

a measure of fit in multiple regression that indicates approximately the proportion of the variation in the dependent variable predicted or “explained” by the independent variables

number of observations (N)

an abbreviation representing the number of observations on which a statistic is based

pure reverse causation

When the DV causes the IV (but the IV does not cause the DV)

simultaneity bias

When the IV causes the DV and the DV causes the IV

lag

*to your predictor variable*: use the value of that variable from the previous time period (t-1) instead of the current time period

omitted variable bias

happens when we fail to control for confounders: mistakenly attributing a causal effect to X when it was really due to Z; failing to detect a causal effect when one really exists

endogeneity

when one or more of your predictor variables causes other predictor variables; for example when X causes Z

instrumental variables

solution to endogeneity: a variable that predicts one of the problem variables but not not the other

experiment basics

1. take a sample of subjects
2. randomly assign a combination of IVs to each subject (might have different levels of IV)
3. expose each subject to the assigned value of the IV
4. measure the outcome of interest (DV)
5. compare across subjects to see how the different IVs affected the DV

factorial design

an experiment in which two or more variables (factors) are manipulated

reactive measurement effect

an effect in which participants’ awareness of being studied produces changes in how they ordinarily would respond

lab experiments

an experiment conducted in a controlled environment

survey experiments

an experiment embedded in a survey in which respondents are given different, randomly assigned versions of survey questions

natural experiment

a study in which the main IV is not manipulated by researchers, but instead varies naturally in a way that allows the data to be analyzed as though it were in an experiment

internal validity

evidence that rules out the possibility that factors other than the manipulated independent variable are responsible for the measured outcome

external validity

the extent to which experimental findings may be generalized to other settings, measurements, populations, and time periods

field experiments

an experiment conducted in a natural setting

manipulation checks

procedure used to provide evidence that participants interpreted the manipulation of the independent variable in the way intended

list experiment

a questionnaire design technique used to mitigate respondent social desirability bias when eliciting information about sensitive topics

audit study

a study that examines racial and other forms of discrimination by sending matched pairs of individuals to apply for jobs, purchase a car, rent an apartment, and so on

complications of experiments

often the IV has more than 1 level of “treatment”; multiple IVs

When small-N research is most useful

1. for inductive theory building
2. when the research question involves a very rare event (inherently small cases)
3. when measuring key variables is difficult
4. for investigating causal mechanisms

process tracing

divide the cause-effect link that connects IV and DV into smaller steps; then look for evidence of each step

empirical tests in process tracing

straw in the wind, hoop, smoking gun, doubly decisive

straw in the wind

suggests that one theory is more plausible than another, but on its own does not prove anything; neither necessary nor sufficient

hoop tests

a theory has to “jump through the hoop” to be viable, but jumping through the hoop does not prove the theory; necessary but not sufficient

smoking gun tests

if you are caught holding the smoking gun, you are presumed guilty, however, if you are not caught with a gun, you are not presumed innocent; sufficient but not necessary

doubly decisive tests

tests confirm one theory and eliminate all others; necessary and sufficient

pitfalls when it comes to case selection

1. selecting on the DV (when you choose cases to study based on the value of the DV)
2. selection bias/selection effects

ways of gathering evidence

* archives: collections of original unpublished material or primary sources
* field research
* interviews
* participant observation
* nonparticipant observation

challenges when using archives or reading archival research

1. incomplete records
2. government redaction/limitation of access
3. cherry-picking

field research/when to do it

when you can’t get the information you need at your home institution

techniques for obtaining data in field research

1. interviews
2. participant observation
3. non-participant observation

types of interviews

structured, unstructured, and semi-structured

peer review process

* idea, theory, hypothesis
* research and write
* manuscript (proposed article)
* send to journal editor
* reviewed by 3-5 experts in the field
* decision is made to accept, revise, or reject

4 sets of research methods

1. survey research
2. other observational large-N
3. experiments
4. small-N

unintentional/unconscious bias

confirmation bias + motivated reasoning; seeing what you want to see

confirmation bias

the tendency to search for, interpret/evaluate, and recall information in a way that supports what we already believe

motivated reasoning

seeing what you want to see; the unconscious tendency of individuals to fit their processing of information to conclusions that suit some end or goal

misconduct (p-hacking/fraud)

* the practice of reanalyzing data in many different ways and only presenting the preferred results (“fishing expeditions”)
* outright making things up and making fraudulent claims

What tends to do best in peer review?

* research that is (or appears) scientifically rigorous on an important topic
* research that has statistically significant results
* research that reaches novel conclusions