POL222 - Experiments

Goal of research design

use data to evaluate the prediction(s) derived from our theory

Experiments

Research design in which the researcher controls the setting (lab, stimulus, timing, etc.), the selection of participants, and random assignment of participants to treatments (i.e. value of the independent variable)

random assignment of treatment

Allows us to isolate the causal effect of the treatment bc it controls for all confounding variables.

How does random assignment control for all confounding variables?

Since participants are randomly assigned to treatments, there
will be no systematic differences between the treatment and
control groups (descriptive statistics should be the same)

What if there is no random assignment

there will be differences in results that have more to do with confounding variables than with the treatment.

Hurdles of Causality

Causal mechanism: Must come from theory
Reverse causality: Effect does not cause treatment
Covariance: Identifies effect size
Spuriousness: Random assignment rules out other factors

Internal Validity

To what extent are we certain that the independent variable, or treatment, manipulated by the researcher is the sole source or cause of systematic variation in the dependent variable?

External Validity

The extent to which a causal relationship, once identified in a particular setting with particular research participants, can safely be generalized to other times, places, and people.

Threats to internal validity

compromised random assignment, variables beyond the researcher's control that prevented everything from being held constant (e.g., lab, time)

Types of experiments

Lab experiment
Field experiment (is not really an experiment)
Survey experiment (is not really an experiment)
Natural experiment (is not really an experiment)

how do we evaluate evidence with experiments?

Compare treatment group to control group, relying on the idea of hypothesis testing

Null hypothesis

What we expect if the theory is incorrect

Alternative hypothesis

What we expect the results to be if our theory is correct

Negative stereotyping example: what is H0?

There is no difference between the control and treatment groups

Negative stereotyping example: what is H1?

The treatment and control group both performed differently from one another

When the p-value is small

We reject the null

when the p-value is large

we fail to reject the null hypothesis.

Limitations of experiments

• Type of questions you can ask: not all interesting Xs are controllable

• External validity

• Ethics (Milgram experiment, Stanford Prison, Tuskegee, War-torn regions)

• Expensive

• Emphasis--the selected variable may not actually be important