Week 4. Experimental Designs

What is a “third variable problem”?

Another variable causes changes in both variables, creating a correlation without causation.

Why do we conduct experiments?

To test whether an IV (A) causes a DV (B).

What are IV and DV in an experiment?

1. IV = manipulated cause

2. DV = measured outcome.

What 3 conditions must be met to claim A causes B?

1. A precedes B (takes place before)

2. A and B covary (need to correlate)

3. No other variable explains the change (control).

What is temporal order?

The cause must happen before the effect.

What is covariation?

When A changes, B changes (they move together).

What is the “control” condition for causality?

Showing the effect is due to A, not other factors.

What is a control group?

A group not exposed to the experimental treatment.

What is random assignment?

Participants are randomly assigned to conditions, not by choice.

Why do we use random assignment?

To make groups comparable and reduce selection bias.

What does “X” mean in experimental design notation?

Treatment/intervention/independent variable.

What does “O” mean in experimental design notation?

Observation/measurement.

What does “R” mean in experimental design notation?

Randomization (random assignment).

What are O1 and O2?

O1 = pretest, O2 = posttest.

What is a one-group pretest-posttest design?

O1 X O2 (same group measured before and after treatment).

Why is one-group pretest-posttest weak for causality?

Changes could happen anyway or due to other influences.

What is a two-group pretest-posttest design?

Experimental: O1 X O2; Control: O1 O2

What is a two-group random assignment pretest-posttest design?

R O1 X O2 and R O1 O2

Why add groups without a pretest?

The pretest itself can influence the posttest (testing effect).

What problem does the Solomon four-group design address?

The pretest might change participant behavior/performance.

What are the 4 groups in Solomon design (idea)?

Two groups get pretest + posttest, two groups only posttest; one treated and one control in each pair.

When is Solomon design useful?

When you suspect pretesting influences outcomes.

What is a factorial design?

An experiment with two or more IVs combined (“crossed”).

What does “2×2” mean?

Two IVs, each with two levels.

Why use factorial designs?

To test interaction effects between IVs.

What is an interaction effect in a factorial design?

The effect of IV1 depends on the level of IV2.

What is a between-subjects design?

Each participant experiences only one condition/level.

What is a within-subjects design?

Each participant experiences multiple (often all) conditions.

What is a mixed design?

At least one IV is between-subjects and at least one IV is within-subjects.

What is internal validity?

Confidence that the IV caused the DV (not other factors).

What is a spurious relationship?

A relationship that appears real but is actually coincidence or not meaningful.

What is selection bias?

Groups differ at the start because of non-random assignment or self-selection.

How do you reduce selection bias?

Random assignment + large sample; if needed, pretest.

What is attrition?

Dropout from the study (especially problematic if unequal across conditions).

What is diffusion?

Participants across conditions share information outside the experiment.

What is maturation (threat)?

People naturally change over time (not because of the treatment). (Knowledge, skills, beliefs, attitudes etc.)

How do control groups help with maturation?

They show whether change happens without the treatment too. You look for a comparable group (to which you do not read stories to everyday). 

What is repeated testing (testing threat)?

People improve because they’ve seen the test/tasks before.

What is experimenter bias / demand effects?

Researcher cues influence participant behavior (consciously or unconsciously).

How do you reduce experimenter bias?

Use a blind experimenter (data collector unaware of the research purpose and experimental conditions).

What is external validity?

How well results generalize to the real world/population.

What is ecological isomorphism (ecological validity issue)?

Experiment conditions are unnatural, so behavior may not match real life.

What is the Hawthorne effect?

People change behavior because they know they’re being observed.

Why can student samples reduce external validity?

Volunteers may not represent the broader population.

Two advantages of online experiments?

1. Fast, cheap data collection

2. Reduces experimenter bias.

What are common challenges in online experiments?

• High attrition rates: decision to quit is easier

• Inattentive participants: they may not always read instructions well or may even take a break, resulting in lower quality and noisier data. 

• Non-representative participant samples (external validity): sample will be biased to typical internet users.

• Noisy measurement of timed responses

• Inconsistency of question lay-out across devices: e.g., part of scale may be hidden for users with smaller phones.

What is an ex post facto (natural experiment) design?

Observing outcomes after a naturally occurring event (no manipulation).

What is a field experiment?

Manipulating an IV in a real-world setting and observing outcomes.

Example: manipulating a variable (study conditions) and observing results (test scores) for the study group vs. the rest of the class (who we assume are studying individually).

Why don’t natural/field comparisons always prove causality?

Groups may differ already; lack of baseline measures and control.

Example: There is no baseline measurement of student performance. The difference in scores may not be due to the group vs. individual study conditions at all. Therefore, you need to compare the students’ performance before they studied under the 2 different conditions, as well as after they did this. 

You don’t know how they study outside your research

What is a time-series design?

Many observations over time before and after a treatment (…O O O X O O O…).

What is a manipulation check?

A measure verifying participants perceived the IV as intended.

Why are manipulation checks important?

Effects can fail because participants didn’t interpret the manipulation correctly.

What do all experiments have in common?

They manipulate an IV to see its effect on a DV.

Biggest weakness of experiments?

Artificial conditions may not resemble real life (ecological validity issue).