Chapter 5 - Experimental Research

Criteria for Causal Claims

1. Covariation of Cause and Effect

2. Temporal Precedence

3. Ruling out alternative explanations

Covariation of Cause and Effect

-When presumed cause is present, effect is more likely to occur (or stronger)

-When presumed cause is absent, effect is less likely to occur (or is weaker on average)

Temporal Precendence

Cause must precede the effect

Ruling out alternative explanations

Cause is isolated from any other potential cause, plausible alternative explanations are ruled out in causing the observed effect

Experiment

Study designed to test whether a causal relationship between variables

-True experiment involves manipulation of the IV, random assignment to conditions/condition sequences, and experimental control to minimize alternative explanations

Confounding Variable

Extraneous variable that systematically varies with the IV, causally influences the DV

-Provides alternative explanation for the IV-DV relationship, threatening internal validity

Between Subject Design

Each participant is assigned to ONE level/condition of the IV

-Looks at difference in DV score between different groups of people

-Groups are as similar as possible on all extraneous variables BEFORE intervention; only difference should be levels of IV

Between Subject Design - Threat

Pre-existing group differences (personality, baselines, sleep, motivation, etc)

-Differences may become confounds (increasing noise)

Dealt with by Random Assignment, Standardization, and Pre/Posttest

Random Assignment

Uses a chance process to decide which participants are in each group, creates an equal chance of each participant to be in any group

-Groups are therefore comparable in pre-existing conditions, but NOT perfectly balanced

-Guarantees assignment is independent of pre-existing characteristics by design, but groups can still differ by chance

Standardization

Procedure is the same (environment, instructions, etc) for every participant regardless of condition

-Changes in these procedures may introduce extraneous variables

Performing Standardization

1. Written Protocol - Script of everything

2. Automated Instructions - Using software, videos to deliver instructions

3. Using double-blind procedures where neither participant nor researcher know which experimental condition participant has been assigned to

PreTest and PostTest Design

Participants assigned to different conditions measured on the DV before and after IV manipulation

-Assesses baseline difference between groups, accounts for them when evaluating treatment effect

PreTest/PostTest Protocol

1. Measure DV at baseline (pretest)

2. Assign participants to groups/conditions

3. Manipulate the IV

4. Measure DV again after IV manipulation

Change Score = Posttest - Pretest

Pros of Pretests

1. Similar Groups - Allows researchers to assess baseline difference between groups, accounting for difference in their analyses

2. Selective Attrition - If people drop out of your study, pretest data helps you see if dropouts were systematically different from those who stayed

3. Small Sample - Pretest is a good way to check if you should also account for some of these extraneous variables statistically

Risks of Pretests

1. Reactivity - Pre-test might tip off participants to your hypothesis/manipulation, causing them to change their behaviour or expectation

2. Internal Validity Threat - Participants may react differently to treatment/IV manipulation because they were sensitized by the pre-test

3. External Validity Threat - May not generalize to the “real world” where people don’t usually get pre-tested before encountering a stimulus

Within-Subjects Design

Each participant experiences every level/condition of the IV

-Measures same person after each condition, rather than comparing groups

Advantage of Within-Subjects

1. Groups are identical in terms of participant characteristics (no need to worry about pre-existing conditions)

2. Differences across conditions are less likely due to participant difference, so design is more statistically sensitivity (variability due to individual difference is reduced)

3. Requires fewer people

Disadvantage of Within-Subjects

1. Order Effects - Sequence of conditions can become a confound (carryover effect, practice effect, fatigue effect, context/contrast effect)

2. Reactivity - Participants see all conditions, making it easier for them to guess your hypothesis and change their behaviour

Counterbalancing

Systematically varying order across participants to reduce Order Effects (like how random assignment balances groups, counter-balancing balances orders)

1. Complete Counterbalancing - Equal number of participants complete each possible order of conditions (50% do cond 1, then cond 2 and 50% do cond 2 then cond 1)

2. Partial Counterbalancing - Small set of carefully selected sequence

• Latin Square, each condition appears equally often in each position (first, second, third)

Latin Square Example

Latin Square, each condition appears equally often in each position (first, second, third)

Consider Within-Subjects Design If:

1. Population is scarce (small sample size)

2. Individual differences are substantial

3. Manipulation of IV is temporary and reversible, effect won’t alter later performance

• Ex; Caffeine, alcohol, memorization tasks

• Anti-Ex; Therapy, training, invasive procedures

4. Task is brief/low burden and fatigue/boredom/practice unlikely to distort results meaningfully

Consider Between-Subjects Design If:

1. Manipulation has lasting, irreversible effects

2. Carryover effects are likely and difficult to control

3. Experiencing multiple conditions could reveal hypothesis/change participants’ behaviour (or reactivity)

Levels of Measurement

1. Nominal level

2. Ordinal level

3. Ratio level

4. Interval level

Nominal Level

Used for categorical variables

-No ordering among responses, NO comparison

Ex; Asking participants their favourite color

Ordinal Level

Assigning scores so that they represent the rank order

Ex; Participants ranking how they feel today (happy, sad, very happy, neutral)

Interval Level

Uses numerical scales where intervals reflect real differences

-NO true zero

-Ex; Between 30 degrees and 40 degrees difference is the same as 80 degrees and 90 degrees

Ratio Level

Assigning scores in such a way that there is a true zero point

Ex; Measuring in metres/kg, money

Experimenter Expectancy Effect

Experimenter’s expectations on how participants “should” behave in the experiment affects how the participant actually behaves

Demand Characteristics

Subtle cues that reveal to participants how the researcher expects them to respond

Carryover Effects

Practice Effect - Performing a task better in later conditions due to practice

Fatigue Effect - Doing worse in a task because they become tired/bored

Context/Contrast Effect - Participants change how they perceive stimuli/interpret task in later conditions

Types of Control Conditions

1. No-Treatment Control - Participants receive no treatment

• Downside: Existence of placebo effect

2. Placebo-Control Condition - Participants receive placebo

• Any improvement in treatment group over control group must have been caused by treatment, not participant’s expectation to improve

3. Wait-List Control Condition - Participants wait to receive treatment, allows researchers to compare those who have received vs those waiting but expect to improve

Improving Questionable Measurement Practices

Reporting all measurement decisions transparently

-Lack of this prevents the internal, external, statistical and construct validity

Questions to Promote Transparent Reporting of Measurement

1. What is your construct

2. Why/how did you select your measure

3. What measure did you use to operationalize the construct

4. How did you quantify your measure

5. Did you modify the scale? How/why?

6. Did you create a measure on the fly?

Jingle and Jangle Fallacy

Jingle Fallacy - Two instruments measure the same construct because they have similar names

Jangle Fallacy - Two measures assess different constructs because they have different names