Comprehensive Study Guide to Simple Experimental Designs and Causal Validity

Fundamentals of Simple Experiments and Causal Claims

• Definition of an Experiment:

  • In psychological science, an experiment occurs when researchers manipulate at least one variable and measure another.
  • Experiments can take place in laboratory settings, schools, or field environments wherever variable manipulation and measurement occur.
  • Manipulated Variable: A variable whose levels are systematically managed by the researcher, who assigns participants to specific values/levels (e.g., coin flips or experimental conditions).
  • Measured Variable: Records of behavior, attitudes, self-reports, behavioral observations, or physiological responses recorded after the experimental environment is established.

• Causal Claims vs. Association Claims:

  • Causal claims replace non-causal verbs (such as "related to", "associated with", or "linked to") with action-oriented verbs like "makes", "influences", or "affects".
  • Causal claims provide the empirical justification for applied interventions, treatments, and policy changes.
  • Experiments represent the sole research design capable of supporting causal claims.

• Classifying Experimental Variables:

  • Independent Variable (IV): The manipulated, causal variable. The term stems from the researcher's independence in assigning participants to levels.
  • Levels / Conditions: The specific operationalized values or variations of the independent variable. An independent variable must possess at least two levels.
  • Dependent Variable (DV): The measured outcome variable. A participant's response on this variable depends on the level of the independent variable administered.
  • Graphical Placement: When graphing experimental results, the independent variable is placed on the x-axis, and the dependent variable is placed on the y-axis.
  • Multiple Dependent Variables: An experiment must have at least one IV and one DV, but can include multiple DVs measured on distinct scales (e.g., number of button presses and total play duration in seconds).

• Control Variables:

  • Definition: Any extraneous factor that a researcher deliberately keeps constant across all experimental conditions.
  • Operational Function: Control variables are not true variables because their levels do not vary. They isolate the proposed active ingredient by holding potential third/nuisance variables constant, thereby ruling out alternative explanations and establishing internal validity.

• Detailed Case Study 1: Publicizing Diversity Norms

  • Theoretical Framework: Social norms are common beliefs and standards for behavior within a group. Highlighting widely held pro-diversity norms can foster an inclusive campus environment.
  • Baseline Prevalence Measurement: A survey of University of Wisconsin-Madison (UW-Madison) students established that $93\%$ agreed with the statement: "We embrace diversity and welcome people from all backgrounds into our UW-Madison community", and $84\%$ consented to having their photograph displayed on a poster bearing this message.
  • Experimental Design (Murrar, Campbell, & Brauer, 2020):
  • Sample: $163$ UW-Madison undergraduate students recruited to a laboratory waiting room.
  • Independent Variable: Poster condition in the waiting room with two levels:
    • Norms Poster: High-quality poster displaying the pro-diversity message and survey statistics.
    • Control Poster: A flu shot informational poster.
  • Exposure & Task: Participants spent $5$ to $7\text{ minutes}$ in the waiting room (leaving phones and backpacks in another room), completed a filler memory task, and filled out an attitude survey.
  • Dependent Variable: "Inclusive Climate" score—a composite measure combining self-report scales evaluating attitudes toward out-group members, internal motivation to avoid prejudice, and sense of belonging/inclusion on campus.
  • Control Variables: Waiting room location, waiting room duration ($5$ to $7\text{ minutes}$), filler task, and experimenter interactions.
  • Outcome: Participants exposed to the norms poster scored higher on the inclusive climate scale than those exposed to the flu shot poster.
  • Replications & Generalization: Four follow-up experiments (one conducted in undergraduate classrooms and three using a video format) yielded identical results, demonstrating that exposure to pro-diversity norms causes higher inclusive climate scores.

• Detailed Case Study 2: Adult Persistence Modeling in Infants

  • Theoretical Framework: Investigating whether infants learn perseverance by observing adult models work through challenges.
  • Experimental Design (Leonard et al., 2017):
  • Sample: Over $100$ infants aged $13-18\text{ months}$ recruited at the Boston Children's Museum play lab. Parents sat next to infants but were instructed not to assist.
  • Independent Variable: Model's demonstrated effort level, manipulated via coin flip:
    • Effort Condition: Adult model attempted to extract a toy frog from a plastic box, verbalizing prompts ("How do I get this out?") for $30\text{ seconds}$ before opening a secret door ("I got it out!"). The model then tried to unhook a toy from a carabiner, asking ("How do I get this off?") for $30\text{ seconds}$ before releasing it ("Yay!").
    • No-Effort Condition: Model interacted with the same toys for the exact same duration ($30\text{ seconds}$ each), but effortlessly extracted the frog three times in a row and easily detached the toy from the carabiner three times.
  • Control Variables: Identical toys (frog-in-box and carabiner), cheerful voice tone, demonstration length ($30\text{ seconds}$ per toy), female model gender, seating chair, and play cubicle.
  • Dependent Variable: Persistence, measured by the number of times the infant pressed a large white, inert button on a novel cube-shaped musical toy during a $2\text{-minute}$ testing period.
  • Outcome: Infants in the effort condition pressed the inert button a median of $22.5\text{ times}$ (averaging approximately $11\text{ times}$ more presses) compared to a median of $12\text{ times}$ in the no-effort condition.

Establishing Causation: The Three Criteria

• Satisfying Covariance:

  • Group Differences: Covariance requires that levels of the independent variable are systematically associated with distinct outcomes on the dependent variable (evidenced by statistically significant mean differences between groups).
  • The "Compared to What?" Principle: Personal experience lacks comparison conditions. Experiments establish covariance by including structured comparison groups.
  • Flawed Empirical Example: A study claiming dogs dislike being hugged examined internet photos and found $82\%$ of hugged dogs displayed stress signs (Coren, 2016). Because it lacked a comparison group of dogs not being hugged, covariance could not be evaluated.
  • Types of Comparison Conditions:
  • Control Group: A level of the IV intended to represent a "no treatment" or neutral baseline.
  • Treatment Group(s): The active level or levels of the IV being tested.
  • Placebo Group (Placebo Control Group): A control group exposed to an inert intervention (e.g., a sugar pill).
  • Comparison Group: Alternative experimental conditions that differ in meaningful ways without requiring a true zero-treatment control (e.g., Harlow's 1958 monkey study comparing wire mothers to cloth mothers).

• Satisfying Temporal Precedence:

  • Experimental Ordering: The experimenter manipulates the causal independent variable before measuring the outcome dependent variable.
  • Advantage Over Correlational Designs: Correlational studies collect data in snapshot fashion where all variables are measured concurrently, making directionality ambiguous. Experimental manipulation guarantees the cause precedes the effect.

• Satisfying Internal Validity and Eliminating Confounds:

  • Definition of Internal Validity: The ability to rule out alternative explanations (third variables) for the observed relationship between the IV and DV.
  • Design Confounds: An experimenter's failure to control extraneous variables, allowing an alternative variable to covary systematically along with the intended independent variable. (e.g., if a norms poster was colorful with photos while a control poster was monochromatic text, poster color/design would represent a design confound).
  • Systematic vs. Unsystematic Variability:
  • Systematic Variability: Differences in participant or environmental traits that vary selectively with the levels of the IV, creating a confound.
  • Unsystematic Variability: Random noise or individual differences (e.g., infant tiredness, baseline mood, attention spans) distributed evenly across all conditions. Unsystematic variability does not constitute a design confound, though it can obscure true effects.
  • Selection Effects:
  • Definition: A threat to internal validity occurring when the kinds of participants in one level of the IV are systematically different from those in another level.
  • Mechanisms: Selection effects arise when participants self-select into conditions or when experimenters non-randomly assign specific demographics to specific conditions.
  • Empirical Example of a Selection Effect: A trial evaluating an intensive $40\text{ hours/week}$ autism therapy compared an intensive treatment group to a treatment-as-usual group (Lovaas, 1987). Parents chose their group assignment based on geographical distance or personal preference. The intensive group showed greater symptom improvement, but high parent motivation in the self-selected group created a selection effect confound.
  • Countermeasures Against Selection Effects:
  • Random Assignment: Assigning participants to IV levels via coin flips, die rolls, or random number generators gives every participant an equal chance of being placed in any group. Random assignment desystematizes participant characteristics, equalizing groups prior to manipulation. Simulations demonstrate random assignment creates comparable groups up to $98\%$ of the time, even with sample sizes as small as $n = 4$ per group.
  • Matched Groups (Matching): A technique where participants are measured on an influential variable (e.g., hometown size or IQ), sorted rank-order, paired into sets, and then randomly assigned within each pair to experimental conditions. Matching guarantees exact group equality on key variables but requires extra pre-testing time and resources.

Experimental Design Structures: Independent-Groups vs. Within-Groups

• Independent-Groups (Between-Subjects / Between-Groups) Designs:

  • Structural Definition: Distinct, separate groups of participants are placed into different levels of the independent variable.
  • Posttest-Only Design (Equivalent Groups, Posttest-Only):
  • Procedure: Participants are randomly assigned to independent variable groups and measured on the dependent variable exactly once following exposure.
  • Evaluation: Satisfies covariance (detecting mean differences between groups), temporal precedence (IV administered first), and internal validity (when random assignment eliminates selection effects and control variables eliminate design confounds).
  • Example: Murrar et al. (2020) norms poster study.
  • Pretest/Posttest Design (Equivalent Groups, Pretest/Posttest):
  • Procedure: Participants are randomly assigned to at least two groups and tested on the key dependent variable twice—once before and once after exposure to the independent variable.
  • Example: A study on cognitive performance randomly assigned $48$ students to a $2\text{-week}$ mindfulness class or a $2\text{-week}$ nutrition class (Mrazek et al., 2013). Verbal reasoning GRE tests were administered $1\text{ week}$ prior to and $1\text{ week}$ after the classes. Mindfulness participants improved significantly on posttest GRE scores relative to pretest, whereas nutrition participants showed no significant change.
  • Purpose & Utility: Used to verify that random assignment produced equal baseline scores across groups and to track individual change over time.
  • Selecting Between Posttest-Only and Pretest/Posttest Designs:
  • Pretest/posttest designs confirm pre-existing equality, but repeated exposure to the DV may induce pretest sensitization, revealing hypotheses or altering subsequent behavior.
  • Posttest-only designs are preferable when pretesting risks inducing demand characteristics or participant suspicion.

• Within-Groups (Within-Subjects) Designs:

  • Structural Definition: Each participant is presented with all levels of the independent variable.
  • Repeated-Measures Design:
  • Procedure: Participants are measured on a dependent variable more than once, after successive exposures to each level of the independent variable over time.
  • Empirical Example: A study evaluated whether shared experiences heighten intensity without verbal interaction (Boothby et al., 2014). Female participants ($n = 23$) sat silently beside a female confederate and tasted dark chocolates. Participants tasted one chocolate while the confederate simultaneously tasted chocolate (shared condition), and another chocolate while the confederate viewed a painting (unshared condition). The chocolates were identical. Chocolate liking ratings were significantly higher in the shared condition.
  • Concurrent-Measures Design:
  • Procedure: Participants are exposed to all levels of an independent variable at roughly the same time, registering a single attitudinal or behavioral preference as the DV.
  • Empirical Example: A puppy cognition study evaluated whether $8\text{ to }10\text{-week-old}$ puppies follow human pointing cues (Bray et al., 2020). Two inverted cups were presented simultaneously (one pointed at, one control). Puppies selected the pointed-at cup $69.4\%$ of the time.
  • Additional Examples: Harlow's (1958) study presenting baby monkeys with wire and cloth mothers simultaneously; commercial preference taste tests.
  • Core Advantages of Within-Groups Designs:
  • Participants Act as Their Own Controls: Individual differences (e.g., baseline dark chocolate preference, personality traits) are held constant across conditions.
  • Reduced Measurement Error: Holding unsystematic variability constant increases statistical power and precision in detecting IV effects (analogous to dampening background noise at a party).
  • Sample Efficiency: Requires fewer total participants. To obtain $n = 50$ responses per condition across two levels, an independent-groups design requires $100$ total participants, whereas a within-groups design requires only $50$ participants.
  • Structural Distinction Between Pretest/Posttest and Repeated-Measures:
  • In a pretest/posttest independent-groups design, participants see only one level of the independent variable (tested twice on the DV).
  • In a repeated-measures within-groups design, participants see all levels of the independent variable (tested on the DV after each level).

Internal Validity Threats and Countermeasures in Within-Groups Designs

• Order Effects:

  • Definition: A threat to internal validity occurring when exposure to one condition alters participant responses to subsequent conditions in a within-groups design.
  • Types of Order Effects:
  • Practice Effects (Testing Effects): Performance improves on later tasks due to repetition, familiarity, or learning.
  • Fatigue Effects: Performance deteriorates on later tasks due to tiredness, boredom, or cognitive overload.
  • Carryover Effects: Contamination carries over from one condition to the next (e.g., tasting orange juice immediately after brushing teeth).

• Counterbalancing:

  • Definition: Presenting levels of the independent variable to participants in different sequences so that order effects cancel out when aggregated.
  • Procedure: Participants are randomly assigned to condition sequences (e.g., sequence $A \rightarrow B$ vs. sequence $B \rightarrow A$).
  • Full Counterbalancing: All possible condition orders are represented. Used when an IV has two or three levels:
  • For $2$ conditions ($A, B$): $2! = 2$ orders ($A \rightarrow B$; $B \rightarrow A$).
  • For $3$ conditions ($A, B, C$): $3! = 6$ orders ($A \rightarrow B \rightarrow C$; $A \rightarrow C \rightarrow B$; $B \rightarrow A \rightarrow C$; $B \rightarrow C \rightarrow A$; $C \rightarrow A \rightarrow B$; $C \rightarrow B \rightarrow A$).
  • For $k$ conditions, total orders equal $k!$. An IV with $4$ conditions requires $24$ sequence orders.
  • Partial Counterbalancing: Only a subset of all possible condition orders is presented. Used when condition numbers would require prohibitive participant sample sizes:
  • Randomized Order: A computer generates a distinct random sequence for each participant.
  • Latin Square: A formal square matrix system ensuring that every condition appears in each ordinal position at least once.

• Additional Disadvantages of Within-Groups Designs:

  • Permanent State Changes: Inappropriate when exposure to an IV level permanently alters the participant, making baseline resets impossible (e.g., comparing bike-riding instruction Method A vs. Method B, or assessing mindfulness training impacts on GRE performance).
  • Demand Characteristics (Experimental Demand): Cues that enable participants to deduce the study's hypothesis, causing them to alter their spontaneous behavior (e.g., rating attractiveness across photos of Black, Latine, and White individuals sequentially).

Interrogating Causal Claims with the Four Big Validities

• Interrogating Construct Validity:

  • Dependent Variables: Evaluating the reliability and validity of measurement instruments.
  • Scale Example: The Internal Motivation to Respond Without Prejudice Scale (IMS; Plant & Devine, 1998) used in Murrar et al. (2020).
  • IMS Psychometric Properties: Internal consistency (Cronbach's alpha $> .80$); convergent validity (negative correlation with self-reported racism, positive correlation with humanitarianism); discriminant validity (uncorrelated with social interaction anxiety); criterion validity (predicts negative affect when imagining acting prejudiced).
  • Independent Variables: Evaluating manipulation efficacy.
  • Manipulation Checks: An extra dependent variable inserted into an experiment to collect empirical proof that the manipulation produced the intended mental or emotional state (Chester & Lasko, 2021).
  • Empirical Example: Kaplan & Pascoe (1977) manipulated lecture humor (serious vs. humorous) to measure memory. A manipulation check required students to rate how "funny" and "light" the lecture was, confirming the humor condition was perceived as funnier.
  • Pilot Studies: A separate study conducted before (or after) the main experiment using a distinct participant sample to test manipulation effectiveness.
  • Empirical Example: Murrar et al. (2020) conducted a pilot study showing the norms poster to minoritized and nonminoritized students, confirming it was authentic and non-offensive.
  • Theory Testing: Aligning variables with conceptual definitions.
  • Empirical Example: Boothby et al. (2014) tested the theory that shared experiences intensify emotional experiences. Study 1 used sweet chocolate (positive experience). Study 2 used bitter $90\%$ dark chocolate (negative experience). Shared tasting rendered bitter chocolate less liked than unshared tasting, and increased absorption ratings across both studies, verifying construct validity.

• Interrogating External Validity:

  • Generalizing to Other People: Evaluated by examining sampling methods (random sampling vs. convenience sampling).
  • Convenience Sample Limits: Murrar et al. used $163$ UW-Madison undergraduates, leaving generalizability to broader populations unknown.
  • Subgroup Analysis: Recruiting large, diverse samples enables separate statistical analyses within minoritized and nonminoritized subgroups. Murrar et al.'s follow-up studies verified norms intervention efficacy across both demographic groups.
  • Generalizing to Other Situations: Evaluated via environmental variations (e.g., testing norms via posters vs. videos; moving from laboratory settings to real undergraduate classrooms).
  • Validity Trade-offs: Experimenters frequently prioritize internal validity (experimental control and confound elimination) over external validity (real-world representativeness). Unsystematic variability is minimized by using homogeneous student samples in controlled environments.

• Interrogating Statistical Validity:

  • Effect Size Expression:
  • Original Units: Evaluates practical impact using raw scale measurements (e.g., infants pressing a button $22.5$ vs. $12$ times; or an increase of approximately $\frac{2}{3}$ notch on a $9\text{-point}$ Likert scale in the norms study).
  • Standardized Effect Size ($d$ or Cohen's $d$): Measures the distance between group means relative to the spread/standard deviation of scores within groups.
  • Distribution Overlap Dynamics: Standardized effect size $d$ decreases as group means get closer or as within-group variability increases (higher overlap = smaller $d$).
  • Guidelines and Benchmarks for Cohen's $d$:
  • Standard Benchmarks (Cohen, 1992): $d = 0.2$ (small), $d = 0.5$ (moderate), $d = 0.8$ (large). (Corresponding correlation benchmarks: $r = .10$, $r = .30$, $r = .50$).
  • Domain-Specific Benchmarks: Educational intervention benchmarks average $d = 0.27$ (Hill et al., 2008).
  • Observed Study Values: Inclusive climate norms study $d = 0.37$; infant persistence study $d = 0.56$.
  • Confidence Intervals ($95\%$ CI) and Statistical Significance:
  • CI Precision: Reflects estimate precision based on sample size (larger samples yield narrower CIs).
  • Zero Exclusion Rule: A $95\%$ CI that excludes zero is statistically significant ($p < .05$), allowing researchers to rule out a true population difference of zero.
  • Observed CI: Murrar et al. calculated $d = 0.37$ with a $95\%$ CI of $[0.04, 0.71]$.
  • Replication:
  • Collecting multiple population estimates via exact or conceptual replications.
  • Forest Plots: Graphical displays illustrating effect sizes and $95\%$ CIs across multiple studies.
  • Empirical Replications: Murrar et al. published $5$ studies ($2$ poster, $3$ video), all showing $95\%$ CIs above zero. Leonard et al. conducted a direct replication with $120$ additional infants, duplicating the original persistence effect.

• Interrogating Internal Validity:

  • Primary Checklist: Verify the absence of design confounds (accidental covariation), selection effects (controlled via random assignment or matching), and order effects (controlled via counterbalancing).

Practice Scenarios and Empirical Applications

• Application Scenario 1: Hand Temperature and Friendliness Ratings (Max's Experiment)

  • Methodology: Participants shook hands with a female experimenter whose hands were pre-cooled in cold water or pre-warmed in warm water, using an identical standard greeting. Participants rated her friendliness on a self-report measure.
  • Findings: Warm hands condition generated significantly higher friendliness ratings than cold hands condition.
  • Temporal Precedence: Handshake preceded friendliness rating evaluation.
  • Control Variables: Experimenter identity (same female actor) and standardized greeting phrase.
  • Experimental Design Classification: Posttest-only independent-groups design.
  • Selection Effect Control: Random assignment to warm or cold conditions.
  • Construct Validity Question: Evaluating how well the self-report friendliness rating operationalized actual participant impressions.

• Application Scenario 2: Bystander Intervention in Staged Accidents (Latané & Darley, 1969)

  • Methodology: Participants filled out questionnaires in a lab room. An experimenter next door staged a loud accident (pre-recorded audio of falling off a chair and groaning). Participants were assigned via coin flip to be alone or accompanied by a passive confederate who sat impassively.
  • Dependent Variable: Participant helping behavior (stopping the questionnaire and investigating the accident room).
  • Control Variables: Identical lab room setup and identical audio recording playback.
  • Findings: $70\%$ of participants in the alone condition attempted to help, compared to only $7\%$ in the passive confederate condition.
  • Experimental Design Classification: Posttest-only independent-groups design.