Week 2 — Reliability, Validity & The IAT Case Study

Week 2 Overview & Learning Outcomes

• Transition: moving from last week’s conceptual introduction to psychological testing to this week’s focus on two psychometric “usability” properties – reliability and validity.

• Five foundational properties of a psychological test (re-visited)

1. Standardised administration

2. Appropriate norms

3. Accurate scoring rules

4. Practical utility

5. Good psychometric properties (reliability & validity) ⇦ focus of Week 2

• Four learning outcomes for the week
– LO 1: Define reliability & explain its importance
– LO 2: Describe multiple forms of reliability
– LO 3: Define validity & explain its importance
– LO 4: Describe multiple forms of validity

Assigned Reading Logistics

• Weekly reading citations appear on lecture slides.
• Ebook users: page numbers in the Monash Library e-version differ from those on slides. ➜ Navigate by section headings supplied in Moodle folders.
• Hard-copy textbook users: slide page numbers remain accurate.

Video Sequence & Pedagogical Rationale

1. Case-study video (current): builds intuition for why reliability & validity matter.

2. Mini-lecture on reliability (next).

3. Mini-lecture on validity (last).

Case Study: The Implicit Association Test (IAT)

Background & Influence

• Introduced in 1998 by Greenwald et al.
• >16 000 citations; heavily referenced in academic, corporate, and public-policy contexts.
• Often integrated into HR diversity training and anti-bias workshops.

Intended Construct

• Measures implicit (unconscious) racial prejudice – automatic attitudes or associations individuals may hold without conscious endorsement.
• Implicit biases are empirically documented, contribute to stereotyping, prejudice, and micro-aggressions, and can produce real-world harms.

Task Structure

1. Phase 1 – Familiarisation/Baseline
   a. Face categorisation: press E for White, I for Black.
   b. Word categorisation: press E for bad words (e.g., “despise”), I for good words (e.g., “joy”).

2. Phase 2 – Critical Blocks (Bias Measurement)
   • Mixed trials present either a face or a word.
   • Example key mapping (varies across blocks):
   – E = White or Bad word
   – I = Black or Good word
   • Metric: Slower reaction times (RTs) and increased errors when required to pair Black + Good relative to pairing Black + Bad are interpreted as stronger anti-Black bias.

Criticism 1 – Poor Reliability (Instability Over Time)

Why Reliability Matters

• A psychometric instrument must yield stable scores so that an observed score reflects the person’s true standing on the construct.
• Without stability, high (or low) scores could be artefacts of transient error.

Weight-Scale Analogy

– Unreliable scale: 69.3 kg → 89.2 kg → 74.3 kg for the same individual (no confidence in any single reading).
– Reliable scale: 67.5 kg → 67.4 kg → 67.4 kg (high confidence the true weight is ≈ 67.4–67.5 kg).

Quantifying Stability: Test–Retest Reliability

• Correlation between scores at Time 1 and Time 2.
• Bounded: $0 \leq r<em>{tt} \leq 1$ – $r</em>{tt} \approx 0$ = completely unstable
– $r<em>{tt} \approx 1$ = perfectly stable • Conventional cutoff for “usable” tests: $r</em>{tt} \geq 0.70$ to $0.80$.

IAT’s Empirical Estimate

• Meta-analytic test–retest reliability: $r_{tt} = 0.44$ → labelled “unacceptable.”
• Consequence: A person obtaining a “high bias” score may or may not actually possess high implicit prejudice; the reading could simply be measurement noise.

Criticism 2 – Questionable Validity (What Is Actually Measured?)

Divergent Behavioural Findings

• Study A: White police officers with high anti-Black IAT scores were faster to shoot African-American targets.
• Study B: Similar officers with high anti-Black IAT scores were slower to shoot African-American targets.
• Such contradictory patterns undermine confidence that the IAT captures a single, well-defined construct of racial prejudice.

Alternative Construct Interpretation

• IAT may index familiarity with or awareness of cultural stereotypes, rather than personal endorsement of those stereotypes.
– Example: An activist who battles racism daily may have strong cognitive accessibility of negative stereotypes and therefore generate a “high bias” RT profile despite personally rejecting those beliefs.

Real-World Consequences of Using a Flawed Test

• Corporate HR: High scorers singled out for mandatory sensitivity training – could misallocate resources if scores are unreliable/invalid.
• Police Reform: Anti-racism programs evaluated via IAT change scores; interventions often show gains that fade within 24 hours, suggesting inadequate outcome metrics.
• Policy & Funding Decisions: Over-reliance on an unstable measure may mask true effectiveness of racism-reduction initiatives.

Key Takeaways – Why Reliability & Validity Are Non-Negotiable

1. The more socially or ethically important a construct (e.g., implicit bias), the more critical it is to have precise, dependable, and accurate measurement tools.

2. Reliability (consistency) is a prerequisite for validity (measuring the intended construct) but does not guarantee it.

3. Psychometric flaws are not merely academic; they ripple into hiring decisions, police conduct assessments, and societal attitudes toward inequality.

4. Recognising measurement limitations allows researchers and practitioners to refine tools, design better interventions, and avoid unintended harms.

What Comes Next

• Mini-Lecture 2 will dive into technical definitions, formulas, and multiple forms of reliability (e.g., internal consistency, inter-rater, parallel forms).
• Mini-Lecture 3 will unpack validity frameworks (content, criterion, construct, and argument-based validity) and how to assemble evidence for each.