PSY2041 Reliability and Validity Notes (Week 2)

What is a psychological test?

Psychological tests are samples of behaviour that are measured using objective procedures.
They produce quantitative scores.
They have objective reference points for interpretation.
They must have good psychometric properties to be useful.
Definitional properties vs. usability properties.

Overview: Reliability and validity (Week 2) – Daniel Bennett

Learning outcomes focus on defining reliability and validity, how to assess them, and the different kinds of validity.
Weekly readings: Shum et al., Ch. 4 (pp. 71–84) and Ch. 5 (pp. 85–93, 98–101).

Mini-lecture 1: A case study — The Implicit Association Test (IAT)

The IAT (Greenwald et al., 1998) measures implicit bias by assessing the strength of unconscious associations.
It is used to measure implicit racial prejudice and has influenced public and policy discourse.
Implicit bias is an unconscious association, belief, or attitude toward a social group, contributing to stereotypes and prejudiced behavior.
Criticisms of the IAT have arisen in the literature: Arkes & Tetlock (2004); Gawronski (2019); Mitchell (2010); Schimmack (2021).

The Implicit Association Test (IAT) – Phase 1 and Phase 2

Phase 1 (example mapping): Categorize stimuli (e.g., White/Black faces) with target attributes (e.g., Bad/Despise, Good).
Phase 2: Merge categories (e.g., White with Good) and measure response times/errors.
Slower/more error-prone categorization when African-American faces and “good” words share the same response key indicates implicit bias.
The IAT is a behavioural measure of implicit bias, not a direct questionnaire.
Source link for IAT: https://implicit.harvard.edu/implicit/

Reliability criticisms of the IAT

One major critique: IAT scores are not stable over time (poor test–retest reliability).
Without stability, a high score on one occasion does not guarantee a high level of the underlying construct on another occasion.
Example reliability comparisons:
- Scale 1 (weight): consecutive measures 69.3, 89.2, 74.3 kg; another person = 85.5 kg → poor stability.
- Scale 2 (weight): consecutive measures 67.5, 67.4, 67.4 kg; another person = 85.5 kg → more stable, closer to actual value.
IAT test–retest reliability estimate: $r_{tt} ext{ (IAT)} \approx 0.44 ext{ (unacceptable)}$
Consequences: without stable measurements, high IAT scores may not reflect stable underlying bias.

Reliability: what it is and how we measure it

Reliability is a measure of consistency: does the test yield the same score across repeated measurements?
Unreliable scales give fluctuating scores for the same person; reliable scales give similar scores on repeated measurements.
Examples repeated from IAT discussion:
- Unreliable: 69.3, 89.2, 74.3 kg for the same person; different person 85.5 kg; unlikely to reflect true weight.
- Reliable: 67.5, 67.4, 67.4 kg for the same person; 85.5 kg for a different person reflects a plausible weight.

Four kinds of reliability

Test-retest reliability: does the test give consistent results over time?
Interrater reliability: does the test give consistent results across different administrators?
Internal consistency: do the items of the test measure the same construct in a coherent way?
Equivalent forms reliability: are different versions of the test consistent with each other?

Test-retest reliability

Measured by the correlation between test scores at two time points for the same participants.
Interpretation: high correlation means scores are stable over time; low correlation indicates instability.
Typical benchmark: a test should have r_{tt} > 0.70 ext{ or } 0.80 to be considered usable.
Interpretations: 0.60–0.70 = questionable; 0.50–0.60 = poor; < 0.50 = unacceptable.
Use when the underlying construct is stable over the interval between tests (e.g., height) rather than transient states (e.g., mood).
Practical concerns: practice effects, interventions between tests, time-of-day/seasonality can affect stability.

Interrater reliability

Assesses consistency between different test administrators scoring the same behaviour/performance.
High interrater reliability means different administrators give similar scores.
Most relevant when scoring involves subjective judgment; depends on data type.

Internal consistency

Measures whether items on a test are all assessing the same underlying construct.
Important for multi-item tests that aim to measure one latent construct.
Methods to assess:
- Split-half reliability: divide items into two halves, score each half, and correlate the halves.
- Cronbach’s alpha: a global estimate of internal consistency across all items.

Split-half reliability

Procedure: split the test into two halves (A and B), compute scores for each half, then assess the correlation between halves.
Used as an estimate of internal consistency, but depends on how items are split.

Cronbach’s alpha

Cronbach (1951) introduced alpha to overcome split-half limitations by averaging across all possible splits (and applying a correction).
Concept: it estimates the proportion of total score variance that is due to the common construct rather than random error.
Interpretation: range from 0 to 1; higher values indicate greater internal consistency.
Common benchmark: values above 0.70–0.80 are acceptable for research; values above .95 may indicate redundancy among items.
Formula (standard form):
$\alpha = \frac{k}{k-1} \left(1 - \frac{\sum{i=1}^{k} \sigmai^2}{\sigmaX^2}\right)$ where k is the number of items, \sigmai^2 is the variance of item i, and \sigma_X^2 is the variance of the total test score.
Spearman–Brown correction is used to adjust reliability from a split-half estimate to the full test, with the common form:
$r{SB} = \frac{2 r{half}}{1 + r_{half}}$

Equivalent forms reliability

Are different versions of a test (e.g., version A vs version B) consistent with each other?
Especially relevant when the same construct is measured with alternate forms (e.g., weekly MCQs with different question sets).
Good equivalent-forms reliability means scores are stable across versions for the same person.

Equivalent forms vs. internal consistency

Internal consistency pertains to consistency among items within a single form.
Equivalent-forms pertains to consistency across different forms aimed at measuring the same construct.

Generalizability theory (G-theory)

Framework asking which reliability index to use depends on the kind of generalization we want to make from test scores.
Generalization across assessors/judges → interrater reliability.
Generalization across versions → equivalent forms reliability.
Generalization from one item to others within the test → internal consistency.
Generalization over time → test-retest reliability.

How much is enough? reliability considerations

Standard guideline: test-retest reliability above $0.70-0.80$ for usable tests.
Thresholds: 0.60–0.70 questionable; 0.50–0.60 poor; < 0.50 unacceptable.
Very high reliability is not always desirable: Cronbach’s alpha > 0.95 may indicate item redundancy.
In clinical settings, high reliability is crucial when decisions directly impact individuals (e.g., capacity to drive after injury).
For measuring group differences, extremely high reliability may be less critical than for measuring individual differences.

Validity: what it is and why it matters

Validity is about how well a test measures what it is intended to measure.
Examples: Ishihara color plates are valid for testing red-green color blindness; a toe length test is not valid for color blindness.
Many psychological constructs are latent (not directly observable): anxiety, empathy, self-esteem, intelligence.
Validity is not binary; evidence accumulates to judge extent of validity in a given context.
There is no perfect method to establish validity; multiple forms of validity evidence are used.

Historical and practical validity evidence (overview)

Binet’s validity criteria for intelligence tests: higher scores for identifying brighter children, and older children should score higher than younger ones; items selected based on these criteria.
Four kinds of validity (assessment evidence):
1) Face validity: does the test appear to measure the intended construct to the test-taker?
2) Content validity: does the test cover all components of the construct?
3) Predictive validity: do test scores predict external indicators of the construct?
4) Construct validity: are test assumptions about the construct theoretically justified?
Evidence for validity can be gathered via convergent and discriminant evidence.

Face validity

Does the test appear to measure the target construct to test-takers?
Examples: spelling aloud vs listing TV shows.
Face validity is the weakest form of validity evidence.
A test can be invalid despite high face validity (e.g., Myers–Briggs); a test can be valid despite low face validity (e.g., finger-tapping for concussion severity).
High face validity can improve test-taker motivation and perceived credibility.

Content validity

Ensures all components of the construct are represented in the test.
Example: Generalized Anxiety Disorder (GAD) includes feelings of anxiety, excessive worry, restlessness, fatigue, concentration difficulties, irritability, muscle tension, sleep disturbance; a test only asking about anxiety feelings would lack content validity.

Predictive validity

Are test scores predictive of external criteria related to the construct?
Examples: driving test predicting safe driving; marital satisfaction test predicting divorce 12 months later; ATAR predicting university performance; new anxiety scale correlating with clinician ratings.
Note: some researchers use predictive validity only when the criterion is measured after the test; others use concurrent validity when criterion is measured at the same time.

Construct validity

Are the test’s assumptions about the construct justified by theory?
Constructs are latent and inferred from behaviour.
Assess construct validity using convergent and discriminant evidence:
- Convergent evidence: scores correlate with measures they should correlate with.
- Discriminant evidence: scores do not correlate with measures they should not be related to.
Example: depression criteria (low mood, anhedonia, guilt, sleep disturbance, etc.) should correlate with related measures of mood and symptoms, and not correlate with unrelated constructs (e.g., colour vision).

How reliability and validity are related

Reliability is necessary for validity: an unreliable test cannot be valid because it does not measure something consistently.
A test can be reliable but not valid: consistent measurement of an irrelevant construct (e.g., toe-length test for colour blindness).
In short: validity requires reliability, but reliability alone does not guarantee validity.

Known-groups validity and criteria quality

Known-groups validity: predictive validity where external groups are known to differ on the construct (e.g., chronic pain severity among treated vs. healthy controls).
A good criterion should be:
1) Reliable itself (cannot establish validity with an unreliable criterion).
2) Theoretically appropriate (the criterion should represent the construct behaviorally).
3) Not contaminated by the test items (avoid criterion that overlaps with test items to inflate validity).

Convergent and discriminant validity in practice

Convergent evidence: expect correlations with related constructs (e.g., depression tests correlating with mood-related measures).
Discriminant evidence: expect low correlations with unrelated constructs (e.g., colour-blindness tests with general visual acuity).

Reliability vs validity: summary

Reliability: consistency of measurement across time, items, raters, or forms.
Validity: whether the measurement actually assesses the intended construct.
They are distinct but interrelated; reliability is a prerequisite for validity; validity is not guaranteed by high reliability alone.

Quiz concept recap (from the slides)

A criticism of the IAT concerns a form of validity: its predictive validity regarding actual behaviour is weak or inconsistent across studies.
Question types include identifying which validity form is being questioned (Face, Content, Predictive, Construct).

Key formulas and numeric guidelines mentioned

Test-retest reliability threshold: $r_{tt} \gtrsim 0.70-0.80\,$ is typically desired; values around 0.44 (as cited for IAT) are considered unacceptable.
Internal consistency (Cronbach’s alpha) ranges from 0 to 1; higher is better, but >0.95 may indicate redundancy.
Cronbach’s alpha formula (standard):
$\alpha = \frac{k}{k-1} \left(1 - \frac{\sum{i=1}^{k} \sigmai^2}{\sigmaX^2}\right)$ where k is the number of items, \sigmai^2 is the variance of item i, and \sigma_X^2 is the total score variance.
Spearman–Brown correction (to estimate full-test reliability from a split-half estimate):
$r{SB} = \frac{2 r{half}}{1 + r_{half}}$

Case-based implications for practice

IAT and implicit bias research illustrates the importance of evaluating both reliability and validity before drawing conclusions about real-world behavior or policy implications.
In psychometrics, choose reliability forms based on what generalization you intend to make (G-theory guidance):
- Interrater reliability for generalization across judges.
- Equivalent forms for generalization across test versions.
- Internal consistency for generalization across items.
- Test-retest for generalization across time.

Summary takeaways

A test must be reliable to be valid, but reliability alone does not guarantee validity.
There are four main types of reliability and four main types of validity, plus related concepts like convergent/discriminant evidence and known-groups validity.
The IAT provides a concrete example of how reliability and validity issues can influence interpretations of a psychological measure, highlighting the ongoing debate about what such tests actually measure and predict.
Practical considerations include choosing the right reliability/validity evidence for the test’s intended use, and balancing the desire for high reliability with the risks of item redundancy and clinical utility.