Test Validity – Criterion-Related Validity

What is criterion-related validity used for?

To predict or estimate scores on another measure (criterion)

What are the two types of criterion-related validity?

Concurrent validity and predictive validity

What is concurrent validity?

Evaluating predictor and criterion scores at about the same time to estimate current status

What is predictive validity?

Evaluating predictor scores before obtaining criterion scores to predict future status.

What does the criterion-related validity coefficient indicate?

The accuracy of predictor scores in predicting criterion score. ranging from -1 to +1. The closer to ±1, the better the prediction.

What does squaring the validity coefficient tell us?

The amount of variability in the criterion explained by the predictor.

What happens during cross-validation?

Predictor is validated on a new sample

what could cause the validity coefficient to shrink in criterion-related validity?

Small sample sizes and/or many predictors.

When does shrinkage occur in criterion related validity?

when chance factors inflate initial validity

What happens when the predictor and/or criterion have low reliability?

The validity coefficient is attenuated (reduced) due to measurement error.

What is correction for attenuation used for?

To estimate the maximum validity coefficient if reliability = 1.0 (perfect reliability).

What is clinical utility?

The extent to which a test is useful for clinical purposes, assessed through incremental validity and diagnostic efficiency.

What is incremental validity?

The increase in prediction accuracy when adding a new predictor to existing methods.

What happens when you raise the predictor cutoff score?

fewer true positives and false positives, but more false negatives.

What happens when you lower the predictor cutoff score?

more true positives and false positives, but fewer false negatives.

What is diagnostic efficiency?

test’s ability to correctly distinguish between those with and without a disorder or characteristic

What is sensitivity?

identifies true positives

What is specificity?

identifies true negatives.

Higher prevalence

higher positive predictive value, lower negative predictive value

Lower prevalence

lower positive predictive value, higher negative predictive value

What limits the validity coefficient of a predictor?

The predictor’s reliability places a ceiling on its validity

What does the term 'base rate' refer to?

The proportion of people in the population who meet the criterion.

How does a low base rate affect predictions?

Reduces the test’s positive predictive value and More false positives occur.

How does a high base rate affect predictions?

Increases the test’s positive predictive value (PPV). Fewer false positives occur.

What is the hit rate in a diagnostic test?

The percentage of correct predictions, including true positives and true negatives.

False positives

People incorrectly predicted to meet the criterion

False negatives

People incorrectly predicted to NOT meet the criterion.

What is the selection ratio?

The proportion of applicants selected for a position

How does a low selection ratio impact predictions?

Increases prediction accuracy because fewer candidates are selected → reduces false positives.

How does a high selection ratio impact predictions?

Reduces prediction accuracy because more candidates are selected → increases false positives.

What is the Taylor-Russell table used for?

To determine the incremental validity of a test by considering base rate, selection ratio, and validity coefficient.

What is utility analysis?

Assesses the cost-benefit ratio of using a predictor for decision-making.

What is a true positive?

A correct prediction that someone meets the criterion

What is a true negative?

A correct prediction that someone does NOT meet the criterion

What does the term 'criterion contamination' mean?

When the criterion is influenced by predictor scores, artificially inflating the validity coefficient.

What is construct validity?

The degree to which a test measures the theoretical construct it claims to measure.

Convergent Validity

Test correlates with related measures.

Divergent Validity

Test does NOT correlate with unrelated measures.

How can you improve criterion-related validity?

Use a reliable criterion, a representative sample, and a sufficiently large sample size.

What does a low validity coefficient indicate?

Poor prediction accuracy due to factors like low reliability, contamination, or sampling issues.