Chapter 7 – Test Utility and Utility Analysis

Flashcards cover definitions, formulas, decision-theory concepts, cut-score methods, and practical considerations related to test utility and utility analysis.

What is test utility in psychometrics?

The usefulness or practical value of using a test (or testing program) to improve decision-making efficiency.

Which three broad questions illustrate comparative, treatment, and diagnostic utility?

1) How useful is this test vs. another test? 2) Does using this test lead to better intervention results? 3) How well does this neurological test classify cases?

Name the three major factors that affect a test’s utility.

Psychometric soundness, costs, and benefits.

How does utility differ from reliability and validity?

Reliability is consistency of measurement; validity is whether the test measures what it purports to; utility concerns the practical value of the information for making better, cost-effective decisions.

Why can a test be valid yet have little utility?

If it is impractical, too costly, easily tampered with, or its results are not used to make different decisions (e.g., hiring everyone regardless of score).

Give an example of a valid tool with limited utility from the notes.

The sweat-patch test for cocaine showed 92% agreement with urine tests but had limited utility because participants often tampered with the patch.

What kinds of ‘costs’ are considered in utility analysis?

Economic (money, staff time, facilities) and noneconomic (safety risks, loss of public confidence, pain, suffering).

Provide two noneconomic benefits that may arise from personnel testing.

Fewer workplace accidents and decreased employee turnover.

Define utility analysis.

A family of cost–benefit techniques designed to determine the practical value or usefulness of a test, training program, or intervention.

List four decisions a utility analysis can inform when evaluating a test.

Whether (1) one test is preferable to another, (2) a test is better than another assessment tool, (3) adding a test to a battery helps, or (4) no testing is preferable.

What are expectancy data in utility analysis?

Information (often displayed in expectancy tables) showing the likelihood that test-takers scoring in certain ranges will achieve specific criterion outcomes.

What do the Taylor-Russell tables estimate?

The percentage of employees who will be successful using a given test, based on its validity, the selection ratio, and the base rate.

What limitation do Taylor-Russell tables share with Naylor-Shine tables?

They require a linear relationship between predictor and criterion and often dichotomize performance into ‘successful’ vs ‘unsuccessful.’

State the Brogden-Cronbach-Gleser (BCG) formula in words.

Utility gain equals (number hired × tenure × validity × SD of job performance × mean z-score of selectees) minus (number tested × cost per test).

What is ‘productivity gain’ in the modified BCG formula?

An estimate of percent increase in work output obtained by replacing SD in dollars (SDy) with SD of productivity (SDp).

Define base rate in personnel selection.

The proportion of applicants who would be successful if all were hired (success rate without testing).

What is the selection ratio?

The proportion of applicants to be hired relative to the total number who apply.

Explain a ‘hit,’ ‘false positive,’ and ‘false negative’ in decision theory terms.

Hit: applicant passes test and succeeds on job. False positive: passes test but fails job. False negative: fails test but would have succeeded.

Why can high selection ratios inflate the false-positive rate?

Because more applicants are hired, increasing the chance of selecting unqualified candidates.

Describe top-down selection.

Filling positions by ranking applicants on test scores and hiring from the highest downward until slots are filled.

What is adverse impact in top-down selection?

Unintended discriminatory effect on protected groups due to hiring strictly by highest scores.

Contrast relative and fixed (absolute) cut scores.

Relative (norm-referenced) cut scores depend on group performance (e.g., top 10%); fixed cut scores are preset minimum proficiency levels independent of group scores.

What are multiple cut scores?

Two or more cutoff points on one predictor (or across predictors) that categorize applicants into several classifications (e.g., A, B, C, D, F).

Define a multiple-hurdle selection process.

A multistage procedure where applicants must meet a specific cut score at each stage to proceed to the next assessment stage.

What is a compensatory model of selection?

A model in which high scores on one predictor can offset low scores on another; predictors are differentially weighted and summed.

Name two practical issues that can distort utility estimates.

Assuming an unlimited applicant pool and assuming all selected applicants accept job offers.

What is the Angoff method for cut scores?

Experts estimate the probability that minimally competent individuals will answer each item correctly; averaged probabilities across items set the passing score.

Describe the known-groups (contrasting groups) method.

Administer test to groups known to possess and not possess the trait; set the cut score at the point of least overlap between their score distributions.

Explain the item-mapping method (IRT-based).

Items are grouped by difficulty; experts select the column representing the lowest difficulty that minimally competent examinees should answer correctly, establishing the cut score.

What is the bookmark method?

Experts place a ‘bookmark’ in an ordered booklet of test items (easiest to hardest) at the point where minimally competent examinees would answer items correctly 50% of the time.

Define return on investment (ROI) in testing.

The ratio of monetary benefits gained from using a test to the costs incurred in implementing the test.

Why might productivity terms be preferred over financial terms in presenting utility gains?

In public-sector or non-profit contexts where direct dollar values are hard to estimate or politically sensitive, increases in output may be a clearer metric.

How does job complexity affect utility analysis?

More complex jobs show greater variance in performance, which can increase the potential utility of valid selection tests.

Give an example of economic cost vs noneconomic cost in testing decisions.

Economic: $100 for an X-ray; Noneconomic: risk of undiagnosed child abuse if X-ray is not taken.

What does sensitivity measure in a selection test?

The proportion of truly qualified applicants correctly identified (hit rate).

What does specificity measure?

The proportion of truly unqualified applicants correctly rejected (true negative rate).

Explain positive predictive value (PPV).

Among those who pass the test, the proportion who are actually successful on the criterion.

Explain negative predictive value (NPV).

Among those who fail the test, the proportion who are actually unsuccessful on the criterion.

What is utility gain?

The estimated benefit—monetary, productivity, or other—resulting from using a specific test or selection method.

How can a high base rate limit the usefulness of a test?

If most applicants would succeed without testing, the incremental gain from testing is small, reducing utility.