Utility

Factors Affecting a Test's Utility

• Utility: Usefulness or practical value of testing to improve efficiency.

• Psychometric Soundness: Reliability and validity of a test.- A test is psychometrically sound if reliability and validity coefficients are acceptably high.

  • Reliability index indicates how consistently a test measures what it measures.

  • Validity index tells whether a test measures what it's supposed to measure.

  • Utility index indicates the practical value of information from test scores.

• Test scores have utility if their use leads to better, cost-effective decisions.

• A psychometrically sound test may have low utility if test-takers don't follow directions carefully.

Costs and Benefits

• Costs: Disadvantages, losses, or expenses (economic and noneconomic).- Economic costs: Financial expenditures for the test (blank test protocols), computerized processing, scoring, interpretation, personnel, facility rental, insurance, legal, accounting, and licensing.

  • Noneconomic costs: Emotional harm.

• Benefits: Profits, gains, or advantages.- Economic perspective: Financial returns from a successful testing program.

  • Noneconomic perspective: Increased work quality and quantity in industrial settings, good work environment.

• The cost of testing is justified by significant noneconomic benefits.

Utility Analysis

• A cost-benefit analysis assessing the usefulness and practical value of an assessment tool.

• Used to evaluate if the benefits of a test outweigh the costs.

• Helps decide if one test is preferable to another, or if adding tests improves the process, or if training programs and interventions are useful, by making it more effective and efficient.

• The goal is to make an informed decision about the optimal course of action.

How Utility Analysis is Conducted

Expectancy Data

• Expectancy Table: Indicates the likelihood of a test-taker scoring within a specific interval on a criterion measure (e.g., passing, acceptable, failing).

• Taylor-Russell Table: Estimates the improvement in selection from including a test in the selection system.- Estimates the percentage of successful hires using a particular test, considering test validity, selection ratio, and base rate.

  • Validity Coefficient: r_{xy} (correlation between test and job performance).

  • Selection Ratio: The ratio of positions filled to applicants (\frac{\text{Positions}}{\text{Applicants}}).

  • Base Rate: Percentage of successful hires under the existing system.

  • Limitation: Assumes a linear relationship between the predictor (test) and the criterion (job performance).

• Naylor-Shine Table: Determines the increase in average score on a criterion measure by comparing the means of selected and unselected groups.

Brogden-Cronbach-Gleser Formula

• Calculates the dollar amount of utility gain from using a selection instrument under specified conditions.- Utility Gain: An estimate of the benefit (monetary or otherwise) of using a particular test or selection method.

  • A modification exists for prioritizing productivity over financial gains:

  • Productivity Gain: Estimated increase in work output.

Cut Scores

• A reference point used to classify data, guiding actions or inferences.- Relative Cut Score: Based on norm-related considerations, referencing the performance of a group (norm-referenced).

  • Fixed Cut Score: Set with reference to a judgment concerning a minimum level of proficiency.

• Absolute cut scores.

• Multiple Cut Scores: Using two or more cut scores with reference to one predictor.- Example: Academic success (A+, B+, C, D, F).

• Multiple Hurdle: A multi-stage decision-making process where achieving a cut score on one test is required to advance.- Assumes a minimum level of knowledge, skills, or ability is needed for each attribute measured.

• Compensatory Model of Selection: High scores on one attribute can compensate for low scores on another.- Strengths in some areas can offset weaknesses in others.

Methods of Setting Cut Scores

Angoff Method

• Averaging expert judgments to determine cut scores.

• Experts judge how a minimally competent individual would respond to test items.

• Scores at or above the cut score indicate sufficient ability or trait level.

• Achilles heel: Low inter-rater reliability and disagreement on testtaker responses.

Known Groups Method

• Collecting predictor data from groups known to possess or lack a trait/ability.

• Setting a cut score that best discriminates the two groups’ test performance.

• Disadvantage: Cutoff score is affected by the composition of the groups.

Item Response Theory (IRT)-Based Methods

• Classical test score theory for setting cut scores.

• Cut scores are typically set based on testtakers’ performance across all the items on the test

• Test-takers must meet the minimum score to pass.

• Each item is associated with a difficulty level.

• To pass, test-takers must answer items above a minimum difficulty level that experts determine.

  • IRT methods: Item-mapping method entails the arrangement of items in a histogram, with each column containing items deemed to be of equivalent value

Bookmark Method

• Experts are trained on the minimal knowledge/skills needed to pass.

• Experts review items in ascending order of difficulty and place a bookmark to separate those who meet the standard from those who do not.

• The bookmark serves as the cut score.

• Drawbacks: Training issues, floor and ceiling effects, item booklet length.- Floor Effects: A high percentage of participants score the minimum (test too difficult).

  • Ceiling Effects: A high percentage score the maximum (test too easy).

Method of Predictive Yield

• Developed by R.L. Thorndike.

• Considers the number of positions, likelihood of offer acceptance, and applicant