4.4 Utility Analysis

Utility

Usefulness or practical value of testing to improve efficiency. Reliability < Validity < ___

Higher utility

Higher criterion-related validity means?

Psychometric Soundness, Costs, & Benefits

Factors that affect utility

Psychometric Soundness

The reliability and validity of a test or tool

Costs

Disadvantages, losses, or expenses, both economic and noneconomic

Benefits

Profits, gains, or advantages

Utility Analysis

It is a family of techniques that entails a cost-benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment

Utility Analysis

It must be undertaken for the purpose of evaluating whether the benefits of using a test outweigh the costs

Expectancy Tables, Taylor-Russell Tables, Selection Ratio, Naylor-Shine Tables, Brogden-Cronbach-Gleser Formula, & Decision Theory

Expectancy Tables

Utility analysis method that provides an indication that a test taker will score within some interval of scores on a criterion measure (passing, acceptable, failing)

Taylor-Russell Tables

Utility analysis method that provides an estimate of the extent to which inclusion of a particular test in the selection system will improve selection

Selection Ratio

Utility analysis method that involves a numerical value that reflects the relationship between the number of people to be hired and the number of people available to be hired

Naylor-Shine Tables

Utility analysis method that entails obtaining the difference between the means of the selected and unselected groups to derive an index of what the test is adding to already established procedures

Brogden-Cronbach-Gleser Formula

Utility analysis method used to calculate the dollar amount of a utility gain resulting from the use of a particular selection instrument

Decision Theory

Utility analysis method where developers examine the usefulness of test scores in helping the test user make decisions

Relative Cut Score

Reference point based on norm-related considerations

Fixed Cut Score

Set with reference to a judgement concerning minimum level of proficiency required

Multiple Cut Scores

Two or more cut score with one predictor

Multiple Hurdle

A multi-stage selection process, a cut score is in place for each predictor

Compensatory Model of Selection

Hold the assumption that high scores on one attribute can compensate for lower scores

Angoff Method, Known Groups Method, IRT-Based Methods, Method of Predictive Yield, & Discriminant Analysis

Methods for Setting Cut Score

Angoff Method

Method for setting a cut score that involves setting fixed score. It requires expert judges

Known Groups Method

Method of setting a cut score that is based on a collection of data on the predictor of interest from groups known to possess, and not to possess, a trait, attribute, or ability of interest

IRT-based Methods

Cut scores are typically set based on test taker’s performance across all the items on the test

Item-mapping method

IRT method Method wherein items are arranged in a histogram, with each column containing items with deemed to be equivalent value

Bookmark Method

IRT method where an expert places a “bookmark” between the two pages that are deemed to separate test takers who have acquired the minimum knowledge, skills, and/or abilities from those who are not

Method of Predictive Yield

Method of setting a cut score that takes into account the number of positions to be filled, projections regarding the likelihood of offer of acceptance, and the distribution of applicant scores

Discriminant Analysis

Method of setting a cut score that shed lights on the relationship between identified variables and two naturally occurring groups