(15) TEST UTILITY

Utility

usefulness or practical value of testing to improve efficiency

Utility

Can tell us something about the practical value of the information derived from scores on the test

Utility

Helps us to make decisions

Utility

Reliability < Validity < Utility

Utility

Higher criterion-related validity = higher utility

Psychometric Soundness

the reliability and validity of a test or tool

Costs

disadvantages, losses, or expenses, both economic and noneconomic

Benefits

profits, gains, or advantages

Utility Analysis

it is a family of techniques that entails a cost-benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment

Utility Analysis

It must be undertaken for the purpose of evaluating whether the benefits of using a test outweigh the costs

Expectancy Tables

provide an indication that a test taker will score within some interval of scores on a criterion measure (passing, acceptable, failing)

Taylor-Russell Tables

provide an estimate of the extent to which inclusion of a particular test in the selection system will improve selection

Selection Ratio

numerical value that reflects the relationship between the number of people to be hired and the number of people available to be hired

Naylor-Shine Tables

entails obtaining the difference between the means of the selected and unselected groups to derive an index of what the test is adding to already established procedures

Brogden-Cronbach-Gleser Formula

used to calculate the dollar amount of a utility gain resulting from the use of a particular selection instrument

Decision Theory

developers examine the usefulness of test scores in helping the test user make decisions

Practical Considerations

The pool of job applicants

Practical Considerations

The complexity of the job

Practical Considerations

The cut score (reference point derived as a result of a judgment and used to divide a set of data into two or more classifications) in use

Relative Cut Score

reference point based on norm-related considerations

Fixed Cut Score

set with reference to a judgement concerning minimum level of proficiency required

Multiple Cut Scores

two or more cut score with one predictor

Multiple Hurdle

multi-stage selection process, a cut score is in place for each predictor

Compensatory Model of Selection

assumption that high scores on one attribute can compensate for lower scores

Methods for Setting Cut Score - Angoff Method

setting fixed score, requires expert judges

Methods for Setting Cut Score - Known Groups Method

collection of data on the predictor of interest from groups known to possess, and not to possess, a trait, attribute, or ability of interest

Methods for Setting Cut Score - IRT-based Methods

cut scores are typically set based on test taker’s performance across all the items on the test

Methods for Setting Cut Score - Item-mapping Method

arrangement of items in histogram, with each column containing items with deemed to be equivalent value

Methods for Setting Cut Score - Bookmark Method

expert places “bookmark” between the two pages that are deemed to separate test takers who have acquired the minimum knowledge, skills, and/or abilities from those who are not

Methods for Setting Cut Score - Bookmark Method

Methods for Setting Cut Score - Method of Predictive Yield

took into account the number of positions to be filled, projections regarding the likelihood of offer of acceptance, and the distribution of applicant scores

Methods for Setting Cut Score - Discriminant Analysis

shed lights on the relationship between identified variables and two naturally occurring groups