M.25 UTILITY PSYCH ASSESS 2

huh

utility

The practical value of testing to improve efficiency.

factors affecting utility

Psychometric soundness, costs, and benefits.

psychometric soundness

Higher the criterion-related validity of test scores, the higher the utility of the test. Exceptions exist as many factors may enter into an estimate of a test’s utility, and there are variations in the ways in which the utility is determined. Valid tests are not always useful tests.

costs

One of the most basic elements of utility analysis is the financial cost associated with a test.

cost in the context of test utility

Refers to disadvantages, losses, or expenses in both economic and noneconomic terms.

economic costs

May include purchasing a particular test, a supply of blank test protocols, and computerized test processing.

other economic costs

The cost of not testing or testing with an inadequate instrument are more difficult to calculate.

noneconomic costs

Include costs related to human life and safety.

benefits

The benefits of testing should be weighted against the costs of administering, scoring, and interpreting the test.

benefits definition

Profits, gains, or advantages.

successful testing programs

Can yield higher worker productivity and profits for a company.

potential benefits

Increase in the quality and quantity of workers’ performance, a decrease in the time needed to train workers, a reduction in the number of accidents, and a reduction in worker turnover.

noneconomic benefits

May include a better work environment and improved morale.

utility analysis

A family of techniques that entail a cost–benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment.

utility tests

Address the question of “Which test gives us the most bang for the buck?”

utility analysis endpoint

Yields an educated decision as to which of several alternative courses of action is most optimal (in terms of costs and benefits).

expectancy data

The likelihood that a test taker will score within some interval of scores on a criterion measure.

taylor-russell tables

Provide an estimate of the percentage of employees hired by the use of a particular test who will be successful at their jobs.

taylor-russell variables

The test’s validity, the selection ratio used, and the base rate.

validity

The validity coefficient.

selection ratio

A numerical value of the relationship between the number of people to be hired and the number of people available to be hired.

base rate

The percentage of people hired under the existing system for a particular position.

naylor-shine tables

Help obtain the difference between the means of the selected and unselected groups to derive an index of what the test (or some other tool of assessment) is adding to already established procedures.

validity coefficient

Both Taylor-Russell and Naylor-Shine tables comes from concurrent validation procedures.

brogden-cronbach-gleser formula

Used to calculate the dollar amount of a utility gain resulting from the use of a particular selection instrument under specified conditions.

utility gain

Refers to an estimate of the benefit (monetary or otherwise) of using a particular test or selection method.

decision theory and utility

Cronbach and Gleser (1965) presented a classification of decision problems, various selection strategies ranging from single-stage processes to sequential analyses, quantitative analysis of the relationship between test utility, the selection ratio, cost of the testing program, and expected value of the outcome, and a recommendation that in some instances job requirements be tailored to the applicant’s ability instead of the other way around (adaptive treatment).

pool of job applicants

Some utility models are based on the assumption that there will be a ready supply of viable applicants from which to choose and fill positions.

pool of job applicants considerations

Some jobs require such expertise or sacrifice that the pool of qualified candidates may be very small. The economic climate also affects the size of the pool. The top performers on a selection test may not accept a job offer.

complexity of the job

The same kind of utility models are used for a variety of positions, yet the more complex the job, the bigger the difference in people who perform well or poorly.

cut score

Relative to the scores achieved by a targeted group.

fixed cut scores

Made on the basis of having achieved a minimum level of proficiency on a test.

multiple cut scores

The use of _____ for a single predictor for the purpose of categorizing test takers.

multiple hurdles

Achievement of a particular cut score on one test is necessary in order to advance to the next stage of evaluation in the selection process.

angoff method

The judgments of the experts are averaged to yield cut scores for the test.

angoff method use

Can be used for personnel selection based on traits, attributes, and abilities.

angoff method problem

Problems arise if there is disagreement between experts.

known groups method

Entails collection of data on the predictor of interest from groups known to possess, and not to possess, a trait, attribute, or ability of interest.

known groups method process

Based on the analysis of data, a cut score is set on the test that best discriminates the groups’ test performance.

known groups method issue

There is no standard set of guidelines for choosing contrasting groups.

IRT-based methods

In an IRT framework, each item is associated with a particular level of difficulty.

IRT-based passing criterion

In order to “pass” the test, the test taker must answer items that are deemed to be above some minimum level of difficulty, which is determined by experts and serves as the cut score.

IRT-based methods examples

Make use of the item-mapping method and the bookmark method.

method of predictive yield

A norm-referenced method proposed by R. L. Thorndike (1949) that took into account the number of positions to be filled, projections regarding the likelihood of offer acceptance, and the distribution of applicant scores.

discriminant analysis

A family of statistical techniques used to shed light on the relationship between identified variables (such as scores on a battery of tests) and two (or more) naturally occurring groups (such as persons judged to be successful at a job and persons judged unsuccessful at a job).