Share 07 PSM108

Psychological Testing and Assessment Utility

Intended Learning Outcomes

  • Define utility and apply utility analysis principles, emphasizing its importance in setting cut scores for assessments.

  • Analyze utility analysis methods and evaluate their relevance in establishing cut scores for assessments.

  • Demonstrate competence in understanding utility, applying utility analysis, and critically assessing methods for setting cut scores in evaluation of assessments.

Utility Defined

  • Utility

    • In everyday language, utility refers to the usefulness of something or a process.

    • In psychometrics, utility (also referred to as test utility) indicates how useful a test is.

    • It relates to the practical value of using a test to aid in decision-making.

    • The effectiveness of testing in improving efficiency is a critical aspect of utility.

Factors that Affect a Test’s Utility

  • Psychometric Soundness

    • A test is psychometrically sound if its reliability and validity coefficients are acceptable.

    • Reliability Index: Measures the consistency of what a test measures.

    • Validity Index: Evaluates if the test measures what it is intended to measure.

    • Utility Index: Reflects the practical value of information derived from test scores.

  • Cost

    • Refers to disadvantages, losses, or expenses linked to the testing process, both economic and non-economic.

    • Example costs include purchasing tests, supplies, and computer processing services.

  • Benefits

    • Involves profits, gains, or advantages derived from testing.

    • When well-implemented, test administration costs can be minimal compared to significant economic benefits.

    • Non-economic benefits may include improved workplace environments.

Utility Analysis

  • Utility Analysis

    • A collection of techniques for cost-benefit analysis that assess the usefulness of an assessment tool.

    • It is an umbrella term for various methods that require different data input and yield diverse outputs.

  • Conducting Utility Analysis

    • Utilizes expectancy data to convert scatterplots of test data into expectancy tables.

    • Expectancy Table: Indicates scoring likelihood on a criterion measure.

  • Expectancy Tables

    • Taylor-Russell Tables: Estimate how a test inclusion improves selection systems.

    • Naylor-Shine Tables: Assess the mean differences between selected and unselected groups to determine the value added by the assessment tool.

  • Brogden-Cronbach-Gleser Formula

    • Calculates the dollar amount of utility gain from using a specific selection instrument under prescribed conditions.

    • Utility Gain: Benefits (monetary or otherwise) derived from using a specific test or selection method.

    • Productivity Gain: Increase in work output attributable to specific tools or methods.

Some Practical Considerations in Utility Analysis

  • The pool of job applicants.

  • Job complexity.

  • The cut score in use.

    • Cut Score: A numerical reference point derived from judgments that classifies data into different actions or inferences.

Cut Scores in Utility Analysis

  • Relative Cut Score

    • Set based on norm-related considerations rather than test scores' correlation with a criterion.

  • Fixed Cut Score

    • Based on judgments about minimum proficiency levels necessary for inclusion in classifications.

  • Multiple Cut Score

    • Employs two or more cut scores to categorize test takers based on one predictor.

  • Multiple Hurdle

    • Elements within a multistage decision-making process requiring specific cut score achievements on tests to advance to following evaluation stages.

  • Compensatory Model of Selection

    • Assumes that high scores on one attribute can offset or compensate for low scores on another attribute in selection processes.