The Incremental Validity of Psychological Testing and Assessment Notes

The Incremental Validity of Psychological Testing and Assessment

Introduction

  • Authors: John Hunsley (University of Ottawa), Gregory J. Meyer (University of Toledo).

  • Purpose: Evaluate incremental validity in applied psychology; examine research design, statistical, and measurement issues.

Overview of Incremental Validity

  • Definition: Incremental validity refers to whether a measure adds to the prediction of a criterion above what can be predicted by other data.

  • Importance: Understanding how measures improve predictions leads to complexities such as power, sensitivity, specificity, and predictive efficacy in clinical decision-making.

  • Clinical contexts for assessment include:

    • Diagnosing disorders

    • Developing case conceptualizations

    • Treatment planning

    • Treatment monitoring

    • Treatment outcome evaluation

  • Incremental validity can vary by application; a measure may be valid in one scenario and not in another.

  • Cost considerations: Financial and human resource costs should be weighed against incremental validity in applied settings.

Historical Context

  • 1950s Development: Test validity became a significant focus, leading to discussions around the incremental validity of tests for personnel decisions (Cronbach & Gleser, 1957).

  • Sechrest (1963): Articulated the concept, emphasizing that new tests must show improved prediction compared to existing data.

  • Wiggins (1973): Emphasized the context-specific nature of incremental validity and the necessity of alternative measures to be considered.

Key Contributors and Findings

  • Anastasi (1988): Discussed how incremental validity is influenced by base rates and selection ratios.

  • Garb (1984): Conducted a review on incremental validity within clinical assessments, noting inconsistencies and lack of cumulative research.

  • Incremental validity studies span various psychological domains, including:

    • Anxiety sensitivity (Lilienfeld, 1997)

    • Cognitive ability self-reports (Schwartz et al., 1996)

    • Use of response latencies in various predictions (Holden & Hibbs, 1995).

  • Notable focus on personnel psychology and meta-analytic studies (e.g., Huffcutt et al., 1996; Schmidt & Hunter, 1998).

Research Design Considerations

Conceptual Overview

  1. Incremental Validity of Testing Instruments

    • Used to evaluate how new measures contribute to predicting relevant clinical criteria.

    • Studies often employ regression analysis to determine variance accounted for by new data.

    • Example: Watkins and Glutting (2000) examined cognitive subtest profiles for predicting academic achievement.

  2. Incremental Validity of Test-Informed Clinical Inferences

    • Focuses on how clinician interpretations based on test data improves predictive validity.

    • Example: Schwartz & Wiedel (1981) found MMPI results significantly improved diagnostic accuracy in neurology residents.

  3. Incremental Validity as Validation for New Measures

    • Examines new assessments against existing measures to justify their development.

    • Example: Lilienfeld (1996) compared MMPI–2 scales and established criteria, finding differences in measured constructs.

Issues in Design and Analysis

  • Predominantly correlational designs are used; experimental designs are rare.

  • Possible Methodologies:

    • Random assignment in manipulated assessments allows for examining data contribution.

    • Within-subject designs can evaluate changes in prediction accuracy as data accumulates.

    • Hierarchical multiple regression is commonly utilized where variable entry order is critical.

Statistical Issues

  1. Reliability and Incremental Validity

    • A measure might show increment due to reliability rather than unique variance.

    • Sechrest’s correction for attenuation method can help determine true incremental contributions.

  2. Regression Analysis Approaches

    • Hierarchical Regression: Researcher specifies entry order.

    • Stepwise Regression: Order based on associations, can capitalize on sampling error, and is less reliable.

  3. Meaningfulness of Validity Increments

    • Significance does not equate to practical importance; size and context of increments for clinical utility should be evaluated.

    • SOS effects approach may provide insights into shared versus unique variance in predictions.

  4. Semipartial r: Useful statistic that quantifies contribution size in multiple regression contexts.

Criterion Problems in Incremental Validity Research

  • Issues with criterion selection and measurement impact validity findings.

  • Reliable criteria are crucial to avoid artificial inflation in predictor associations.

  • Criterion Contamination: Occurs when predicted outcomes influence the predictor measures.

  • Source Overlap Artifact: Common information source leads to inflated validity estimates.

Implications for Research and Clinical Practice

  • The necessity for cumulative research in incremental validity across different studies and contexts.

  • Incremental validity evidence does not always predict improved clinical decisions; psychologists require user-friendly strategies to integrate findings effectively.

  • Focus on non-redundant, convergent data to enhance clinical assessment accuracy.

  • Ongoing assessment will require a re-evaluation of the incremental validity beyond initial data collection phases.

  • Additional research is essential to understand the contribution of new assessment practices in clinical settings.

References

  • Comprehensive list as provided in the transcript, focusing on noteworthy assessments, psychometric theories, and historical developments in psychology.