Chapter 6 Practice: Validity

Concept and Scope of Validity

Traditional and Modern Frameworks of Validity

  • The Trinitarian View: Traditionally, measurement specialists, such as Guion (1980), conceptualized validity in three categories:

    1. Content Validity: Based on the subjects, topics, or content covered by test items.

    2. Criterion-Related Validity: Obtained by evaluating the relationship between test scores and other measures/tests.

    3. Construct Validity: Arrived at through a comprehensive analysis of how scores relate to other measures and how they fit into a theoretical framework.

  • Umbrella Validity: In the trinitarian view, construct validity is seen as the "umbrella" under which all other types of validity fall.

  • The Unitary View: Messick (1995) criticized the trinitarian model as fragmented and called for a unitary view that includes the societal implications and consequences of test use.

  • Ecological Validity: This refers to how well a test measures a variable (behavior, cognition, emotion) at the time and place it actually occurs in real life. It is closely related to Ecological Momentary Assessment (EMA), which evaluates variables in naturalistic contexts to avoid the pitfalls of retrospective self-report.

Face Validity

  • Definition: Face validity is a judgment of how relevant the test items appear to be on the surface to the testtaker.

  • Perspective: Unlike other validity types, face validity is often viewed from the perspective of the testtaker rather than the test user.

  • The "Rodney Dangerfield" of Psychometrics: Like the comedian known for "getting no respect," face validity is often dismissed by researchers but is crucial for “buy-in” and motivation.

  • Consequences of Low Face Validity:

    • Decreased cooperation or motivation from testtakers.

    • Unwillingness of managers to use an assessment (buy-in).

    • Objections from parents or legislators.

    • Potential lawsuits from disgruntled parties.

.

Criterion-Related Validity

  • Criterion Definition: A standard against which a test or score is evaluated. A criterion can be a test score, a behavior, a rating, or even an index like alcohol intoxication.

  • Characteristics of an Adequate Criterion:

    • Relevance: Pertinent to the matter at hand.

    • Validity: The criterion itself must be a valid measure.

    • Uncontaminated: Criterion contamination occurs when the criterion is based, in part, on predictor measures. For example, if a guard's opinion is used to create a violence potential score (the predictor) and the same guard's opinion is used to validate that score (the criterion), the results are meaningless.

  • Two Types of Criterion-Related Validity:

    1. Concurrent Validity: Test scores and criterion measures are obtained at the same time. This is used to estimate current standing. Example: Using the Beck Depression Inventory (BDI) to differentiate depressed vs. non-depressed adolescents.

    2. Predictive Validity: Test scores are obtained at one time and compared to criterion measures at a future time (e.g., SAT scores predicting freshman GPA).

Hit Rates and Prediction Errors

  • Base Rate: The proportion of a trait or behavior existing in the population.

  • Hit Rate: The proportion of people the test accurately identifies as possessing a trait.

  • Miss Rate: The proportion of people the test fails to identify accurately.

    • False Positive: The test predicts the person has the attribute when they do not.

    • False Negative: The test predicts the person does not have the attribute when they actually do.

Statistical Indices of Validity

  • Validity Coefficient: A correlation coefficient (usually Pearson rPearson\,r) describing the relationship between test scores and the criterion.

    • Affected by Restriction of Range: If the sample only includes highly qualified people (self-selection), the validity coefficient may be artificially low.

  • Incremental Validity: The degree to which an additional predictor explains something about the criterion that is not already explained by existing predictors.

    • Calculated using Hierarchical Regression.

    • Example: Emotional Intelligence (EI). While early claims were exaggerated, research suggests EI has modest incremental validity over traditional IQ and personality tests for academic and mental health outcomes.

Construct Validity Evidence

  • Construct Definition: An unobservable, scientific idea (e.g., anxiety, intelligence, bigotry) hypothesized to describe behavior.

  • Techniques for Evidence Gathering:

    • Homogeneity: How uniform a test is in measuring one concept. Increased by correlating subtest scores with total scores or item-analysis.

      • Marital Satisfaction Scale (MSS): Developers retained 48 items with correlation coefficients greater than 0.500.50 to ensure homogeneity.

    • Changes with Age: If a construct (like reading rate) grows with age, test scores must also increase with age for the test to be valid.

    • Pretest-Posttest Changes: Scores should change in predicted directions after an intervention (e.g., therapy or education).

    • Method of Contrasted Groups: Demonstrating that scores vary predictably between distinct groups (e.g., hospitalized depressed patients vs. non-depressed adults).

    • Convergent Evidence: High correlations with established tests measuring the same or similar constructs. (Example: .79.79 correlation between MSS and the Locke-Wallace Marital Adjustment Test).

    • Discriminant Evidence: Little to no (statistically insignificant) relationship between scores and variables with which the construct should not theoretically correlate. (Example: MSS not correlating with the Marlowe-Crowne Social Desirability Scale).

Structural Analysis: Multitrait-Multimethod Matrix and Factor Analysis

  • Multitrait-Multimethod Matrix: A table correlating different traits measured by different methods.

    • Method Variance: Correlation due to the same method being used rather than trait overlap.

  • Factor Analysis: Mathematical procedures to identify common factors or dimensions.

    • Exploratory Factor Analysis: Extracting and rotating factors.

    • Confirmatory Factor Analysis: Testing how well a hypothetical model fits actual data.

    • Factor Loading: A metaphor for how much a factor determined a specific score.

    • Naming Factors: This is a matter of expert judgment and verbal abstraction rather than mathematics.

Bias and Fairness in Assessment

  • Distinctions: Validity, bias, and fairness are separate issues.

  • Test Bias: A factor inherent in a test that systematically prevents impartial measurement.

    • Intercept Bias: Consistent underprediction or overprediction of a specific group's performance.

    • Slope Bias: A predictor has a weaker correlation with the outcome for one group.

  • Rating Errors:

    • Leniency (Generosity) Error: Rater is too easy.

    • Severity Error: Rater is too harsh.

    • Central Tendency Error: Rater avoids extremes, clustering ratings in the middle.

    • Halo Effect: Failure to discriminate between independent aspects of behavior due to an overall positive impression.

  • Techniques for Correction: Using Rankings instead of ratings can overcome restriction-of-range errors.

  • Test Fairness: Rooted in values. Defined as the impartial, just, and equitable use of a test.

    • Adverse Impact: When valid tests result in different group outcomes.

    • Section 106 of the Civil Rights Act of 1991: Prohibits employers from adjusting test scores (e.g., race-norming) on the basis of race, color, religion, sex, or national origin for hiring or promotion.

Psychometric Techniques for Affirmative Action and Remedying Adverse Impact

Technique

Description

Addition of Points

Adding a constant number of points to the scores of a specific group members to reduce group differences.

Differential Scoring

Scoring specific items for one group but not another (empirical keying).

Item Elimination

Removing items that favor one group over another based on Differential Item Functioning (DIF).

Differential Cutoffs

Setting lower passing thresholds for specific groups (e.g., 65 vs 70).

Separate Lists

Ranking group members on separate lists and alternating selections.

Within-Group Norming

Converting raw scores to percentiles based only on the testtaker's own group (also known as race-norming).

Banding

Treating all scores within a specific range (e.g., stanines 1-9) as equivalent.

Preference Policies

Choosing a member of a previously excluded group when scores are equal.