Study Notes on Content and Criterion Validity
Validity Foundations and Definitions
- Previous Definition: The degree to which a method accurately measures what it is intended to measure.
- Current Definition: The degree to which evidence and theory support the interpretation of test scores entailed by proposed uses of tests.
- Operational Basis: Validity is a property of test score interpretations, uses, and resulting inferences, rather than a static property of the test itself.
- Rationale for Redefinition: The traditional definition enabled overly broad application; generalized usage can produce misleading conclusions when tests are applied outside their intended context.

Sources of Validity Evidence
- Content Validity: Evidence demonstrating that test content adequately samples and represents the construct without over- or under-representation. Most effectively evaluated with concrete attributes (observable, measurable behaviors).
- Response Processes Evidence: Evidence concerning the cognitive mechanisms engaged by test takers; evaluated by observing or interviewing test takers or observers.
- Construct Validity: Evidence examining internal test structure to verify that unintended constructs are not being captured. Best suited for abstract attributes.
- Criterion-Related Validity: Evidence evaluating correlations with external variables; related tests should demonstrate correlation, whereas unrelated tests should not. Well-suited for abstract attributes and outcome prediction.
- Consequences of Testing Evidence: Examination of intended and unintended social or operational outcomes, assessing whether test applications produce biased or inaccurate consequences.
Developing and Evaluating Content Validity
- Construct: Any concept or characteristic that an assessment is engineered to measure.
- Development Phase (Qualitative Evidence / Non-Numerical):
- Testing Universe: The comprehensive body of knowledge, skills, or behaviors represented by an assessment.
- Test Specifications: A documented plan detailing the assessment's operational design and designated content areas (the specific subject matter to be assessed).
- Test Format Selection: Deciding between a written test (paper-and-pencil series of questions) and a practical test (active skill demonstration in applied settings).
- Item Generation: Formulated using job analysis (identifying the knowledge, skills, abilities, and other characteristics [KSAOs] required for a job) or competency modeling (identifying organization-wide KSAOs critical for success; generally less scientifically rigorous). Serves as a standard defense against employment-related legal challenges.
- Post-Development Evaluation (Quantitative Evidence / Numerical):
- Content Validity Ratio (CVR): Quantifies panel agreement among subject matter experts rating item essentialness:
CVR=2Nne−2N
where ne represents the number of experts rating an item as essential and N is the total number of evaluating experts.
- Face Validity: The degree to which test items appear relevant to the testing purpose from the perspective of the test taker; functions similarly to social validity and can impact test-taker performance.
- Criterion: An external standard or performance measure correlated with predictor test scores.
- Predictive Method:
- Administer the predictor test to a large sample.
- Withhold and conceal test results without making operational decisions based on them.
- Measure criterion performance across the same cohort following a predetermined time interval.
- Strong relationships between predictor and criterion establish predictive validity evidence.
- Evaluated using a validity coefficient (rxy).
- Restriction of Range: When high test performance is required for selection along with other valid screening tools, lower-scoring individuals are excluded from hiring. This artificially restricts score variance, attenuating the observed correlation.
- Concurrent Method:
- Collect predictor test scores and criterion performance metrics simultaneously or within close temporal proximity.
- Functions as a practical, lower-cost alternative to predictive studies by evaluating current status-quo performance.
Criterion Selection, Evaluation, and Unreliability
- Criterion Classifications:
- Objective Criteria: Observable, directly quantifiable metrics.
- Subjective Criteria: Judgment-based evaluations (e.g., supervisor or peer ratings). Subjective criteria can be preferable when capturing multifaceted behavioral dimensions.
- Criterion Contamination: A threat to validity occurring when a criterion measure captures extraneous dimensions beyond what the predictor test is designed to assess.
- Validity Coefficient (rxy) Evaluation:
- Tests of Significance: Determine if a test-criterion relationship is statistically non-zero while adjusting for group size and sampling error.
- Coefficient of Determination (r2): The proportion of variance shared between the predictor test and criterion measure, calculated by squaring rxy.
- Confidence Limitations: Validity coefficients are subject to chance, sampling error, and situation-specific factors, necessitating ongoing empirical evaluation.
- Attenuation Due to Unreliability: Random measurement error within the criterion, compounded by measurement error in the predictor, depresses (attenuates) the observed validity coefficient below its true theoretical value.
Statistical Prediction Models
- Linear Regression: Statistical model using one test score (X) to predict a single criterion outcome (Y′):
Y′=a+bX
where a is the y-axis intercept and b is the regression slope (b weight), representing the expected change in Y′ for each one-unit change in X.
- Multiple Regression: Statistical procedure using multiple test scores (X1,X2,X3,…,Xn) to predict a criterion (Y′):
Y′=a+b1X1+b2X2+b3X3+⋯+bnXn
- Coefficient of Multiple Determination: Statistic evaluating the total variance accounted for by the combined predictor battery.
- Multicollinearity: Statistical overlap and redundancy occurring when predictor variables are intercorrelated.
Questions & Discussion
- Reliability vs. Validity Operational Independence: Reliability reflects the internal consistency and precision of the test instrument itself, whereas validity is not a static property of the test but rather depends on the inferences, interpretations, and applications drawn from the scores in specific contexts.
- Value of the Five-Source Model: Moving beyond the classical tripartite model (content, criterion-related, construct) to five distinct sources provides a more comprehensive, evidentiary framework for defending test usage and examining cognitive processes and testing consequences.
- Role of the Testing Universe: The testing universe defines the complete scope of knowledge or behaviors under evaluation; content validity requires demonstrating that test items form an unbiased, representative sample of that universe.
- Mitigating Restriction of Range: Restriction of range is minimized when an organization does not use the test or other correlated selection hurdles to truncate the applicant pool, preserving the full spectrum of predictor and criterion scores.
- Impact of Unreliability on Validity: Unreliability introduces random error into the test scores, the criterion, or both; this measurement error attenuates the validity coefficient, placing a statistical ceiling on empirical test-criterion correlations.
- Linear Regression vs. Multiple Regression: Both models predict an outcome criterion using weighted predictor scores; linear regression utilizes a single predictor variable (X), whereas multiple regression integrates multiple predictor variables (X1,X2,…,Xn) and must account for multicollinearity.
Administrative Dates & Requirements
- Test 1: Scheduled for October 17 (administered at home).
- Critical Review Paper: Due on October 20.
- Open-Ended Assignments: Continue compiling responses across instructional units.
- Assessment Presentations: At least one multiple-choice examination item will be directly derived from group assessment presentations.