Lecture 11 Validity

Validity Overview

• Validity: Traditionally defined as the extent to which a test measures what it was designed to measure.

• A test can possess various types of validity depending on its intended use, target population, administration conditions, and methods for determining validity.

Determining Validity

• Validity can be assessed through:

  • Content Analysis: Examining the test content.

  • Correlation with Criteria: Comparing test scores against relevant performance criteria.

  • Construct Characteristics: Investigating the psychological constructs measured by the test.

• Consider evaluating incremental validity: It reflects how much additional predictive power a test offers compared to existing prediction methods.

Relationship Between Validity and Reliability

• A test may be reliable without being valid.

• A test cannot be valid without also being reliable.

Content Validity (I)

• Face Validity: The initial appearance of a test relative to its intended purpose is crucial for marketing.

• Content validity assesses whether the test yields responses representative of the entire skill range it aims to measure.

Content Validity (II)

• Achievement Tests: Content validity is critical in evaluating achievement tests, measures of aptitude, interests, and personalities.

• Evaluation involves analyzing the test composition to ensure alignment with instructional objectives.

Content Validity (III)

• Compare the test content to a table of specifications to confirm alignment with the subject matter.

• Expert agreement on the test's appearance and function indicates content validity, involving cognitive processes used when answering.

Criterion-Related Validity

• Definition: Compares test scores of a group with actual performance ratings or classifications.

• Types include:

  • Concurrent Validity

  • Predictive Validity

Criterion-Related Validity: Concurrent Validity

• Used when tests are administered across different population categories to see distinctions in scores.

• Example: MMPI used for identifying mental disorders through test score variations among diagnostic groups.

Criterion-Related Validity: Predictive Validity

• Focuses on how well test scores can forecast future performance, assessed through correlations with future performance indicators.

• Particularly relevant for aptitude and intelligence tests, correlating with grades and achievement scores.

Factors Affecting Criterion-Related Validity

• Incremental Validity: Evaluates the added accuracy of predictions made possible by a specific test versus other methods like interviews.

• Group Differences: Characteristics (e.g. age, sex) may sway test correlations.

• Test Length: Generally, longer tests yield higher predictive validities compared to shorter ones.

• Criterion Contamination: Validity impacted by the measurement methods of the criterion itself, which can introduce errors or biases.

Construct Validity (I)

• Refers to how well an instrument measures a specific psychological construct (e.g., anxiety, extroversion).

• Involves thorough investigations to confirm whether a test measures the intended personality variable.

Construct Validity (II): Evidence

• Establishing construct validity utilizes:

  • Expert judgments on content pertains to the construct.

  • Internal consistency analyses.

  • Studies examining relationships between test scores and other variables on which the groups differ.

  • Correlations with scores from other tests expected to relate closely.

  • Questioning examinees on their thought processes during responses.

Construct Validity (II): Convergent and Discriminant Validation

• A well-validated construct instrument displays:

  • High correlations with other measures that capture the same construct (convergent validity).

  • Low correlations with measures of different constructs (discriminant validity).

• Typically assessed through a matrix of methods including same constructs using different methods and vice versa.