Chapter 5: Reliability and Validity

Introduction to Reliability and Validity

Definitions
  • Reliability: Refers to the consistency of a test or measure. It indicates that if a test is taken multiple times by the same person, similar results should be obtained each time.

    • Analogy: Comparable to a bathroom scale that gives the same reading every time you step on it, assuming no changes in weight.

  • Validity: Pertains to the accuracy of a test. It assesses whether the test measures what it is intended to measure.

    • Analogy: Similar to a bathroom scale that may be consistently reading 10 pounds too heavy; it is reliable but not valid.

Importance of Reliability and Validity
  • Both reliability and validity are crucial for evaluating the quality of a test or measure.

  • A test can be reliable without being valid, although it is unlikely to be valid if it is not reliable. This is because consistency of results is a prerequisite for accuracy.

Reliability

What is Reliability and Why Does It Matter?
  • Definition: Reliability refers to the repeatability of a measure. If a measurement taken at different times yields similar results, it is considered reliable.

  • Characteristics of Reliable Measures: Less error and more consistency in results.

    • Example: A personality test that yields drastically different results on different days is unreliable.

How to Determine Reliability
  • Testing Methods: Various approaches can be employed to assess the reliability of a measure.

Test-Retest Reliability
  • Definition: Evaluates consistency over time by re-administering the same test to the same individuals after a specific interval.

  • Key Considerations:

    • Stable Construct: The trait being measured should not change significantly in a short time frame.

    • Replicable Conditions: The testing environment should be similar across both testing occasions.

    • Minimal Practice Effects: Previous exposure to the test items should not bias the results.

  • Example 1: Hand-grip strength test conducted two weeks apart shows consistent results.

  • Example 2: A numeric-keypad test yielding similar scores one week apart demonstrates reliability.

When Not to Use Test-Retest Reliability
  • Considerations: Avoid if expected change in behavior is anticipated, if there are strong practice or memory effects, or if there are unmanageable external context shifts.

  • Example: A quiz on probability reused after four weeks of relevant instruction; significant score changes indicate growth rather than reliability issues.

Alternate-Forms (Parallel-Forms) Reliability
  • Definition: Assesses whether different versions of a measure yield comparable results.

  • Procedure: Develop two forms of a test that are similar in content and difficulty, administer them to the same set of individuals, and compare the scores.

  • When to Use: Valuable for longitudinal studies or situations where retaking an identical test could lead to memorization effects.

  • Example: A professor gives two different exam forms with parallel questions, and scores correlate well, indicating strong reliability.

Split-Half Reliability
  • Definition: An internal consistency estimate derived by splitting a single test into two halves, then comparing the scores of both halves.

  • Application: Useful when there is only one opportunity to assess the measure.

  • Example: An employer administering a 100-item measure finds high correlation between scores of odd and even numbered items, indicating good split-half reliability.

Internal Consistency (Cronbach's Alpha)
  • Definition: Evaluates how closely related a set of items are as a group. It checks if items measuring the same construct yield similar scores.

  • Analysis: Higher values (0-1 scale) indicate stronger consistency among items. Generally, values above 0.70 are considered acceptable.

Inter-Rater Reliability
  • Definition: Assesses consistency in ratings made by different judges based on the same performance or criteria.

  • Example: Two instructors score essays independently, demonstrating high inter-rater reliability when their scores align closely.

Error Considerations in Testing

Sources of Measurement Error
  1. Candidate-Related Error: Factors intrinsic to the test-taker that cause score variability.

    • Examples:

      • Fatigue affecting performance (e.g., all-nighter prior to a test).

      • Caffeine boosting scores temporarily.

  2. Test-Related Error: Issues with the test itself that lead to inaccuracies.

    • Examples: Ambiguous questions or unclear print outs.

  3. Procedural & Environmental Error: External factors relating to testing conditions affecting performance.

    • Examples: Distractions from the environment, incorrect room temperature affecting concentration.

Types of Error:
  • Systematic Error: Consistent bias affecting scores in a predictable direction.

  • Random Error: Unpredictable deviations creating variability in scores.

Validity

Definitions and Importance of Validity
  • Validity: The degree to which a test measures what it claims to measure. Validity impacts the effectiveness of decisions made based on test scores.

  • Real-World Implications: Validity failure can have severe consequences in practical applications such as employment or clinical assessments.

Types of Validity to Evaluate
Content Validity
  • Definition: Ensures the measure includes all relevant domains of the construct it claims to assess.

  • Example: Examination content matches course objectives proportionately.

Criterion Validity
  • Definition: Measures how closely the test scores correlate with an external criterion representing what the test is designed to predict (can be predictive or concurrent).

  • Example: A driving simulator test correlating with accident rates demonstrates high criterion validity.

Construct Validity
  • Definition: Establishes whether the test accurately represents the construct it is intended to assess.

  • Evidence Gathering: Involves showing strong correlations with other measures of the same construct.

  • Example: Evidence suggests that a simulator accurately measures decision-making under threat, validated through various measures and correlation evidence.

Other Validity Considerations
  • Internal Validity: Reflects the degree to which the test accurately captures the causal relationships within the study.

  • External Validity: Assesses whether findings can be generalized from one population to another.

  • Face Validity: This is based on the subjective judgment of observers regarding whether the assessment seems appropriate for the intended purpose.

Applied Examples

Example 1: Personality Test for Student Jobs
  • Reliability: Consistent results across test administrations and forms.

  • Error: Issues with ambiguous questions and environmental disruptions.

  • Validity: Broad coverage of relevant traits and predictors of success in job performance.

Example 2: 360-Degree Review Process
  • Reliability: Consistent inter-rater and test-retest reliability across evaluations.

  • Error: Considerations of candidate state prior to evaluation and rater understanding of the evaluation scale.

  • Validity: Assessment based on job analysis demonstrating relevant content and predictive validity.

Conclusion
  • Evaluating a measure's reliability and validity is essential to ensure accurate assessments and effective decision-making. Without both, data can lead to misleading conclusions.