Chapter 5: Reliability and Validity
Introduction to Reliability and Validity
Definitions
Reliability: Refers to the consistency of a test or measure. It indicates that if a test is taken multiple times by the same person, similar results should be obtained each time.
Analogy: Comparable to a bathroom scale that gives the same reading every time you step on it, assuming no changes in weight.
Validity: Pertains to the accuracy of a test. It assesses whether the test measures what it is intended to measure.
Analogy: Similar to a bathroom scale that may be consistently reading 10 pounds too heavy; it is reliable but not valid.
Importance of Reliability and Validity
Both reliability and validity are crucial for evaluating the quality of a test or measure.
A test can be reliable without being valid, although it is unlikely to be valid if it is not reliable. This is because consistency of results is a prerequisite for accuracy.
Reliability
What is Reliability and Why Does It Matter?
Definition: Reliability refers to the repeatability of a measure. If a measurement taken at different times yields similar results, it is considered reliable.
Characteristics of Reliable Measures: Less error and more consistency in results.
Example: A personality test that yields drastically different results on different days is unreliable.
How to Determine Reliability
Testing Methods: Various approaches can be employed to assess the reliability of a measure.
Test-Retest Reliability
Definition: Evaluates consistency over time by re-administering the same test to the same individuals after a specific interval.
Key Considerations:
Stable Construct: The trait being measured should not change significantly in a short time frame.
Replicable Conditions: The testing environment should be similar across both testing occasions.
Minimal Practice Effects: Previous exposure to the test items should not bias the results.
Example 1: Hand-grip strength test conducted two weeks apart shows consistent results.
Example 2: A numeric-keypad test yielding similar scores one week apart demonstrates reliability.
When Not to Use Test-Retest Reliability
Considerations: Avoid if expected change in behavior is anticipated, if there are strong practice or memory effects, or if there are unmanageable external context shifts.
Example: A quiz on probability reused after four weeks of relevant instruction; significant score changes indicate growth rather than reliability issues.
Alternate-Forms (Parallel-Forms) Reliability
Definition: Assesses whether different versions of a measure yield comparable results.
Procedure: Develop two forms of a test that are similar in content and difficulty, administer them to the same set of individuals, and compare the scores.
When to Use: Valuable for longitudinal studies or situations where retaking an identical test could lead to memorization effects.
Example: A professor gives two different exam forms with parallel questions, and scores correlate well, indicating strong reliability.
Split-Half Reliability
Definition: An internal consistency estimate derived by splitting a single test into two halves, then comparing the scores of both halves.
Application: Useful when there is only one opportunity to assess the measure.
Example: An employer administering a 100-item measure finds high correlation between scores of odd and even numbered items, indicating good split-half reliability.
Internal Consistency (Cronbach's Alpha)
Definition: Evaluates how closely related a set of items are as a group. It checks if items measuring the same construct yield similar scores.
Analysis: Higher values (0-1 scale) indicate stronger consistency among items. Generally, values above 0.70 are considered acceptable.
Inter-Rater Reliability
Definition: Assesses consistency in ratings made by different judges based on the same performance or criteria.
Example: Two instructors score essays independently, demonstrating high inter-rater reliability when their scores align closely.
Error Considerations in Testing
Sources of Measurement Error
Candidate-Related Error: Factors intrinsic to the test-taker that cause score variability.
Examples:
Fatigue affecting performance (e.g., all-nighter prior to a test).
Caffeine boosting scores temporarily.
Test-Related Error: Issues with the test itself that lead to inaccuracies.
Examples: Ambiguous questions or unclear print outs.
Procedural & Environmental Error: External factors relating to testing conditions affecting performance.
Examples: Distractions from the environment, incorrect room temperature affecting concentration.
Types of Error:
Systematic Error: Consistent bias affecting scores in a predictable direction.
Random Error: Unpredictable deviations creating variability in scores.
Validity
Definitions and Importance of Validity
Validity: The degree to which a test measures what it claims to measure. Validity impacts the effectiveness of decisions made based on test scores.
Real-World Implications: Validity failure can have severe consequences in practical applications such as employment or clinical assessments.
Types of Validity to Evaluate
Content Validity
Definition: Ensures the measure includes all relevant domains of the construct it claims to assess.
Example: Examination content matches course objectives proportionately.
Criterion Validity
Definition: Measures how closely the test scores correlate with an external criterion representing what the test is designed to predict (can be predictive or concurrent).
Example: A driving simulator test correlating with accident rates demonstrates high criterion validity.
Construct Validity
Definition: Establishes whether the test accurately represents the construct it is intended to assess.
Evidence Gathering: Involves showing strong correlations with other measures of the same construct.
Example: Evidence suggests that a simulator accurately measures decision-making under threat, validated through various measures and correlation evidence.
Other Validity Considerations
Internal Validity: Reflects the degree to which the test accurately captures the causal relationships within the study.
External Validity: Assesses whether findings can be generalized from one population to another.
Face Validity: This is based on the subjective judgment of observers regarding whether the assessment seems appropriate for the intended purpose.
Applied Examples
Example 1: Personality Test for Student Jobs
Reliability: Consistent results across test administrations and forms.
Error: Issues with ambiguous questions and environmental disruptions.
Validity: Broad coverage of relevant traits and predictors of success in job performance.
Example 2: 360-Degree Review Process
Reliability: Consistent inter-rater and test-retest reliability across evaluations.
Error: Considerations of candidate state prior to evaluation and rater understanding of the evaluation scale.
Validity: Assessment based on job analysis demonstrating relevant content and predictive validity.
Conclusion
Evaluating a measure's reliability and validity is essential to ensure accurate assessments and effective decision-making. Without both, data can lead to misleading conclusions.