ch 4 kaplan
Measurement of Ability and Reliability
In human ability assessments, discrepancies often arise between measured and true abilities.
Example: A gymnast scores 8.9 but is capable of 9.5.
Common in various fields: Students feel tests don't reflect true knowledge or abilities.
Measurement errors refer to inaccuracies in ability assessments.
In psychological testing, "error" implies an inevitable inaccuracy not necessarily a mistake.
History and Theory of Reliability
Conceptualization of Measurement Error
Physical sciences allow precise measurements (e.g., width of an object).
In psychology, measurements often involve complex traits (e.g., intelligence).
Lack rigid yardsticks results in using adaptable measures.
Comparisons made to carpenters using unreliable measuring tapes.
Measurement error is prevalent across sciences (e.g., economics, medicine), but concern for reliability is strong in psychology.
Spearman's Early Contributions
Charles Spearman advanced reliability assessment in 1904.
Combined ideas of sampling error and correlation coefficients.
Historical developments include:
Abraham De Moivre (1733) - foundational notion of sampling error.
Karl Pearson (1896) - product moment correlation.
Kuder and Richardson's introduction of reliability coefficients (1937) and subsequent refinements by Cronbach and associates.
Contemporary reliance on mathematical models for latent variables based on multiple measures.
Classical Test Theory
True Score and Error Concept
According to classical test theory:
Each individual possesses a true score for an ability or trait.
Observed score (X) comprises true score (T) plus error (E):
Errors are assumed to be random in classical test theory.
Distribution of Scores
The distribution of observed scores due to measurement error resembles a bell curve, centering around the true score.
Figure 4.1 illustrates this distribution.
The standard deviation of errors indicates the extent of measurement error, termed the standard error of measurement (SEM):
Domain Sampling Model
Represents constructs through limited items rather than complete assessments (e.g., spelling ability).
As the sample size increases, it yields a higher reliability estimate and reduces error.
Item Response Theory (IRT)
A newer assessment tool that shifts from classical test theory due to its methodology co-dependent on item difficulty.
Aims to efficiently gauge an individual’s ability level through sequential item adjustments based on responses.
Relies on a bank of pre-evaluated items which raises tests' precision.
Models of Reliability
Reliability Coefficient Definition
Defined as the ratio between the variance of true scores to the variance of observed scores:
Reflects the percentage variation attributable to true score variance.
Categories of Test Reliability Methods
Test-retest involves consistency over time.
Parallel forms compare different forms measuring the same attribute.
Internal consistency evaluates performance across similar items from the same test.
Common methods include:
Split-half reliability (comparing scores from two halves of the same test).
KR20 and Coefficient Alpha for dichotomous and non-dichotomous scored tests.
Sources of Error and Their Assessments
Situational Factors Affecting Reliability
Errors may arise from testing conditions (e.g., room temperatures, participant health).
Methods of assessing reliability based on time sampling (test-retest), item sampling (parallel forms), and internal consistency are critical.
Test-Retest Reliability
Evaluates consistency over repeated tests, suitable for stable traits like IQ.
Influences include carryover effects, where past test results affect later ones, and practice effects, where improvement is due to familiarity rather than true change.
Validity decreases for measures affected significantly by mood or environment variations.
Alternate Forms Method
Uses different test formats to assess reliability through performance comparison.
Ideal for measuring with two or more forms with a consistent structure.
Split-Half Method
Divides a single test into halves, correlating scores to estimate reliability, corrected via the Spearman-Brown formula:
Coefficient Alpha (Cronbach's Alpha)
General measure of internal consistency, applicable to all item formats, crucial in tests lacking dichotomous response systems.
Formula:
Reliability of Difference Scores
Formed by subtracting two test scores, the reliability of difference scores is less than that of individual test scores due to added error:
Behavioral Observation and Reliability
Observational Reliability Metrics
Reliability tied to inter-rater consistency or agreements across different observers measuring the same behavior.
Kappa statistic as a correction method to account for chance agreement in subjective measures, recognizing agreement ranges from poor (<0.40) to excellent (>0.75).
Practical Application of Reliability Assessment
Standards and Guidelines
American Psychological Association has established guidelines encouraging clear reporting of reliability metrics (e.g., confidence intervals, estimates for subscales) for robust reliability assessments.
Using Reliability Data
Reliability’s importance varies by application; clinical settings necessitate higher reliability than research contexts.
The standard error of measurement serves as a practical utility for establishing confidence in observed scores:
Decision structures guide the balance between reliability improvement costs and necessary outcomes, influencing the eventual test deployment.