ch 4 kaplan

Measurement of Ability and Reliability

  • In human ability assessments, discrepancies often arise between measured and true abilities.

    • Example: A gymnast scores 8.9 but is capable of 9.5.

    • Common in various fields: Students feel tests don't reflect true knowledge or abilities.

  • Measurement errors refer to inaccuracies in ability assessments.

    • In psychological testing, "error" implies an inevitable inaccuracy not necessarily a mistake.

History and Theory of Reliability

Conceptualization of Measurement Error

  • Physical sciences allow precise measurements (e.g., width of an object).

  • In psychology, measurements often involve complex traits (e.g., intelligence).

    • Lack rigid yardsticks results in using adaptable measures.

    • Comparisons made to carpenters using unreliable measuring tapes.

  • Measurement error is prevalent across sciences (e.g., economics, medicine), but concern for reliability is strong in psychology.

Spearman's Early Contributions

  • Charles Spearman advanced reliability assessment in 1904.

    • Combined ideas of sampling error and correlation coefficients.

  • Historical developments include:

    • Abraham De Moivre (1733) - foundational notion of sampling error.

    • Karl Pearson (1896) - product moment correlation.

    • Kuder and Richardson's introduction of reliability coefficients (1937) and subsequent refinements by Cronbach and associates.

  • Contemporary reliance on mathematical models for latent variables based on multiple measures.

Classical Test Theory

True Score and Error Concept

  • According to classical test theory:

    • Each individual possesses a true score for an ability or trait.

    • Observed score (X) comprises true score (T) plus error (E):
      X=T+EX = T + E

  • Errors are assumed to be random in classical test theory.

Distribution of Scores

  • The distribution of observed scores due to measurement error resembles a bell curve, centering around the true score.

  • Figure 4.1 illustrates this distribution.

  • The standard deviation of errors indicates the extent of measurement error, termed the standard error of measurement (SEM):
    SEM=SDimesext√(1−r)SEM = SD imes ext{√}(1 - r)

Domain Sampling Model

  • Represents constructs through limited items rather than complete assessments (e.g., spelling ability).

  • As the sample size increases, it yields a higher reliability estimate and reduces error.

Item Response Theory (IRT)

  • A newer assessment tool that shifts from classical test theory due to its methodology co-dependent on item difficulty.

  • Aims to efficiently gauge an individual’s ability level through sequential item adjustments based on responses.

  • Relies on a bank of pre-evaluated items which raises tests' precision.

Models of Reliability

Reliability Coefficient Definition

  • Defined as the ratio between the variance of true scores to the variance of observed scores:
    r=σ2<em>Tσ2</em>Xr = \frac{\sigma^2<em>{T}}{\sigma^2</em>{X}}

  • Reflects the percentage variation attributable to true score variance.

Categories of Test Reliability Methods

  • Test-retest involves consistency over time.

  • Parallel forms compare different forms measuring the same attribute.

  • Internal consistency evaluates performance across similar items from the same test.

    • Common methods include:

    • Split-half reliability (comparing scores from two halves of the same test).

    • KR20 and Coefficient Alpha for dichotomous and non-dichotomous scored tests.

Sources of Error and Their Assessments

Situational Factors Affecting Reliability

  • Errors may arise from testing conditions (e.g., room temperatures, participant health).

  • Methods of assessing reliability based on time sampling (test-retest), item sampling (parallel forms), and internal consistency are critical.

Test-Retest Reliability

  • Evaluates consistency over repeated tests, suitable for stable traits like IQ.

  • Influences include carryover effects, where past test results affect later ones, and practice effects, where improvement is due to familiarity rather than true change.

  • Validity decreases for measures affected significantly by mood or environment variations.

Alternate Forms Method

  • Uses different test formats to assess reliability through performance comparison.

  • Ideal for measuring with two or more forms with a consistent structure.

Split-Half Method

  • Divides a single test into halves, correlating scores to estimate reliability, corrected via the Spearman-Brown formula:
    rcorrected=2×r1+rr_{corrected} = \frac{2 \times r}{1 + r}

Coefficient Alpha (Cronbach's Alpha)

  • General measure of internal consistency, applicable to all item formats, crucial in tests lacking dichotomous response systems.

  • Formula:
    α=N⋅σ2<em>X∑σ2+(N−1)⋅σ2</em>y\alpha = \frac{N \cdot \sigma^2<em>{X}}{\sum \sigma^2 + (N - 1) \cdot \sigma^2</em>{y}}

Reliability of Difference Scores

  • Formed by subtracting two test scores, the reliability of difference scores is less than that of individual test scores due to added error:
    ReliabilityofDifference=(r<em>1+r</em>2)−2r<em>121−r</em>12Reliability of Difference = \frac{(r<em>{1} + r</em>{2}) - 2r<em>{12}}{1 - r</em>{12}}

Behavioral Observation and Reliability

Observational Reliability Metrics

  • Reliability tied to inter-rater consistency or agreements across different observers measuring the same behavior.

  • Kappa statistic as a correction method to account for chance agreement in subjective measures, recognizing agreement ranges from poor (<0.40) to excellent (>0.75).

Practical Application of Reliability Assessment

Standards and Guidelines

  • American Psychological Association has established guidelines encouraging clear reporting of reliability metrics (e.g., confidence intervals, estimates for subscales) for robust reliability assessments.

Using Reliability Data

  • Reliability’s importance varies by application; clinical settings necessitate higher reliability than research contexts.

  • The standard error of measurement serves as a practical utility for establishing confidence in observed scores:
    SEM=SD1−rSEM = SD \sqrt{1-r}

  • Decision structures guide the balance between reliability improvement costs and necessary outcomes, influencing the eventual test deployment.