Psychometrics and Reliability Concepts

Test Length and Reliability Dynamics

  • Sample Size and Reliability Principles:

    • Shorter tests are inherently less reliable than longer tests.

    • Conceptual Analogy: Evaluating student opinions on campus housing policy at ECSU (such as whether first-year students receive priority on dorm selection) by surveying 22 students yields low confidence, whereas surveying 100100 students provides a larger, more representative sample and higher statistical confidence.

    • Because an infinite pool of potential questions exists for any given construct, a 22-item test provides a far less reliable measurement than a 100100-item test.

  • Splitting Tests and Reliability Underestimation:

    • Splitting a test in half reduces the effective length of the assessment being evaluated, leading to an underestimation of the test's full-length reliability when calculating correlation between the two halves.

    • To adjust for this reduction in test length, the Spearman-Brown Prophecy Formula is applied to estimate full-test reliability or determine necessary length adjustments to reach a desired reliability coefficient.

  • Spearman-Brown Prophecy Formula Applications:

    • Calculates the required multiplication factor for test length to achieve a target reliability coefficient (rxxr_{xx}).

    • Multiplication Factor Example: If a test currently has a reliability of 0.500.50 and the desired target reliability is 0.800.80, the test length must be multiplied by 44:

      • A 5050-item test must be expanded to 200200 items.

      • A 44-item test must be expanded to 1616 items.

    • In split-half reliability applications, the formula treats the half-test correlation as if the test length were doubled back to its original full size.

  • Limitations of Split-Half Reliability:

    • Requires selecting a single specific split method (e.g., comparing the first 1010 items against the last 1010 items, or comparing odd-numbered items against even-numbered items).

    • Different split configurations produce different correlation coefficients, with no objective statistical basis for determining which split yields the single "correct" reliability value.

Advanced Internal Consistency Estimators

  • Kuder-Richardson Formula 20 (KR-20):

    • Published in 1937 by G. Frederic Kuder and M. W. Richardson as Formula #20 in a psychometric paper outlining multiple formulas.

    • Evaluates internal consistency by calculating the split-half reliability across every possible split combination of a test and taking the average of all resulting split-half coefficients.

    • Specifically designated for tests composed of dichotomous items (items scored with definitive right or wrong / correct or incorrect answers).

  • Cronbach's Coefficient Alpha:

    • Developed by Lee Cronbach (frequently referred to simply as Coefficient Alpha).

    • Extends the mathematical logic of KR-20 to continuous and non-dichotomous items (e.g., Likert response scales without single right or wrong answers).

    • Calculates the average reliability across all possible split-half combinations and inherently incorporates the Spearman-Brown correction into its mathematical formulation.

  • Empirical Demonstration: Heatwave Behavioral Adaptation Study:

    • Study Context: Survey evaluating how individuals perceive heatwave threats and adapt their behavior during a week-long heat wave scenario.

    • Item Pool: Participants indicated agreement on statements covering behavioral adaptations:

      • Having trouble sleeping.

      • Spending less time outdoors.

      • Increasing water consumption.

      • Spending more time at home.

      • Reducing scheduled time with other people.

      • Eating more junk food as a coping mechanism.

      • Monitoring energy/electricity bills.

      • Reducing outdoor physical activity.

      • Increasing indoor physical activity to compensate for reduced outdoor activity.

      • Binge-watching media.

      • Spending maximum time in air-conditioned environments.

      • Minimizing effort to avoid heat generation.

    • Factor Analysis: Statistical technique applied to examine item interrelationships and identify underlying scale dimensions, identifying 33 potential scales.

Scale Reliability Benchmarks and Evaluation

  • Standard Score Review:

    • Z-Score: Standard score defined by a mean of 00 and a standard deviation of 11 (μ=0,σ=1\mu = 0, \sigma = 1).

    • T-Score: Capital TT standard score defined by a mean of 5050 and a standard deviation of 1010 (μ=50,σ=10\mu = 50, \sigma = 10). Distinct from the tt-statistic used in hypothesis testing.

  • Reliability Thresholds:

    • Research Applications: A reliability coefficient (α\alpha or rxxr_{xx}) of 0.700.70 or higher is deemed acceptable.

    • High-Stakes / Decision-Making Applications: Hiring, selection, or high-stakes diagnostic decisions require a reliability coefficient of 0.900.90 or higher.

  • Empirical Scale Evaluation Results:

    • Scale 1 (Sedentary / Isolation Behaviors):

      • Items: Trouble sleeping, spending less time with others, eating junk food, reducing physical activity, sedentary behaviors, doing as little as possible, social media usage.

      • Cronbach's Alpha: α=0.8166667\alpha = 0.8166667

      • Assessment: Exceeds the 0.700.70 threshold; acceptable for research purposes.

    • Scale 2 (Indoor Containment Behaviors):

      • Items (44 items): Less time outdoors, drinking more water, spending more time at home, spending more time in air conditioning.

      • Cronbach's Alpha: α=0.76\alpha = 0.76

      • Assessment: Exceeds the 0.700.70 threshold; acceptable for research purposes.

    • Scale 3 (Energy Management and Outdoor Timing):

      • Items (44 items): Reducing energy use, increasing indoor physical activity, going outdoors when possible, going outdoors early/late.

      • Cronbach's Alpha: α=0.56\alpha = 0.56

      • Assessment: Unacceptable reliability (<0.70< 0.70). The items fail to cluster as a cohesive scale and must be analyzed as 44 separate individual items. Future scale construction requires increasing item length to achieve acceptable reliability.

Taxonomy of Reliability Estimation Methods

  • Internal Consistency:

    • Most widely utilized method for assessing reliability in psychological and educational testing.

    • Primary Advantage: Requires only a single test administration.

    • Comparison to Alternative Methods:

      • Test-Retest Reliability: Requires administering the same test to the same cohort (e.g., over 200200 participants) across two time points separated by weeks or months (e.g., 66 weeks or 66 months). Suffers from participant attrition and logistical barriers.

      • Alternate Forms Reliability: Requires constructing two distinct, equivalent forms with different item wording, doubling item development workload.

  • Measurement Scale Categorization:

    • Behavioral sciences classify Likert-type response scales as Interval scales based on the assumption of equal intervals between response points (e.g., the distance between "disagree" and "slightly agree" equals the distance between "disagree" and "strongly disagree"), despite lacking an absolute zero.

    • (Note: Pure mathematics discipline conventions classify Likert items as Ordinal scales).

  • Inter-Rater (Inter-Score) Reliability:

    • Applied when assessment scoring depends on human judgment or subjective evaluation (e.g., grading essays or evaluating presentations via a scoring rubric).

    • Objective: Ensures independent evaluators produce consistent, concordant scores rather than subjective personal judgments.

    • Procedure: Two or more independent raters score the same sample of performances, and the correlation between rater scores is calculated.

    • Target Correlation: High positive correlation in the 0.800.80 to 0.900.90 range.

Classical Test Theory and Standard Error of Measurement

  • Classical Test Theory Model:

    • Observed Score=True Score+Error\text{Observed Score} = \text{True Score} + \text{Error}

    • X=T+EX = T + E

    • Where XX is the observed score, TT is the true score, and EE is measurement error.

    • Reliability estimation quantifies the proportion of variance attributable to error (EE), allowing for inferences regarding the True Score (TT).

  • Standard Error of Measurement (SEM):

    • Quantifies the expected variation of observed scores around a true score.

    • Formula:         SEM=σ1−rxxSEM = \sigma \sqrt{1 - r_{xx}}         Where σ\sigma is the standard deviation of test scores from a single administration, and rxxr_{xx} is the test reliability coefficient.

  • Confidence Intervals for Observed Scores:

    • In a normal distribution, approximately 95%95\% of scores fall within ±1.96\pm 1.96 standard deviations of the mean (z=1.96z = 1.96, often approximated as 22).

    • 95%95\% Confidence Interval Formula:         CI95%=Observed Score±1.96×SEM\text{CI}_{95\%} = \text{Observed Score} \pm 1.96 \times SEM

    • Interpretation Example: An observed score evaluated with a calculated SEM yielding an interval calculation from 8181 to 9898 establishes 95%95\% confidence that the true score lies within that span.

Educational and High-Stakes Decision Policy

  • Flaws in Rigid Cut-Score Applications:

    • High-stakes testing environments often set arbitrary cut-scores (e.g., a required threshold score of 110110 to pass or avoid grade repetition).

    • Treating an observed score of 109109 as a strict, error-free absolute score falsely equates the observed score (XX) directly to the true score (TT).

  • Confidence Interval Application to Borderline Scores:

    • Assuming an observed score of 109109 with an estimated SEM yielding a confidence interval margin of ±3\pm 3 points:         CI95%=109±3  ⟹  [106,112]\text{CI}_{95\%} = 109 \pm 3 \implies [106, 112]

    • Because the 95%95\% confidence interval (106106 to 112112) includes scores at and above the passing cut-score of 110110, there is a statistical probability that the student's true score meets or exceeds the passing standard.

  • Policy Implications in Educational Assessment:

    • To account for measurement error near decision boundaries, public school systems (e.g., Virginia state educational policies) have established re-testing provisions for students scoring within the confidence interval of borderline cut-scores.

    • Borderline students within the standard error band are provided a secondary test administration to obtain a score reflecting true performance above the cutoff.