Psychometrics and Reliability Concepts
Test Length and Reliability Dynamics
Sample Size and Reliability Principles:
Shorter tests are inherently less reliable than longer tests.
Conceptual Analogy: Evaluating student opinions on campus housing policy at ECSU (such as whether first-year students receive priority on dorm selection) by surveying students yields low confidence, whereas surveying students provides a larger, more representative sample and higher statistical confidence.
Because an infinite pool of potential questions exists for any given construct, a -item test provides a far less reliable measurement than a -item test.
Splitting Tests and Reliability Underestimation:
Splitting a test in half reduces the effective length of the assessment being evaluated, leading to an underestimation of the test's full-length reliability when calculating correlation between the two halves.
To adjust for this reduction in test length, the Spearman-Brown Prophecy Formula is applied to estimate full-test reliability or determine necessary length adjustments to reach a desired reliability coefficient.
Spearman-Brown Prophecy Formula Applications:
Calculates the required multiplication factor for test length to achieve a target reliability coefficient ().
Multiplication Factor Example: If a test currently has a reliability of and the desired target reliability is , the test length must be multiplied by :
A -item test must be expanded to items.
A -item test must be expanded to items.
In split-half reliability applications, the formula treats the half-test correlation as if the test length were doubled back to its original full size.
Limitations of Split-Half Reliability:
Requires selecting a single specific split method (e.g., comparing the first items against the last items, or comparing odd-numbered items against even-numbered items).
Different split configurations produce different correlation coefficients, with no objective statistical basis for determining which split yields the single "correct" reliability value.
Advanced Internal Consistency Estimators
Kuder-Richardson Formula 20 (KR-20):
Published in 1937 by G. Frederic Kuder and M. W. Richardson as Formula #20 in a psychometric paper outlining multiple formulas.
Evaluates internal consistency by calculating the split-half reliability across every possible split combination of a test and taking the average of all resulting split-half coefficients.
Specifically designated for tests composed of dichotomous items (items scored with definitive right or wrong / correct or incorrect answers).
Cronbach's Coefficient Alpha:
Developed by Lee Cronbach (frequently referred to simply as Coefficient Alpha).
Extends the mathematical logic of KR-20 to continuous and non-dichotomous items (e.g., Likert response scales without single right or wrong answers).
Calculates the average reliability across all possible split-half combinations and inherently incorporates the Spearman-Brown correction into its mathematical formulation.
Empirical Demonstration: Heatwave Behavioral Adaptation Study:
Study Context: Survey evaluating how individuals perceive heatwave threats and adapt their behavior during a week-long heat wave scenario.
Item Pool: Participants indicated agreement on statements covering behavioral adaptations:
Having trouble sleeping.
Spending less time outdoors.
Increasing water consumption.
Spending more time at home.
Reducing scheduled time with other people.
Eating more junk food as a coping mechanism.
Monitoring energy/electricity bills.
Reducing outdoor physical activity.
Increasing indoor physical activity to compensate for reduced outdoor activity.
Binge-watching media.
Spending maximum time in air-conditioned environments.
Minimizing effort to avoid heat generation.
Factor Analysis: Statistical technique applied to examine item interrelationships and identify underlying scale dimensions, identifying potential scales.
Scale Reliability Benchmarks and Evaluation
Standard Score Review:
Z-Score: Standard score defined by a mean of and a standard deviation of ().
T-Score: Capital standard score defined by a mean of and a standard deviation of (). Distinct from the -statistic used in hypothesis testing.
Reliability Thresholds:
Research Applications: A reliability coefficient ( or ) of or higher is deemed acceptable.
High-Stakes / Decision-Making Applications: Hiring, selection, or high-stakes diagnostic decisions require a reliability coefficient of or higher.
Empirical Scale Evaluation Results:
Scale 1 (Sedentary / Isolation Behaviors):
Items: Trouble sleeping, spending less time with others, eating junk food, reducing physical activity, sedentary behaviors, doing as little as possible, social media usage.
Cronbach's Alpha:
Assessment: Exceeds the threshold; acceptable for research purposes.
Scale 2 (Indoor Containment Behaviors):
Items ( items): Less time outdoors, drinking more water, spending more time at home, spending more time in air conditioning.
Cronbach's Alpha:
Assessment: Exceeds the threshold; acceptable for research purposes.
Scale 3 (Energy Management and Outdoor Timing):
Items ( items): Reducing energy use, increasing indoor physical activity, going outdoors when possible, going outdoors early/late.
Cronbach's Alpha:
Assessment: Unacceptable reliability (). The items fail to cluster as a cohesive scale and must be analyzed as separate individual items. Future scale construction requires increasing item length to achieve acceptable reliability.
Taxonomy of Reliability Estimation Methods
Internal Consistency:
Most widely utilized method for assessing reliability in psychological and educational testing.
Primary Advantage: Requires only a single test administration.
Comparison to Alternative Methods:
Test-Retest Reliability: Requires administering the same test to the same cohort (e.g., over participants) across two time points separated by weeks or months (e.g., weeks or months). Suffers from participant attrition and logistical barriers.
Alternate Forms Reliability: Requires constructing two distinct, equivalent forms with different item wording, doubling item development workload.
Measurement Scale Categorization:
Behavioral sciences classify Likert-type response scales as Interval scales based on the assumption of equal intervals between response points (e.g., the distance between "disagree" and "slightly agree" equals the distance between "disagree" and "strongly disagree"), despite lacking an absolute zero.
(Note: Pure mathematics discipline conventions classify Likert items as Ordinal scales).
Inter-Rater (Inter-Score) Reliability:
Applied when assessment scoring depends on human judgment or subjective evaluation (e.g., grading essays or evaluating presentations via a scoring rubric).
Objective: Ensures independent evaluators produce consistent, concordant scores rather than subjective personal judgments.
Procedure: Two or more independent raters score the same sample of performances, and the correlation between rater scores is calculated.
Target Correlation: High positive correlation in the to range.
Classical Test Theory and Standard Error of Measurement
Classical Test Theory Model:
Where is the observed score, is the true score, and is measurement error.
Reliability estimation quantifies the proportion of variance attributable to error (), allowing for inferences regarding the True Score ().
Standard Error of Measurement (SEM):
Quantifies the expected variation of observed scores around a true score.
Formula: Where is the standard deviation of test scores from a single administration, and is the test reliability coefficient.
Confidence Intervals for Observed Scores:
In a normal distribution, approximately of scores fall within standard deviations of the mean (, often approximated as ).
Confidence Interval Formula:
Interpretation Example: An observed score evaluated with a calculated SEM yielding an interval calculation from to establishes confidence that the true score lies within that span.
Educational and High-Stakes Decision Policy
Flaws in Rigid Cut-Score Applications:
High-stakes testing environments often set arbitrary cut-scores (e.g., a required threshold score of to pass or avoid grade repetition).
Treating an observed score of as a strict, error-free absolute score falsely equates the observed score () directly to the true score ().
Confidence Interval Application to Borderline Scores:
Assuming an observed score of with an estimated SEM yielding a confidence interval margin of points:
Because the confidence interval ( to ) includes scores at and above the passing cut-score of , there is a statistical probability that the student's true score meets or exceeds the passing standard.
Policy Implications in Educational Assessment:
To account for measurement error near decision boundaries, public school systems (e.g., Virginia state educational policies) have established re-testing provisions for students scoring within the confidence interval of borderline cut-scores.
Borderline students within the standard error band are provided a secondary test administration to obtain a score reflecting true performance above the cutoff.