1/47
Comprehensive vocabulary flashcards covering Classical Test Theory, reliability estimation methods, exploratory factor analysis, item analysis, and test validity concepts from Exam 2 review slides.
Classical Test Theory (CTT)
Posits that every obtained test score is a composite determined by this equation
X = T+E
X is the observed score
T is the true score
Observed Score (X)
The actual numerical score recorded or obtained from an assessment instrument
(e.g., total points earned on a test)
fluctuates across occasions due to varying error scores
True Score (T)
The score a respondent would obtain if measurement were completely free of error
Represents the average score we would obtain across infinite repeated testings with the test-taker’s memory reset each time
purely hypothetical because error-free measurement cannot be directly observed in practice
Error Score (E)
Unsystematic (random) fluctuations that cause an observed score (X) to deviate from the true score (T)
Error Independence
Because measurement error is assumed to be unsystematic in CTT:
individual errors cancel out over repeated measurements, eventually yielding a mean error score of zero
error scores have zero correlation to true scores or other error scores from separate test administrations (this is known as error independence)
When is E positive or Negative?
If X > T, E is positive
If X < T, E is negative
Theoretical Reliability Index (ρXT)
The correlation between true scores and observed scores
I.e., how closely do our measurements reflect the true score?
Purely hypothetical and cannot be calculated in the real world (lol)
Reliability Coefficient (ρXX′)
Tells us what percent of the variance in observed scores is attributable to variance in true scores; ranges from 0 to 1
The leftover observed-score variance that ISN’T attributed to true-score variance (i.e., 1-Pxx) is associated with random error
Cronbach's Alpha (α)
Measures the interrelatedness of items across a test (i.e., the test’s internal consistency/reliability)
the higher the coefficient alpha, the more intercorrelated the items are
Effect of Adding Items on Alpha
Adding items to a test generally increases coefficient alpha, provided the newly added items correlate at least as well with existing items as the existing items already correlate with each other.
Interrelatedness vs. Unidimensionality
The principle that high coefficient alpha cannot be automatically interpreted as evidence of unidimensionality, as a multidimensional test with correlated subdimensions can still yield a very high alpha.
Correlated Measurement Errors
A violation of CTT's error-independence assumption occurring when multiple items share a common element (e.g., reading passage or chart), causing shared error variance to masquerade as true-score variance and artificially inflate alpha.
Kuder-Richardson Formulas
Measures of internal consistency designed specifically for tests composed of dichotomously scored items (scored correct/incorrect with no in-between).
KR-20
The exact algebraic equivalent of Cronbach's coefficient alpha applied to binary data, where the term piqi represents item variance.
KR-21
A simplified version of KR-20 estimating reliability using only the test mean, variance, and number of items under the restrictive assumption that all items have equal difficulty; it underestimates KR-20 reliability if item difficulties vary.
Sample Variability Effect on Reliability
Internal consistency estimates increase when the sample has greater variability on the construct because genuine differences expand true-score variance relative to error and prevent restriction of range.
Reliability Standards
Benchmarks calibrated to testing consequences: approximately 0.70 is an acceptable guideline for early test development, whereas clinical or high-stakes contexts require coefficients of 0.90 or higher.
Nature of Reliability
Reliability is not an inherent property of a test itself, but a characteristic of the scores obtained from a particular sample under particular administration conditions.
Objective Item Formats (Reliability Advantage)
Item formats (e.g., multiple choice, true/false) that yield higher internal consistency than subjective formats by maximizing domain sampling efficiency (allowing more items) and minimizing human scoring error.
Exploratory Factor Analysis (EFA)
A statistical method that models correlations among variables as functions of underlying, latent factors that they hold in common, simplifying large numbers of messy items into fewer hidden dimensions.
Principal Component Analysis (PCA)
A data-reduction technique that analyzes full variance across all variables using a standard correlation matrix to create weighted composite summary scores.
Standard Correlation Matrix
A matrix summarizing all pairwise inter-variable relationships across a sample, featuring diagonal values of 1.0 representing total variance (common variance + unique variance + random error).
Reduced Correlation Matrix
A correlation matrix that replaces the diagonal values of 1.0 with communality estimates (h2), discarding unique item variance so factor extraction algorithms analyze only shared common variance.
Communality (h2)
The proportion of variance in a single measured item or variable that is shared with the common factors.
Factor Extraction
The mathematical process of pulling out underlying, latent dimensions from a correlation or covariance matrix to account for as much shared covariation as possible using the fewest dimensions.
Principal Axis Factoring (PAF)
An EFA extraction method that extracts latent dimensions from the reduced correlation matrix using eigenanalysis, yielding initial factors that are completely uncorrelated (orthogonal) prior to rotation.
Maximum Likelihood (ML) Extraction
An EFA factor extraction method estimating parameters most likely to have produced the observed sample correlation matrix; it crucially assumes multivariate normality across variables.
Unweighted Least Squares (ULS) Extraction
An EFA factor extraction method that determines factor estimates by minimizing the sum of squared differences between observed and reproduced correlations, without requiring multivariate normality.
Kaiser Criterion
A factor retention rule stating that any factor with an eigenvalue greater than 1.0 should be retained; its major flaw in EFA is that it typically overestimates the number of factors.
Factor Loading Matrix
A matrix in EFA where rows represent observed variables, columns represent latent factors, and entries represent the relationship or loading between a variable and a factor.
Uniqueness Matrix
A diagonal matrix in EFA representing leftover variance (u2) in each observed variable unexplained by common factors, where off-diagonal entries are set to zero under the assumption that item uniquenesses are uncorrelated.
Model Fit (EFA)
An evaluation of how well an extracted factor model accounts for actual relationships by comparing observed correlations against reproduced correlations to determine if residuals are close to zero.
Item Difficulty (P)
The proportion of examinees who answer a dichotomously scored item correctly (P=Total number of responsesNumber of correct responses), which functions as an index of item easiness.
Item Variance (s2)
The spread of responses for a dichotomous item, calculated as P(1−P) (or pq), which peaks at its maximum of 0.25 when P=0.50 and drops to zero when P=1.0 or P=0.
Item Discrimination
The degree to which an individual test item distinguishes between respondents who possess high versus low levels of the attribute, knowledge, or skill being measured.
Internal Criterion
The use of total scores from the same test to rank examinees and establish upper and lower comparison groups for item discrimination analysis.
External Criterion
The use of an outside measure (e.g., scores on another validated test, clinical diagnoses, or job performance ratings) to form comparison groups for item discrimination analysis.
Discrimination Index (D)
A metric quantifying item discrimination, calculated as D=PU−PL, where PU is the proportion correct in the upper group and PL is the proportion correct in the lower group.
Negative Item Discrimination
A scenario where lower-scoring examinees answer an item correctly more often than higher-scoring examinees (PU<PL, yielding D<0), most commonly caused by miskeying or multiple plausible correct answers.
Validity
The degree to which empirical evidence and theoretical rationales support the interpretations and proposed uses of test scores, functioning as a chain of reasoning rather than an inherent, unchanging property of the test.
Content Evidence
Validity evidence evaluating whether test items provide a representative and balanced sample of the domain, evaluated by subject-matter experts for relevance, importance, clarity, and possible bias.
Construct Underrepresentation
A threat to validity occurring when an assessment is defined too narrowly and omits important aspects of the target construct from the test.
Construct-Irrelevant Variance
A threat to validity occurring when test scores are influenced by factors unrelated to the target construct, consisting of construct-irrelevant difficulty and construct-irrelevant easiness.
Construct-Irrelevant Difficulty
Features of an item or testing format that make it artificially harder for reasons unrelated to the target skill (e.g., overly strict time limits causing rushed answers on a free-response test).
Construct-Irrelevant Easiness
Features of an item or testing format that make it artificially easier for reasons unrelated to the target skill (e.g., correct options consistently being the longest and most detailed choice).
Think-Aloud Protocol
A primary investigation method for response process validity where examinees verbalize their thought process step-by-step in real time while solving a problem.
Internal Structure Evidence
Validity evidence evaluating whether actual statistical relationships (correlations) among test items align with theoretical expectations of unidimensionality or distinct subdimensions.
Confirmatory Factor Analysis (CFA)
A factor analytic method requiring researchers to prespecify every factor and item relationship in advance based on strong theory to test model fit, unlike exploratory approaches.