Psychometrics - Exam 2

0.0(0)
Studied by 0 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/47

flashcard set

Earn XP

Description and Tags

Comprehensive vocabulary flashcards covering Classical Test Theory, reliability estimation methods, exploratory factor analysis, item analysis, and test validity concepts from Exam 2 review slides.

48 Terms

1
New cards

Classical Test Theory (CTT)

  • Posits that every obtained test score is a composite determined by this equation

    • X = T+E

    • X is the observed score

    • T is the true score


2
New cards

Observed Score (XX)

  • The actual numerical score recorded or obtained from an assessment instrument

    • (e.g., total points earned on a test)

  • fluctuates across occasions due to varying error scores


3
New cards

True Score (TT)

  • The score a respondent would obtain if measurement were completely free of error

  • Represents the average score we would obtain across infinite repeated testings with the test-taker’s memory reset each time

    • purely hypothetical because error-free measurement cannot be directly observed in practice


4
New cards

Error Score (EE)

  • Unsystematic (random) fluctuations that cause an observed score (X) to deviate from the true score (T)


5
New cards

Error Independence

Because measurement error is assumed to be unsystematic in CTT:

  1. individual errors cancel out over repeated measurements, eventually yielding a mean error score of zero

  2. error scores have zero correlation to true scores or other error scores from separate test administrations (this is known as error independence)


6
New cards

When is E positive or Negative?

If X > T, E is positive

If X < T, E is negative

7
New cards

Theoretical Reliability Index (ρXT\rho_{XT})

The correlation between true scores and observed scores

  • I.e., how closely do our measurements reflect the true score?

  • Purely hypothetical and cannot be calculated in the real world (lol)


8
New cards

Reliability Coefficient (ρXX′\rho_{XX'})

  • Tells us what percent of the variance in observed scores is attributable to variance in true scores; ranges from 0 to 1

  • The leftover observed-score variance that ISN’T attributed to true-score variance (i.e., 1-Pxx) is associated with random error


9
New cards

Cronbach's Alpha (α\alpha)

  • Measures the interrelatedness of items across a test (i.e., the test’s internal consistency/reliability)

    • the higher the coefficient alpha, the more intercorrelated the items are


10
New cards

Effect of Adding Items on Alpha

Adding items to a test generally increases coefficient alpha, provided the newly added items correlate at least as well with existing items as the existing items already correlate with each other.

11
New cards

Interrelatedness vs. Unidimensionality

The principle that high coefficient alpha cannot be automatically interpreted as evidence of unidimensionality, as a multidimensional test with correlated subdimensions can still yield a very high alpha.

12
New cards

Correlated Measurement Errors

A violation of CTT's error-independence assumption occurring when multiple items share a common element (e.g., reading passage or chart), causing shared error variance to masquerade as true-score variance and artificially inflate alpha.

13
New cards

Kuder-Richardson Formulas

Measures of internal consistency designed specifically for tests composed of dichotomously scored items (scored correct/incorrect with no in-between).

14
New cards

KR-20

The exact algebraic equivalent of Cronbach's coefficient alpha applied to binary data, where the term piqip_i q_i represents item variance.

15
New cards

KR-21

A simplified version of KR-20 estimating reliability using only the test mean, variance, and number of items under the restrictive assumption that all items have equal difficulty; it underestimates KR-20 reliability if item difficulties vary.

16
New cards

Sample Variability Effect on Reliability

Internal consistency estimates increase when the sample has greater variability on the construct because genuine differences expand true-score variance relative to error and prevent restriction of range.

17
New cards

Reliability Standards

Benchmarks calibrated to testing consequences: approximately 0.700.70 is an acceptable guideline for early test development, whereas clinical or high-stakes contexts require coefficients of 0.900.90 or higher.

18
New cards

Nature of Reliability

Reliability is not an inherent property of a test itself, but a characteristic of the scores obtained from a particular sample under particular administration conditions.

19
New cards

Objective Item Formats (Reliability Advantage)

Item formats (e.g., multiple choice, true/false) that yield higher internal consistency than subjective formats by maximizing domain sampling efficiency (allowing more items) and minimizing human scoring error.

20
New cards

Exploratory Factor Analysis (EFA)

A statistical method that models correlations among variables as functions of underlying, latent factors that they hold in common, simplifying large numbers of messy items into fewer hidden dimensions.

21
New cards

Principal Component Analysis (PCA)

A data-reduction technique that analyzes full variance across all variables using a standard correlation matrix to create weighted composite summary scores.

22
New cards

Standard Correlation Matrix

A matrix summarizing all pairwise inter-variable relationships across a sample, featuring diagonal values of 1.01.0 representing total variance (common variance + unique variance + random error).

23
New cards

Reduced Correlation Matrix

A correlation matrix that replaces the diagonal values of 1.01.0 with communality estimates (h2h^2), discarding unique item variance so factor extraction algorithms analyze only shared common variance.

24
New cards

Communality (h2h^2)

The proportion of variance in a single measured item or variable that is shared with the common factors.

25
New cards

Factor Extraction

The mathematical process of pulling out underlying, latent dimensions from a correlation or covariance matrix to account for as much shared covariation as possible using the fewest dimensions.

26
New cards

Principal Axis Factoring (PAF)

An EFA extraction method that extracts latent dimensions from the reduced correlation matrix using eigenanalysis, yielding initial factors that are completely uncorrelated (orthogonal) prior to rotation.

27
New cards

Maximum Likelihood (ML) Extraction

An EFA factor extraction method estimating parameters most likely to have produced the observed sample correlation matrix; it crucially assumes multivariate normality across variables.

28
New cards

Unweighted Least Squares (ULS) Extraction

An EFA factor extraction method that determines factor estimates by minimizing the sum of squared differences between observed and reproduced correlations, without requiring multivariate normality.

29
New cards

Kaiser Criterion

A factor retention rule stating that any factor with an eigenvalue greater than 1.01.0 should be retained; its major flaw in EFA is that it typically overestimates the number of factors.

30
New cards

Factor Loading Matrix

A matrix in EFA where rows represent observed variables, columns represent latent factors, and entries represent the relationship or loading between a variable and a factor.

31
New cards

Uniqueness Matrix

A diagonal matrix in EFA representing leftover variance (u2u^2) in each observed variable unexplained by common factors, where off-diagonal entries are set to zero under the assumption that item uniquenesses are uncorrelated.

32
New cards

Model Fit (EFA)

An evaluation of how well an extracted factor model accounts for actual relationships by comparing observed correlations against reproduced correlations to determine if residuals are close to zero.

33
New cards

Item Difficulty (PP)

The proportion of examinees who answer a dichotomously scored item correctly (P=Number of correct responsesTotal number of responsesP = \frac{\text{Number of correct responses}}{\text{Total number of responses}}), which functions as an index of item easiness.

34
New cards

Item Variance (s2s^2)

The spread of responses for a dichotomous item, calculated as P(1−P)P(1 - P) (or pqpq), which peaks at its maximum of 0.250.25 when P=0.50P = 0.50 and drops to zero when P=1.0P = 1.0 or P=0P = 0.

35
New cards

Item Discrimination

The degree to which an individual test item distinguishes between respondents who possess high versus low levels of the attribute, knowledge, or skill being measured.

36
New cards

Internal Criterion

The use of total scores from the same test to rank examinees and establish upper and lower comparison groups for item discrimination analysis.

37
New cards

External Criterion

The use of an outside measure (e.g., scores on another validated test, clinical diagnoses, or job performance ratings) to form comparison groups for item discrimination analysis.

38
New cards

Discrimination Index (DD)

A metric quantifying item discrimination, calculated as D=PU−PLD = P_U - P_L, where PUP_U is the proportion correct in the upper group and PLP_L is the proportion correct in the lower group.

39
New cards

Negative Item Discrimination

A scenario where lower-scoring examinees answer an item correctly more often than higher-scoring examinees (PU<PLP_U < P_L, yielding D<0D < 0), most commonly caused by miskeying or multiple plausible correct answers.

40
New cards

Validity

The degree to which empirical evidence and theoretical rationales support the interpretations and proposed uses of test scores, functioning as a chain of reasoning rather than an inherent, unchanging property of the test.

41
New cards

Content Evidence

Validity evidence evaluating whether test items provide a representative and balanced sample of the domain, evaluated by subject-matter experts for relevance, importance, clarity, and possible bias.

42
New cards

Construct Underrepresentation

A threat to validity occurring when an assessment is defined too narrowly and omits important aspects of the target construct from the test.

43
New cards

Construct-Irrelevant Variance

A threat to validity occurring when test scores are influenced by factors unrelated to the target construct, consisting of construct-irrelevant difficulty and construct-irrelevant easiness.

44
New cards

Construct-Irrelevant Difficulty

Features of an item or testing format that make it artificially harder for reasons unrelated to the target skill (e.g., overly strict time limits causing rushed answers on a free-response test).

45
New cards

Construct-Irrelevant Easiness

Features of an item or testing format that make it artificially easier for reasons unrelated to the target skill (e.g., correct options consistently being the longest and most detailed choice).

46
New cards

Think-Aloud Protocol

A primary investigation method for response process validity where examinees verbalize their thought process step-by-step in real time while solving a problem.

47
New cards

Internal Structure Evidence

Validity evidence evaluating whether actual statistical relationships (correlations) among test items align with theoretical expectations of unidimensionality or distinct subdimensions.

48
New cards

Confirmatory Factor Analysis (CFA)

A factor analytic method requiring researchers to prespecify every factor and item relationship in advance based on strong theory to test model fit, unlike exploratory approaches.