Psychometric Testing: Validity, Scaling Methods, and Item Analysis

Vocabulary practice flashcards covering test validity models, decision theory, scaling methods, item construction rules, and statistical item analysis indices.

Validity

A fundamental psychometric property of tests indicating the degree to which a test accurately measures what it is designed to measure.

Systematic Error

A consistent bias that causes a measurement to inaccurately represent what it is supposed to measure, such as an intelligence test that inadvertently assesses English vocabulary proficiency.

Unsystematic Error

Unpredictable or random factors (such as fatigue, distractions, or mood) that cause measurements to vary inconsistently across occasions, primarily threatening test reliability.

Trinitarian Model

The most widely used framework to describe validation procedures, comprising three major categories: content validity, criterion-related validity, and construct validity.

Content Validity

The degree to which the items on a test represent a comprehensive sample of the domain or universe of skills and behaviors the test is intended to measure.

Face Validity

The appearance of test appropriateness from the perspective of the test taker based on item content; it is unrelated to whether the test is truly psychometrically valid.

Criterion-Related Validity

The extent to which scores on a test correlate with non-test behaviors or external outcome variables known as criterion measures.

Concurrent Validity

A form of criterion-related validity established when a test score is correlated with a criterion measure that is collected at the same time as testing.

Predictive Validity

A form of criterion-related validity established when a test score is correlated with a criterion measure that will become available in the future.

Standard Error of Estimate ($SE_{\text{est}}$)

A statistic indicating the expected margin of error in criterion prediction, representing the average difference between the actual criterion score ($Y$) and the predicted criterion score ($Y'$).

Decision Theory

A framework in testing where test scores are used directly as tools for decision-making and personnel selection, often summarized in expectancy tables.

Empirical Expectancy Table

A table showing data from predictive validation studies displaying the distribution of criterion outcomes across test score brackets, identifying correct classifications and classification errors relative to a cutoff score.

Selection Outcome Decision Matrix

A four-cell classification grid displaying the possible results when a selection test predicts performance: True Positive (Hit), False Positive (Miss), False Negative (Miss), and True Negative (Hit).

False Positive (Miss)

An erroneous selection decision where a test predicts an individual will succeed on the criterion, but the individual fails.

False Negative (Miss)

An erroneous selection decision where a test predicts an individual will fail on the criterion, but the individual actually succeeds.

Construct Validity

The most comprehensive type of validity (subsuming content and criterion-related validity) assessing whether test scores behave consistently with the theoretical framework of the construct being measured.

Incremental Validity

The amount of gain in predictive accuracy achieved by adding a new test to an existing battery of assessments.

Expert Rankings

A scaling method providing ordinal data by assigning numbers according to set criteria that indicate where an examinee falls on a construct (e.g., the Glasgow Coma Scale).

Method of Equal Appearing Intervals

A scaling technique developed by Thurstone to construct an interval scale from attitude statements by having judges sort items into $11$ categories ranging from extremely favorable to extremely unfavorable.

Guttman Scale

An ordinal scaling method assuming that endorsing a statement in an ordered sequence implies endorsement of all milder or lesser statements below it.

Likert Scale

An attitude scaling method, often termed a summative scale, that typically employs an odd number (such as $5$ or $7$) of ordered response options from strongly agree to strongly disagree.

Central Tendency Bias

A distortion observed in Likert-type scaling wherein respondents disproportionately select middle or neutral options instead of extreme ratings.

Empirical Keying (Criterion Keying)

A scale construction method where test items are selected solely because they contrast an empirical criterion group from a normative sample, without regard to theoretical rationale or expert judgment.

Testing Time Guidelines

Standard pacing allowances: True/False items require $30\text{ seconds}$ per item, Multiple Choice items require $45\text{--}60\text{ seconds}$ per item, and Essay questions require $6\text{--}8\text{ minutes}$ per $\frac{1}{2}\text{ page}$.

Stem and Distractors

The structural parts of a multiple-choice item, where the stem contains the question or problem statement, and distractors are plausible incorrect answer options.

Item Reliability Index

A statistic assessing an item's consistency with the overall trait being measured, calculated as $s_i r_{iT}$, where $s_i$ is the item standard deviation and $r_{iT}$ is the point-biserial correlation between item score and total test score.

Item Validity Index

A statistic assessing an item's contribution to predicting an outside criterion, calculated as $s_i r_{iC}$, where $s_i$ is the item standard deviation and $r_{iC}$ is the correlation between item score and criterion score.

Item Difficulty Index ($p$)

The proportion of examinees who answer a question correctly, calculated as $p = \frac{\text{number correct}}{\text{number who took the item}}$; values range from $0$ to $1$, where higher values denote easier items.

Optimal Item Difficulty ($p_{\text{optimal}}$)

The ideal difficulty level adjusted for guessing chance ($g$), calculated using the formula $p_{\text{optimal}} = \frac{1 + g}{2}$ (e.g., $p_{\text{optimal}} = .625$ for a 4-option item where $g = .25$).

Item Discrimination Index ($d$)

A measure comparing upper- and lower-scoring groups on item performance, calculated as $d = \frac{U - L}{N}$, where $U$ and $L$ are the correct counts in upper and lower groups and $N$ is group size; ranges from $-1$ to $+1$.

Distractor Analysis

An item analysis technique that checks whether incorrect options effectively attract lower-scoring examinees while not leading high-scoring examinees away from the keyed answer.