Psychological Testing: Concepts, Reliability, Validity, and Test Development

Comprehensive vocabulary flashcards covering the nature, history, norming, reliability, validity, scaling methods, and item analysis procedures in psychological testing.

Aptitude test

A test designed to estimate an individual's potential for future learning or performance, in contrast to measuring prior learning.

Achievement test

A test designed to measure what an individual has already learned.

Neuropsychological test

A test that assesses cognitive, emotional, or behavioral effects associated with brain functioning.

Standardization

The procedure of administering a test with identical instructions, materials, timing, and scoring so that scores are comparable and interpretable against appropriate norms.

Brass instruments era

An early testing period associated with Francis Galton, who utilized sensory and physical measures such as reaction time and physical strength to investigate individual differences.

Army Alpha and Army Beta

Group tests developed during World War I to classify military recruits; Army Alpha was predominantly verbal, whereas Army Beta was constructed for individuals with limited English literacy or reading ability.

Kurtosis

A descriptive statistical property that characterizes how peaked or flat a distribution is.

Positive skew

An asymmetric distribution characterized by a tail that extends toward the higher scores.

Negative skew

An asymmetric distribution characterized by a tail that extends toward the lower scores.

Norm-referenced test

A test that interprets an individual's score by comparing it directly to the performance of a defined norm group.

Criterion-referenced test

A test that evaluates an individual's performance against a predefined standard, skill level, or cut-off criterion rather than comparing it to peers.

Percentile rank

The percentage of individuals in the normative group who score below a specific test score; it does not indicate the percent of items answered correctly.

$z$ score

A standard score expressing how many standard deviations an individual score is from the distribution mean, having a typical mean of $0$ and a standard deviation of $1$.

$T$ score

A transformed standardized score scale established with a typical mean of $50$ and a typical standard deviation of $10$.

Stanine

A standardized scoring system that expresses scores with a typical mean of $5$ and a typical standard deviation of $2$.

Classical test theory

A psychometric framework that conceives of an observed score ($X$) as the sum of a true score ($T$) and unsystematic measurement error ($E$): $X = T + E$.

Systematic error

A consistent bias that leads a test to inaccurately represent the target construct, directly threatening validity.

Unsystematic error

Unpredictable, random fluctuations stemming from factors such as fatigue, mood, distraction, or environment that primarily reduce score reliability.

Coefficient alpha

An internal consistency reliability metric calculated for test items that possess multiple scoring options or non-dichotomous responses.

KR-20

An internal consistency formula used specifically for tests composed of dichotomously scored items (e.g., right/wrong).

Standard error of measurement (SEM)

A statistic estimating the typical amount of error around an observed score, computed as $\text{SEM} = \text{SD} \times \sqrt{1 - r}$.

Point-biserial correlation

A correlation coefficient computed between one continuous variable and one genuinely dichotomous variable.

Biserial correlation

A correlation coefficient computed between one continuous variable and an artificial dichotomy created by splitting a continuous variable.

Speed test

A test composed of relatively easy items administered under strict time limits, where overall performance is determined by the number of items completed.

Power test

A test administered with ample time containing items of increasing difficulty designed to evaluate the highest level of ability or difficulty an examinee can master.

Item response theory (IRT)

A psychometric framework that models the mathematical probability of answering an item correctly as a function of the examinee's underlying trait level.

Trinitarian model

A traditional, widely used framework describing validation procedures across three primary categories: content validity, criterion-related validity, and construct validity.

Content validity

The degree to which test items representatively sample the broader universe of behaviors, skills, or domain that the test is designed to measure.

Face validity

The superficial appearance of test appropriateness from the viewpoint of the examinee; it does not reflect the objective empirical validity of the instrument.

Concurrent validity

A form of criterion-related validity where test scores are correlated with an external criterion measure collected at the same time as testing.

Predictive validity

A form of criterion-related validity where test scores correlate with an external criterion measure that will only become available in the future.

Standard error of estimate ($SE_{\text{est}}$)

A statistical index indicating the expected error or margin of inaccuracy in predicting a criterion score from a test score, representing the average value of $Y - Y'$.

Construct validity

The overarching form of validity that assesses whether test scores behave consistently with the theoretical framework of the psychological attribute being measured.

Incremental validity

The specific increase in predictive accuracy or decision effectiveness achieved by incorporating an additional test into an existing assessment protocol.

Expectancy table

A practical decision-theory table that presents the empirical probability of various criterion outcomes based on specific test scores or cut-off thresholds.

Method of Equal Appearing Intervals

A Thurstone scaling technique where judges sort attitude statements into 1 of 11 ordered categories (from extremely favorable to extremely unfavorable) to calculate item means and standard deviations.

Guttman scale

An ordinal scaling approach based on the assumption that endorsement of a particular item in an ordered sequence guarantees endorsement of all milder or less extreme items.

Likert scale

An attitude measurement format using an odd number of ordered responses (e.g., 5 or 7 options), which can be vulnerable to central tendency bias.

Empirical keying

A test development method where items are chosen solely based on their ability to differentiate an established criterion group from a normative sample, without regard to theoretical rationale.

Item reliability index

A statistical index computed as $s_i r_{iT}$, representing an item's standard deviation ($s_i$) multiplied by its correlation with the total test score ($r_{iT}$).

Item validity index

A statistical index computed as $s_i r_{iC}$, representing an item's standard deviation ($s_i$) multiplied by its correlation with an external criterion ($r_{iC}$).

$p$ index

A metric of item difficulty calculated as the proportion of examinees who answered an item correctly: $p = \frac{\text{number correct}}{\text{number who took the item}}$.

Optimal item difficulty ($p_{\text{optimal}}$)

The targeted item difficulty level that accounts for guessing probability ($g$), calculated using the formula $p_{\text{optimal}} = \frac{1 + g}{2}$.

$d$ index

A measure of item discrimination calculated as $d = \frac{U - L}{N}$, where $U$ and $L$ are the counts of correct responses in upper and lower groups and $N$ is the group size.

Distractor analysis

The evaluation of incorrect multiple-choice alternatives to ensure they attract lower-scoring test takers while not drawing away high scorers from the keyed answer.