Classical Test Theory and Validity: Key Concepts and Reliability Measures

What is the core equation of Classical Test Theory?

X (Observed Score) = T (True Score) + E (Error)

What is the main goal of Classical Test Theory?

Maximize T (True Score) and minimize E (Error).

What does high reliability indicate?

The variability in observed scores reflects real, meaningful differences among examinees, not random error.

How is reliability estimated in CTT?

By examining the correlation between two parallel tests.

What is the range of reliability coefficients?

0 to 1, with closer to 1 indicating more reliability.

What is Alternate-Form Reliability?

Using two different forms of the same test administered to the same group.

What is a key advantage of Alternate-Form Reliability?

It minimizes memory/practice effects.

What does Split-Half Reliability measure?

It correlates performance on two halves of the same test to measure internal consistency.

What is Cronbach's Alpha?

The average of all possible split-half reliability coefficients, indicating internal consistency.

What does Test-Retest Reliability assess?

It correlates scores on the same test at two different time points to measure stability.

What is Inter-Rater Reliability?

The correlation between two judges administering or scoring the test, indicating the level of agreement.

What reliability coefficient is considered excellent?

≥ .90

What is the Standard Error of Measurement (SEM)?

It quantifies error in an individual's observed score.

What is the empirical rule in a normal distribution?

68% within ±1 SD, 95% within ±2 SD, and 99.7% within ±3 SD.

What does a z-score represent?

The number of standard deviations a data point is from the mean.

What is the mean and standard deviation of a T-score?

Mean = 50, SD = 10.

What is the difference between norm-referenced and criterion-referenced tests?

Norm-referenced tests are interpreted relative to others, while criterion-referenced tests use an absolute standard.

What is the Angoff Method used for?

To estimate cut scores by having SMEs assess item difficulty for minimally competent individuals.

What does a correlation coefficient (r) measure?

The strength of the linear relationship between two variables.

What is the significance of a reliability coefficient of ≥ .80?

It is considered good and suitable for screening.

What is the difference between convergent and discriminant validity?

Convergent validity refers to strong correlations between measures of the same or related constructs, while discriminant validity refers to low correlations between measures of different constructs.

What is face validity?

It assesses whether a test appears valid to examinees.

What is content validity?

It evaluates whether test items adequately represent the construct domain.

What is criterion-related validity?

It compares test scores to an external criterion.

What is predictive validity?

It assesses how well a test predicts future performance.

What is the purpose of factor analysis?

To reveal the internal structure of a test and identify latent factors underlying item correlations.

What does a higher Cronbach's Alpha indicate?

More internal consistency in the test.

What is the core formula of Classical Test Theory (CTT)?

X = T + E, where X is the observed score, T is the true score, and E is the error.

What does a reliability coefficient range from?

0 to 1, with higher values indicating more consistent tests.

What is an example of Alternate-Form Reliability?

The correlation between SAT Form A and SAT Form B.

What does Split-Half Reliability measure?

The correlation between two halves of the same test, such as odd and even items.

What is Cronbach's Alpha (α)?

The average of all possible split-half correlations, indicating internal consistency.

What is Test-Retest Reliability?

The correlation of the same test administered at two different time points.

What does Inter-Rater Reliability assess?

The agreement between different judges or raters.

What reliability coefficient is considered high enough for high-stakes decisions?

≥ .90

What is the Standard Error of Measurement (SEM)?

An estimate of the amount of error in an observed score.

What is Face Validity?

The extent to which a test appears to measure what it claims to measure.

What is Content Validity?

The degree to which test items represent the entire construct being measured.

What is Construct Validity?

Evidence that a test truly measures the theoretical construct it claims to measure.

What is Convergent Validity?

High correlation with similar constructs.

What is Discriminant Validity?

Low correlation with unrelated constructs.

What does Predictive Validity assess?

How well a test predicts future performance, such as SAT scores predicting college GPA.

What is the formula for calculating Sensitivity?

Sensitivity = True Positives / (True Positives + False Negatives)

What is Factor Analysis used for?

To reveal latent traits that explain item correlations.

What does EFA stand for in factor analysis?

Exploratory Factor Analysis.

What is the Empirical Rule in statistics?

68% of data falls within ±1 SD, 95% within ±2 SD, and 99.7% within ±3 SD.

What is the mean and standard deviation for IQ scores?

Mean = 100, SD = 15.

What does a T-Score of 70 indicate?

2 SD above the mean, indicating clinically significant results.

What is a major issue with developmental scores?

They can be misinterpreted as placement rather than representing a score relative to peers.

What is the difference between Norm-Referenced and Criterion-Referenced tests?

Norm-Referenced compares scores to peers, while Criterion-Referenced measures against a standard.

Observed score in Classical Test Theory

The observed score (X) is equal to the true score (T) plus the error (E), represented as X = T + E.

High reliability in testing

A test with high reliability indicates that most variability in scores reflects true score differences rather than measurement error.

Alternate-form reliability

This type of reliability involves two different but equivalent versions of a test.

Cronbach's alpha

Cronbach's alpha measures the internal consistency of a test, indicating how closely related a set of items are as a group.

Test-retest reliability

This reliability method is most affected by memory or practice effects, as it involves administering the same test to the same group at different times.

Reliability coefficient of .90

A reliability coefficient of .90 is considered excellent for high-stakes decisions.

Standard Error of Measurement (SEM)

SEM is used to estimate the true score confidence band, indicating the range within which the true score is likely to fall.

Face validity

Face validity refers to the extent to which a test appears on the surface to measure what it claims to measure.

Construct underrepresentation

Construct underrepresentation occurs when test items do not cover the full domain of the construct being measured.

Convergent validity

Convergent validity is demonstrated when two similar constructs correlate strongly with each other.

Example of concurrent validity

An example of concurrent validity is when a reading test correlates with a reading diagnostic given on the same day.

Sensitivity in testing

Sensitivity refers to the probability of a positive test result given that the disorder is present (Pr(+ test | disorder)).

Positive Predictive Value (PPV)

PPV increases when the base rate of the condition being tested is high.

Factor analysis purpose

Factor analysis is primarily used to determine underlying latent traits in a set of variables.

Factor loading

A factor loading represents the correlation between an item and a factor, indicating how much variance in the item is explained by the factor.

WISC-IV to WISC-V revision findings

Factor analysis revealed that perceptual reasoning actually contained two separate abilities in the WISC-IV to WISC-V revision.

Normal distribution characteristics

In a normal distribution, the mean, median, and mode are all equal.

Z-score of +1 percentile

A z-score of +1 corresponds approximately to the 84th percentile.

T-score of 70

A T-score of 70 is considered two standard deviations above the mean.

Norm-referenced interpretation

A norm-referenced interpretation indicates that a student scored better than 84% of peers.

Major problem with grade equivalents

A major problem with grade equivalents is that they assume linear growth in student performance.

Correlation of r = .50

A correlation of r = .50 between two different constructs is considered medium.

Developmental score interpretation

A developmental score stating a child performs at the '6.4 grade level' means their raw score matches the median of 6th graders in month 4.

Criterion-referenced score interpretation

Criterion-referenced score interpretation focuses on comparing performance to a standard rather than ranking students against peers.

Angoff Method estimation

In the Angoff Method, Subject Matter Experts (SMEs) estimate the probability that a minimally competent person would answer each item correctly.

What is a perfectly normal distribution?

A distribution where the mean, median, and mode are all equal.

What percent of scores fall within ±1 standard deviation in a normal distribution?

Approximately 68% of scores.

What does a z-score represent?

The number of standard deviations a data point is from the mean.

What percentile corresponds to a z-score of -1?

Approximately the 16th percentile.

What is the z-score for a student scoring at the 98th percentile?

Roughly z = +2.

How do you calculate the z-score for X = 120, M = 100, and SD = 10?

The z-score is 2.

Which standard score system has a mean of 50 and a standard deviation of 10?

T-scores.

What does a scaled score of 7 correspond to on a WISC subtest?

1 standard deviation below the mean.

What z-score corresponds to a T-score of 70?

A z-score of +2.

What percentage of the population is above a z-score of +1 in a normal distribution?

Approximately 16%.

What is the z-score for a score of 650 if M = 500 and SD = 100?

The z-score is 1.5.

What percentile does a z-score of -2 correspond to?

Approximately the 2nd percentile.

Which standard score has a mean of 100 and a standard deviation of 15?

IQ standard scores.

How many standard deviations above the mean is an IQ of 130?

2 standard deviations.

What best describes the shape of a normal curve?

Symmetrical and unimodal.

What does a z-score of 0 correspond to?

The mean of the distribution.

If X = 40, M = 50, and SD = 5, what is the z-score?

The z-score is -2.

What is the percentile rank for a T-score of 60?

Approximately the 84th percentile.

Which statement about percentiles is true?

A percentile compares you to a norm group.

What percentage of scores fall between z = -1 and z = +1?

Approximately 68%.

What percentile corresponds to a z-score of 1.5?

Approximately the 93rd percentile.

What is the formula for a z-score?

z = (X - M) / SD.

What is the z-score if a raw score is equal to the mean?

The z-score must be 0.

What does a T-score of 30 indicate?

It is 2 standard deviations below the mean.

Why do standard scores exist?

To allow comparison across different tests.