PTPR 4: Modern Test Theory

classical test theory evaluation

Advantages:

• intuitive and easy to apply
• SPSS, Excel
• no large sample size/no. of items needed

Disadvantages:

• focus on test, not on items
• test properties depend on the population (reliability, difficulty)
• person properties depend on the test (sum score is higher if the test is easy, lower if difficult)

modern test theory

"specify a measurement model in which we mathematically link the item scores to the construct -> latent variable/latent trait/factor

Assumptions:

• unidimensionality - only measure 1 construct

"

item response theory

"form of modern test theory with specific link between item and latent variable:
individual's response to test item is influenced by qualities of the individual and qualities of the item

𝜃 - variable
𝑠 - subject
P - probability of correct response
i - item

Logistic fuction/S-shaped function: mathematical link between observed variable and probability of correct response - item characteristic curve
"

Rasch model (1PL)

"applies only to dichotomous items

𝜷_i - item difficulty/mean;
max positive/negative no. - position relative to 0
Blue item is more difficult than red (larger 𝜷).

• probability of response depends on
• latent variable
• item difficulty

ideal item difficulty - 0.5; maximal variability

"

2 Parameter logistic model (2PL)

"𝛽_i - parameter difficulty
𝛼_i - item discrimination - steepness of S-shaped curve, how well item discriminates between two people

• positive number; negative for contraindicative item

similar to item-total correlation in CTT

"

3 Parameter logistic model (3PL)

"𝛽_i - parameter difficulty
𝛼_i - item discrimination - steepness of S-shaped curve, how well item discriminates between two people 
𝑐_i - guessing value, probability of a correct answer purely on chance (no. from 0 to 1) - lower bound of slope

"

graded response model (GRM)

"for Likert scale items! and other polytomous items
separate item characteristic curve for each option.

you don't look at probability of correct response, but at probability that it's larger than j
difficulty for each category - 𝛽_ij - difficulty of response option j,
"

what can item response theory do

• scale analysis: see how well items are doing, how well scale is, are there enough medium, easy, difficult items
• test construction
• study item fairness: differential item functioning
• computerised adaptive testing
• person fit: can indicate cheating, random responding, low motivation, cultural bias, misinterpretation, other errors

test construction

"item information 

• little information in flat ranges of related variable
• much information in steep ranges
• peak of information is at the difficulty

item information function = tells you how much information about the latent variable you get at a certain level of the related variable

items give information in different ranges

Item information curve:

• height of curve - amount of info item provides
• highest point - trait level at which item provides most info (coincides with difficulty level)
• items differ in points at which they provide best info

-> scale/test information function : adds all of the information you get in the items

• norm referenced test: compare score to all levels of population, need a lot of information in the entire range
• criterion referenced test: determine whether someone passes a cut-off, need most information in cutoff point!, discrimination not very important

"

differential item functioning (DIF)

"whether item is fair towards different groups.

if no differences between groups are expected for the latent variable, there should be no differences in the item scores

difficulty 𝛽 is different for students of the same level (𝜃)
mean doesn't matter in interpretation, looking at people of same level of latent variable.
OR
worse discrimination for some groups, more false positives/negatives

guessing is not considered as a source of DIF"

computerised adaptive testing (CAT)

"use item information and scale information 

Computerised Adaptive Testing estimates level of the latent variable, selects item that will give the most information for that position, to discriminate.

• uses an item bank (many items of which we know the difficulty and discrimination)
• computer selects most informative item for individual test-taker (usually one where 𝛽 = 0.5)
• if correct, more difficult item; if incorrect, easier item
• continues until estimate for latent trait stops changing

"

item response theory evaluation

"Advantages:

• population and test statistics are independent
• focus on items, not test

Disadvantages:

• more statistically complex, can't calculate latent variable, estimate it using computer
• needs larger sample - need more people to estimate discrimination and difficulty

"

factors and models

""

initial estimates of variable levels/item difficulties

""

CTT vs. IRT

CTT: one single estimate of reliability for set of test scores
IRT: test is more reliable for some than others