1/70
Looks like no tags are added yet.
What two types of Item Response Theory
Dichotomous and Non-Dichotomous Item Response Theory.
What is Factor Analysis used for?
To determine the quality of items and identify which items load onto a specific latent construct.
What is Item Response Theory (IRT) used for?
To determine which items are best for specific groups and examine how items operate within those groups.
Why is IRT useful across different demographic groups?
not all items are created equally; some work better for certain groups (e.g., men vs. women, different ages).
What is the Classical testing theory equation?
X = T + E
What is the major limitation of Classical testing theory?
All error is lumped into one term (E), so we cannot separate random error from systematic error.
It cannot identify sources of error at the item level.
How does IRT differ from CTT?
IRT examines scores on each item and uses hierarchical mathematical modeling to determine scoring patterns
What major advantage does IRT have over CTT?
IRT allows error to be examined at the item level.
What are the two components of Item Analysis?
Item Difficulty and Item Discriminability.
What is a test floor?
A test with many easy items.
What is a test ceiling?
A test with many hard items
When would you design a test with a high ceiling?
When testing medical students because the test should contain difficult items.
When would you design a test with a floor?
When testing special-needs students because the test should contain easier items
What standardized test is an example of a test with a high ceiling?
The SAT
What does test sensitivity measure?
How sensitive a test is in determining the presence of a disorder.
What happens when a test has low sensitivity?
It produces many false positives
What is an example of a low-sensitivity test discussed in class?
Home COVID tests (approximately 50-60% sensitivity).
What happens when a test has high sensitivity?
It produces few false negatives.
If a highly sensitive test says "negative," what can you conclude?
You can trust that negative result
What are examples of highly sensitive tests?
pregnancy tests
What important caution did the professor give about sensitivity?
Sensitivity only matters if the test is administered properly
What does a negative skew indicate?
(Tail on the left)
An easy test where most examinees score high.
What does a positive skew indicate?
(Tail on the right)
A difficult test where most examinees score low.
What determines the direction of skew?
The tail. Tail = skew.
How is item difficulty defined (Dichotomous?
The percentage of people who answer an item correctly.
If 84% of people answer an item correctly, what is its difficulty value?
.84
In item difficulty, what does a higher value mean?
The item is easier
In item difficulty, what does a lower value mean?
The item is harder
What is the ideal average item difficulty for most tests?
.50
What item difficulty range is generally preferred?
Between .30 and .70.
(So it averages to .50 for entire test)
Why might medical school exams have difficulty values around .20-.30?
The questions need to be difficult to ensure competency
Why might tests for disabled students have values around .80-.90?
Easier items are needed for the population being tested
What does item discriminability measure?
Whether people who do well on an item also do well on the entire test
What are the groups in discriminability for dichotomous
Upper
Middle
Lower
How is discrimination calculated using the extreme group method? (Dichotomous)
U - L
What does a higher discrimination value indicate? (Dichotomous)
Greater distinction between upper and lower groups
How is item difficulty calculated in the extreme groups example? (Dichotomous)
U + M + L
What is item discriminability? (Dichotomous)
It determines whether people who do well on an item also do well on the entire test
What does a higher discrimination index mean? (Dichotomous)
etter discrimination between high and low performers.
How is discriminability found in dichotomous scoring (methods)
extreme groups method
Point Biserial method
What does point biserial range from?
-1 to +1
What does a point biserial value that is positive and closer to 1 tell us?
Those scoring higher on the exam got this question correct
What does a negative point biserial value tell us?
There is likely a problem with the item
Can point biserial values be seen as percentages?
No
What does a good discriminating ICC curve look like? (Dichotomous)
A smooth upward curve where higher scorers are more likely to answer correctly.
What does a flat ICC curve indicate? (dichotomous)
No discrimination across score groups
What does the "overthinking" ICC pattern show? (dichotomous)
Lower and middle scorers improve, but upper scorers perform worse.
What types of constructs are measured with non-dichotomous IRT?
Personality traits, attitudes, depression, and other continuous psychological constructs
Why doesn't the traditional bell curve approach work well for non-dichotomous constructs?
Psychological traits like depression do not fit neatly into lower, middle, and upper groups.
How are non-dichotomous groups typically conceptualized?
Mild, moderate, and severe levels of symptom severity
What is an Item Response Function?
A graph showing the relationship between trait level and probability of endorsing an item
What is the basic unit of non-dichotomous IRT?
The Item Response Function
What indicates item difficulty on a non-dichotomous ICC curve?
The inflection point of the curve
Which curve represents the most difficult item?
(non-dichotomous)
The curve furthest from the Y-axis.
What does a higher level of difficulty mean in
non-dichotomous IRT?
A person must have a higher level of the trait/severity to endorse the item
How is discrimination determined on a nondichotomous ICC?
Steepness of the slope
What does a steeper slope indicate in nondichotomous ICC ?
Greater discrimination
What is being discriminated in non-dichotomous IRT?
Different levels of symptom severity (e.g., mild vs. severe).
What does the A parameter represent?
(Non-dichotomous)
Item discrimination
how well a question can tell the difference between people with high ability/severity and people with low ability/severity
What does a higher A value indicate (non-dichotomous)
Higher discrimination.
What does a higher B value indicate (non-dichotomous)
Higher difficulty or greater severity
What does the B parameter represent?
Item difficulty
the degree of severity the question is.
For dichotomous tests, what is the key advantage of IRT?
Item difficulty remains stable across different groups of test-takers.
Difficulty is matched to the aptitude of the test-takers ability
For non-dichotomous tests, what is the key advantage of IRT?
It allows comparison of how different demographic groups respond to the same items and helps identify test bias.
(across gender, age, other demographic affiliations)
What is test bias?
A scoring difference between demographic groups.
What does ICC stand for?
Item characteristic curves
What are you looking for when looking at ICC curves (discriminability wise)
Are the groups performing as expected?
What points does Item response theory hit?
analyze items and scales
measure psychological constructs
compare individuals on psychological constructs
(AMC)
Why are the groups changed for Dichotomous and Nondichotomous.
Mild, moderate, severe, maps better onto psychological constructs (nondichotomous)
What would happen if we were to harness IRT in nondichotomous scoring systems?
We could get over test bias - we aren’t there yet in the field
Reduce over pathologizing people and provide accurate diagnoses based on demographic affiliations