A. 3. Common Psychometric (Measurement) Theories

Homogeneous Test Items

Test items designed to measure one factor or construct that indicates high internal consistency

Heterogeneous Test Items

Test designed to measure multiple factors or construct that indicates low internal consistency

Dynamic Characteristics

A trait, state, or ability presumed to be ever-changing because of situational factors (e.g mood)

Static Characteristics

A trait, state, or ability presumed to be relatively unchanging (e.g pesonality)

Restricted range

Occurs when scores are too similar (e.g from a homogenous group). This results in little to no variation which lessens the reliability coefficient

Inflated range

Occurs when scores are too different (e.g from a heterogenous group). This means the range is too wide which inflates the reliability coefficient

Power Test

A test where the time limit is long enough or even non-existent to allow test takers to attempt all items. The items increase in difficult the higher they go

Speed Test

Test that contains items of uniform level of difficulty but with a time limit. The goal is to complete as many items as fast as possible

Norm-Referenced Test

Test that compares the score of an examinee to the scores of other people or a group of people

Criterion-Referenced Test

Test that compares the performance of an examinee to a predetermined standard or competency level, usually through certain cut-off scores

Classical Test Theory

Theory that is referred to as true score (or classical) model of measurement and believes that everyone has a "true score" on a test

True Score

A value that, according to CTT, genuinely reflects an individual’s ability (or trait) level as measured by a particular test

Formula for the CTT

X = T + E

Domain Sampling Theory

Core concept in the CTT that states that any test is a subset of items drawn from an infinite universe of potential questions measuring a single trait. Assumes that all items come from the same single definition and tests under this theory have high internal consistency

Domain of Behavior

The universe of items that could measure a certain behavior

Generalizability Theory

Theory that believes that error is not a single pool (Unlike CTT). Rather, it believes that error has different sources that needs to be taken into account

Terminologies in the Generalizability Theory

Universe

Facets

Universe Score

Generalizability Study

Decision Study

Universe

In Generalizability Theory, it it the testing environment. It describes the context/details of the particular test situation.

Facets

In Generalizability Theory, it is the possible sources of error

Universe Score

In Generalizability Theory, it is equivalent to the true score. The average of an individual’s scores across all possible conditions in a universe of generalization

Generalizability Study (G-Study)

In Generalizability Theory, it examines how generalizable scores from a particular test are if the test is administered in different situations

Decision Study

In Generalizability Theory, it is when developers use the G-Study to make decisions on how to change or improve a test

Item-Response Theory (Latent-Trait Theory)

The Modern Test Theory. It models how individuals interact or respond to a particular test question

Latent Trait

The unobservable Trait

Manifestation Trait

The observable Trait

Dichotomous Test Items

Test items with only two possible responses

Three Main Models of IRT (Parameters) for dichotomous items

Parameter A: Discrimination

Parameter B: Difficulty

Parameter C: Pseudoguessing/Guessing

Polytomous Test Items

Test items with more than two responses

Parameter A: Discrimination

In Item-Response Theory, it measures how an item differentiates among people with higher or lower levels of the trait/ability that is being measured

Parameter B: Difficulty

In Item-Response Theory, it is the most basic parameter that tries to evaluate how high the trait/ability is in a certain test taker

Parameter C: Pseudoguessing/Guessing

In Item-Response Theory, it is the parameter that accounts how likely a test taker will guess the correct item

Formula for probability of guessing

Number of correct options / Total number of choices

Standard Error of Measurement/Score

Provides an estimate of the amount of error inherent in an observed score of measurement

Confidence Interval

A range or band of the test scores that is likely to contain the true score

68% confidence interval (±1 SEM)

This means there's a 68% chance that the individual's true score falls within that range

95% confidence interval (±1.96 SEM often approximated as ±2 SEM)

This means there’s a 95% chance that the individual’s true score falls within that range

Standard Error of the Difference

Statistical measure that aids in determining whether the difference between two test scores is due to true variance or error variance

Standard Error of Estimate

Statistical measure that predicts how much the actual data differs from a prediction line in regression