1.3 Nature of Tests & Reliability Models and Theories

Homogeneous Test Items

Test designed to measure one factor (high internal consistency)

Heterogeneous Test Items

Test designed to measure multiple factors (low internal consistency)

Dynamic Characteristics

A trait, state, or ability presumed to be ever-changing as function of situational and cognitive experience

Static Characteristics

A trait, state, or ability presumed to be relatively unchanging

Restricted

Lower correlation coefficient

Inflated

Higher correlation coefficient

Power Test

When the time limit is long enough to allow test takers to attempt all items and if some items are so difficult that no test takers can obtain a perfect score

Speed Test

Contains items of uniform level of difficulty so that, when given generous time limits, all test takers should be able to complete all the test items correctly

Criterion-Referenced Test

Test assigned to provide an indication of where a test taker stands with respect to some variable or criterion

Classical Test Theory

Theory that is referred to as true score (or classical) model of measurement and believes that everyone has a "true score" on a test

True Score

A value that, according to CTT, genuinely reflects an individual’s ability (or trait) level as measured by a particular test

Domain Sampling Theory

Seeks to estimate the extent to which specific sources of variation under defined conditions are contributing to the test score

Domain of Behavior

The universe of items that could conceivably measure that behavior, can be thought of as a hypothetical construct

Generalizability Theory

Theory that believes a person’s test scores vary from testing to testing because of variables in the testing situation

Terminologies in the Generalizability Theory

Universe, Facets, Universe Score, Generalizability Study, & Decision Study

Universe

In Generalizability Theory, it describes the details of the particular test situation

Facets

In Generalizability Theory, it is the number of items in the test, the amount of training the test scores have had, and the purpose of test administration

Universe Score

In Generalizability Theory, it believes that the exact same conditions of all the facets in the universe should obtain the exact same test score

Generalizability Study

In Generalizability Theory, it examines how generalizable scores from a particular test are if the test is administered in different situations

Decision Study

In Generalizability Theory, developers examine the usefulness of test scores in helping the test user make decisions

Item-Response Theory (Latent-Trait Theory)

Provides a theory way to model the probability of that a person with X ability will be able to perform a level of Y

Latent Trait

Unobservable Trait

Manifestation Trait

Observable Trait

Discrimination

In Item-Response Theory, an item differentiates among people with higher or lower levels of the trait, ability, etc., that is being measured

Dichotomous Test Items

Test items with only two possible responses

Polytomous Test Items

Test items with more than two responses

Standard Error of Measurement

Provides an estimate of the amount of error inherent in an observed score of measurement

Standard Error of Score

Index of the extent to which one individual’s score vary over tests presumed to be parallel

Confidence Interval

A range or band of the test scores that is likely to contain the true score

68% confidence interval (±1 SEM)

This means there's a 68% chance that the individual's true score falls within that range

95% confidence interval (±1.96 SEM

often approximated as ±2 SEM)