1.1 Reliability

Psychometric Properties

Refer to the technical qualities and characteristics of psychological tests and assessments that determine their scientific soundness and practical usefulness

Psychometric Properties

These properties are crucial for ensuring that a test is measuring what it intends to measure, that its results are consistent, and that the scores can be meaningfully interpreted

Reliability

Consistency in measurement

Reliability Coefficient

An index of reliability, a proportion that indicates the ratio between the true score variance on a test and the total score, ranging from 0 to 1

More reliable

The greater the proportion of the total variance attributed to the true variance, the test is considered…

True Score Formula

True Score = Rxx(x − x) + x

Goals of Reliability

Estimate errors in psychological measurement. Device techniques to improve testing so errors are reduced

Classical Test Theory Reliability Formula

True variance/True score variance + error variance

Classical Test Theory

A score on an ability is presumed to reflect not only the test taker’s true score on the ability being measured but also the error

Error

The component of observed test score that does not have to do with the test taker’s ability

Foundational Equation of the Classical Test Theory

X = T (true score) + E (error)

Variance

Useful in describing sources of test score variability

True & Error

Types of variance

True Variance

Variance from true difference

Error Variance

Variance from irrelevant random sources

Measurement Error

All of the factors associated with the process of measuring some variables, other than variables being measured

Random Error

Source of error in measuring a targeted variable caused by unpredictable fluctuations and inconsistencies of other variables in the measurement process. It is unavoidable

Systematic Error

Source of error in measuring a variable that is typically constant or proportionate to what is presumed to be the true value of the variable being measured. It is avoidable if corrected

Test Construction & Test Administration

Sources of Error

Item Sampling/Content Sampling

Variation among items within a test as well as to variation among items between tests to avoid errors in test construction

Test Environment

Variables such as room temperature, level of lighting, and the amount of ventilation and noise that can contribute to errors in test administration if not controlled

Test taker variable

Variables in the test taker such as emotional problems, physical discomfort, lack of sleep, effects of drug, formal learning experiences, casual life experiences, therapy, illness, and changes in mood or mental state that can contribute to errors in test administration

Examiner-related variables

Variables such as the examiner’s physical appearance and demeanor, nonverbal gestures, and professionalism that can contribute to test administration errors on the test administrator's part

Internal Reliability & External Consistency

Two types of reliability

Internal Reliability

Type of reliability that has consistency within itself (e.g., “I hate Mondays” and “I don’t like Mondays” are worded differently but have one meaning)

External consistency

Type of reliability that shows how results compare between individuals and across time (e.g., pretest and posttest, interrater difference)