Week 4 Module 2 Reliability

What is reliability?

The extent to which a measurement is consistent and free from error

Reproducibility

Dependability

Measurements are rarely perfectly reliable- all instruments are _______ to some extent

Fallible

What is systematic error?

Predictable errors of measurement

Constant and biased

Occur in one direction, consistently over or under the true scoring

What is random error?

Due to chance

Can affect a measurement/score in an unpredictable way from trial to trial

Reliability focuses on the degree of…

Random error that is present within a measurement system

What are the three main sources of measurement error?

The individual taking the measurements (the tester or rater)

The measuring instrument

Variability of the characteristic being measured

Sources of error can be limited by…

Careful planning

T raining

Clear operational definitions

Inspection of equipment

What should clearly describe the method of measurement and how it was performed across trials?

Testing protocol

What are some sources of error that are difficult to control?

Background noise and random fluctuations in instruments

Personal characteristics

Environmental characteristics

A construct or response that is inherently unstable

What does ‘regression toward the mean’ mean?

Over time and multiple measurements, we should expect an extreme measurement/score to move closer, or regress, toward the group average

What is the ‘regression fallacy‘?

It’s when someone incorrectly attributes a return to the normal scores to be specific to the intervention, while ignoring the natural tendency of extreme values to regress towards the mean

What is variance?

A measure of the variability or differences among scores within a sample

The larger the variance the…

Larger the dispersion of scores

The smaller the variance the…

More homogenous the scores

What are the levels of reliability (value wise)?

Poor reliability- <0.50

Moderate reliability- between 0.50 and 0.75

Good reliability- > 0.75

What is a correlation?

It reflects the degree of association between two sets of data

Correlation is affected by _______ error but not _______ error

Random, systematic

What is the essence of reliability?

It is agreement between two assessments and not just correlation

What is test-retest reliability?

Used to establish that an instrument is capable of measuring a variable with consistency

A reliable instrument should achieve similar results with repeated administrations of the test

What is rater reliability?

Estimates the consistency with which a single or group of ragers applies the measurement method

What are the different types of rater reliability?

Intrarater reliability

Interrater reliability

What is intrarater reliability?

The stability of data recorded by one individual across two more trials

What is rater bias?

Is a person’s second observation influenced by their first observation

What is interrater reliability?

Variation between two or more raters who measure the same group of subjects

What are alternate forms reliability?

Used to compare different versions of the same assessment

What is internal consistency?

The extent to which items measure various aspects of the same characteristic and nothing else; homogeneity of test items

What is split-half reliability?

Tests internal consistency by splitting a single test into two portions and administering to a single group at a single time poi t

What are reliability coefficients for internal consistency?

Cronbach’s alpha

Spearman-Brown prophecy statistic

Item-to-total correlation

What is minimal detectable difference? (MDD)

The smallest amount of difference that passes the threshold of error for a specific instrument and application

What is population-specific reliability?

Reliability established for one population cannot be automatically generalized to another population; established for one set of raters may not be generalizable to another set of raters

What is the generalizability theory?

Every individual score/measurement should be thought of as a sample from a universe of possible scores that have been obtained under the same testing conditions