Reliability

Psychological Testing Unit 5

Reliability

Consistency in MEASUREMENT, produces similar results

Reliability Coefficient

Quantifies reliability ranging from 0 (not at all reliable) to 1 (perfectly reliable)

Measurement Error

Inherent uncertainty associated with any kind of measurement

• Always apparent

• Estimations of a quantity may differ even when procedures are perfectly followed

True Score

Long term average of MANY measurements

• Free of carryover effects

• Can never be observed directly

• Tied to the measurement instrument used

• Factor of reliable tests

Interference factors of accurate measurement

1. Time elapses between measurements

2. Act of measurement itself

Carryover Effect

Measurement processes that alter what is being measured

Construct Score

A person’s standing on a theoretical value

• Independent of any particular measurement

• Factor of valid tests

Variance

How much individual scores differ from one another and from the group average

• More total variance attributing to true variance = more reliability

Random Error

Unpredictable fluctuations + inconsistencies of other variables in the measurement process

• Can INCREASE or DECREASE test scores unpredictably

• Random scores tend to cancel each other out

• Examples: Spontaneous thunderstorms, power outages, drop in test-taker’s blood pressure

Systematic Error

Influence test scores in a consistent manner

• Can consistently INFLATE or DELATE scores

• Because of this, it is more easily fixable

• Example: A 12-inch ruler is found to be 1/10th of an inch longer than 12 inches, so every measurement taken with the ruler is consistently 1/10th longer than 12 inches

Item/Content Sampling

Variation among items WITHIN a test + variation among items BETWEEN tests

Test-Retest Method

Using the same instrument to measure the SAME THING at two points in time in order to gauge the instrument’s reliability

Test-Retest Reliability

An estimate of reliability obtained by correlating pairs from the SAME PEOPLE on two different administrations of the same test

• INCREASE of time interval between administrations = DECREASE of correlation between scores

Coefficient of Stability

When the interval between testing is GREATER than six (6) months

Coefficient of Equivalence

Degree of the relationship between various FORMS of a test

• Example: A makeup exam where the questions are not all the same as the test you initially took

Parallel Forms

For each form of the test, the means and variances of the observed test scores are EQUAL

Alternate Forms

Different versions of a test that have been constructed to be parallel

• Typically designed to be equivalent in content and difficulty

Split-Half Reliability

Correlating two pairs of scores obtained from equivalent halves of a single test, administered ONCE

• Useful when access to administering two tests/one test twice is impractical

• Factors such as differing amounts of fatigue, test anxiety, and item difficulty make splitting a test difficult

Odd-Even Reliability

Assigning odd-numbered items to one half of a test and even-numbered items to the other half

Inter-item Consistency

Degree of correlation among all items on a scale

Inter-Scorer Reliability

Degree of agreement between two or more scorers with regard to a specific measure

• Often used when coding nonverbal behavior

• Potential bias is guarded against by using more than one scorer

Standard Error of Measurement (SEM)

Provides an estimate of the amount of error in an observed measurement

• Used to estimate the extent to which an observed score DEVIATES from a true score

• Inverse relationship between SEM and reliability: HIGHER reliability, LOWER SEM

Domain Sampling Theory

Seeks to estimate the extent to which specific sources of variation are contributing to the test score

Generalizability Theory

A person’s test scores VARY from testing to testing because of variables in the testing situation

Dichotomous Test Items

Questions that can only be answered with TWO answers – Yes-No, True-False, etc.