Reliability

Consistency in measurement; a reliable test yields similar results under consistent conditions.

What is reliability in psychological testing?

A numerical index showing the proportion of true score variance to total test variance.

What is a reliability coefficient?

A large portion of the score variance reflects true differences rather than error.

What does a high reliability coefficient indicate?

statistical measure of variability in test scores, which includes both true and error variance.

What is variance in testing?

The portion of total variance caused by actual differences in the trait being measured.

What is true variance?

The part of total variance caused by factors unrelated to the measured trait (random or systematic errors).

What is error variance?

Total Variance = True Variance + Error Variance

Formula: What makes up total variance?

The test is more reliable, with less interference from random or systematic error.

What does high true variance mean in a test?

Any influence on a score that is not part of what the test is trying to measure.

What is measurement error?

Random error and systematic error.

Two types of measurement error?

Unpredictable, inconsistent influences on test scores (e.g., testtaker's sudden illness).

What is random error in testing?

A testtaker experiencing a sudden blood pressure surge during testing.

What is an example of random error?

A consistent or proportional error that affects scores in the same direction.

What is systematic error?

A ruler that is always one-tenth inch too long, causing consistent mismeasurement.

Example of systematic error?

To reduce error variance and improve the reliability and accuracy of score interpretations.

Why is understanding error important in testing?

Test construction, test administration, and test scoring/interpretation.

What are the three main sources of error variance?

Variability in test scores caused by differences in the test items selected or constructed.

What is item or content sampling?

A testtaker may perform better or worse depending on which items were included in the test.

How does item sampling affect scores?

Scoring higher on a test that includes familiar or expected questions versus a different version of the same test

Give an example of item sampling error.

Environmental distractions like noise, lighting, or temperature affecting testtaker performance.

What is a source of error in test administration?

Fatigue, illness, emotional distress, or lack of sleep.

What are testtaker variables that can cause error?

Errors caused by the examiner deviating from test procedures or influencing responses.

What are examiner-related variables?

An examiner unintentionally nodding to indicate a correct answer during an oral test.

Example of examiner-related error?

Poor lighting, ventilation, or uncomfortable settings can distract the testtaker.

How can the testing environment introduce error?

Technical glitches, subjective judgments, or inconsistencies in applying scoring rules.

What is a source of error in test scoring?

When scorers must interpret open-ended or creative responses without clear criteria.

What increases error in subjective scoring?

Different scorers disagreeing on which creative block designs deserve credit.

Example of subjective scoring error?

By using standardized, computerized, or well-documented scoring procedures.

How can objectivity reduce scoring error?

To improve test reliability and ensure accurate, fair assessment outcomes.

Why is identifying sources of error variance important?

The consistency or stability of test scores over time, forms, or items.

What does reliability measure in psychometrics?

A value from 0 to 1 that reflects the proportion of true score variance in observed scores.

What is a reliability coefficient?

90% of score variance is due to true differences; 10% is due to error.

What does a reliability coefficient of 0.90 mean?

Inversely proportional — as reliability increases, error variance decreases.

What is the relationship between reliability and error variance?

Correlation between scores on two test administrations with the same individuals.

What is test-retest reliability?

For stable traits like intelligence or personality.

When is test-retest reliability appropriate?

Test-retest reliability when the time gap between administrations is over 6 months.

What is the coefficient of stability?

Longer intervals reduce reliability due to greater chances for error variance.

How does time affect test-retest reliability?

Forms with equal means and variances that measure the same construct.

What are parallel forms of a test?

Correlation between different but equivalent versions of a test.

What is alternate-forms reliability?

Reliability estimate between alternate or parallel forms of a test.

What is coefficient of equivalence?

Item sampling, motivation, fatigue, practice, and therapy effects.

What errors affect alternate-form reliability?

The degree of correlation among items on the same test.

What is internal consistency reliability?

When evaluating how well test items measure the same construct.

When is internal consistency most useful?

Correlating scores of two halves of a single test administered once.

What is split-half reliability?

Spearman-Brown formula.

What formula adjusts split-half reliability?

To estimate the reliability of a full-length test based on split-half data.

What is the Spearman-Brown formula used for?

A type of split-half reliability where odd-numbered items are compared to even-numbered ones.

What is odd-even reliability?

It may introduce bias due to fatigue, anxiety, or uneven item difficulty.

Why avoid splitting a test in the middle for split-half reliability?

For tests with dichotomous (right/wrong) items with varying difficulty.

When is KR-20 used?

When all test items are assumed to have equal difficulty.

When is KR-21 used?

A reliability index for tests with nondichotomous items, measuring internal consistency.

What is coefficient alpha (Cronbach's alpha)?

0.70–0.90 is acceptable; over 0.90 may indicate redundancy.

What is a good Cronbach's alpha value?

It generally increases — longer tests tend to be more reliable.

What happens to alpha as the number of items increases?

Internal consistency based on score differences between items.

What does Average Proportional Distance (APD) measure?

≤ 0.20 is considered excellent; ≤ 0.25 is acceptable.

What is an excellent APD value?

Agreement between two or more scorers on the same test.

What is inter-scorer reliability?

When tests involve subjective judgment (e.g., essays, creativity tasks).

When is inter-scorer reliability most important?

An estimate of how much a test score deviates from the true score due to measurement error.

What is Standard Error of Measurement (SEM)

SEM is inversely related to reliability — higher reliability means lower SEM.

How is SEM related to reliability?