Ch. 5 Notes: Reliability in Psychological Testing

Flashcards covering key concepts of reliability, Classical Test Theory, methods for estimating reliability, and factors influencing it, based on Chapter 5 notes.

What is reliability in the context of psychological testing?

It refers to the consistency of scores; if re-testing, it means obtaining similar results.

What is a reliability coefficient?

It is a preferred statistical measure to show a test's consistency in measuring each person in the same way, every time it's administered.

How can the number of items affect a test's reliability?

Adding more items that measure the same construct can increase reliability.

What value represents perfect reliability?

1.0.

According to Classical Test Theory, what two components make up a person's test score?

A true score (T) and random error (E).

In Classical Test Theory, what is a true score?

It is a measure of the amount of the trait that a test is designed to measure, assumed to be infinite and universal.

What is random error (E) in Classical Test Theory?

It causes a person's test score to vary from one administration of the test to the next, is defined as the difference between a person's observed score and their true score, and is assumed to cancel itself out after repeated testing.

What are the characteristics of random error according to Classical Test Theory?

It is normally distributed and uncorrelated with the true score.

What is systematic error?

A single source of error that consistently increases or decreases a person's true score by the same amount, such as practice effects or order effects.

Which method of estimating reliability involves giving the same test to the same group on two separate occasions?

The test-retest method.

What is the term for test-takers benefiting from practicing a test if given multiple times?

Practice effect.

What does the parallel forms (or alternate forms) method of reliability involve?

Administering two or more different but equivalent forms of a test.

What are order effects in testing?

Changes in test scores resulting from the sequence in which test items are presented.

What does internal consistency reliability measure?

How related the items on a test are to one another.

How is reliability estimated using the split-half method?

The test is split into two halves, and the scores on one half are compared with the scores on the second half.

Which formula is used to adjust for the halving of a test in the split-half method?

The Spearman-Brown formula.

What is the difference between homogeneous and heterogeneous test items?

Homogeneous items measure a single trait, whereas heterogeneous items measure multiple traits.

When are internal consistency reliability estimates less appropriate, and why?

For heterogeneous tests, because these tests measure multiple traits, resulting in lower expected reliability coefficients.

What does inter-scorer agreement measure?

The amount of consistency among scorers' ratings.

What does inter-rater agreement refer to?

The reliability of how consistently different scorers rate or make decisions.

What is a confidence interval in the context of test scores?

A range of scores within which we are confident the test taker's true score lies, typically at 68%, 95%, or 99.7% confidence levels.

What four main factors can influence the reliability of a test?

The test itself, test administration, test scoring, and test takers.

How does the length of the interval between test administrations affect the reliability coefficient?

Generally, the longer the interval between tests, the lower the reliability coefficient.

Who developed Generalizability Theory, and what does it concern?

Cronbach, Gleser, Nanda, and Rajaratnam (1972) developed it, and it concerns how and under what conditions one can generate an estimate of the reliability of test scores from one test administration to another.