1.2 Reliability Estimates

Test-Retest Reliability

Reliability obtained by correlating pairs of scores from the sample people on two different administrations

Coefficient of Stability

In test-retest reliability, the longer the time that passess, the greater the likelihood that the reliability coefficient will be lower

Pearson Product-Moment Correlation Coefficient (r)

Typically calculated between the scores from the two administrations in test-retest reliability

Carryover effects

Happens when test-retest interval is short, wherein the second test is influenced by the first test because the content of the first test was remembered

Practice Effect

Happens in test-retest wherein scores on the second session are higher due to their experience of the first session of testing

Test Sophistication

In test-retest, items are remembered by the test takers, especially the difficult ones/items that got highlighted as confused

Test Wiseness

In test-retest, it might inflate the abilities of test takers

Mortality

Problems in absence in second session oof the test-retest session (just remove the first test of the absents)

Parallel Forms Reliability

Reliability in which item sampling and other errors have affected scores on versions of the same test

Parallel Forms

Each form of the test means that the variance of observed test scores is equal

Alternate Forms Reliability

Reliability that is an estimate of the extent to which these different forms of the same test have been affected by item sampling error, or other errors

Alternate Forms

Simple, different versions of the test that have been constructed so as to be parallel

Counterbalancing

A technique to avoid carryover effects for parallel forms, by using different sequence for groups

Pearson product-moment correlation coefficient (r)

In parallel forms, what is coefficient is calculated using the scores on the two parallel forms?

Internal Consistency (Inter-Item Consistency)

Degree of correlation among all the items on a scale

Homogeneity

Single factor test measure

Heterogeneity

Multiple factor test measure

Kuder-Richardson Formula 20

Used exclusively for tests where items are dichotomously scored (i.e., items with only two possible responses, such as correct/incorrect, true/false, or yes/no) with different degrees of difficulty

KR21

Dichotomous items

Cronbach’s Alpha (α)

Arguably the most widely used and reported measure of internal consistency, especially for tests with items that are scored on a continuous scale (e.g., Likert-scale items, multiple-choice items with partial credit) ●

McDonald’s Omega (ω)

A measure of internal consistency reliability, similar to Cronbach's Alpha, but often preferred for its ability to handle more complex factor structures

McDonald’s Omega (ω)

It is a way to assess how well the items on a questionnaire or scale consistently measure a single underlying construct

Average Proportional Distance

A measure used to evaluate internal consistency of a test that focuses on the degree of difference that exist between item scores

Split-Half Reliability

Reliability that correlates two pairs of scores obtained from equivalent halves of a single test administered once

Spearman-Brown Formula

Estimates internal consistency reliability from a correlation of two halves of a test

Spearman-Brown Prophecy Formula

Estimates how many more items are needed in order to achieve the target reliability

Steps of the Spearman-Brown Formula

1. Divide test into equivalent halves 2. Calculate the correlation coefficient (Spearman’s rank correlation or Pearson correlation if the data is interval/ratio) between the scores of the two halves 3. Adjust the half-test reliability using Spearman-Brown formula

Rulon’s Formula (Flanagan-Rulon formula)

Counterpart of the Spearman-Brown formula, a method for assessing the consistency of a test by comparing the scores on two halves of the test

Steps of Rulon's Formula

1. Split the test: Divide the test into two equivalent halves (e.g., odd and even questions) 2. Calculate variances Calculate the variance of the differences between each person's scores on the two halves Calculate the variance of the total scores for each person 3. Apply the formula

Odd-Even Reliability

Reliability that assigns odd-numbered items to one half of the test and even-numbered items to the other half

Interrater Reliability

Reliability that involves the degree of agreement or consistency between two or more scorers with regards to particular measure

Coefficient of Interrater Reliability

Coefficient that is the way of determining the degree of consistency among scorers in the scoring of a test is to calculate a coefficient of correlation

Kappa Statistics

Used formula for nominal data

Fleiss & Cohen's

Types of Kappa Statistics

Fleiss Kappa

Agreement between multiple raters (three or more)

Cohen’s Kappa

Agreement between two raters

Kendal’s W

Used for rankings or ordinal data