5.0 reliability

Every measurement is confunded

• by error

Error can be due to

• Transitory effects

• Specificity effects

• Random effects

Transitory effects

• caused by personal or situational effects that change from trail to trail (from measure to measure).

• effects the performance of the whole session - makes u feel underperformed

Specificity effects

• caused by the content of the ítems.

Random effects

• other than transitory & specificity

• They cannot be attributed to either to personal, situational (trial) or specificity factors

Error in measurement example

Because every measurement is subject to error

• Observed score ≠ True score

Observed score ≠ True score

• We believe every that person has a true score, that is: a ‘true value’ for a given the attribute.

• We need to figure out how much observed scoresdeviate from those true scores.

• Classical Test Theory (CTT) attempts to do this.

A mathematical assumption is a

• proposition believed to be truth that will help us guide our actions

  • What follows are the main assumptions of the CTT.

 Scores follow a linear model:

is the

• observed score of person i on test j

Xi,j= observed score of subject i on test j → considering different tests with the same lenght & contents specifications

• forms or parallel forms

• true score of subject i

• error measurement for subject i on test j

• unknown

Note that Ti does not have a j subscript because

• the true score of a person is independent from the test form they are taking

ei, j relates to the

• person & not the test

Some examples of sources of error when administering tests:

• the test-take

• the test

• the application

From the test-taker

• lack of motivation

• lack of concentration

• tiredness, impulsiveness

• participant reactivity

From the test

• poorly worded items

• reverse-worded items

• test length

From the application

• unclear instructions

• time of the day

• uncomfortable exam room

CTT does not differentiate between different sources of error:

X i , j = Ti + ei,j

Ti = E (X i,j )

• The true score (Ti) equals the expected value of the observed score (Xi , j )

If we were to test person i 

• an infinite number of times to assess her true level on a trait

• the sum of all her observed scores added together would be equal to the true score (= true value on that trait).

If we were to test person i an infinite amount of times to assess her true level on a trait, the sum of all her observed scores added together would be equal to the true score (= true value on that trait). → WHY??

• Because we expect that random error will sometimes make a person’s score higher than it is, while other times it will make it lower than it is

• We expect that all errors added together, in the long run, will cancel each other out.

The expected value of the errors is equal to zero 

CTT formula

X = T + E

• 0

Given that the errors from a form are not systematic (so are random)

• they are independent from the True score

• 0

• the errors from a form are independent from the errors from a different form

• those errors are independent from the true score of a different test (other construct).

validity vs reliability

• poor validity

• good reliability

validity vs reliability

• poor reliability

• poor validity

validity vs reliability

• good reliability

• good validity

Reliability definition

• consistency of scores of a test

• to the extent to which the measure yields the same value (or score) when we measure the same subject repeatedly.

• is the ability to reproduce a measure consistently

If we could remove all sources of error

• we would have perfect reliability

To calculate reliability

• we need to choose a procedure to compare scores

Depending on which scores we choose to compare, we will talk about different types of reliability:

• Parallel or alternate forms reliability

• Test-retest reliability

• Inter-rater reliability

• Internal consistency

parallel or alternate forms → how it works

• We build two parallel (or alternate) forms of the same test, we administer to a sample and then we calculate the correlation between the scores.

• If correlation is high, we conclude that both tests are measuring the same thing

Parallel tests are 

• test that are similar in content

• that cover the same range and level of difficulty in the items

We assume that both tests (parallel tests) are measuring

• the trait that we intend to measure

In practice → parallel tests (we do)

• We don’t create alternate forms of a test just to check for reliability.

• We usually create them because we want several forms of the same test.

Example: Parallel or alternate forms

Test-retest reliability → how it works

• We test the same group of people twice, allowing some time to go by between the first and the second application of the test

• The correlation between the scores of both administrations indicate how stable the measure is in time

Test-retest reliability → The psychological trait (or ability) that we are measuring has to be

• stable in time

Test-retest reliability → The interval of time between first & second application needs to be

• long enough so subjects don’t remember what they answered in the first application

Test-retest reliability → it is important to control for

• learning effects

• practice

• variance due to personal situation of the subject

Example: Test-retest

Inter-rater reliability → how it works

• when using a measure that depends on the judgment of experts (for instance constructed response or neuropsychological scales with tasks)

• we assess reliability by the degree of agreement amongst raters

Inter-rater reliability → differences in ratings are to be

• expected

• as there will be more strict or more lenient raters

• we still need a certain degree of agreement to consider the measurement reliable

Inter-rater reliability → If we are dealing with nominal or ordinal data

• we need to introduce a new coefficient to assess agreement → Cohen’s kappa.

• Inter-rater reliability → For ratio or interval type data

• we can still use Pearson’s correlation to calculate the strength of the relationship among raters.

Example: Inter-rater reliability