slide deck 8: reliability of measure

modules: 1 ( measurement error), 2 ( reliability), 3 ( pearson's correlation coefficient), 4 (test-retest reliability), 5 ( alternative forms reliability), 6 (inter-observer reliability), 7 (internal consistency)

no perfect measure

even precise instruments contain measurement error

systematic error (bias)

consistent, predictable errors that shift all scores in one direction

random error (noise)

unpredictable fluctuations that vary from one measurement to another

measurement error

difference between the observed (measured) value and the true value of a variable

reliability

getting the same result under the same conditions

high reliability

there is no error

low reliability

there is an error; scores vary

Pearson correlation coefficient

statistical measure that calculates the strength and direction of a linear relationship between two continuous variables

measuring reliability

calculate Pearson correlation coefficient ( r ), between two sets of scores, by measuring the same construct using the same instrument.

interpreting Reliability (r)

• Strong/High reliability, r = .70 – 1.00

• Moderate reliability, r= .30 - .69

• No or weak reliability, r = 0 - .29

correlation

correlation near 0 means low reliability or no consistent relationship

test- retest reliability

repeatedly of a measure. will the test produce the same if given to same person under same conditions, at different times

practice effect

if given the same test, participants become familiar with format or content.

fatigue/ boredom effect

when given the same test twice, may become tired, distracted or less motivated during 2nd administration.

unstable construct

depression, happiness not stable over time, the longer the interval the less reliable the score.

cross language context

test takers speak different languages. to ensure fairness make the same test in different languages

different backgrounds or abilities

test takers have different education/ cultural background (verbal and non-verbal version)

alternate-form (parallel form) reliability

method of testing reliability by giving two different but equivalent versions of test (version a and b) to the same person different time

interrater (interobserver) reliability

reflects the degree of agreement or consistency between 2 or more observers who independently rate/ code the same behavior/ phenomenon.

inter- observer (interrater) reliability

degree of agreement between 2 or more independent judges who evaluate/ score the same behavior or event

reliability is low

judges may misunderstand the scoring, apply the rules inconsistently, rating categories may be vague or overlapping

increase the inter-rater reliability

define every construct operationally, include examples and counter examples, provide description for each level.

internal consistency

how well a set of survey questions or test items measure the same concept/ behavior.

homogeneity

items targeting the same trait produce more stable and reliable data

intern- item correlation

items should tightly group together rather than measuring unrelated concepts

split- half reliability

split all items two sets and compute correlation of the two sets

cronbach’s alpha

how closely related a set of items are as a group

how to increase internal consistency

add more high quality items that measure the same construct