Reliability, Validity and Error

What is meant by reliability?

the extent to which a measure is consistent or repeatable

What is test-retest reliability?

consistency of test overtime

measures whether the same results are obtained when the test is administered to the same group of people on two different occasions

How to assess test-retest reliability?

1. Administer the test at one point in time

2. Wait one day then administer the same test again to the same ppts

3. Correlate the scores from both tests using a statistical method (Pearson’s)

4. High correlation between the scores (close to 1) suggests high test-retest reliability

What is inter-rater reliability?

Level of agreement of scores between different raters or observers rating the same target

What is one issue with correlation?

correlation suggests are relationship not a cause

What is internal consistency reliability?

extent to which different items on a test are consistent with each other and whether different parts of the same test produce similar results

How do we assess internal consistency reliability?

1. Split-half technique

   • split items into 2 sets, compute scores for each set and determine relationship between them

   • r value of equal to or more then 0.8 = good internal consistency

What is meant by measurement error?

refers to the difference between the true value of a quantity and the value obtained by a measurement

it can affect the accuracy and reliability of measurements

What is meant by systematic error?

consistent, predictable errors that occur in the same direction every time

What is meant by random error?

unpredictable errors that vary in magnitude and direction

how can we improve these errors?

• calibration- regularly calibrating equipment to ensure accurate measurements

• standardisation- using standardised procedures and protocols to minimise variability

• training- ensuring all individuals performing measurements are properly trained and follow consistent techniques

What is the standardised error of measurement?

statistical measure that quantifies the amount of variability or dispersion of sample means around the population mean

What is validity?

extent to which a test or measuring tool accurately measures what its intended to measure

What are the types of validity?

• content

• concurrent

• face

• criterion

• construct

• predictive

• ecological

what is content validity?

extent to which a test or measurement represents all aspects of the intended content domain

assesses whether the test items adequately cover the topic or concept

what is face validity?

test appears to measure what it claims too

what is criterion validity?

assesses the performance of a test based on its correlation with a known external criterion or outcome

can be further divided into concurrent and predictive validity

what is concurrent validity?

extent to which a new assessment correlates with an existing, established measure os the same construct taken at the same time

what is predictive validity?

extent to which a test or measurement can accurately predict future outcomes or behaviours

what is construct validity?

degree to which a test measures a hypothetical construct

what is ecological validity?

extent to which the results can be generalised to real life