Reliability: assessing and improving

What is reliability?

it refers to how consistent the findings from an investigation are. It is reliable if it produces consistent results every time

What’s the difference between reliability and validity?

Reliability = consistency; Validity = accuracy.

What are the two ways of assessing reliability?

test re-test and inter-observer reliability

What does test–retest reliability check?

Whether results are consistent over time.

What is test re-test and how is it done?

you administer the same test or questionnaire to the same person on different occasions. If a construct is reliable, similar results should be achieved every time

Where does test re-test commonly apply?

to questionnaires, psychological tests and interviews

What does inter-observer reliability check?

Whether two or more observers record behaviour in the same way.

What is inter-observer reliability and how is it done?

when researchers observe the same behaviour, independently, and then compare their data. If the data collected is similar then it is reliable

What value must the agreements be at to be reliable?

0.8

How do observers typically reach 0.8?

by doing the number of agreements/ number of observations

How can you improve reliability in questionnaires?

if a questionnaire appears to have low reliability (through test re-test, less than 0.8) then some of the questions should be removed or reworded. If questions are ambiguous, they should be turned into closed questions to make them more clear

How can you improve reliability in interviews?

the best way to improve reliability when using interviews is to use the same interviewer each time. If that is not possible or practical, the interviewers should be properly trained. They shouldn’t ask leading or ambiguous questions so structured interviews are good in improving reliability

How can you improve reliability in experiments?

lab experiments are described as reliable because researchers can control aspects of the procedure such as instructions or conditions. Experiments should be easy to replicate and participants should be tested under exact conditions each time

How can you improve reliability in observations?

make sure that behavioural categories have been properly operationalised (e.g turning aggression into kicking/punching/spitting). Categories shouldn’t overlap and all behaviours should be covered otherwise observers will report inconsistently