Reliability

Reliability

How consistent a measuring device is. Includes psychological tests or observations with measure behaviour

Test-retest reliability

A method of measuring the reliability of a questionnaire or psychological test by assessing the same person on 2 separate occasions.

This shows to what extent the test produces the same answers on 2 separate occasions.

Inter-observer reliability

The extent to which there is agreement between 2 or more observers involved in observations of behaviour.

Measured by correlating the observations of 2 or more observers.

A general rule is that if (total number of agreements) / (total number of observations) > 0.80, the data has high inter-observer reliability.

How to establish reliability

Using a correlational analysis. In test-retest and inter-observer reliability, the 2 sets of scores are correlated.

The correlation coefficient must exceed +0.80 for reliability.

Ways to improve reliability

Questionnaires

Interviews

Observations

Experiments

Questionnaires

Reliability can be measured using the test-retest method.

Comparing 2 sets of data should produce a correlation that exceeds +0.80.

A questionnaire may require some questions to be removed or rewritten if there is low test-retest reliability. e.g. if some questions are complex or ambiguous they can be interpreted differently by different people.

Replace open questions with closed and fixed-choice alternatives which are less ambiguous.

Interviews

Best way of ensuring reliability is to use the same interviewer each time.

All interviewers must be properly trained so one particular interviewer is not asking questions that are too leading or ambiguous.

Easier to avoid in structured interviews - Interviewer’s behaviour is more controlled by fixed questions.

Unstructured interviews are less likely to be reliable.

Observations

Make sure that behavioural categories have been operationalised properly so they are measurable and self-evident.

Categories should not overlap and all possible behaviours should be covered on the checklist.

If categories overlap, are absent or not operationalised well, different observers have to make their own judgements of what to record where and may have different records.

If reliability is low, observers may need further training in using behavioural categories and wish to discuss decisions with each other so they can apply categories more consistently.

Experiments

The procedures are the focus of reliability.

To compare the performance of different participants the procedures must be the same every time.

An experimenter is concerned about standardised procedures.