Reliability and Validity

What is reliability?

The consistency of a measure.

What is test-retest reliability?

The consistency of a measure over time.

What is inter-rater reliability?

The consistency of a measure across different individuals using the same measure.

What does a lack of reliability mean?

A lack of objectivity, leading to invalid results.

How to assess test-retest reliability.

1. Psychologist conducts a study using the measure

2. Psychologist waits an appropriate amount of time for participants to forget the details of the measure

3. Psychologists get the same participants to complete the same measure again

4. The two sets of results are put through a statistical test measuring correlation to see how similar they are

5. There must be a strong positive correlation of 0.8 or higher to suggest the reliability of the measure is high

How to assess inter-rater reliability.

1. Psychologist conducts a study where more than one person independently observes and classifies the same behaviour of the same participants at the same time

2. All the sets of results are put through a statistical test of correlation to see how similar they are

3. There must be a strong positive correlation of 0.8 or higher to suggest the reliability of the measure is high

How can external reliability be improved?

1. Effective operationalisation of variables

2. Training to ensure inter-rater reliability

3. Careful transportation of data

4. Use of standardised instructions

What is effective operationalisation of variables?

Objective measuring of DV or co-variables. (E.g. using behavioural categories)

How does operationalisation of variables improve reliability?

The more objective a measure is, the more likely it is to be interpreted the same way by participants and researchers. This means the same behaviour will be counted no matter who the measure is used by.

What does training to ensure inter-rater reliability mean?

Providing the same training to all researchers who will use the behavioural categories to ensure they know what category means and how to recognise them.

How does training increase reliability?

Each researcher will make their judgement on the same basis, making the results more objective. The more objective the judgment is, the more likely it is that different people will classify behaviours in the same way.

What does careful transportation of data mean?

To be meticulously when copying over the raw results from the data collection sheets.

What does ‘use of standardised instructions’ mean?

To use an unchanging, standardised set of directions on what participants should do in a study.

What is validity?

Whether the measure accurately measures what is intends to measure.

What is face validity?

When the measure subjectively appears to accurately measure what it means to.

What is concurrent validity?

When the measure corresponds to a previously established measure of the same behaviour.

How can face validity be assessed?

The researcher examines the measure closely and checks if there are obvious flaws. If not, the researcher can accept its validity.

How can concurrent validity be assessed?

The participants undertake the measure once. The researcher then uses a previously established measure of the same behaviour and gets the same participants to undertake this measure. The two sets of data are placed in a statistical test measuring correlation. There must be a strong positive correlation of 0.8 or above between the two test scores to suggest the validity of the measure is high.

In what ways can internal validity be improved?

1. Generally - tightly controlling the study

2. Single Blind Technique - participants are unaware of true aim of study

3. Double Blind Technique - both participants and researchers are unaware of true aim of study

What are the different types of external validity?

1. Ecological validity - results generalised to real life

2. Temporal validity - results generalised to different time periods

3. Population validity - results generalised to individuals beyond the sample

How can ecological validity be improved?

The situation in which the behaviour is being measured should be as similar to real-life as possible.

How can temporal validity be improved?

The situation in which the behaviour is being measured should not be context-bound.

How can population validity be improved?

Make the sample as representative as the target population as possible. (Stratified sampling)