Validity and Reliability

Validity

The extent to which an observed effect is genuine

Reliability

How consistent the findings from an investigation or measuring device are

Types of validity

• Population validity

• Internal validity

• Temporal validity

• External validity

• Ecological validity

Mundane realism

The extent to which the tasks and materials of the investigation resemble those of the real world

Population validity

The extent to which the results of an investigation can be generalised to other groups of the wider population

Internal validity

Extent to which the findings of the study are due to a change in the independent variable rather than extraneous variables

Temporal validity

The extent to which the findings of a study can be generalised to other historical times and eras

External validity

The extent to which the findings from a research study can be applied outside the context of that study

Ecological validity

The extent to which the findings from a research study can be generalised to other settings and situations

Ways of measuring validity

• Face validity

• Predictive validity

• Concurrent validity

Face validity

Measures whether a test looks like it measures what it set out to measure

Predictive validity

How well a study or test can predict future behaviour

Concurrent validity

Measures how closely two different tests of the same behaviour/skill agree with each other

How can validity be improved in experimental research? (2)

• Standardise procedures to limit investigator effects, and participant reactivity.

• Using single-blind or double-blind trials achieves the same aim.

How can validity be improved in observations? (2)

• Using covert observations.

• Ensuring behavioural categories aren’t too broad, overlapping or ambiguous.

How can validity be improved in questionnaires?(2)

• Using a lie scale to assess the consistency of answers.

• Submitting data anonymously to remove social desirability bias.

Types of reliability

• Internal reliability

• External reliability

Internal reliability

The extent to which a measure is consistent with itself

External reliability

The extent to which a measure is consistent over time

Ways of measuring reliability (2)

External reliability: Test-retest method

Internal reliability: Split-half method

Test-retest method

A method for assessing the reliability of a questionnaire or psychological test by assessing the same person on two different occasions.

Split-half method

• The researcher splits the test in half and analyses the responses given to the first half of the questionnaire compared to the second half of the questionnaire

• If similar responses are given in both halves, then internal reliability is established

What is inter observer reliability?

The extent to which there is agreement between two or more observers involved in observations of the same behaviour.

How to improve reliability of observations (3)

Ensuring behavioural categories are:

• operationalised

• measure only observable behaviour

• are distinct, with no overlapping or ambiguity

How to improve reliability of questionnaires (2)

• running the test-retest method and rewriting or excluding any questions which do not show consistency in scores

• replacing open questions with closed, fixed choice alternatives that are less ambiguous

How to improve reliability of interviews (3)

• using the same interviewer for each interview

• using trained interviewers (which is particularly important for the cognitive interview used with eyewitnesses)

• ensuring that leading questions or ambiguous, double-barreled questions are omitted from the questioning

How can reliability of an observation be assessed through inter observer reliability? (4)

• The two observers would agree on behavioural categories.

• The two would make independent observations

• The two would compare to check for agreement.

• The researchers would accept the +0.8 correlation as reliable.