assessing and improving reliability and validity

reliability

• the extent to which a test or measurement procedure has consistent results - if a test is repeated using the same method, design etc. and gets the same results every time then it can be said to be reliable.

• internal reliability - the extent to which something is consistent within itself.

• external reliability - the extent to which a test measures consistently over time.

assessing reliability of observations

• inter-observer reliability - two or more observers carry out an observation and their observation schedules or records are compared to see if they are similar by correlating the observer’s scores.

• if there is a correlation of 0.8, then you can say you have inter-observer reliability.

improving observational reliability

• always have more than one observer.

• have clearly defined/operationalised and separate observational criteria.

• train observers so they know exactly what to look for.

• do a pilot study so you know the observers are applying the observational categories properly.

assessing reliability of self-report techniques

• split-half method - assess internal reliability by splitting a test in two and making the same participants do both halves. If the results are the same for both halves then it indicates the test has internal reliability.

• test-retest method - assesses external reliability by giving participants the same test on two occasions with normally a week or two apart, so they don’t remember the answers. If results are the same then external reliability is established. This can also be done with interviews to test the reliability of the interviewer.

improving reliability of self-report techniques

• make questions clear and precise, e.g. closed and more reliable than open.

• pilot a questionnaire beforehand to check if the questions are clear enough.

• use the same interviewer with each participant or fully train if using more than one.

assessing reliability of experiments

• see if procedure, instructions and conditions are all carefully controlled, and that participants were all tested under the same conditions.

• ways to measure the DV.

improving reliability of experiments

• use exactly the same procedures for all participants.

• use the same conditions for all participants.

• if repeated by other researchers then it needs to be replicated exactly the same way as the original.

validity

• the degree to which something measures what it claims to - ‘legitimacy’ or ‘accuracy’.

• internal validity - whether the results are due to manipulation of IV and not confounding variables.

• external validity - the extent to which results can be generalised to other settings.

• population validity - whether results can be generalised to other groups of society.

• temporal validity - whether results can be generalised beyond the time period of the study.

• ecological validity - whether results can be generalised to real-life settings.

assessing validity

• to know if what you are measuring is actually measuring what you think it is.

• face validity - a quick eyeballing or intuitive measure, where you look over the test to see if on the face of it is measuring what you think it is.

• concurrent validity - when a well-established, validated test is used to compare with your new test, and participants take part in both tests and if they get similar scores then this confirms the concurrent validity of the test.

improving validity of questionnaires

• review questionnaires or tests if when assessed they have low face or concurrent validity.

• assure responses are anonymous.

improving validity of experimental research

• use a control group so that the researcher is better able to assess whether changes in the DV were due to the the IV.

• standardise procedures to reduce investigator effects and participant reactivity.

• reduce demand characteristics by using double-blind or single-blind procedures.

improving validity of observations

• tend to have high ecological validity, especially if covert.

• make sure that behaviour categories are not too broad, overlapping or ambiguous.

improving validity of qualitative methods

• interviews and case studies that produce qualitative data are said to be higher in ecological validity because of the depth and detail involved in reflecting the participants’ reality.

• interpretive validity (extent to which researcher’s interpretation of events matches the participant’s) can be demonstrated by using direct quotes and being coherent in reporting.

• triangulation can be used to improve validity using different sources as evidence, e.g. interviews with friends, family, personal diaries, observations etc.