1/23
What is reliability?
The consistency of a measure - if it produces similar results each time.
What are the two types of reliability?
Test-retest reliability
Inter-observer/rater reliability
What is test-retest reliability?
The same test or interview is given to the same participants on two occasions to see if the same results are obtained.
There must be sufficient time between the test and the retest
What should the correlational coefficient be to show good high reliability?
0.8 or above
How do you check test-retest reliability?
The researcher gives a test/questionnaire to a group of participants.
The SAME participants complete the SAME test again on a different occasion (e.g. one month later).
Each participant's first and second scores are correlated and can be plotted on a scattergram.
The strength and direction of the correlation are assessed using an appropriate statistical test (Spearman's Rho/Pearson's r).
The correlation coefficient is compared against a statistical table to determine whether the correlation is statistically significant.
A strong positive correlation (+0.80 or above) suggests good test-retest reliability, meaning the test produces consistent results over time.
What is inter-rater reliability?
The extent to which there is agreement between two or more observers involved in observations of a behaviour.
What is the correlation ‘test’?
Two sets of data are compared to identify whether there is a relationship between them.
Spearman's Rho or Pearson's r is used to calculate a correlation coefficient.
The coefficient ranges from -1 to +1, indicating the direction and strength of the relationship (+0.80 or above indicates good inter-rater reliability).
Statistical tables are used to determine whether the correlation is statistically significant.
Improve reliability in observations
Operationalise all behavioural categories so that they’re measurable and self-evident
Ensure that the categories do not overlap (e.g. hugging and cuddling), as different observers have to make their own judgement of what to record, leading to inconsistent records
Improve reliability in questionnaires
Low test-rest may be due to poorly written questions, researchers need to reduce ambiguity
Removing open questions (can cause misinterpretation) and fixed choice questions (less ambiguous) may increase reliability
Improve reliability in interviews
Ensure that the interview is structured
Ensure the same interviewer is used every time
Interviewer should be trained not to ask leading questions
Improve reliability in experiments
Ensuring procedures are standardised
So that performance of different participants can be easily compared
Use a laboratory experiment - more control over extraneous variable
What is validity?
The ability of a test to measure what it is intended to measure, i.e. its 'truthfulness', if it can be generalised beyond the research setting within which it was found.
What are the types of validity?
Internal
External
Ecological
Temporal
Face validity
Concurrent validity
What is internal validity?
Whether the effects observed in an experiment are due to manipulation of the independent variable and not some other factor.
A threat to internal validity is that participants may respond to demand characteristics.
What is external validity?
A measure of whether data can be generalised to other situations outside the research environment, such as other settings, other populations of people and other eras.
What is ecological validity?
The extent to which findings from a research study can be generalised to other settings and situations - predominately real life.
What is temportal validity?
The extent to which findings from a research study can be generalised to other historical times/eras.
What is face validity?
The extent to which test items look like what the test claims to measure. Conducted by a specialist in the given area, if they believe the instrument is valid, this is a good indicator of validity.
What is concurrent validity?
Establishing validity by comparing it to an existing test or questionnaire with the one you are interested in. If the results are very close/match those obtained on another recognised and well-established test, that can be an indicator of validity.
What is mundane realism?
Whether the task used to measure the dependent variable represents real world events/use.
Improve validity in observations
Covert observations can increase validity, as the behaviour is more likely to be natural and authentic
Ensure that behavioural categories do not overlap, are not ambiguous and are clearly defined (operationalised)
Improve validity in questionaires
Incorporate lie scales (to identify participants who may be dishonest) to eliminate demand characteristics such as social desirability bias
Tell participants that results will remain anonymous so participants are less likely to give socially desirable answers
Improve validity in experiment
Ensure control groups are present to assess whether the changes of the DV is definitely due to the effects of the IV
Standardised procedures to minimise participant reactivity and investigator bias (double and single blind procedures)
Improve validity in qualitative methodology
Direct quotes - demonstrate interpretative reality (the extent to which the researcher's interpretation of events matches those of their participants)
Triangulation - using various different resources for evidence (interviews with family members, diaries and observations.)