Week 11: Reliability and validity

What is validity in research?

The extent to which an effect demonstrated in research is genuine, not produced by bogus variables, and not limited to a specific context.

What is internal validity?

In experimental research, it refers to the extent to which an effect found in a study can be taken to be genuinely caused by manipulation of the independent variable (IV).

What is a causal relationship in research?

A situation where manipulation of the independent variable directly causes changes in the dependent variable (DV).

What is an example of internal validity? Vincent and Lewycky’s (2009) insomnia treatment study

Vincent and Lewycky's (2009) 'Better Sleep' trial, a randomised controlled trial for insomnia treatment.

• Randomised controlled trial

• 5-week online treatment for insomnia

• 118 adults, randomly allocated to either:

• Online treatment

• Waiting list control group

-> 33% of pps dropped out

-> people with higher levels of insomnia dropped out

Name some threats to internal validity

1. Attrition/mortality

2. History

3. Sampling

4. Maturation

5. Testing and instrument issues

6. Participant expectancy

7. Demand characteristics

8. Observer/experimenter expectancy effect

What is attrition in research?

The dropout of participants from a study, which can threaten internal validity.

What does 'history' refer to in the context of internal validity?

Events that occur between measurements that may affect the results, such as political events or natural disasters.

How does sampling affect internal validity?

The method used for sampling can lead to non-random assignment of participants to conditions, affecting the validity of the results.

What is maturation in research?

Changes in participants over the course of the study that can affect the outcomes, particularly relevant in studies involving children.

What are testing and instrument issues?

Problems that arise from repeated testing using the same measures, which can lead to order effects and practice effects.

What is participant expectancy?

The effect of participants' expectations about what they think is supposed to happen in a study.

What are demand characteristics?

Cues in the study that help participants figure out what is expected of them.

What is the Hawthorne Effect?

When individuals modify their behaviour in response to their awareness of being observed.

What is the observer/experimenter expectancy effect?

When a researcher's expectations affect their behaviour and how participants respond.

What is the Rosenthal effect?

self-fulfilling prophecy where higher expectations placed on someone lead to improved performance, while lower expectations lead to worse performance

 

Name 3 ways of improving validity

1. Standardised procedures

2. Counterbalancing

3. Blinding

What is single blinding in research? Give some research examples

When either the participant or the researcher is unaware of the condition being tested.

• Meda et al.’s (2009) alcohol and driving simulator task: placebo, moderate level, high level

• Wan et el. (2018): aerobic exercise versus stretching control group in individuals with Schizophrenia: psychiatrist didn't know but pps did

What is double blinding? Give some research examples

When both the participant and the research assistant are unaware of the condition being tested.

• Standard for clinical trials (RCTs).

• Klaassen et al. (2013) caffeine and fMri scans.

• Weyantet al. (2018) the effect of a study drug (Aderall) on student working memory.

What is external validity?

The degree to which results generalise beyond the experimental context.

What is ecological validity?

The extent to which research findings can be generalised across different settings.

What is construct validity?

How well a test measures the psychological constructs that are abstract and not directly observable.

What is face validity?

The extent to which a measure is subjectively considered to be a plausible operationalisation of the conceptual variable.

→ The extent to which a measure appears, on the surface, to assess what it claims to measure

What is content validity?

The extent to which a measure contains all the parts related to the theory of what is being measured.

What is reliability in research?

The extent to which a measurement is reproducible or consistent over time.

What is test-retest reliability?

The correlation of people's scores at one time and at a later time.

What is inter-observer reliability?

The extent to which different researchers agree in their ratings or codings.

What is internal reliability?

The internal consistency of a test, where participants tend to score similarly across multiple items of a construct.

What is Cronbach's Alpha?

A measure of internal reliability that calculates how closely related a set of items are as a group.

→ Range from 0 - 1. 0.7 critical value (Kline, 1999)

Why is reliability and validity important in research?

They ensure the credibility of research, scientific rigor, and that findings do not conflict with each other.

What are the characteristics of experiments and RCTs? Is validity and reliability high or low?

• Lab-based and fully controlled

• Experimental manipulation of the IV

• Standardised procedures

• Random allocation of pps to conditions

-> Internal validity = high, external validity = low

What are correlational studies used for? Is validity and reliability high or low?

Used to determine if one factor is related to another factor.

–Non-manipulated variables

–Non-causal hypotheses.

-> Internal validity = low, external validity = high

Is validity and reliability high or low within quasi experiments?

• Random assignment into treatment and control groups: non-equivalent groups

• Full control over the IV

-> Internal validity = moderate, external validity = higher

What is the importance of questionnaires in research? What types of validity can be addressed?

They are frequently used data collection methods that can assess multiple traits and require validity checks.

-> Face validity: checking that the questions are measuring what they are intending to measure

-> Content validity: does the content of the questions fit the aim?

-> Criterion validity: can it distinguish between different levels of criterion

-> Construct validity: are we measuring the actual construct? e.g. depression