ensuring validity

Ways to minimise sampling error

• Choose your sample randomly

• Avoid bias

• Use proper sampling methods

• Use a large sample size

• Avoid/minimise non-response bias

• Conduct pilot (practice) studies

• Question reversal/rephrasing

Identifying misleading statistics

• Small sample size

• Biased sample

• Lack of source or transparency

• Graphs with misleading scales/axes

• Confusing correlation with causation

• Cherry-picked results

• Ambiguous or vague language

• No margin of error or confidence interval

• Over-generalisation

• Too good to be true

Principles of experimental studies required for a causal claim

• How were the participants put into groups?

• Were the variables of interest measured?

• Is the control group evident?

• Was a placebo used?

• Were the treatments compared between the treatment groups, or the treatment and control groups?

• Were ethical considerations taken into account?

• Who was the experiment conducted on?

• Is the explanatory variable evident?

• Is the response variable evident?

• What is the treatment group?

• Is blinding evident?

Sources of non-sampling bias

• Observational/measurement error

• Data entry/processing error

• Non-response bias

• Response bias

• Coverage error or under-coverage biased

Behavioural factors that influence non-sampling error

• Social desirability bias

• Lack of motivation or engagement

• Memory/recall bias

• Interviewer influence

• Fear of judgement or lack of trust

• Fatigue or survey length

• Question misinterpretation

• Conflict of interest

Reducing non-sampling error

• Use neutral language (avoid emotionally-charged language)

• Avoid leading questions (e.g. “Don’t you agree that…”)

• Keep questions clear and concise

• Ask one question at a time

• Provide balanced response options

• Include a “don’t know” or “prefer not to answer” option

• Avoid assumptions

• Use inclusive language (be mindful of diversity in gender, culture, background, etc)

• Randomise response order

• Focus on personal experience to make the study/data realistic

• Consider ethics

• Use mutually-exclusive responses (avoid answers that overlap)

• Ensure confidentiality and anonymity

• Increase response rates'

• Check and clean data carefully

• Shorten survey length