Lecture Notes: Critical Thinking in Statistics (1.1–1.2)

The Sixth Sense for Thinking Statistically

• Six core principles to apply when evaluating statistics (the “sixth sense”):

  • Data beat anecdotes; belief is not a substitute for arithmetic. Always ask: Is there data backing the claim? Who are the authors/credentails? Is it peer-reviewed?
  • Aim for a bird's eye view (data from many) rather than a worm's eye view (one anecdote). Consider broad, representative data rather than a single story.
  • Beware the data source: where data come from matters. Distinguish random samples (more representative) from opt-in polls (potential bias).
  • Watch for lurking variables (alternative explanations). Confounding factors can explain apparent relationships.
  • Variation is everywhere; uncertainty is a given. Distinguish common-cause variation from special-cause variation; expect random error.
  • Conclusions are not certain; report and consider confidence/uncertainty. Use ranges (e.g., a
    $CI$) rather than a single point estimate when possible.
  • Data reflect social and political values. Measurement choices and what gets studied can change over time and across cultures.

• Examples used to illustrate the six sense:

  • Power lines and cancer debate: anecdotal claims vs large case-control studies; importance of distinguishing causation from correlation; media use of anecdotes vs data.
  • Birth month and height: large observational studies can show statistically significant differences that may be practically negligible; consider sample context and potential confounders.
  • Left-handedness and longevity: potential confounding (historical practices forcing hand-switching) can distort conclusions; sample composition matters.
  • OECD obesity graph: self-reported vs measured data differ; context of population and data collection method matters.
  • Bus arrival times: illustrated random variation vs special causes (accidents) and the need to separate them.
  • Mammograms and mortality risk: single figure vs confidence interval; snapshots vs over-time interpretation.
  • WEIRD and data representativeness: most data are from Western, educated, industrialized, rich democracies; avoid overgeneralizing to other populations.
  • Data sources like social media or app-based data can introduce sampling bias (who uses the platform).
  • Potholes app in a city can over-represent wealthier areas if uptake is higher there.
  • Lies, damn lies, and statistics: data sources matter; some numbers in the wild are wrong or misinterpreted.

• Key takeaway: develop a habit of asking what data exist, where they came from, who is included, what is being measured, and what might be missing or confounded.

Data provenance, sampling, and representativeness

• Data origins matter for trustworthiness:

  • Market research firms (random samples) vs website polls (opt-in, self-selected).
  • Random samples aim for representativeness; opt-in polls risk bias.
  • Reporting should clearly label polls as scientific or non-scientific when presenting results.

• Missing and unrepresentative data:

  • Historically, many studies focused on men; data sets may be WEIRD (Western, Educated, Industrialized, Rich Democracies).
  • Big data sources (social media, streaming subscribers) exclude non-users and can misrepresent populations.
  • Real-world examples show bias in data collection (e.g., pothole reporting skewed toward wealthier areas).

• Practical note on data credibility:

  • Data from reputable, randomly sampled sources are generally more trustworthy than opportunistic online polls.
  • Always check whether data are drawn from a random sample and whether the sample is representative of the population of interest.

Hidden variables and alternatives explanations

• Lurking variables can explain apparent associations:

  • Example: owning at least two cars correlates with longevity, likely due to SES/income confounding.
  • Confounding factors can masquerade as causal relationships.

• Observational studies vs experiments:

  • Observational evidence can show associations but not causation; experiments are needed for causal claims.

• Anecdotes vs data:

  • Individual stories are compelling but not reliable evidence of general patterns.

Variation, uncertainty, and how to read results

• Variation is inherent in data:

  • Random (common-cause) variation vs special-cause variation (e.g., accidents).
  • Expect bus arrivals to vary around a typical time; occasional deviations happen.

• Uncertainty in conclusions:

  • Example: mammogram study reported a
    $26\%$ reduction in mortality in a snapshot; best practice is to report a confidence interval around the estimate, not a single point value.
  • A single figure is less reliable than a range of plausible values.

• Data interpretation depends on context:

  • Does the reported difference matter in practice (practical significance) as well as statistically significant?

Data and values: social and political influences

• Data measurement reflects values and priorities:

  • How poverty, unemployment, homelessness, bullying, and workplace safety are defined and measured changes over time.
  • COVID-19 death counting practices changed during the pandemic, affecting death rate estimates.

• The role of funding and governance:

  • Longitudinal studies (e.g., Dunedin) can face funding pressures linked to political winds, which can influence research continuity.

• Questions to reflect on in reading and evaluating data:

  • Why do we hear so much bad news? How are perceptions formed?
  • Do funders influence research topics or methods?
  • Do media portray a lopsided view of reality?
  • How do we spot fake or misleading statistics in reporting?

• Reading and resources mentioned for critical scrutiny:

  • Guardian article: nine-point guide to spotting a dodgy statistic.
  • TED talk: three ways to spot a bad statistic.

Guidelines and worry questions for evaluating a study (overview)

• A two-page document titled Guidelines and Worry Questions for Evaluation of a Study is provided for Week 1:

  • Aim: build a structured approach to critically assess studies.
  • Eight steps in the guidelines, used to evaluate a study.

• Step 1: Determine the type of study. Types listed include: sample survey, experiment, observational study, anecdote, or combinations thereof.

• Step 2: Identify the seven critical components (to be stepped through). This section indicates there are eight steps total and that step 2 covers seven components.

• Note: The details of steps 3–8 are not listed in the transcript, but the framework emphasizes systematic evaluation of study design and evidence.

Short recap and upcoming focus

• Today and tomorrow: focus on reading the news critically and evaluating statistical claims in articles, which will feed into Assignment 1 (due end of Week 6).
• Readings and in-class discussions will cover identifying data sources, sampling, bias, and appropriate interpretation of results.

Reading and assignments reminder

• Article referenced for Week 1 page: critical reading of news items and statistical claims.
• Practice expectations: ability to distinguish data-driven conclusions from anecdotes, assess sampling, and interpret uncertainty.