Notes on Causality, Evidence Hierarchy, and EBM

Causality, Evidence, and the Evidence-Based Medicine (EBM) Hierarchy

• Topic context: examining causality and evidence in medicine, and how EBM attempts to define the best way to establish causal claims.
• Key premise: evidence-based medicine promotes a specific methodological relationship with evidence, raising questions about its universality across disciplines.
• Barrowman (2014) point: some causal relationships may have no correlations; correlation is not necessary for causation in some cases.

Causality and Non-Correlation: Illustrative Examples

• Classic view: causality is often linked to correlation (covariation).
• Barrowman’s provocative claim: there are causal relationships without measurable correlation in some contexts.
• Example proposed in class (Causation without correlation): Burning fuel (oil) keeps a house warm. Inside-temperature changes may show little correlation with the amount of oil burned if there is a dynamic equilibrium.
  • This highlights how causal relationships can operate via mechanisms that keep a system at a stable level despite changes in a driver.
• Buffer solution example (Ilona’s suggestion): adding acid to a buffer causes chemical reactions (causal interactions) but may not change the overall pH due to a dynamic equilibrium (Le Châtelier’s principle).
  • Emphasizes that causal chains can exist even if an observable outcome (pH) remains constant.
  • Applications to biology: blood pH must be tightly regulated; radical pH changes are harmful.
• Additional buffer example (Expii/ScienceRevisions UK): adding acid to buffer may produce a chain of reactions with no net pH change, illustrating causal interactions that are kept in check by equilibrium.
• Takeaway: Causation and correlation can diverge; causal processes can be masked by system dynamics.

Evidence-Based Medicine (EBM) and the Big Picture

• EBM foregrounds a particular methodological path to evidence: not just correlations or simple experiments, but randomized controlled trials (RCTs) and aggregations of RCTs.
• Bradford Hill criteria historically influenced causality discussions in medicine, but they have limitations.
• Question raised: Is RCT-based evidence the very best form of science for all fields?
• EBM vision: a hierarchy of evidence that places RCTs and systematized reviews at the top; clinical judgment is redefined by high-quality evidence.

The Evidence Hierarchy: The Pyramid

• The hierarchy (one widely used version) includes eight levels:
  1. Systematic reviews & meta-analyses of RCTs
  2. RCTs: Randomized controlled trials
  3. Controlled trials without randomization (quasi-experimental)
  4. Single non-experimental studies (cases, correlations) – analytic
  5. Systematic reviews of descriptive studies
  6. Single descriptive study
  7. Authority & expert statements
  8. Mechanistic reasoning/explanations
• Note: There are several pyramid variants; some omit level 8, but many interpretations treat level 8 as a key target in mechanistic explanations.
• Visual cue: the top (level 1) is the strongest form of evidence; the bottom (level 8) includes mechanistic reasoning and expert opinion.

1) Systematic reviews & meta-analyses of RCTs

• Purpose: synthesize all RCTs on a treatment to obtain a broader, less biased estimate of efficacy.
• Rationale: aggregate data across trials to overwhelm biases from individual studies.
• Requires careful interpretation by human readers and interpreters; meta-analysis power and biases depend on study quality and heterogeneity.
• Ioannidis and related debates: strength and limitations of meta-analysis; the method’s power vs bias concerns.
• Key caveat: even systematic reviews can propagate bias if included studies are biased; quality control remains essential.

2) Randomized controlled trials (RCTs)

• Core idea: randomize subjects into treatment vs control groups to balance known and unknown confounders.
• Randomization aims to ensure that differences post-intervention can be attributed to the treatment rather than other factors.
• Blinding is typically used so researchers and participants do not know who is in which group, reducing bias.
• Terminology note: “controlled” typically refers to keeping all other conditions constant apart from the intervention; RCTs are often treated as quantitative with clearly defined outcomes.
• Historical touchpoints:
  • Fisher’s work on randomization (agriculture) laid groundwork for modern randomization concepts.
  • Bradford Hill popularized randomization in medical trials (1948); he emphasized it as a safeguard against bias.
• Context: RCTs are not without limitations; they require careful design to balance groups and to interpret results in light of real-world variability.

3) Controlled trials without randomization (quasi-experimental)

• Used when randomization is unethical or impractical (e.g., social interventions, public health policies).
• Still aims at causal inference, but with greater potential bias than RCTs.
• Examples include some public health and education interventions where random assignment isn’t feasible.

4) Single non-experimental studies (analytic)

• Includes cases, cohorts, cross-sectional studies; correlations, etc., sometimes with informal control groups or retrospective data.
• These are observational in nature and can be useful for exploring associations and generating hypotheses, but are more vulnerable to bias and confounding.

5) Systematic reviews of descriptive/qualitative studies

• Synthesize descriptive or qualitative data to draw broader inferences about phenomena.
• Descriptive studies can be large and global; they may capture broad patterns but are less tightly controlled.

6) Single descriptive study

• No intervention; no predefined comparisons.
• Can be long-term and provide useful information about effects over time, side-effects, genetic differences, etc.
• Serves as a basis for generating future experimental work.

7) Authority & expert statements (opinion, committee reports)

• Historically influential but criticized for over-reliance on expert opinion without empirical support.
• Not unique to medicine; similar issues arise in other fields (ecology, conservation biology, etc.).
• Recent debates highlight the need to ground expert opinions in higher levels of evidence when possible.

8) Mechanistic reasoning/explanations

• Involves causal mechanisms: what components are involved, what they do, and in what order.
• A mechanistic account explains the “how” and “why” behind observed outcomes, filling in the “black box.”
• In RCTs, mechanistic explanations are often treated as weaker evidence because they are inferences, not observed effects.
• Debate: mechanistic explanations can be strong (informing trial design, interpretation, generalization) but can also mislead if the mechanism is context-specific or speculative.
• Example visualization: a speculative mechanism for ivermectin’s action against COVID-19; such diagrams illustrate mechanistic thinking but may lack robust evidence and can be retracted when data contradict them.

The Oxford EBM Table and the Place of Mechanisms

• The Oxford table (major EBM center) shows mechanisms and case-based reasoning at lower levels; still acknowledged as part of the broader evidence landscape.
• The bottom levels historically include case studies and mechanism-based reasoning; the placement of mechanisms in the pyramid has been debated.
• Acknowledgment: we will revisit mechanisms in Week 6; some argue that mechanisms should be integrated higher in the evidential hierarchy.

The Historical and Philosophical Context

• EBM has been framed as part of a broader shift between rationalist (mechanistic explanations) and empiricist (data-driven) approaches in medicine.
• Historical pendulum: medicine has swung between emphasis on explanatory mechanisms and emphasis on observation and data, sometimes depicted as a pendulum.
• Some readings argue that EBM represents a late-empiricist stage; others argue for a more integrated view that includes both mechanism and data.
• For further reading: discussions on rationalism vs empiricism, and the possible paradigm shifts in medicine.

What Is Evidence? Data, Information, Knowledge, and Evidence in EBM

• Core distinctions (Damman & Smart, 2019; summarized in lecture slides):
  • Data: symbols (numbers, text, images, sounds) representing phenomena.
  • Information: data in a specific context, processed for a purpose.
  • Evidence: information used to evaluate explanations, models, hypotheses; comparisons across bodies of information against explanations.
  • Knowledge: evidence-based understanding, justified by evaluation and community consensus; broader than facts.
• Important distinction: data, information, evidence, and knowledge are related but not identical; evidence in EBM is about the evaluation of explanations, often requiring comparison and synthesis across studies.
• Methods and evaluation interplay with evidence: methods gather data; evaluation judges how well data support explanations.

Why a Pyramid? Qualities of Evidence Across Levels

• The pyramid is often portrayed because high-quality evidence is rarer and more resource-intensive to obtain than lower-level evidence.
• Common questions: what makes evidence strong or weak? What counts as bias, and how does it influence the hierarchy?
• The answer is not fixed: criteria for strength and weakness can vary (quality, bias, relevance, generalizability).
• The hierarchy is a heuristic, not an absolute rule; context matters for what counts as appropriate evidence in a given field.

Bias, Causality, and the Role of Randomization

• Randomization is designed to identify and mitigate confounding factors by balancing unknown differences across groups.
• Eliminative induction: a logical approach to ruling out alternatives to isolate the effect of interest (e.g., “not this, not that, only this”).
• Limits of randomization: perfect balance is never guaranteed in the real world; actual RCTs may deviate from ideal conditions.
• Some contexts may benefit more from case studies and mechanistic reasoning than from RCTs alone; broadening evidence sources can be valuable in certain decision contexts.

Limits and Critiques of the Hierarchy

• Borgerson’s critique (2009):
  • The hierarchy may fail to rank methods that are better at identifying causality or less biased.
  • The hierarchy may overstate the objectivity of higher levels and understate the value of mechanistic reasoning and other approaches.
• The strongest claims of EBM (the “strong version”): only randomized trials establish genuinely causal relationships; lower levels capture mere correlations.
• In essays and debates, distinguishing between assertion (claims without reasoning) and argument (claims supported by reasoning and evidence) is crucial.
• There are calls for a more nuanced approach: EBM+ advocates incorporating mechanistic studies alongside probabilistic evidence to improve causality inferences.

Meta-Analyses, Systematic Reviews, and Their Critiques

• Meta-analyses and systematic reviews sit at the top of the pyramid but raise concerns:
  • They pool biases of individual studies, potentially amplifying biases if the included studies are flawed.
  • Some accompanying summaries or press materials steer readers away from critical engagement with the data.
• Notable cautionary examples: homeopathy and ADHD meta-analyses illustrating how biased selections can mislead conclusions.

Supplementary Readings: Expanding the Debate

• Cartwright (2007): Are RCTs the gold standard? A nuanced defense of when RCTs work well and when they do not; introduces hypothetico-deductivism and other philosophical concepts.
• Greenhalgh et al. (2022): Advocates for EBM+; argues that RCTs alone are not sufficient for decision making, especially in rapid health decisions; suggests integrating mechanistic studies.
• Jureidini & McHenry: Critiques of industry-funded trials and the broader RCT paradigm; highlights external (non-methodological) influences on trial outcomes.
• Streptomycin TB example (Clarke et al.): Demonstrates how RCTs can mislead if mechanistic resistance is ignored; illustrates need for mechanism-aware interpretation.
• Worrall and other philosophers’ contributions: discuss limits of randomization and the role of explanatory reasoning in identifying causality.

Practical Implications: How to Use Evidence Wisely

• RCTs are essential but not sufficient in many cases; a broader evidence base improves decision making, especially in public health and complex interventions.
• Public health perspectives argue for broader evidence sources beyond RCTs when deciding on population-level interventions.
• In medicine and health policy, a balanced approach (EBM+) that includes mechanistic reasoning, observational data, and tailored context is increasingly advocated.

Writing and Academic Practice

• Thesis and argumentation: when outlining essays, start with a clear thesis, then structure arguments and counterarguments, citing main readings and supplementary readings.
• Distinguish between data, evidence, and knowledge; cite primary sources rather than relying solely on lectures.
• Use a structured outline, avoid overreliance on passive constructions, and allow time for revision and tutor feedback.
• Extensions and special considerations: academic extensions exist; use special considerations for longer deadlines when needed.

Concluding Outlook: Week 6 and Beyond

• The course will explore Big Data and its relationship to theory and causal explanation, continuing the debate about how best to establish causality across disciplines.
• The ongoing conversation about EBM, its hierarchy, and the role of mechanisms suggests a pluralistic, context-sensitive approach to evidence in science and health.

Key Terms and Concepts (Glossary)

• Evidence-Based Medicine (EBM): a movement promoting the primacy of randomized trials and their aggregations as the basis for clinical decision-making.
• Bradford Hill criteria: a set of considerations used to infer causality from observational data, including temporality, strength, consistency, specificity, dose-response, plausibility, coherence, experimentation, analogy.
• Randomized Controlled Trial (RCT): a study design in which participants are randomly assigned to a treatment or control group, often with blinding and predefined outcomes.
• Quasi-experimental: non-randomized controlled trials used when randomization is unethical or impractical.
• Mechanistic reasoning: explanations that describe the biological or physical mechanisms by which an intervention produces an effect.
• Eliminative induction: a method of reasoning that rules out alternative explanations to isolate a causal claim.
• Mechanistic vs probabilistic explanations: mechanistic explanations describe how a system works; probabilistic explanations describe the likelihood of outcomes given certain factors.
• EBM+: an approach that integrates mechanistic and probabilistic evidence to inform causal inferences.
• Bias: systematic error that distorts study results; includes selection bias and ascertainment bias; mitigated by allocation concealment and blinding.
• Meta-analysis: a statistical technique for combining results from multiple studies to derive a pooled estimate of effect.
• Systematic review: a comprehensive, pre-planned review of the literature using explicit methods to minimize bias.
• Knowledge, Information, Data: Data are raw symbols; information is data in context; evidence is information used to evaluate explanations; knowledge is justified, consensus-based understanding.