Systematic Reviews & Meta-analysis – N7000 Lecture

Objectives

• Describe various literature review types with focus on Systematic Reviews (SR) and Meta-analyses (MA)

• Identify purposes, similarities, and differences between SR and MA

• Understand principles of critical appraisal of SR/MA

Evidence Hierarchy

• Oxford Centre for Evidence-Based Medicine (2011) Levels of Evidence

  • Level 1: $\text{Systematic reviews of RCTs} \rightarrow \text{highest for therapy/benefits}$

  • Grading can be downgraded for: study quality, imprecision, indirectness, inconsistency, small absolute effects

  • Can be upgraded when effect is very large (dramatic effect)

• Practical takeaway: Well-conducted SR/MA of randomized controlled trials (RCTs) usually sits at/near the top of the evidence pyramid

Case Example (Medication Adherence)

• Community clinician notices poor adherence among chronic disease patients

• Wonders whether text-message reminders will help

• Literature search reveals:

  • Several RCTs with conflicting results

  • A narrative literature review

  • A systematic review

  • A meta-analysis

• Leads to questions:

  • How are these review types different?

  • How to reconcile conflicting primary studies?

  • Where do they sit in the evidence hierarchy?

Types of Reviews & Key Features

• Narrative Literature Review

  • Broad, potentially comprehensive description of a topic

  • Search strategy often not stated → not reproducible

  • No formal quality appraisal; synthesis is qualitative

• Systematic Review (SR)

  • A priori protocol registered (e.g., PROSPERO)

  • Transparent, reproducible search across multiple databases + grey literature

  • Pre-specified inclusion/exclusion criteria; dual independent screening

  • Formal risk-of-bias assessment (e.g., Cochrane RoB, JBI tools)

  • Synthesizes findings qualitatively (words) and may include MA

• Meta-analysis (MA)

  • Statistical pooling of quantitative data from studies included in an SR

  • Provides overall effect estimate (e.g., pooled $\text{OR}$, $RR$, $\beta$)

  • Increases statistical power, detects small but meaningful effects

  • Must assess heterogeneity & publication bias; conduct sensitivity analyses

• Integrative Review

  • Broader question; can include experimental, observational, and qualitative studies

  • Example: “Which interventions most improve treatment compliance in liver transplant recipients?”

• Scoping Review

  • Broadest mapping of key concepts, evidence volume, and gaps

  • Often precursor to SR; uses PRISMA-ScR reporting guidelines

  • Quality appraisal optional

• Umbrella Review

  • “Review of reviews” that synthesizes evidence from multiple SRs

• Meta-synthesis

  • Aggregates findings from qualitative studies to develop higher-order themes

At-a-Glance Comparison (based on slide table)

Fundamental difference: Narrative reviews lack the bias-reducing elements required in other review types.

Systematic Review: Step-by-Step Process

1. Define clinical question (often PICO format)

2. Draft & register protocol (defines methods a priori)

3. Comprehensive literature search

   • Multiple databases

   • Grey literature, conference proceedings

   • Reference list hand-search

   • Librarian assistance recommended

4. Study selection

   • Clear inclusion/exclusion criteria

   • Dual independent screening with consensus arbitration

5. Critical appraisal / risk-of-bias assessment

   • Tools: Cochrane RoB 2, JBI, Newcastle-Ottawa, etc.

   • Conducted independently by ≥2 reviewers

6. Data extraction

   • Standardized form; dual independent extraction

   • Captures study design, participants, interventions, outcomes, effect sizes

7. Synthesis

   • Qualitative: narrative description of similarities/differences

   • Quantitative (if appropriate): meta-analysis

8. Interpretation

   • Place findings in clinical context

   • Discuss limitations, heterogeneity, publication bias, applicability

9. Reporting

   • PRISMA 2020 checklist & flow diagram

Critical Appraisal Principles (Three Discrete Questions)

1. Are the results valid?

   • Methodological rigor, risk of bias, adherence to protocol

2. What are the results?

   • Effect size, precision (confidence intervals), consistency

3. Can I apply the results to patient care?

   • External validity, feasibility, cost–benefit, patient values

JBI Checklist for SR/MA (Condensed)

• Clear review question & appropriate inclusion criteria

• Suitable search strategy & adequate sources

• Dual independent appraisal & data extraction with error minimization

• Appropriate methods for combining studies (fixed vs. random effects)

• Assessment of publication bias (e.g., funnel plot, Egger test)

• Recommendations supported by data; identification of research gaps

Meta-Analysis Essentials

• Definition: Statistical synthesis of data from multiple studies to produce a pooled estimate of effect size

• Advantages

  • $\uparrow$ power ⇒ detect small effects

  • Resolves uncertainty when individual studies conflict

  • Quantifies heterogeneity

• Key Statistical Concepts

  • Effect measures: $\text{OR}$, $RR$, $HR$, $\text{SMD}$, mean difference

  • Models

  • Fixed-effect: assumes one true underlying effect

  • Random-effects: assumes distribution of true effects; accounts for between-study variability

  • Heterogeneity statistics

  • $Q$ test ((\chi^2))

  • $I^2$ (percentage of total variability due to heterogeneity); $I^2 = 50\%$ = moderate, >75\% = high

  • Publication bias detection

  • Funnel plot asymmetry

  • Egger’s or Begg’s tests

  • Sensitivity & subgroup analyses to explore robustness

• Interpreting Confidence Intervals

  • If 95% CI for an OR excludes $1.0$ ⇒ statistically significant

  • Narrow CI ⇒ precise estimate; wide CI ⇒ imprecision

  • Example (from slides): $\text{OR}=2.11\; (95\%\;CI\;1.52–2.93)$ indicates significantly higher odds of adherence with texting

Forest Plot Anatomy

• Horizontal line = CI for each study; square = point estimate (size reflects weight)

• Diamond at bottom = pooled estimate; width = pooled CI

• Vertical line = “line of no effect” (e.g., $OR = 1$)

• Visual tool for heterogeneity & direction of effects

Publication Bias

• Tendency for significant/positive results to be published more often than null results

• Consequence: Overestimation of intervention effects in SR/MA

• Detection

  • Funnel plot symmetry (should resemble inverted funnel if no bias)

  • Statistical tests (Egger, Trim-and-Fill)

• Slide schematic: inclusion of unpublished null studies can

  • Alter statistical significance

  • Change clinical relevance

  • Potentially reverse direction of pooled effect

Example Meta-Analysis: Text Messaging & Medication Adherence

• Source: Thakkar et al., JAMA Intern Med, $2016$

• Methods

  • Search: MEDLINE, EMBASE, CENTRAL, PsycINFO, CINAHL (inception–Jan 15, 2015)

  • Inclusion: RCTs assessing text-message interventions for adult chronic disease adherence

  • $n=16$ RCTs, $N=2742$ patients; median age $39$ yr; $50.3\%$ female

  • Intervention characteristics

  • Personalization (5/16)

  • Two-way communication (8/16)

  • Daily message frequency (8/16)

  • Median duration $12$ weeks

  • Outcome measure: mostly self-reported adherence

• Results

  • Pooled effect $\text{OR}=2.11$; $95\%\;CI\;1.52–2.93$; P<.001

  • Heterogeneity moderate ( $I^2=62\%$ )

  • Sensitivity analysis (higher-quality studies): $\text{OR}=1.67$; $95\%\;CI\;1.21–2.29$

  • After publication-bias adjustment: $\text{OR}=1.68$; $95\%\;CI\;1.18–2.39$

  • Translating odds to absolute terms: adherence rises from $50\%$ to $67.8\%$ ⇒ absolute increase $17.8\%$

• Limitations

  • Short intervention duration

  • Reliance on self-report measures

  • Moderate heterogeneity; publication bias possible

• Implications

  • Text messaging is promising, scalable, cost-effective strategy to improve adherence

  • Need long-term trials, objective adherence metrics, identification of optimal message features

Applying Evidence to Practice (Generalizability Checklist)

• Population similarity: Are study participants comparable to your patients (age, disease type, tech literacy)?

• Feasibility: Can your health system deliver automated, secure texting?

• Costs vs. benefits: Infrastructure vs. potential reduction in morbidity & health-care utilization

• Ethical/privacy considerations: Patient consent, data security

• Implementation: Staffing, scheduling, language customization

Strengths & Limitations of SR/MA

• Strengths

  • Summarize large bodies of evidence

  • Provide transparent, reproducible assessments

  • Identify knowledge gaps

  • Inform guidelines & policy

• Limitations

  • Quality limited by included primary studies

  • Poorly conducted SR/MA can mislead (methodological shortcuts, selective outcome reporting)

  • Heterogeneity may preclude pooling

Bonus Concepts (For deeper appraisal)

• Study heterogeneity tests guide decision to pool or not

• Sensitivity analysis: Remove studies one-by-one to gauge influence

• Subgroup analysis: Explore effect moderators (e.g., disease type, message personalization)

• Trim-and-Fill method: Estimates effect size after accounting for missing (unpublished) studies

• GRADE approach: Rates overall certainty of evidence (High, Moderate, Low, Very Low)

Practical Tips for Reading SR/MA Papers

• Check for protocol registration (PROSPERO #) early in Methods

• Inspect PRISMA flow diagram for comprehensiveness

• Look for dual independent processes (screening, appraisal, extraction)

• Verify risk-of-bias tables & how they inform synthesis (e.g., sensitivity analysis excluding high-risk trials)

• Assess heterogeneity statistics and authors’ rationale for chosen meta-analytic model

• Examine funnel plot & authors’ commentary on publication bias

• Apply JBI or AMSTAR-2 critical appraisal tool for structured evaluation

Key Equations & Statistical Notations (LaTeX)

• Odds Ratio example: $OR = \frac{\text{odds in intervention}}{\text{odds in control}}$

• Confidence interval (95%): $\hat{\theta} \pm 1.96 \times SE(\hat{\theta})$

• $I^2$ formula: $I^2 = \left( \frac{Q - df}{Q} \right) \times 100\%$

• Fixed-effect pooled estimate (inverse-variance weighted): $\hat{\theta}<em>{FE} = \frac{\sum w</em>i \theta<em>i}{\sum w</em>i}$ where $w<em>i = 1/SE</em>i^2$

• Random-effects weight: $w<em>i = \frac{1}{SE</em>i^2 + \tau^2}$ ((\tau^2) = between-study variance)

Ethical & Philosophical Considerations

• Transparency & reproducibility combat research waste

• Publication bias and selective reporting undermine scientific integrity

• Clinicians rely on SR/MA for evidence-based decisions – meticulous methodology is an ethical imperative

Quick-Reference Checklist for Planning Your Own SR/MA

• [ ] Formulate specific PICO question

• [ ] Develop & register protocol (PRISMA-P)

• [ ] Engage information specialist/librarian

• [ ] Conduct exhaustive search (≥3 databases + grey literature)

• [ ] Duplicate screening & extraction with consensus plan

• [ ] Choose appropriate risk-of-bias tool and analytic model

• [ ] Assess heterogeneity, conduct sensitivity & subgroup analyses

• [ ] Evaluate publication bias (funnel plot, Egger’s)

• [ ] Grade overall certainty (GRADE)

• [ ] Follow PRISMA 2020 for transparent reporting