Analysis and Meta-Analysis in Systematic Reviews

Types of Analysis in Systematic Reviews

• Qualitative or Narrative Analysis: This involves a structured summary and discussion of the characteristics and findings of the included studies. It serves as a descriptive overview rather than a statistical calculation.

• Quantitative Analysis (Meta-analysis): This is a statistical analysis that involves the combination of results from two or more different studies. It focuses on a pairwise, head-to-head comparison of interventions, such as an intervention group versus a control group, or an intervention versus another comparator.

Core Framework for Synthesizing Findings

Regardless of whether the analysis is narrative or quantitative, researchers utilize a framework consisting of three primary questions to synthesize the findings of included studies:

1. What is the direction and size of the effect?

2. Is the effect consistent across the various studies?

3. What is the strength of the evidence supporting the observed effect?

Key Stages and Components of Meta-analysis

• Definition of Effect: The difference observed in outcomes between the intervention and control/comparator groups is known as the "effect."

• Pooled (Summary) Effect: In meta-analysis, the observed effects from individual studies are pooled to produce a weighted average, representing the summary effect of all included studies.

• Process Steps:

  • Step 1: Estimating individual effect sizes: Calculate the effect size for every individual study included in the review.

  • Step 2: Assigning weights: Each study is assigned a weight based on the amount of information it contributes. Studies with larger amounts of data (often due to larger sample sizes) are weighted more heavily.

  • Step 3: Calculating the summary estimate: Determining the weighted average of the individual results.

  • Step 4: Assessing consistency: Evaluating whether the effect remains consistent across the included studies (heterogeneity assessment).

Anatomy of a Forest Plot

A forest plot is a graphical representation used in meta-analyses to display individual study results alongside the combined summary result.

• Case Study Example: A 2013 Cochrane systematic review by Palmer titled "Antiplatelet agents for chronic kidney disease" reviewed the effects of antiplatelet treatment for preventing cardiovascular and adverse kidney outcomes.

• Title and Metadata: The plot header indicates the comparison (e.g., "Antiplatelet agents versus control") and the outcome (e.g., "Fatal or non-fatal myocardial infarction").

• Study Identification: The first column lists the name and year of the included studies.

• Event Data: The subsequent columns list the number of events per trial arm:

  • Intervention Arm: Number of events over total participants in the intervention/antiplatelet group.

  • Control/Placebo Arm: Number of events over total participants in the placebo or no-treatment group.

• Effect Size Representation:

  • Calculated as a Risk Ratio ($RR$), Odds Ratio ($OR$), or Mean Difference ($MD$).

  • Point Estimates: Represented by a solid block (square). The size of the block indicates the weight assigned to the study.

  • Confidence Intervals (CI): Indicated by horizontal lines extending from either side of the block.

  • Numerical Data: The numerical values for the relative risk and $CI$ are typically listed on the far right of the plot.

• The Summary Diamond:

  • Lower section of the plot shows a diamond representing the pooled estimate.

  • The center (vertical tips) of the diamond indicates the summary effect size.

  • The width of the diamond (horizontal tips) represents the confidence interval for the summary estimate.

  • Significance Test: Located at the bottom left, typically expressed as a $z$-score with a corresponding $p$-value. (Example from Palmer review: $z = 2.16$, $p = 0.03$).

  • Line of No Effect: A vertical line. If the summary diamond's confidence intervals do not cross this line, the result is considered statistically significant.

Precision and Weighting

• Precision: Driven primarily by sample size. Higher precision results in narrower confidence intervals.

• Weighting Logic: More precise studies (those with larger samples) are assigned more weight. This is reflected visually by larger square blocks in the forest plot, which dominate the final summary effect calculation.

Understanding Heterogeneity

Heterogeneity is the variation in characteristics between studies regarding populations, interventions, outcomes, and methods.

• Sources of Heterogeneity:

  • Populations: Variations in disease severity (e.g., mild vs. severe).

  • Interventions: Differences in types of interventions, drug dosages, durations, or intensity of therapy.

  • Comparators: Use of placebos versus active comparators, or varying doses of active comparators.

  • Outcomes: Differences in how outcomes are defined or measured.

• Decision to Pool: Meta-analysis should only be performed if the studies are sufficiently homogeneous to provide a meaningful summary.

Assessing Heterogeneity: Methods and Metrics

1. The "Eyeball Test":

   • Inspect the forest plot for the degree of overlap between the confidence intervals of individual studies.

   • Poor overlap suggests the presence of statistical heterogeneity.

   • Observe if the direction and magnitude of effects are similar across studies.

2. Chi-squared ($\chi^2$) Test and $p$-value:

   • Assesses whether variation is due to chance.

   • If $p$ is small, the null hypothesis (that variation is due to chance) is rejected, indicating significant heterogeneity.

   • Note on Power: The $\chi^2$ test has low power when studies are few or have small sample sizes. Therefore, a significance threshold of p < 0.1 is often used instead of the standard $0.05$.

3. I-squared ($I^2$) Test:

   • Quantifies the percentage of variability in effect estimates due to heterogeneity rather than chance.

   • Interpretation Guildines (Cochrane):

     • $0\% \text{ to } 40\%$: Might not be important.

     • $30\% \text{ to } 60\%$: May represent moderate heterogeneity.

     • $50\% \text{ to } 90\%$: May represent substantial heterogeneity.

     • $75\% \text{ to } 100\%$: May represent considerable heterogeneity.

Investigative Techniques for Heterogeneity

• Subgroup Analysis:

  • Stratified analysis exploring the same outcome by grouping participants into categories (e.g., sex, disease severity, or timing of intervention like surgery within $6 \text{ hours}$ versus later).

  • Used to investigate sources of variation or answer specific questions about particular groups.

  • Example: Cochrane review on antibiotics for sore throat categorized data by positive throat swabs, negative swabs, and untested participants. It found higher efficacy in those with positive swabs.

• Sensitivity Analysis:

  • Selectively removing studies from the meta-analysis to see if the summary estimate changes significantly.

  • Criteria for exclusion include study quality, publication date, or identifying outlier studies causing heterogeneity.

  • Example: The sore throat review conducted sensitivity analysis based on date (pre-$1975$ vs. later) and blinding (blinded vs. unblinded).

• Methodological Rule: Criteria for subgroup and sensitivity analyses must be specified in the review protocol a priori (before the analysis is conducted).