Synthesizing Evidence from Literature

Synthesizing Evidence from Literature

Challenges in Synthesizing Evidence

  • Multiple Answers: Research-heavy topics (e.g., COVID, asthma, hypertension, diabetes) often yield multiple answers.
  • Combining Answers: Need to consolidate answers into a single estimate for the study.

Uncertainty and Sensitivity Analysis

  • Uncertainty Introduction: Combining estimates introduces uncertainty and a margin of error.
  • Ideal Approach: Ideally, identify the best, most relevant answer and focus on it, citing others for context.
  • Sensitivity Analysis: Essential to assess if the chosen parameter, given its uncertainty, will affect the study's reliability.

Addressing Heterogeneity

  • Heterogeneity Assessment: Check if studies are done similarly and yield similar results.
  • Random Effects: Differences may be due to statistical noise.
  • Study Variations: Investigate differences in studies with divergent results.
    • Small population.
    • Non-domestic study.
    • Funding source (e.g., drug or device manufacturer).

Factors Influencing Study Outcomes

  • National Differences: Study populations vary outside the industrialized Western world; assess relevance.
  • Study Endpoints:
    • Longer studies vs. short-term studies: time frame relevance (e.g., diabetes, hypertension vs. cancer care).

Study Design Considerations

  • RCTs (Randomized Clinical Trials):
    • Gold standard but may lack generalizability.
    • High internal validity due to controlled parameters.
    • Smaller, more specific samples.
  • Generalizability:
    • Larger, diverse samples yield more generalizable studies.
    • Consider diversity (populations, ethnicities, genders) to ensure relevance to the study population.

Fixed vs. Random Effects

  • Fixed Effects:
    • Assumption: All studies have the same effect; variations are random.
    • Method: Weighted average of results based on sample size (N).
    • Calculation: weighted average=∑(percentage×N)\text{weighted average} = \sum{(\text{percentage} \times N)}
    • Caution: Regression of results may lead to confounding and low R2R^2 due to variable differences.
  • Random Effects:
    • Assumption: Effects vary but come from a common probability distribution.
    • Method: Straight average of results without population weighting (Z-scored approach).
    • Benefit: Simpler approach if studies are reasonably comparable.

Total Evidence Approach

  • Neither Fixed Nor Random: Focus on maximizing the number of relevant studies.
  • Rationale: A larger data sample increases the likelihood of good quality decisions, despite potential confounding elements.

Goal of Evidence Synthesis

  • Obtain the best possible answer.
  • Identify and combine relevant sources.
  • Applications: ACA (Affordable Care Act), CBA (Cost-Benefit Analysis), cost-utility analysis.

Steps to Incorporate Synthesized Data into Cost-Effectiveness Analysis

  • Synthesize data based on understanding the relationships between study characteristics.
  • Determine if there are bias-free estimates that align with each other.
  • Predict parameter values and use them in a CTA (Cost-effectiveness Analysis) model.

Phases for Synthesizing Evidence

  1. Pre-Analytical Phase:
    • Define the research question and target data.
    • Identify the evidence and develop parameters for a systematic review.
  2. Selection and Review:
    • Extract information from sources.
  3. Qualitative Analysis:
    • Ensure comparability and consistency among sources.
    • Assess risk of bias and study limitations.
    • Consider transferability to other populations or conditions.
  4. Quantitative Analysis:
    • Conduct quantitative analysis (regression, weighted average).
  5. Prediction and Modeling:
    • Predict study parameters and model the expected outcome.
    • Report the process used to derive parameter estimates.
    • Perform a sensitivity analysis to assess the parameter's impact and potential bias.

Gold Book Recommendations

  • Identify Important Parameters:
    • Determine influential or critical parameters for model validity.
  • Estimate Parameters via Evidence Synthesis:
    • Describe the analysis and critique the evidence base.
    • Assess if the evidence is domestic, manufacturer-funded, or uses convenience samples.
  • Quantitative Data Analysis:
    • Model statistical variability (frequency distribution).
    • Assess central tendencies and clustering.
    • Address heterogeneity that may cause noise.
  • Consistent Parameter Estimates:
    • Aim for consistent, well-informed estimates from synthesized data.
  • Address Bias:
    • Be explicit about potential biases in the study, across studies, or within the population.
    • Consult statistical resources for bias correction methods.
  • Explicit Adjustments:
    • Clearly state how adjustments were made to the evidence.
    • Evaluate the impact on the transferability of parameters to other studies.
  • Scenario Analysis for Sensitivity Analysis:
    • Evaluate model structure relative to the research question.
    • Understand the impact of parameter values on the ultimate result.
    • Account for biases or limitations in generalizability.