SB

Critical Appraisal of Economic Evaluation Literature (Video)

Aims of session

  • Provide an understanding of the key stages of an economic evaluation
  • Overview of how to interpret and critically appraise economic evaluations
  • Complete your critical appraisal for Assignment 1
  • Facilitate your ability to design and undertake an economic evaluation
  • Begin thinking about your own protocol for Assignment 2

What is an economic evaluation?

  • A comparative analysis of the costs and consequences associated with competing interventions
  • Types of economic evaluation (listed in the slides):
    • Cost-effectiveness analysis (CEA)
    • Cost-benefit analysis (CBA)
    • Cost-consequences analysis
    • Cost-utility analysis (CUA)
    • Cost-minimization analysis (CMA)

Stages of an economic evaluation

1) Clarification of the study question
2) Identification of all possible resource and health consequences
3) Quantification of important resource and health consequences
4) Valuation of quantified resource and health consequences
5) Analysis
6) Interpretation of the results
7) Use of the results by the decision maker

Critical Appraisal (definition)

  • “Critical appraisal skills enable you to systematically assess the trustworthiness, relevance and results of published papers.”
    • Source: Critical Appraisal Skills Program (CASP)
    • Website: https://casp-uk.net/

Drummond’s Checklist for Critically Assessing Economic Evaluation Literature

1) Was a well-defined question posed in an answerable form?
2) Was a comprehensive description of the competing alternatives given?
3) Was effectiveness of the programs or services established?
4) Were all the important and relevant costs and consequences for each alternative identified?
5) Were costs and consequences measured accurately in appropriate physical units prior to valuation?
6) Were costs and consequences valued credibly?
7) Were costs and consequences adjusted for differential timing?
8) Was an incremental analysis of costs and consequences of alternatives performed?
9) Was uncertainty in the estimates of costs and consequences adequately characterised?
10) Did the presentation and discussion of study results include all issues of concern to users?

Perspective in economic evaluations

  • From whose perspective is the study undertaken? Perspectives include:
    • Societal: all costs and outcomes to whomsoever they accrue
    • Government: all costs and outcomes to the public sector (health, housing, legal)
    • Health Sector: costs and outcomes for government and private health actors (insurers, providers)
    • Payer: costs/outcomes for payers (could include donors)
    • Patients: out-of-pocket costs, travel, waiting time

Defining the patient population and subgroups

  • The population and any relevant subgroups must be adequately defined
  • This ties into the question of whether a comprehensive description of competing alternatives includes subgroups and conditional pathways

Improving research question formulation (example)

  • Problematic: “Is a chronic home care programme worth it?”
  • improved formulation (from slides):
    • “From the viewpoint of the government and patients, is a chronic home-care programme cost-effective compared to the existing programme of institutionalised care in hospitals?”

Description of competing alternatives (as per Drummond)

  • Key elements to describe for each alternative:
    • Important/relevant alternatives omitted? Consider a do-nothing option
    • Include patient subgroups
    • Details needed to tell who did what to whom, where, and how often
    • Professional involved, grade/supervision, training
    • Intervention specifics: dosage, minutes, equipment used, letters sent, etc.
    • Patient group exclusions
    • Access pathway
    • Location (outpatients/home/community, rural/urban)
    • Timing of all parts of the intervention and follow-up

Example: Breast Screening Programme Pathway (description of the pathway)

  • The pathway includes steps across recruitment, screening, assessment, and follow-up services
  • Provides a structured view of where resources are used and where outcomes occur
  • Note: Assumes quality assurance, counselling and data management occur in all stages

Example: Lithotripsy vs. cholecystectomy (gallstone treatment)

  • Resources used by each intervention include:
    • Operating theatre, doctor/nursing time, instruments, overheads
    • Lithotripsy suite, doctor/nursing time, overheads
    • Hospital wards, follow-up
    • Clinical follow-up (outpatient appointments)
  • Health consequences to compare:
    • Pain, digestion problems, overall wellbeing
  • Implications for care utilization:
    • GP visits for pain control, prescriptions, time off work, need for further treatment

Costs and consequences: measured in appropriate physical units

  • Example considerations:
    • Description and justification of data sources for resource use
    • Whether any items were omitted and what that implies
    • Handling joint or shared resource use and special measurement circumstances
    • Clear description of data sources for benefit measurement

Valuation of costs and consequences (credibility)

  • Key questions:
    • Have the sources of values been clearly identified? (market values, patient preferences, policy-maker views, health professional judgment)
    • Were market values used where possible for resource changes?
    • If market values were absent or inappropriate, were adjustments made to approximate market values?
    • Was the valuation appropriate for the chosen analysis type (e.g., CEA, CBA, CUA)?

Timing and discounting of costs and consequences

  • Were costs and consequences adjusted for differential timing?
  • Discounting concept:
    • Future costs/consequences are discounted to present value because people weight future benefits/costs less
    • Common approach uses a discount rate: present value (PV) is calculated to enable comparison on a common basis
  • Note on formula (illustrative):
    • ext{PV} = rac{FV}{(1+r)^t}
    • where FV = future value, r = discount rate, t = time in years
  • Distinction from inflation: discounting is about time preference, not price level changes
  • Justification for the chosen discount rate should be provided

Incremental analysis of costs and consequences

  • Essential question: what are the additional costs required to achieve additional benefits when moving from one alternative to another?
  • Typical output: Incremental cost, incremental effect, and the ICER
  • Example from slides:
    • Per person treated: Control cost $18{,}687; QALYs 3.07
    • Treatment cost $19{,}124; QALYs 3.22
    • Incremental Cost = $19{,}124 - $18{,}687 = $437
    • Incremental QALYs = 3.22 - 3.07 = 0.15
    • ICER = \frac{437}{0.15} = 2{,}913.33\$ per QALY
  • ICER interpretation: whether the additional cost for the additional benefit is acceptable under a chosen willingness-to-pay threshold

Uncertainty in costs and consequences; sensitivity analysis

  • Importance of characterising uncertainty in estimates
  • Approaches to address uncertainty:
    • One-way sensitivity analysis: vary one parameter at a time (e.g., cost, relapse rate)
    • Robustness: results remain unchanged under variation
    • Cost-effectiveness acceptability curves (CEACs): show probability that an intervention is cost-effective across a range of thresholds
  • CEAC interpretation:
    • The curve indicates the probability that the intervention is the most cost-effective option given a threshold value for cost-effectiveness
  • Other practical considerations in sensitivity analyses:
    • If patient-level data are available, use appropriate statistical analyses
    • Justification for the form and range of the sensitivity analyses
    • Explore heterogeneity by presenting subgroup results

Presentation and interpretation of study results to users

  • Considerations for the user audience:
    • Is the analysis based on an overall index or ratio (e.g., a cost-effectiveness ratio)? How should it be interpreted?
    • Compare results with other studies; account for methodological differences
    • Generalisability: applicability to other settings and patient groups
    • Additional factors: distribution of costs/consequences, ethical issues
    • Implementation considerations: feasibility given financial or other constraints
    • Resource redeployment possibilities if the preferred programme is adopted
    • Implications of uncertainty for future decision making and recommendations for further research

Additional considerations: Translatability

  • Assess whether the programme is likely to be equally effective in your context
  • Consider whether costs and benefits are transferable to your setting
  • Decide if it is worth implementing in your setting

Decision-analytic modelling and supporting guidelines

  • Note on quality checklists for decision-analytic modelling in health technology assessment
    • Reference: Philips, Bojke, Sculpher, Claxton, Golder, et al. pharmacoeconomics (2006)
    • Quote: “Good practice guidelines for decision-analytic modelling in health technology assessment: a review and consolidation of quality assessment.”
  • Additional note in slides: Better checklists exist for models; this is a source of guidance

Examples and exercise highlights

  • Exercise: choosing a comparator (Romosozumab for osteoporosis)
    • Which is the most appropriate comparator: a) Placebo, b) Alendronate, c) Teriparatide, d) Calcium and Vitamin D, e) Subsidised exercise program, f) HRT
  • Exercise: Was effectiveness established?
    • Questions include whether evidence comes from an RCT and whether efficacy vs. effectiveness is reflected in practice; how data were collected (systematic review vs. observational) and potential biases

Important implications and practical takeaways

  • A well-formed question sets the stage for the entire analysis (scope, perspective, comparators, and outcomes)
  • A comprehensive description of alternatives ensures the analysis captures realistic options and clinical pathways
  • Accurate measurement and credible valuation of costs and outcomes are crucial for credible results
  • Time preference and discounting must be justified and consistently applied
  • Incremental analysis (ICER) is central to comparing alternatives; interpretation depends on thresholds and context
  • Uncertainty must be quantified and communicated (via sensitivity analyses and CEACs) to inform decision-makers
  • Results should be assessed for generalisability and implementation feasibility; consider ethical and distributional implications
  • Translatability matters: context matters for effectiveness and costs; what works in one setting may not in another

Quick reference: key formulas and terms

  • ICER: \text{ICER} = \frac{CT - CC}{ET - EC} = \frac{\Delta C}{\Delta E}
    • C = costs, E = effects (e.g., QALYs)
  • Present value (discounting): \text{PV} = \frac{FV}{(1+r)^t}
    • r = discount rate, t = time in years
  • Costs and consequences measured in appropriate physical units (e.g., cost per patient, QALYs gained, life-years gained)
  • Sensitivity analyses include one-way, probabilistic, and CEACs for decision uncertainty

Closing reminder

  • The session aims to equip you with a structured framework (Drummond checklist) to critically appraise economic evaluations, identify gaps, assess relevance to your setting, and prepare for assignment work and protocol development for upcoming tasks.