Evaluating Evidence

Evidence Based Practice

• best available evidence

• clinical expertise

• patient preference

Literature Types

• Primary

• Secondary

• Tertiary

Primary literature

direct access to data

Secondary Literature

• abstracting or indexing search engines

• facilitate location of primary and tertiary literature

Tertiary Literature

• indirect access to data

• author interpretation of the literature

• background information on topic

• may be able to answer some (or all) of DI question

• consider source: textbook, lexidrug/drug database, website

Order of literature search for DI question

1. Tertiary

2. Practice Guidelines

3. Systematic Reviews/Meta-analyses

4. Primary Literature

Questions that can be answered with tertiary sources:

• drug’s place in management of condition

• dosing for certain condition

• dose adjustments for impaired renal function

• length of treatment

• level of evidence for use of treatment for condition

Use of AI

• ask AI to provide sources

• double-check information: make sure references are not hallucinated (made up)

• report use of AI

Strongest Evidence (strongest to weakest)

• Clinical Practice Guidelines/ Health Technology Assessments

• Systematic Review

• Meta-Analysis

• study level data

Weakest levels of evidence (from strongest to weakest)

• Randomized controlled trial

• controlled clinical study

• retrospective/prospective cohort

• case report/case series

• expert opinion

• all subject level data except expert opinion

Are all randomized controlled trials better than cohort studies?

• NO

• well-designed cohort study may provide better evidence than a poorly executed randomized controlled trial

• lower level article may be better evidence than higher level one

Practice Guidelines

• step 2 of literature search

• check:

  • authorship/affiliations

    • reputable guidelines usually associated with non-profit orgs

  • funding source

  • date of publication

  • methodology of development

  • level of evidence for recommendations

Systematic Review

• authors collect evidence from previously conducted studies and summarize it

• NO statistics performed

Meta Analysis

• authors collect evidence from previously conducted studies and summarize it

• statistics are performed

• “study of studies”

Advantages of Systematic Reviews and Meta Analyses

• comprehensive

• increased precision of effect (larger sample size)

• enhance statistical power

Disadvantages of Systematic Reviews and Meta-Analyses

• quality of studies included

  • low quality studies may yield inappropriate conclusions

• publication bias

  • overestimation of treatment effects

• confounding factors

• incomplete reporting

Life Cycle of Scientific info

Original research → write up results → submit for publication → peer review → rewrite (if needed) and re-submit for peer review → publish

Peer Review

• process by which something proposed (as for research or publication) is evaluated by a group of experts in the appropriate field

• NOT EVERYTHING in a journal is peer reviewed!

  • only research

What is peer reviewed in a journal?

• original research

• systematic reviews

• meta-analyses

• some types of reviews

What is usually NOT peer reviewed?

• editorial and opinions

• advertisements

• letters to the editor

• reviews (that are not meta analyses or systematic reviews)

• corrections

• notices

• job listings

• continuing education

Factors to Consider when choosing primary literature

• relevance to topic

• strength of study design

• date of publication

• study population

• reputability of journal

Evaluating a journal’s reputability

• recognized as a scholarly publication

• transparent publication process

• respected within discipline

Steps to check if journal is recognized as a scholarly publication

1. Check if it in indexed in medline

2. Check if it is in other reputable indices

Checking if a journal is indexed in medline

1. check NLM catalog for journal title

   1. PubMed is NOT same as Medline

2. Verify the journal title and ISSN match

3. Look for “Currently Indexed in MEDLINE” status

Reputable Indices

• Medline

• Cabells Journalytics & predatory reports

• directory of open access journals (DOAJ)

• journal citation report (web of science)

• scopus (click on sources)

Signs of Transparent Publication Process

• peer-review and editorial-review process clearly described

• respected scholars on editorial board

• use best practices from COPE, DOAJ, and OASPA

COPE

Committee on Publication Ethics

OASPA

Open Access Scholarly Publishers Association

Signs a journal is respected within the discipline

• citation ratios/rank

  • Scopus, Journal Citation Report

  • (# citations to journal articles)/(# of citable articles in journal)

• Impact Factor

  • find on Scopus, JCR (trusted 3rd party)

  • do not trust journal website

After completing search…

1. Evaluate each article

2. Compare and Contrast Articles

3. Summarize findings

Evaluating individual articles

• type of study

• population

• intervention

• statistics

• findings

• does it answer the question?

Questions to ask while evaluating literature

• does the study population apply to my patients?

• are the results presented in an appropriate way?

• were the statistical analyses appropriate?

• did the discussion address any limitations or biases?

• did the authors provide high-quality references?