Week 6 EBP Critical Appraisal in Health Research

Benefits of Critical Appraisal

  • Health professionals need to make decisions based on the best available evidence.

  • Need to determine the quality of evidence to ensure it has a clinically significant impact.

  • The quality of a research study is related to how the research was conducted.

  • Findings should be a true and accurate representation, not due to biased methodology.

Peer Review

  • Check if a research study has been peer-reviewed.

  • Articles are reviewed by qualified researchers familiar with the topic before publication.

  • Many journals require several peer reviews before accepting an article.

  • Check the peer review process on the journal's homepage under author submission guidelines.

  • Peer review does not guarantee good quality research as the level of scrutiny varies.

Key Terms

  • Validity: Relates to research using a quantitative methodology.

  • Rigor: Relates to research using a qualitative methodology.

Applicability of Research

  • Consider whether the findings can be generalized to other settings.

    • In quantitative research, generalizability is referred to as external validity.

    • Control of bias is referred to as internal validity.

  • Researchers may prioritize internal validity (control of bias) at the expense of external validity (generalizability).

  • Ideally, research should be both good quality and able to be generalized.

Undertaking Critical Appraisal

  • Do not just read the abstract and discussion.

  • Critique each part of the reported research.

  • Read the entire article at a superficial level first, then read in detail and make notes.

  • Consider skipping the abstract initially and compare your understanding after detailed reading.

Non-Inferiority Trials

  • Designed to show that a treatment is equivalent to another treatment.

  • Relevant when a new treatment offers advantages like improved safety, convenience, compliance, or cost.

  • Example: Comparing telehealth delivery with in-clinic delivery.

  • Telehealth offers reduced costs and increased convenience but must demonstrate equivalent efficacy.

N-of-1 Trials

  • Also referred to as a single case study, it involves a single participant.

  • Random allocation can be used to determine the order of experimental and control interventions.

  • Used for people with rare health conditions; increasingly recognized for common conditions.

Critical Appraisal Tools

  • Guide you through questions evaluating research.

    • What was the clinical question?

    • Which study design was used and was it appropriate?

    • What were the characteristics of the sample and recruitment procedure?

    • What data were collected and how?

    • What was the independent variable, and how was it administered (for quantitative research)?

    • What other potential sources of bias may have affected the study?

    • What are the results, and are they applicable to practice?

Developing Appraisal Skills

  • With experience, you may not need a tool.

  • Recognize common quality pitfalls in specific areas of practice.

  • Critical appraisal tools ensure a thorough EBP (Evidence-Based Practice) process.

Validity

  • Concerns the integrity of the findings from a study.

  • Health practitioners must be aware of the main types of validity.

Internal Validity

  • Addresses causality: Can we draw a causal relationship between two variables?

  • Dependent variable (y): The variable researchers need to understand or explain why it varies.

  • Independent variable (x): The variable believed to produce variation in the dependent variable.

  • Need to be confident that x causes y and not an extraneous variable.

  • Example: Birth control classes and increased knowledge of birth control options.

  • Eliminate other plausible explanations to ensure the independent variable is responsible.

Post Hoc Fallacy:

  • The logical fallacy that because Y followed X, Y was caused by X.

  • In Latin: Post hoc ergo propter hoc or post hoc for short.

  • Other confounding factors might be responsible for changes detected.

External Validity

  • The degree to which findings can be generalized beyond the study participants.

  • Research is intended to guide practice, so generalizability is important.

  • Asks whether causal relationships can be generalized to different measures, persons, settings, and times.

  • Criticism: Highly controlled studies may only show treatment effectiveness in ideal circumstances.

Ecological Validity

  • Do the findings of a research study reflect real-life settings?

  • Researchers often focus on maximizing internal validity, potentially sacrificing external validity.

  • Academic journals and funding organizations often require high internal validity.

Efficacy Studies

  • Focus on demonstrating internal validity through highly controlled methodology.

Effectiveness Studies

  • Follow efficacy studies and are carried out in less controlled, real-life situations.

Establishing External Validity

  • Consider how well the research, using a sample, can be generalized to the population.

    • How many participants were involved, and was this a sufficient number to generalize to the population?

    • Was the sample size justified?

    • Is a clear description of the key characteristics of the sample provided? For example, the age and gender, and the time of onset of the health condition of interest.

    • If there were multiple groups in the study, were they comparable in terms of size and participant characteristics?

  • Were appropriate inclusion and exclusion criteria described?

Identifying Common Types of Bias

  • Bias can affect the internal and external validity of research.

  • Internal validity: Identifying a causal relationship between independent and dependent variables.

  • External validity: Factors like sample size and comparability of different groups.

  • Bias affects study results in a particular direction, favoring either the treatment or control group.

Categories of Bias

  • Sample or selection bias

  • Measurement or detection bias

  • Intervention or performance bias

Sample or Selection Bias

  • Volunteer or referral bias

  • Attention bias

Measurement or Detection Bias

  • Number of outcome measures used

  • Lack of ‘masked’ or ‘independent’ evaluation

  • Recall or memory bias

Intervention or Performance Bias

  • Contamination

  • Co-intervention

  • Timing of intervention

  • Site of intervention

  • Different administrators of the intervention

Measurement Errors

  • Categorized as random or systematic.

  • Random error: Unpredictable and cannot be controlled.

  • Systematic error: Predictable and can be identified and eliminated (e.g., imperfect calibration).

Types of Bias

Sample or Selection Bias
Volunteer or Referral Bias

  • Description: Participants volunteer for the study, potentially being more motivated.

  • Potential solution: Randomly select participants where possible; invite from waiting lists rather than advertising.

Attention Bias

  • Description: Awareness of the study's intention might cause participants to perform differently (placebo effect).

  • Potential solution: Include a control group and, if possible, a placebo treatment.

Intervention or Performance Bias
Contamination

  • Description: The control group inadvertently receives treatment.

  • Potential solution: Strict protocols for treatment delivery and control group management.

Co-intervention

  • Description: A participant receives another intervention simultaneously, influencing results.

  • Potential solution: Obtain information about medications or other interventions participants are undergoing.

Timing of Intervention

  • Description: Short duration may not allow noticeable change; long duration may lead to maturation.

  • Potential solution: Follow treatment protocols; use a control group to address maturation.

Site of Intervention

  • Description: The intervention site may affect the result.

  • Potential solution: Ensure consistency across all treatment sites.

Different administrators of the intervention

  • Description: Different therapists deliver the treatment, leading to variability.

  • Potential solution: Address therapist variability to ensure more consistent results

Other Limitations in Quantitative Research

  • Sample characteristics

  • Dropouts

  • Method and frequency of measurement

Sample

  • Questions to ask:

    • How many participants were involved, and was this a sufficient number to be able to generalize the results of population?

    • Was the sample size justified? Preliminary studies may involve small sample sizes, but if the researchers aim to demonstrate efficacy, then a larger sample is needed.

    • Was a clear description of the key characteristics of the same provided (e.g. the age and gender of participants, and the onset of the health condition of interest)?

    • If there were multiple groups in the study, were they comparable in terms of size and participant characteristics?

  • Were appropriate inclusion and exclusion criteria described?

Dropouts

  • Also known as experimental mortality or attrition.

  • Questions to ask:

    • Were the number of drops outs reported?

    • Were the reasons for the dropouts documented?

    • How did the researchers manage the analysis of the data to deal with any missing data caused by participants dropping out?

Measurement

  • Questions to ask:

    • How frequently were the outcomes measured; for example, were they measured before and after treatment, or were short-term and long-term follow-up data also collected?

    • Did the researchers report whether the outcomes measures used are well- established as being reliable and valid?