Week 6 EBP Critical Appraisal in Health Research

Benefits of Critical Appraisal

• Health professionals need to make decisions based on the best available evidence.
• Need to determine the quality of evidence to ensure it has a clinically significant impact.
• The quality of a research study is related to how the research was conducted.
• Findings should be a true and accurate representation, not due to biased methodology.

Peer Review

• Check if a research study has been peer-reviewed.
• Articles are reviewed by qualified researchers familiar with the topic before publication.
• Many journals require several peer reviews before accepting an article.
• Check the peer review process on the journal's homepage under author submission guidelines.
• Peer review does not guarantee good quality research as the level of scrutiny varies.

Key Terms

• Validity: Relates to research using a quantitative methodology.
• Rigor: Relates to research using a qualitative methodology.

Applicability of Research

• Consider whether the findings can be generalized to other settings.
  • In quantitative research, generalizability is referred to as external validity.
  • Control of bias is referred to as internal validity.
• Researchers may prioritize internal validity (control of bias) at the expense of external validity (generalizability).
• Ideally, research should be both good quality and able to be generalized.

Undertaking Critical Appraisal

• Do not just read the abstract and discussion.
• Critique each part of the reported research.
• Read the entire article at a superficial level first, then read in detail and make notes.
• Consider skipping the abstract initially and compare your understanding after detailed reading.

Non-Inferiority Trials

• Designed to show that a treatment is equivalent to another treatment.
• Relevant when a new treatment offers advantages like improved safety, convenience, compliance, or cost.
• Example: Comparing telehealth delivery with in-clinic delivery.
• Telehealth offers reduced costs and increased convenience but must demonstrate equivalent efficacy.

N-of-1 Trials

• Also referred to as a single case study, it involves a single participant.
• Random allocation can be used to determine the order of experimental and control interventions.
• Used for people with rare health conditions; increasingly recognized for common conditions.

Critical Appraisal Tools

• Guide you through questions evaluating research.
  • What was the clinical question?
  • Which study design was used and was it appropriate?
  • What were the characteristics of the sample and recruitment procedure?
  • What data were collected and how?
  • What was the independent variable, and how was it administered (for quantitative research)?
  • What other potential sources of bias may have affected the study?
  • What are the results, and are they applicable to practice?

Developing Appraisal Skills

• With experience, you may not need a tool.
• Recognize common quality pitfalls in specific areas of practice.
• Critical appraisal tools ensure a thorough EBP (Evidence-Based Practice) process.

Validity

• Concerns the integrity of the findings from a study.
• Health practitioners must be aware of the main types of validity.

Internal Validity

• Addresses causality: Can we draw a causal relationship between two variables?
• Dependent variable (y): The variable researchers need to understand or explain why it varies.
• Independent variable (x): The variable believed to produce variation in the dependent variable.
• Need to be confident that x causes y and not an extraneous variable.
• Example: Birth control classes and increased knowledge of birth control options.
• Eliminate other plausible explanations to ensure the independent variable is responsible.

Post Hoc Fallacy:

• The logical fallacy that because Y followed X, Y was caused by X.
• In Latin: Post hoc ergo propter hoc or post hoc for short.
• Other confounding factors might be responsible for changes detected.

External Validity

• The degree to which findings can be generalized beyond the study participants.
• Research is intended to guide practice, so generalizability is important.
• Asks whether causal relationships can be generalized to different measures, persons, settings, and times.
• Criticism: Highly controlled studies may only show treatment effectiveness in ideal circumstances.

Ecological Validity

• Do the findings of a research study reflect real-life settings?
• Researchers often focus on maximizing internal validity, potentially sacrificing external validity.
• Academic journals and funding organizations often require high internal validity.

Efficacy Studies

• Focus on demonstrating internal validity through highly controlled methodology.

Effectiveness Studies

• Follow efficacy studies and are carried out in less controlled, real-life situations.

Establishing External Validity

• Consider how well the research, using a sample, can be generalized to the population.
  • How many participants were involved, and was this a sufficient number to generalize to the population?
  • Was the sample size justified?
  • Is a clear description of the key characteristics of the sample provided? For example, the age and gender, and the time of onset of the health condition of interest.
  • If there were multiple groups in the study, were they comparable in terms of size and participant characteristics?
• Were appropriate inclusion and exclusion criteria described?

Identifying Common Types of Bias

• Bias can affect the internal and external validity of research.
• Internal validity: Identifying a causal relationship between independent and dependent variables.
• External validity: Factors like sample size and comparability of different groups.
• Bias affects study results in a particular direction, favoring either the treatment or control group.

Categories of Bias

• Sample or selection bias
• Measurement or detection bias
• Intervention or performance bias

Sample or Selection Bias

• Volunteer or referral bias
• Attention bias

Measurement or Detection Bias

• Number of outcome measures used
• Lack of ‘masked’ or ‘independent’ evaluation
• Recall or memory bias

Intervention or Performance Bias

• Contamination
• Co-intervention
• Timing of intervention
• Site of intervention
• Different administrators of the intervention

Measurement Errors

• Categorized as random or systematic.
• Random error: Unpredictable and cannot be controlled.
• Systematic error: Predictable and can be identified and eliminated (e.g., imperfect calibration).

Types of Bias

Sample or Selection Bias

Volunteer or Referral Bias

• Description: Participants volunteer for the study, potentially being more motivated.
• Potential solution: Randomly select participants where possible; invite from waiting lists rather than advertising.

Attention Bias

• Description: Awareness of the study's intention might cause participants to perform differently (placebo effect).
• Potential solution: Include a control group and, if possible, a placebo treatment.

Intervention or Performance Bias

Contamination

• Description: The control group inadvertently receives treatment.
• Potential solution: Strict protocols for treatment delivery and control group management.

Co-intervention

• Description: A participant receives another intervention simultaneously, influencing results.
• Potential solution: Obtain information about medications or other interventions participants are undergoing.

Timing of Intervention

• Description: Short duration may not allow noticeable change; long duration may lead to maturation.
• Potential solution: Follow treatment protocols; use a control group to address maturation.

Site of Intervention

• Description: The intervention site may affect the result.
• Potential solution: Ensure consistency across all treatment sites.

Different administrators of the intervention

• Description: Different therapists deliver the treatment, leading to variability.
• Potential solution: Address therapist variability to ensure more consistent results

Other Limitations in Quantitative Research

• Sample characteristics
• Dropouts
• Method and frequency of measurement

Sample

• Questions to ask:
  • How many participants were involved, and was this a sufficient number to be able to generalize the results of population?
  • Was the sample size justified? Preliminary studies may involve small sample sizes, but if the researchers aim to demonstrate efficacy, then a larger sample is needed.
  • Was a clear description of the key characteristics of the same provided (e.g. the age and gender of participants, and the onset of the health condition of interest)?
  • If there were multiple groups in the study, were they comparable in terms of size and participant characteristics?
• Were appropriate inclusion and exclusion criteria described?

Dropouts

• Also known as experimental mortality or attrition.
• Questions to ask:
  • Were the number of drops outs reported?
  • Were the reasons for the dropouts documented?
  • How did the researchers manage the analysis of the data to deal with any missing data caused by participants dropping out?

Measurement

• Questions to ask:
  • How frequently were the outcomes measured; for example, were they measured before and after treatment, or were short-term and long-term follow-up data also collected?
  • Did the researchers report whether the outcomes measures used are well- established as being reliable and valid?