health literacy - Comprehensive Guide to Critiquing Quantitative Research and the CASP RCT Tool
Fundamentals of Critiquing Quantitative Research
Critiquing quantitative research is a systematic process used to identify the strengths and weaknesses of a study.
The primary purpose is to determine the overall quality of the research and the level of confidence one should have in the study’s findings.
A central question in the critiquing process is whether the results are worthy of informing clinical practice. This decision is based on the reliability of the findings and their relevance to specific clinical settings.
In contrast to qualitative research, critiquing quantitative research is highly standardized and widely practiced. It is a mandatory requirement when conducting a systematic review of quantitative studies.
The critiquing process focuses heavily on identifying potential sources of bias. In quantitative paradigms, the goal is to minimize or eliminate bias entirely, whereas qualitative research acknowledges and manages it.
Sources of Bias in Quantitative Studies
Allocation Bias: This refers to how participants are assigned to different groups. A major concern is differences in the baseline characteristics of groups. Ideally, groups should be similar at the start so that those characteristics do not influence the response to an intervention.
Performance Bias: This involves differences in how the groups are treated during the study beyond the specific intervention being tested. It includes the quality of the control group development and the presence of confounding factors that might influence outcomes.
Measurement Bias: This concerns how outcomes are determined and measured across different groups. Bias occurs if measurements are not performed consistently or using the same methods for all participants.
Attrition Bias: This relates to participants who drop out of the study. Bias can arise if there are differences between groups in the number of people who withdraw or the reasons for their withdrawal.
Reporting Bias: This occurs when researchers selectively report findings. For example, if multiple ways of measuring strength were used but only the significant result was reported while the others were ignored.
Methodological Nuances across Study Designs
Bias risks vary depending on whether a study is experimental or observational.
Experimental Studies (e.g., Randomized Controlled Trials): Allocation bias focuses on how participants were recruited and randomized to match baseline characteristics. Performance bias focuses on how well the control group matched the experimental group in terms of settings, location, and exposure to clinicians.
Observational Studies (e.g., Cohort or Case-Control Studies): Since researchers do not randomize participants, allocation bias focuses on recruitment methods. In cohort studies, exposed and unexposed groups should be matched on other characteristics. In case-control studies, those with and without a condition should be matched as closely as possible. Performance bias focuses on matching confounding factors so the only difference is the exposure or condition.
Measurement Differences: In case-control studies, measurement bias often involves looking back in time to how data were originally gathered, rather than measuring immediate outcomes at the end of a trial.
Study Design Selection: It is critical to identify the study design before choosing a critique tool. While RCTs, cohort studies, and case-control studies have specific checklists, experimental studies that are cross-sectional have very few available tools.
The Critical Appraisal Skills Program (CASP) Checklist for RCTs
The CASP checklist for Randomized Controlled Trials (RCTs) consists of questions answered with "yes," "no," or "I can't tell."
Section A: Is the Study Design Valid?
Clearly Focused Question: The study must address a clear aim or question regarding the population studied, the intervention given, the comparator used, and the outcomes measured. This information is typically found in the title, abstract, or the first paragraph of the introduction.
Randomization Procedure: The study must state the specific method of randomization used. An appropriate method ensures that neither the participant nor the researcher could predict the group assignment beforehand.
Concealed Allocation: Ideally, the randomization sequence should be generated by someone not involved in recruiting participants. This prevents the recruiter from knowing the next assignment, which could influence recruitment bias.
Participant Accountability: All participants who entered the study must be accounted for at the end. A flowchart should show the number of people at each stage, including dropouts and the reasons for their withdrawal.
A follow-up rate of to is considered appropriate.
Researchers should analyze if those who dropped out differ significantly in characteristics from those who stayed.
Intention to Treat Analysis: Participants should be analyzed in the groups to which they were originally randomized. If data are missing for dropouts, statistical procedures such as imputing data may be used.
Early Study Termination: If a study stopped early, the reasons must be explained. Reasons might include adverse events, external factors (e.g., COVID), or slow recruitment.
Section B: Methodological Soundness
Blinding: Blinding is used to reduce bias by preventing people from treating participants differently or participants from perceiving changes based on expectations. There are three potential groups to blind:
Participants: Difficult if they are aware of the intervention details through informed consent.
Intervention Providers: Especially difficult in physical therapies (e.g., physiotherapy).
Outcome Assessors: This is the most critical group to blind to ensure unbiased data collection.
Single-blinded: Usually outcome assessors or participants.
Double-blinded: Both participants and outcome assessors.
Triple-blinded: Participants, outcome assessors, and intervention providers (highest quality).
Baseline Similarity: Groups should be similar at the start of the RCT. Researchers often provide a table of participant characteristics. While statistical comparisons are common, the focus should be on whether any differences are "meaningful" enough to influence the outcome.
Consistency of Care: Aside from the experimental intervention, all groups should receive the same level of care. This includes matching clinician contact time, researcher contact time, the setting, and the timing of outcome assessments.
Section C: Results and Practical Application
Comprehensive Reporting: Results should include various details:
Power Calculation: Justification for the sample size. For example, a study might state it needs participants ( per group) to achieve power to detect a difference.
Precision of Estimates: Results should be expressed in absolute and relative numbers, including change scores and percentages. All outcomes and groups should be reported at every follow-up interval.
Confidence Intervals: The study should report confidence intervals. A smaller range indicates higher confidence in the result (e.g., a range of to in the USA compared to to in the UK).
Precision of Treatment Effects: Statistical tests must be clearly stated, and p-values or data tables (including mean, standard deviation, median, and interquartile range) should be provided.
Benefits vs. Harms: This is a subjective assessment. One must consider:
Effect Size: The magnitude of the change.
Minimal Clinically Important Difference (MCID): The amount of change required for the outcome to be considered meaningful.
Adverse Events: Any harms or unintended effects must be reported and weighed against the benefits.
Local Applicability: Consider if the study participants are similar to the local population. Even if they differ, assess if those differences would alter the intervention's effectiveness or harm risk.
Clinical Importance of Outcomes: Ensure the measured outcomes are meaningful to the specific patient population. Identify any limitations or missing information (e.g., specific expertise required to deliver the intervention).
Value Comparison: Evaluate if the new intervention provides greater value than existing standard practice. This involves analyzing needed resources like time, costs, skills, and training. A higher initial cost (e.g., a device) might be justified if it saves long-term resources (e.g., staff time).
Critiquing Observational Studies: Case-Control and Cohort Designs
Recruitment Quality: In case-control studies, the focus is on how cases and controls were selected. In cohort studies, the focus is on how the cohort was recruited and if the groups are representative and balanced to minimize bias.
Exposure Measurement: In case-control studies, identifying past exposure accurately is difficult. In cohort studies, follow-up must be long enough to capture health outcomes that take time to develop.
Confounding Variables: Because experimental control and randomization are absent, observational studies must account for confounding factors through meticulous group matching or by noting additional exposures.
Clinical Implications and Limitations of Observational Evidence
A single observational study rarely provides enough robust evidence to change clinical practice or health policy.
Recommendations from observational studies are stronger when supported by other evidence.
Ethical Constraints: For some questions, observational studies are the only option. For example, it is ethically impossible to conduct an RCT where participants are assigned to be sedentary to study the effects of sedentary behavior on cardiovascular disease. Instead, researchers must select people who are already sedentary and then use an experimental design to test the effect of increasing their activity level.