EBM: Limitations

What should the Results section contain?

Findings only (numbers and statistics), with no interpretation; it should match what the Methods planned

Enrollment flowchart: three math checks

Randomized + excluded = screened; group totals = number randomized; allocated − completed = number who didn't finish

In a baseline characteristics table, what p-values do you want to see?

p > 0.05, meaning the groups are similar and randomization worked

The baseline table shows only the total population, not each group. Why does that matter?

You can't confirm randomization worked, so it's a limitation

Outcomes appear in the Results that weren't in the Methods (or vice versa). What does that suggest?

Reporting bias

External validity

Generalizability; usually the #1 RCT limitation. Would my patient have been included? Was the comparator the standard of care?

Internal validity

The study's "insides": design, selection into groups (randomization), and how the intervention was given; whether the groups were treated the same

Random error vs systematic error

Random error = chance, which evens out with a larger sample. Systematic error = bias introduced by researchers' actions or decisions

Grading a bias: low vs high vs unknown

Low: unlikely to change the results. High: could seriously weaken confidence in the results. Unknown: not reported or not enough information

Selection bias

Groups differ at baseline. Checklist: random sequence generation + allocation concealment. Confirm: differences in Table 1

Performance bias

Groups receive different care. Checklist: blinding of participants and personnel. Confirm: one group improves much faster, or side effects reveal assignment

Detection bias

Outcomes are measured differently between groups. Checklist: blinding of outcome assessment. Confirm: subjective outcomes, or study physicians judging outcomes

Attrition bias

Groups drop out at different rates. Checklist: incomplete outcome data. Confirm: different dropout rates or reasons between groups

Reporting bias

Some findings reported, others hidden. Checklist: selective reporting. Confirm: missing or extra outcomes vs the Methods, or every result positive

Examples of "other" bias

Carry-over in crossover trials (check the drug's half-life) and contamination (control patients receiving the intervention)

How should you grade allocation concealment if it isn't reported?

Unknown (this is common)

How do you mitigate open-label bias?

Use objective outcomes (e.g., death from any cause) and blinded outcome assessors or an independent adjudication committee

Is industry funding automatically a limitation?

No; someone has to pay for the study. Look for safeguards like an independent statistical analysis

Statistical vs practical significance

Statistical: p-value and confidence interval. Practical (clinical): the size of the effect (NNT, Cohen's d)

When can practical significance be meaningfully calculated?

Only when the result is statistically significant

Which effect size goes with a dichotomous vs a continuous outcome?

Dichotomous: NNT or NNH. Continuous: Cohen's d

What data do you need to calculate Cohen's d?

The mean and standard deviation for each group

Cohen's d interpretation

Small ≥ 0.2, medium ≥ 0.5, large ≥ 0.8, very large ≥ 1.2, huge ≥ 2.0

How do you state an NNT in words?

"X patients need to be treated for [the study duration] to prevent 1 event"

Traffic light conclusion: red, yellow, green

Red: too many limitations, more studies needed. Yellow: proceed with caution, close monitoring. Green: confident, minimal monitoring

Where in an article do the authors interpret their results?

Discussion (the Results section presents findings only)

Subgroup analysis

An analysis within a subset of participants who share a characteristic (e.g., only obese patients)

A priori vs post-hoc subgroup analysis

A priori: planned before the study, can be useful. Post-hoc: decided after, hypothesis-generating only

Scenario: Table 1 shows the drug group had significantly more diabetics (p = 0.01), and the authors didn't adjust. What does this suggest?

Possible selection bias / confounding; randomization may not have worked, so it's a limitation

Scenario: 17 of 110 dropped out of the drug group vs 3 of 113 on placebo, mostly for "lack of efficacy." Which bias?

Attrition bias (unequal dropout between groups)

Scenario: Patients self-rate pain, and the drug's side effect (dry mouth) reveals who's on it. Which biases?

Performance and detection bias (blinding is broken and the outcome is subjective)

Scenario: The study's own physicians decided whether each hospitalization counted as an outcome. Which bias?

Detection bias (use an independent, blinded adjudication committee instead)

Scenario: The paper only reports positive outcomes; clinicaltrials.gov lists two outcomes never published. Which bias?

Reporting bias

Scenario: A crossover trial uses a drug with a long half-life and a 2-day washout period. Which bias?

Other bias: carry-over effect

Scenario: Patients in the control group got the study drug from an outside pharmacy. Which bias?

Other bias: contamination

Which bias does allocation concealment prevent?

Selection bias

Which biases does blinding prevent?

Performance bias (participants and personnel) and detection bias (outcome assessors)

TRUE or FALSE: Random error is a form of bias.

FALSE. Random error is chance; bias is systematic error

TRUE or FALSE: If ITT and per-protocol results disagree in a superiority trial, trust per-protocol.

FALSE. Go with ITT

TRUE or FALSE: Cohen's d should be calculated for a dichotomous outcome like death.

FALSE. Use NNT/NNH for dichotomous outcomes; Cohen's d is for continuous outcomes

Interpret Cohen's d = 0.6

Medium effect size (≥ 0.5)

Interpret Cohen's d = 0.25

Small effect size (≥ 0.2)

Interpret Cohen's d = 1.3

Very large effect size (≥ 1.2)

A result is not statistically significant. Should you calculate NNT?

No; practical significance only matters if the result is statistically significant