Effect Size and Meta-Analysis

Narrative reviews

a review of the scientific literature that provides a critical synopsis of a given topic

• depend on experience and expertise of the author

• selection of RCTs may be arbitrary 

• subjective judgment 

• lack of assessement of the quality of the studies 

Meta-Analysis

synthesize the results of multiple studies

• quantitative estimate of the effect

purpose of the MA

• collecting studies about certain topic

• determine the magnitude of the effects 

• compare the effect sizes and relate them to different factors - moderators 

usage of MA

• as a part of research process

• in the process of making a decision if a new study is needed

• in defining the research design 

• as an introduction to a new primary study 

• synthesizing  data

advantages of MA

• increased transparency of primary study selection

• summarization of many studies

• statistical summarization

  • each study is weighted

  • calculation of effect size

  • assessment of heterogeneity of outcomes

  • identification of moderators

  • assessment of publication bias

MA - steps 

1. definition of the research question 

2. identification of the primary studies

3. collection of data

4. analysis

PICO model 

→ construction of well-built question

Patient | Population

Intervention 

Comparison 

Outcomes 

Feasibility MA

• must be enough studies

• only empirical research (quantitative results)

• data must be comparable

clues for search replicability and comprehensiveness

• use multiple search engines

• use*

• consult narrative reviews

• consult reference lists

• consult experts

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)

an evidence-based minimum set of items aimed at helping authors to report a wide array of systematic reviews and meta-analyses

• transparent and complete reporting

gray literature

not published, has limited distribution, not available in traditional channels

selection bias

systematic differences between baseline characteristics of participants in the treatment groups, which may be related to the probability of receiving the intervention, rather than the intervention itself

Allocation concealment

he researchers must be not aware of which group the next participant will be assigned to

• centralized randomization 

• opaque sealed envelopes 

Inadequate randomization

→ imbalances in the baseline characteristics

non-compliance with randomization

some participants are assigned to a group different from what was intended

ex.: drop out

performance bias 

Differences between groups in the care or treatment provided, or in exposure to factors other than the intervention being evaluated, that may affect the outcome being measured

→ blinding/masking

detection bias

Differences between groups in outcome assessment that may arise from knowledge of which intervention was received, or from measurement instruments that are not blinded to treatment allocation

outcome reporting bias

Selective reporting of outcomes based on the results, which may be influenced by the statistical significance of the results, clinical relevance, or the study hypothesis

attrition bias

It’s common for participants to drop out of a trial before or in the middle of treatment, and researchers who only include those who completed the protocol in their final analyses are not presenting the full picture

Intention-to-treat analysis

a comparison of the treatment groups that includes all patients as originally allocated after randomization

→ for missing observations, “last value carried forward”

Per-protocol analysis

only subjects who fully and accurately complied with the study protocol are included

→ estimate of the treatment effect under optimal conditions

• doesn’t account for real-world variability

• may be influenced by confounding factors

As-treated analysis

subjects are included

according to the treatment they received

• may be influenced by confounding factors 

effect size

a measure of the magnitude or strength of an effect

→ assessment of practical/clinical significance

Cohen’s d

used to quantify the difference between two groups in terms of their means and standard deviations

• calculated by subtracting the means and dividing the result by the pooled standard deviation

Glass’s

used to quantify the difference between two groups in terms of their means and standard deviations

• uses the standard deviation of the control group (he argues that the standard deviation of the control group should not be influenced)

Hedges’ G

used to adjust for bias that can occur when estimating the population standard deviation using the sample standard deviation (underestimation of the true population SD)

• small samples

• unknown population variances

Standardized Response Mean

quantifies the magnitude of change in a variable over time or in response to an intervention, taking into account the variability of the measure

• size of the change relative to the variability or standard deviation of the measure

odds ratio

measures the odds of an event occurring in one group compared to the odds of it occurring in another group

• logarithm of the OR → to make more symmetric data and to  stabilize variances 

relative risk

measures the risk of an event occurring in one group compared to the risk of it occurring in another group

• similar to the log(OR), taking the natural logarithm of the RR is done to transform the data into a more symmetrical form and to stabilize variances

pearson correlation coefficient

…

obtaining effect sizes

• direct calculation

• equivalent algebraic formulas (t-tests, chi2) 

• true significance value (p-value)

• correlation coefficients

• other estimates of mean difference 

• estimates of pooled standard deviation 

Common Language Effect Size

metric to describe the practical significance of an effect

→ expresses the probability that one random individual from one group will be greater than a random individual from another group

0% to 100%

computation of the global effect size

weighted average of the different effect sizes, which is obtained by giving more value (weight) to studies conducted on large samples

• weigh by the inverse of the variance of the effect (studies in which the effect is less variable are more accurate)

fixed effects model 

assumes that there is one “true” effect size that is common to all studies, and the variability observed between the studies is due to random error

• studies that are exact replicas

random effects model

assumes that there is not just one “true” effect size, but that the true effect size can vary from study to study due to both random error and systematic differences

• studies combined are not exact replicas 

heterogenity

the degree to which the effect sizes vary across different studies

statistical heterogeneity

refers to the variation in effect sizes that is greater than what would be expected due to random sampling error alone

Q-statistic

compares the observed variability in effect sizes with what would be expected if the studies were homogeneous

• significant = presence of statistical heterogeneity

clinical/substantive heterogeneity

differences in the characteristics of the studies themselves, such as variations in study design, populations, interventions, measurement instruments, or other factors

• assessed qualitatively

• → external validity and generalizability

I2 statistic

relative index of heterogeneity

• based on the principle that the more studies there are, the greater the potential heterogeneity

• each study should contribute the same amount of heterogeneity

random effects model

accounts for this variability when estimating the overall effect size

• significant effect size heterogeneity

subgroup analysis

explore whether specific subgroups of studies have different effect sizes

• clinical or substantive heterogeneity is suspected

meta-regression

investigate whether specific study characteristics explain the observed heterogeneity

sensitivity analysis

assessing the impact of individual studies on the overall results by excluding one study at a time

the funnel plot 

In an ideal world without publication bias, you would expect the studies to form a symmetrical, inverted funnel shape on the plot. Studies with smaller sample sizes (lower precision) would have more scattered effect size estimates due to random variation, while studies with larger sample sizes (higher precision) would have more precise estimates, resulting in a narrower spread around the overall effect size

= asymmetry => presence of publication bias 

• used in exploration of the presence of publication bias

fail-safe N

a statistical measure used to estimate how many additional non-significant or null studies would be needed to nullify the observed effect

• effects of publication bias on findings in MA

larger = less likely that there is publication bias