Effects Sizes and Confidence Intervals

Effects Sizes and Confidence Intervals

Overview

  • Importance of effect sizes and confidence intervals in statistics

Confidence Intervals

  • Definition: Confidence intervals (CIs) communicate precision by providing a range of plausible values for the population parameter being estimated.

  • Purpose: Answers the question: "How sure are we about our finding?"

  • Usage: Refers to the range of expected values of the estimate (usually the effect size) if the experiment were rerun with a different sample.

  • Common Confidence Levels: Researchers commonly use confidence levels of either 95% or 99%.

  • Calculation Factors: CIs are based on three factors:

    • Sample size

    • Response distribution

    • Population size

  • Notation Example: Typically written as the confidence percentage followed by the estimated lower and upper limits of the parameter, e.g., an odds ratio of 7.5 might have a confidence interval associated with it like "95% CI [5.32, 10.45]."

Effects Sizes

  • Definition: Effect sizes communicate strength by telling us the magnitude of the experimental effect, or relationship (i.e., correlation), or odds between variables.

  • Purpose: Answers the question: "How big (or small) was the effect or relationship?"

Statistical Significance Testing

  • Definition: Communicates probability by informing how likely the current result would be if the study's null hypothesis were true.

  • Purpose: Answers the question: "Do we think something happened?"

Types of Effect Sizes

Cohen’s d and Hedges’ g (Standardized Mean Difference)

  • Definition: Both Cohen’s d and Hedges’ g estimate the magnitude of standardized differences between two group means.

  • Bias Correction: Hedges’ g is more appropriate for small sample sizes because it provides a bias correction.

  • Scores Range: Scores for Cohen’s d and Hedges’ g range from -1 to 1, with 0 indicating no effect.

Pearson’s r and Point-Biserial Correlation

  • Definition: Pearson’s r measures the strength of a linear relationship between continuous variables (r).

  • Point-Biserial Correlation: Measures the strength of the relationship between one dichotomous and one continuous variable (rpb).

  • Scores Range: Scores range from -1 to 1 with 0 indicating no effect.

Odds Ratio (Proportion)

  • Definition: The odds ratio is the ratio of the probability that an outcome occurs to the probability that the outcome does not occur.

  • Scores Range: Scores range from 0 to infinity with 1 indicating no effect.

Interpretation of Effect Sizes

Cohen’s d and Hedges’ g

  • Effect Size Interpretation:

    • Small: (+/-) 0.2

    • Medium: (+/-) 0.5

    • Large: (+/-) 0.8

Pearson’s r

  • Effect Size Interpretation:

    • Small: 0.1-0.3

    • Medium: 0.3-0.5

    • Large: >0.5

Odds Ratio Interpretation

  • If the exposure is positively related to the disease, the odds ratio is greater than 1.0.

  • If the exposure is negatively related to the disease, the odds ratio is less than 1.0.

Confidence Intervals - Further Implications

Key Concepts of Confidence Intervals

  • Establish the following through confidence intervals:

    • Occurrence of Effect: If an effect occurred.

    • Range of Scores: The possible range of scores if tested with other samples.

    • Direction of Effect: Whether the effect could be negative or positive.

    • Precision of Estimate: Smaller confidence intervals suggest more precision, while larger ones suggest less precision.

Evaluation of Confidence Intervals

  • If the confidence interval range includes the value representing "no effect," then researchers cannot be certain the effect is real.

  • **Examples of Specific CIs: **

    • A standardized mean difference might show: 95% CI [0.1, 0.9]

    • An odds ratio might show: 99% CI [0.7, 3.5]