Applied Data Interpretation and Statistical Inference Study Guide

PeaEffect Sizes: Cohen's d and Eta Squared

  • Definition of Cohen's dd:

    • Cohen's dd is an effect size metric that measures the magnitude of the difference between two group averages relative to the spread of scores within those groups.

    • It expresses the difference between two means in standard deviation (SD\text{SD}) units.

  • Standard Interpretive Benchmarks for Cohen's dd:

    • Small Effect (0.20.2): Represents a slight difference between group averages relative to the overall spread of scores.

    • Medium Effect (0.50.5): Represents a moderate difference between group averages.

    • Large Effect (0.80.8): Represents a substantial, pronounced difference between group averages.

  • Alternative Effect Size Metric: Eta Squared (η2\eta^2 ):

    • Definition: Eta squared (η2\eta^2) quantifies the proportion of total variation in the dependent variable that is explained by or accounted for by the independent variable (or factor).

    • Standard Interpretive Benchmarks for η2\eta^2:

      • Small Effect (0.010.01 or 1%1\%): The independent variable accounts for 1%1\% of the variance in the dependent variable.

      • Medium Effect (0.060.06 or 6%6\%): The independent variable accounts for 6%6\% of the variance in the dependent variable.

      • Large Effect (0.140.14 or 14%14\%): The independent variable accounts for 14%14\% of the variance in the dependent variable.

  • Selection Criteria for Effect Size Metrics:

    • Cohen's dd is utilized when comparing the averages of two groups, typically in conjunction with a tt-test.

    • Eta squared (η2\eta^2) is utilized when evaluating the proportion of variance in a continuous dependent variable explained by a factor, typically in conjunction with an Analysis of Variance (ANOVA\text{ANOVA}).

    • Contextual Caveat: All standard numerical benchmarks (0.20.2, 0.50.5, 0.80.8 for dd; 0.010.01, 0.060.06, 0.140.14 for η2\eta^2) serve merely as general guidelines; true practical importance is entirely dependent on the specific research domain and context.

One-Sample Inference: t-Tests versus z-Tests

  • Criteria for Choosing Between a One-Sample tt-Test and a zz-Test:

    • One-Sample tt-Test: Employed when the population standard deviation (σ\sigma) is unknown and must be estimated directly from the sample data using the sample standard deviation (ss).

    • zz-Test: Employed when the true population standard deviation (σ\sigma) is known beforehand.

  • Null Hypothesis Specification (H0H_0):

    • Both the one-sample tt-test and the zz-test evaluate identical forms of the null hypothesis: that the population mean (μ\mu) is equal to a specified hypothesized value (μ0\mu_0).

    • Illustrative Example: If investigating whether college students sleep an average of 88 hours per night, the null hypothesis states that the true population average equals 88 hours (H0:μ=8H_0: \mu = 8).

    • Analytical Difference: The one-sample tt-test explicitly incorporates and accounts for the additional statistical uncertainty introduced by estimating the unknown population standard deviation from sample data.

Interpreting Null Results: Failure to Reject versus Proving the Null

  • Rationale for the Terminology "Failed to Reject the Null":

    • Obtaining a non-significant result (p>.05p > .05) signifies that the observed empirical data do not provide sufficient statistical evidence to reject the null hypothesis at the specified significance threshold.

    • A non-significant outcome never proves that the null hypothesis is true, nor does it confirm the complete absence of an effect.

  • Methodological Factors Producing Non-Significant Results:

    • Finding no statistically significant difference between two experimental conditions or treatments does not indicate that the treatments exert identical effects.

    • A study may yield a non-significant pp-value due to:

      • Insufficient statistical power stemming from an inadequate or too small sample size.

      • Excessive extraneous variability or measurement error within the sample.

      • A true population effect that is small and difficult to detect with the design employed.

    • Failing to find sufficient evidence of an effect is epistemologically distinct from confirming the absence of an effect.

Distinguishing Statistical Significance from Practical Significance

  • Conceptual Definitions:

    • Statistical Significance: Indicates whether observed sample data provide sufficient evidence to reject the null hypothesis at a predetermined significance level (α\alpha); it establishes whether an effect is unlikely to have occurred solely due to sampling error.

    • Practical Significance: Evaluates whether the observed magnitude of an effect is large, meaningful, or impactful enough to carry genuine real-world importance.

  • Mechanisms Underlying Statistical Significance Without Practical Significance:

    • Because statistical power scales with sample size, extremely large samples allow hypothesis tests to detect even minuscule, trivial differences as statistically significant.

    • Illustrative Scenario: An investigation with a sample size of N=10,000N = 10{,}000 students evaluates a novel instructional method and detects a statistically significant improvement in test scores of only 0.50.5 points.

      • While the massive sample size provides the statistical power necessary to demonstrate that this 0.50.5-point gain is statistically distinguishable from zero (p<.05p < .05), a half-point increase is practically negligible in terms of real-world academic achievement.

    • Guiding Principle: Statistical significance establishes empirical evidence that an effect exists; practical significance determines whether that detected effect matters in an applied context.

Contextual Factors Influencing Significance and Interpretation

  • Divergent Practical Importance Across Identical Effect Sizes:

    • Two distinct studies can report identical quantitative effect sizes yet carry vastly divergent practical significance depending on the real-world consequences and stakes of the outcome.

    • Illustrative Example: Consider two separate medical interventions that both yield a small effect size of Cohen's d=0.2d = 0.2:

      • Treatment A: Slightly improves cosmetic skin appearance.

      • Treatment B: Slightly reduces symptom severity or mortality risks in a life-threatening disease.

      • Conclusion: Despite having equivalent standardized effect sizes (d=0.2d = 0.2), Treatment B carries significantly greater practical importance due to the severity and life-altering nature of the health condition targeted.

  • Divergent Statistical Compellingness Across Identical pp-Values:

    • Two distinct studies can report the exact same pp-value (such as p=.03p = .03), yet one finding may be significantly more reliable and compelling depending on experimental methodology, design, and analysis protocols.

    • Illustrative Example: Comparing two researchers who each obtain an identical result of p=.03p = .03:

      • Researcher 1: Formulated a single, planned, a priori hypothesis and performed exactly one statistical test.

      • Researcher 2: Evaluated 100100 distinct exploratory hypotheses simultaneously and selectively reported only the significant outcome without implementing standard adjustments or corrections for multiple testing (such as Bonferroni corrections).

      • Conclusion: The finding from Researcher 1 is far more robust and statistically convincing. Testing multiple hypotheses concurrently inflates the family-wise error rate, substantially increasing the likelihood of identifying a statistically significant result purely through random chance.

    • Statistical interpretation requires critical evaluation of study methodology, preregistration, and research context, rather than uncritical reliance on nominal pp-values.

Quick Oral Exam Review Reference

  • Effect Sizes (Cohen's dd vs. Eta Squared):

    • Cohen's dd compares two group means scaled in standard deviation units.

    • Eta squared (η2\eta^2) quantifies the proportion of variation in a dependent variable accounted for by an independent factor.

  • Inference Selection (tt-Test vs. zz-Test):

    • One-sample tt-test: Population standard deviation (σ\sigma) is unknown and must be estimated via sample data.

    • zz-test: Population standard deviation (σ\sigma) is known.

  • Null Results:

    • Failing to achieve statistical evidence sufficient to reject the null hypothesis does not equate to proving the null hypothesis true.

  • Significance Types:

    • Statistical significance reflects empirical evidence that an effect exists beyond chance.

    • Practical significance reflects the substantive real-world importance of the effect size.

  • Impact of Context:

    • Numerically identical effect sizes or pp-values do not signify equivalent practical importance or statistical credibility; interpretation is strictly contingent on research context, real-world stakes, and methodological rigor.

Replace eta 2 with Pearson