Understanding Effect Size and Statistical Significance

Effect Size

  • Definition of Effect Size:

    • Effect size indicates the size of a difference and is important for understanding whether a statistically significant difference might also be meaningful. It is not influenced by sample size.

    • Describes how much two populations do not overlap; less overlap implies a larger effect size.

  • Importance of Effect Size:

    • Helps in determining whether a statistical finding has practical significance.

    • Unlike statistical significance, which can result from a large sample size alone, effect size offers a more direct measure of the difference's importance.

    • -

Understanding Statistical Significance

  • Misinterpretation of Statistically Significant Results:

    • "Statistically significant" means the findings are unlikely to occur if the null hypothesis is true, not necessarily that they represent a meaningful difference.

    • Geoff Cumming (2012) critiques hypothesis testing as relying on backward logic, failing to provide direct information about the effect itself.

Sample Size and Statistical Significance

  • Impact of Sample Size:

    • Increasing sample size generally increases the test statistic if all other variables remain constant.

    • Example using the Implicit Association Test (IAT):

    • Sample size of 30: Mean = -0.16, Standard Deviation (σ) = 0.51, Standard Error of Mean (SEM) calculated as:
      σM=σN=0.5130=0.093\sigma_M = \frac{\sigma}{\sqrt{N}} = \frac{0.51}{\sqrt{30}} = 0.093

    • Test Statistic calculated using:
      z=(M−μ<em>M)σ</em>M=−0.16−(−0.14)0.093=−0.215z = \frac{(M - \mu<em>M)}{\sigma</em>M} = \frac{-0.16 - (-0.14)}{0.093} = -0.215

    • Increasing sample size to 200 or higher influences calculation of standard error and test statistic:

    • Sample size of 200 with Mean = -0.14:
      σM=0.51200=0.036\sigma_M = \frac{0.51}{\sqrt{200}} = 0.036
      z=−0.16−(−0.14)0.036=−0.556z = \frac{-0.16 - (-0.14)}{0.036} = -0.556

    • Sample size of 1000:
      σM=0.511000=0.016\sigma_M = \frac{0.51}{\sqrt{1000}} = 0.016
      z=−0.16−(−0.14)0.016=−1.25z = \frac{-0.16 - (-0.14)}{0.016} = -1.25

    • Sample size of 100,000:
      z=(−0.16−(−0.14))0.002=−10.00z = \frac{(-0.16 - (-0.14))}{0.002} = -10.00

  • Conclusion from Statistical Analysis:

    • As sample size increases, standard error decreases, and test statistics become more extreme.

    • A small effect may appear statistically significant with a large sample, raising concerns about interpretation and validity.