Applied Data Interpretation and Statistical Inference Study Guide
PeaEffect Sizes: Cohen's d and Eta Squared
Definition of Cohen's :
Cohen's is an effect size metric that measures the magnitude of the difference between two group averages relative to the spread of scores within those groups.
It expresses the difference between two means in standard deviation () units.
Standard Interpretive Benchmarks for Cohen's :
Small Effect (): Represents a slight difference between group averages relative to the overall spread of scores.
Medium Effect (): Represents a moderate difference between group averages.
Large Effect (): Represents a substantial, pronounced difference between group averages.
Alternative Effect Size Metric: Eta Squared ( ):
Definition: Eta squared () quantifies the proportion of total variation in the dependent variable that is explained by or accounted for by the independent variable (or factor).
Standard Interpretive Benchmarks for :
Small Effect ( or ): The independent variable accounts for of the variance in the dependent variable.
Medium Effect ( or ): The independent variable accounts for of the variance in the dependent variable.
Large Effect ( or ): The independent variable accounts for of the variance in the dependent variable.
Selection Criteria for Effect Size Metrics:
Cohen's is utilized when comparing the averages of two groups, typically in conjunction with a -test.
Eta squared () is utilized when evaluating the proportion of variance in a continuous dependent variable explained by a factor, typically in conjunction with an Analysis of Variance ().
Contextual Caveat: All standard numerical benchmarks (, , for ; , , for ) serve merely as general guidelines; true practical importance is entirely dependent on the specific research domain and context.
One-Sample Inference: t-Tests versus z-Tests
Criteria for Choosing Between a One-Sample -Test and a -Test:
One-Sample -Test: Employed when the population standard deviation () is unknown and must be estimated directly from the sample data using the sample standard deviation ().
-Test: Employed when the true population standard deviation () is known beforehand.
Null Hypothesis Specification ():
Both the one-sample -test and the -test evaluate identical forms of the null hypothesis: that the population mean () is equal to a specified hypothesized value ().
Illustrative Example: If investigating whether college students sleep an average of hours per night, the null hypothesis states that the true population average equals hours ().
Analytical Difference: The one-sample -test explicitly incorporates and accounts for the additional statistical uncertainty introduced by estimating the unknown population standard deviation from sample data.
Interpreting Null Results: Failure to Reject versus Proving the Null
Rationale for the Terminology "Failed to Reject the Null":
Obtaining a non-significant result () signifies that the observed empirical data do not provide sufficient statistical evidence to reject the null hypothesis at the specified significance threshold.
A non-significant outcome never proves that the null hypothesis is true, nor does it confirm the complete absence of an effect.
Methodological Factors Producing Non-Significant Results:
Finding no statistically significant difference between two experimental conditions or treatments does not indicate that the treatments exert identical effects.
A study may yield a non-significant -value due to:
Insufficient statistical power stemming from an inadequate or too small sample size.
Excessive extraneous variability or measurement error within the sample.
A true population effect that is small and difficult to detect with the design employed.
Failing to find sufficient evidence of an effect is epistemologically distinct from confirming the absence of an effect.
Distinguishing Statistical Significance from Practical Significance
Conceptual Definitions:
Statistical Significance: Indicates whether observed sample data provide sufficient evidence to reject the null hypothesis at a predetermined significance level (); it establishes whether an effect is unlikely to have occurred solely due to sampling error.
Practical Significance: Evaluates whether the observed magnitude of an effect is large, meaningful, or impactful enough to carry genuine real-world importance.
Mechanisms Underlying Statistical Significance Without Practical Significance:
Because statistical power scales with sample size, extremely large samples allow hypothesis tests to detect even minuscule, trivial differences as statistically significant.
Illustrative Scenario: An investigation with a sample size of students evaluates a novel instructional method and detects a statistically significant improvement in test scores of only points.
While the massive sample size provides the statistical power necessary to demonstrate that this -point gain is statistically distinguishable from zero (), a half-point increase is practically negligible in terms of real-world academic achievement.
Guiding Principle: Statistical significance establishes empirical evidence that an effect exists; practical significance determines whether that detected effect matters in an applied context.
Contextual Factors Influencing Significance and Interpretation
Divergent Practical Importance Across Identical Effect Sizes:
Two distinct studies can report identical quantitative effect sizes yet carry vastly divergent practical significance depending on the real-world consequences and stakes of the outcome.
Illustrative Example: Consider two separate medical interventions that both yield a small effect size of Cohen's :
Treatment A: Slightly improves cosmetic skin appearance.
Treatment B: Slightly reduces symptom severity or mortality risks in a life-threatening disease.
Conclusion: Despite having equivalent standardized effect sizes (), Treatment B carries significantly greater practical importance due to the severity and life-altering nature of the health condition targeted.
Divergent Statistical Compellingness Across Identical -Values:
Two distinct studies can report the exact same -value (such as ), yet one finding may be significantly more reliable and compelling depending on experimental methodology, design, and analysis protocols.
Illustrative Example: Comparing two researchers who each obtain an identical result of :
Researcher 1: Formulated a single, planned, a priori hypothesis and performed exactly one statistical test.
Researcher 2: Evaluated distinct exploratory hypotheses simultaneously and selectively reported only the significant outcome without implementing standard adjustments or corrections for multiple testing (such as Bonferroni corrections).
Conclusion: The finding from Researcher 1 is far more robust and statistically convincing. Testing multiple hypotheses concurrently inflates the family-wise error rate, substantially increasing the likelihood of identifying a statistically significant result purely through random chance.
Statistical interpretation requires critical evaluation of study methodology, preregistration, and research context, rather than uncritical reliance on nominal -values.
Quick Oral Exam Review Reference
Effect Sizes (Cohen's vs. Eta Squared):
Cohen's compares two group means scaled in standard deviation units.
Eta squared () quantifies the proportion of variation in a dependent variable accounted for by an independent factor.
Inference Selection (-Test vs. -Test):
One-sample -test: Population standard deviation () is unknown and must be estimated via sample data.
-test: Population standard deviation () is known.
Null Results:
Failing to achieve statistical evidence sufficient to reject the null hypothesis does not equate to proving the null hypothesis true.
Significance Types:
Statistical significance reflects empirical evidence that an effect exists beyond chance.
Practical significance reflects the substantive real-world importance of the effect size.
Impact of Context:
Numerically identical effect sizes or -values do not signify equivalent practical importance or statistical credibility; interpretation is strictly contingent on research context, real-world stakes, and methodological rigor.
Replace eta 2 with Pearson