T-Tests, Sampling Distributions, and Effect Sizes – Lecture 3 Notes

Administrative & Housekeeping

• • Reminder from Rana: visit NUCU ➔ read the PDF of proposed class experiments ➔ complete the Qualtrics vote BEFORE Thursday.

• • Thursday’s class = collaborative design of the selected “Class Experiment – the star of the term.”

Road-Map of Today’s Lecture

• • Revisit & clarify several student-raised questions from last week.

• • Topics in order:

  • ◦ Logic of the normality assumption.

  • ◦ Samples vs. populations & the idea of sampling distributions.

  • ◦ Between-subjects vs. within-subjects t tests.

  • ◦ When to choose Student’s, Welch’s, Mann-Whitney, or Wilcoxon tests.

  • ◦ Calculating & interpreting effect size (Cohen’s $d$) and distinguishing it from $p$ values.

Samples, Populations & Distribution Shapes

• • Demonstration website allowed interactive sampling from an imaginary population (mean $\mu = 10$).

• • Small $n$ (e.g., $n=3$)

  • ◦ Sample histogram shows only three dots — looks non-normal.

  • ◦ Sample mean was $\bar{x}=9.6$ (close but not identical to $\mu=10$).

  • ◦ Re-sample with $n=4$ produced $\bar{x}=8$ (≈ 1 SD away).

  • ◦ Point: with small $n$ the sample mean can be very off.

• • Large $n$ (e.g., $n=100$)

  • ◦ Histogram gets “lumpier” but more closely approximates shape of the population.

  • ◦ $\bar{x}$ hugs the true $\mu$ tightly.

• • Key nuance: increasing $n$ does not guarantee that the sample distribution becomes normal — it becomes whatever shape the population really is. Normal-looking samples only occur when the population itself is normal.

Skewed Population Example

• • New population defined with mean $\mu=3$, strong positive skew (analogy: reaction-time ceiling).

• • Taking $n=100$, $n=1{,}000$, $n=10{,}000$:

  • ◦ Sample histograms retain skew.

  • ◦ Sample mean converges to $\mu=3$.

Central Limit Theorem (CLT)

• • Despite sample skew, the sampling distribution of the mean → normal as $n$ ↑.

• • Illustration: ran $100$ separate studies with $n=50$ each; plotted the $100$ sample means:

  • ◦ Overlay ≈ normal curve (dashed line) even though the raw data were skewed.

  • ◦ Reduced $n=4$ widened & distorted the sampling distribution.

• • Formal statement: “For any population with finite variance, the distribution of $\bar{X}$ approaches normal with mean $\mu$ and SE $\sigma/\sqrt{n}$ as $n \to \infty$.”

Why Normality Matters (or Not)

• • Parametric tests (Student’s t, ANOVA, $\chi^2$, etc.) actually assume a normal sampling distribution, not necessarily normal raw data.

• • Implications:

  • ◦ If $n$ is large → CLT renders assumption safe even under non-normal population shapes.

  • ◦ For small $n$ non-normality can distort $p$ values via:

  1. Heavy tails (↑ Type-I error; false positives).

  2. High skew pulling $\bar{X}$ (↑ Type-II error; false negatives).

Decision Tree: Which Two-Group Test?

• • Check assumptions in order:

  1. Normality (Shapiro-Wilk).

  2. Homogeneity of variance (Levene/Brown-Forsythe).

• • If both satisfied → Student’s t.

• • Variances unequal but normal → Welch’s t (separate-variance formula for SE).

• • Non-normal & small $n$ → Mann-Whitney U (between-subjects) or Wilcoxon Signed-Rank (within-subjects).

• • Enormous $n$ means parametric tests tolerate non-normality – CLT saves the day.

Between- vs. Within-Subjects Designs

• • Between: “independent-samples” – each participant provides data for only one group.

• • Within: “paired,” “matched,” “repeated measures” – same participant supplies data for both levels of the IV.

  • ◦ Lower error variance → greater statistical power.

  • ◦ Fewer participants needed; individuals act as their own controls.

Paired-Samples t Test: Mechanics

• • Compute each person’s difference score: $d<em>i = X</em>{\text{angry},i} - X_{\text{happy},i}$.

• • Example numbers:

  • ◦ $\bar{d} = 12 \text{ ms}$.

  • ◦ $SE_{\bar{d}} = 3 \text{ ms}$.

  • ◦ Test statistic: $t = \dfrac{\bar{d}}{SE_{\bar{d}}} = \dfrac{12}{3}=4$ (sign & rounding vary in slides).

• • Assumptions:

  1. Independence of pairs (each participant unrelated to other participants).

  2. Difference-score distribution ≈ normal (again, trivial if $n$ large).

• • Nonparametric fallback: Wilcoxon Signed-Rank.

  • ◦ Categorize pairs where $d<em>i>0$ vs. $d</em>i<0$; significant imbalance indicates effect.

Reporting Format (APA-style)

• • Example: t(df$=29$) = $-3.97$, $p<.001$, $d = 0.65$.

• • Why include df? Lets readers reconstruct the critical value & sample size.

Effect Size: Cohen’s $d$

• • Formula: $d = \dfrac{\bar{X}<em>1 - \bar{X}</em>2}{s_{\text{pooled}}}$.

  • ◦ $s<em>{\text{pooled}} = \sqrt{ \dfrac{(n</em>1-1)s<em>1^2 + (n</em>2-1)s<em>2^2}{n</em>1+n_2-2} }$.

• • Interpretation: number of SDs separating the two means.

• • Benchmarks (Cohen, 1988):

  • ◦ $d \approx 0.2$ → “small” (≈ $83\%$ overlap).

  • ◦ $d \approx 0.5$ → “medium” (≈ $67\%$ overlap).

  • ◦ $d \approx 0.8$ → “large” (≈ $54\%$ overlap).

  • ◦ Example slide showed giant case $d=2$ (means 2 SD apart).

Statistical Power & Sample Size

• • Power = $1 - \beta$ = probability of detecting a true effect.

• • Large $d$ + modest $n$ → still decent power.

• • Small $d$ requires very large $n$ (else Type-II error risk skyrockets).

Tidal Analogy (Metaphor)

• • Water bottle ⇢ pond ⇢ Lake Texcoco ⇢ Lake Superior ⇢ ocean:

  • ◦ Tidal effect truly exists in all bodies but magnitude is tiny relative to small samples.

  • ◦ Only with “huge sample” (ocean-scale) can the ~5 mm change become detectable.

• • Moral: theory may posit a real yet minuscule effect; plan $n$ accordingly or risk missing it.

Advantages of Within-Subjects (Re-framed)

1. Fewer participants (cost & logistics).

2. Drastic reduction of extraneous-variable noise (each subject controls for self).

3. Greater statistical power: smaller SE in denominator of $t$ increases chance of finding real effect.

Common Misconceptions About $p$ Values

• • “Tiny $p$ means huge effect.” ✘ Not necessarily; could be huge $n$.

• • “Rejecting $H_0$ proves my research hypothesis.” ✘ Type-I errors exist.

• • “Failing to reject $H_0$ proves no effect.” ✘ Could be under-powered (Type-II error).

• • “Significant result guarantees a well-designed study.” ✘ Methodological flaws can still yield false significance.

Practical & Ethical Implications

• • Researchers must justify sample size via a priori power analysis (ethical duty to avoid wasted resources/participants).

• • Must test & report assumption checks (transparency, reproducibility).

• • Avoid “p-value fishing” – understand true interpretation of statistical outputs.

Quick Reference Equations

• • Pooled SD: $s<em>{\text{pooled}} = \sqrt{\dfrac{\sum (n</em>k-1)s<em>k^2}{\sum n</em>k - g}}$, where $g$ = number of groups.

• • Independent-samples $t$: $t = \dfrac{\bar{X}<em>1-\bar{X}</em>2}{\sqrt{\dfrac{s<em>1^2}{n</em>1}+\dfrac{s<em>2^2}{n</em>2}}}$.

• • Paired-samples $t$: $t = \dfrac{\bar{d}}{s_d/\sqrt{n}}$.

• • Welch’s adjustment to df: $df^* = \dfrac{ \left(\dfrac{s<em>1^2}{n</em>1}+\dfrac{s<em>2^2}{n</em>2}\right)^2 }{ \dfrac{(s<em>1^2/n</em>1)^2}{n<em>1-1} + \dfrac{(s</em>2^2/n<em>2)^2}{n</em>2-1} }$.

End-of-Lecture Wrap-Up & Next Steps

• • Hardest material (sampling logic) front-loaded; remaining t-test variants straightforward.

• • Thursday: apply today’s concepts while co-designing the class experiment.