Comprehensive Study Notes on Hypothesis Testing and Comparing Means

Logic of Hypothesis Testing

• Combining Probabilities:

  • Multiple non-significant findings, when taken together, can provide strong support for an effect.

  • Combining individual non-significant probability values (e.g., $p = 0.11$ and $p = 0.07$) using statistical probability combination methods results in a combined probability value of $p = 0.045$, which is statistically significant.

• Confidence Intervals and Small Effects:

  • Statistical analyses can never provide a basis for concluding that an effect is exactly zero.

  • Confidence intervals can demonstrate that an effect is likely small. If all values within a confidence interval are small, the effect can be concluded to be small.

  • Example: In an insomnia treatment study comparing mean time to fall asleep between a treatment group and a control group, the treatment group fell asleep an average of $2\,min$ faster, but the difference was non-significant. If the 95% confidence interval for the mean difference ranges from $-4$ to $8\,min$, the benefit can be concluded to be $8\,min$ or less. This does not justify concluding that the null hypothesis is true or supported.

• Four Steps in Significance Testing:

  • Step 1: Specify the Null Hypothesis ($H_0$):

  • For a two-tailed test, $H_0$ typically states that a parameter equals zero ($\mu_1 - \mu_2 = 0$, which is equivalent to $\mu_1 = \mu_2$).

  • For a one-tailed test, $H_0$ states that a parameter is $\ge 0$ or $\le 0$. If predicting $\mu_1 > \mu_2$, the null hypothesis (the reverse of the prediction) is $\mu_2 - \mu_1 \ge 0$ ($\mu_1 \le \mu_2$).

  • Step 2: Specify the Significance Level (Alpha Level, $\alpha$):

  • Typical significance levels are $\alpha = 0.05$ and $\alpha = 0.01$.

  • Step 3: Compute the Probability Value ($p$-value):

  • The $p$-value is the probability of obtaining a sample statistic as extreme as or more extreme than the parameter specified in $H_0$, assuming $H_0$ is true.

  • Step 4: Compare the Probability Value with the Alpha Level:

  • If the $p$-value is less than $\alpha$, reject $H_0$. Lower $p$-values indicate higher confidence that $H_0$ is false.

  • If the $p$-value is greater than $\alpha = 0.05$, findings are considered inconclusive. Failure to reject $H_0$ does not constitute support for $H_0$.

• Significance Testing and Confidence Intervals:

  • If a statistic is significantly different from $0$ at the $0.05$ level, the 95% confidence interval will not contain $0$.

  • All values inside a confidence interval represent plausible parameter values; values outside the interval are rejected as plausible parameter values.

  • In the Physicians' Reactions case study, the 95% confidence interval for the population mean difference ranges from $2.00$ to $11.26$. Because $0$ is outside this interval (less than $2.00$), zero is rejected as a plausible value, resulting in a statistically significant test ($p = 0.0057$).

  • Similarly, significance at the $0.01$ level implies the 99% confidence interval will not contain $0$

  • A significant finding establishes the direction of an effect because all values in the confidence interval will share the same sign (all positive or all negative).

  • When a 95% confidence interval contains zero, the effect is non-significant at the $0.05$ level. Zero cannot be rejected, but an infinite number of other continuous parameter values in the interval also cannot be rejected, which is why non-significant results should never be interpreted as accepting $H_0$.

• Common Misconceptions:

  • Misconception 1: The probability value is the probability that the null hypothesis is false.

  • Correction: The probability value is the probability of obtaining data as extreme as or more extreme than observed, assuming $H_0$ is true ($P(\text{Data}|H_0)$).

  • Misconception 2: A low probability value indicates a large effect.

  • Correction: A low probability value indicates that the sample outcome is unlikely under $H_0$. Small effect sizes can produce very low $p$-values if the sample size is large.

  • Misconception 3: A non-significant outcome means that the null hypothesis is probably true.

  • Correction: A non-significant outcome indicates that the data fail to conclusively demonstrate that $H_0$ is false.

• Statistical Literacy Application (Higgs Boson Research):

  • Initial research in March 2012 found evidence for the Higgs Boson particle, but the result was not statistically significant. Investigators noted tantalizing evidence existed without being strong enough for definitive statements. Subsequent studies collected sufficient data to reach statistical significance.

Testing a Single Mean

• Testing a Single Mean When Population Standard Deviation ($\sigma$) is Known:

  • A subliminal message experiment presents $9$ subjects with $100$ picture pairs. The hypothesized population mean under no subliminal effect is $\mu = 50$. The sample score frequencies are:

• Sample mean $M = 51$, representing a difference of $1$ from $\mu = 50$

• Binomial variance for $100$ trials with $\pi = 0.5$:

$\sigma^2 = N \pi (1 - \pi) = 100 \times 0.5 \times (1 - 0.5) = 25$

• Population standard deviation $\sigma = \sqrt{25} = 5$

• The standard error of the mean ($\sigma_M$) formula is:

$\sigma_M = \frac{\sigma}{\sqrt{N}} = \frac{5}{\sqrt{9}} = 1.667$

• One-Tailed Test Calculation:

* Under a normal distribution with $\mu = 50$ and $\sigma_M = 1.667$, the one-tailed probability of obtaining $M \ge 51$ is $0.2743$. Because this outcome is not unlikely under $H_0$, the effect is non-significant.

• Two-Tailed Test Calculation:

* Calculating the probability of $M \le 49$ or $M \ge 51$ yields $p = 0.5486$, which is exactly double the one-tailed probability.

• Standard Normal ($Z$) Transformation:

$Z = \frac{M - \mu}{\sigma_M}$

$Z = \frac{51 - 50}{1.667} = 0.60$

* The area on a standard normal distribution ($\mu = 0$, $\sigma = 1$) above $Z = 0.60$ equals $0.2743$

• Testing a Single Mean When Population Standard Deviation ($\sigma$) is Estimated ($t$-Test):

  • In practical applications, $\sigma$ is almost always unknown and estimated using sample standard deviation $s$.

  • Data from the ADHD Treatment case study ($N = 24$ children tested on a Delay of Gratification task across dosage levels):

• The difference column (Diff) represents performance in the $0.6\,mg$ condition (D60) minus the control condition (D0). Null hypothesis: $\mu_{Diff} = 0$

• General formula for $t$:

• Single mean $t$-test formula:

$t = \frac{M - \mu}{s_M}$

• For the $N = 24$ difference scores: sample mean $M = 4.958$, sample standard deviation $s = 7.5382$, hypothesized $\mu = 0$

• Estimated standard error of the mean ($s_M$):

s_M = \frac{s}{\sqrt{N}} = \frac{7.5382}{\sqrt{24}} = 1.54

• Test statistic calculation:

$t = \frac{4.96 - 0}{1.54} = 3.22$

• Degrees of freedom $df = N - 1 = 24 - 1 = 23$

• Two-tailed probability for $|t| \ge 3.22$ is $p = 0.0038$. The null hypothesis is rejected, concluding that the drug condition yields higher scores than the placebo condition.

  • Assumptions of Single-Mean $t$-Tests:

• 1. Each value is sampled independently from every other value.

• 2. Values are sampled from a normally distributed population.

Testing Differences Between Two Means (Independent Groups)

• Experimental Setup and Research Question:

  • Evaluation of ratings (on a 7-point scale) of whether animal research is wrong from the Animal Research case study:

• Females rated animal research as more wrong ($M = 5.353$, $s^2 = 2.743$, $n = 17$) than males ($M = 3.882$, $s^2 = 2.985$, $n = 17$). Mean difference = $1.4705$

  • Assumptions for Independent-Groups $t$-Tests:

• 1. Homogeneity of Variance: The two populations have equal variance ($\sigma_1^2 = \sigma_2^2$).

• 2. Normality: Both populations are normally distributed.

• 3. Independence: Each value is sampled independently from every other value. Each subject must contribute exactly one score.

• Note: Small-to-moderate violations of normality and homogeneity of variance do not severely invalidate the test, but independence must not be violated.

  • Computational Steps for Equal Sample Sizes:

• Step 1: Compute the difference between sample means ($M_1 - M_2 = 5.3529 - 3.8824 = 1.4705$

• Step 2: Calculate Pooled Variance / Mean Square Error ($MSE$):

$MSE = \frac{s_1^2 + s_2^2}{2} = \frac{2.743 + 2.985}{2} = 2.864$

• Step 3: Compute the estimated standard error of the difference between means ($s_{M_1 - M_2}$):

s_{M_1 - M_2} = \sqrt{\frac{2 MSE}{n}} = \sqrt{\frac{2 \times 2.864}{17}} = 0.5805

• Step 4: Compute $t$:

t = \frac{1.4705}{0.5805} = 2.533

• Step 5: Determine degrees of freedom and probability value:

df = (n_1 - 1) + (n_2 - 1) = 16 + 16 = 32

* Two-tailed $p = 0.0164$; one-tailed $p = 0.0082$

• Formatting Data for Computer Analysis:

  • Statistical software requires data structure organized by grouping and dependent variables:

• Computations for Unequal Sample Sizes (Optional):

  • When sample sizes differ, $MSE$ weights the larger group more via Sum of Squares Error ($SSE$):

• Group 1: $3, 4, 5$ ($M_1 = 4$, $n_1 = 3$); Group 2: $2, 4$ ($M_2 = 3$, $n_2 = 2$)

$SSE = \sum (X - M_1)^2 + \sum (X - M_2)^2$

$SSE = (3-4)^2 + (4-4)^2 + (5-4)^2 + (2-3)^2 + (4-3)^2 = 1 + 0 + 1 + 1 + 1 = 4$

• Degrees of freedom error: $df_e = (n_1 - 1) + (n_2 - 1) = (3 - 1) + (2 - 1) = 3$

• Calculate $MSE$:

$MSE = \frac{SSE}{df_e} = \frac{4}{3} = 1.333$

• Harmonic mean of sample sizes ($n_h$):

n_h = \frac{2}{\frac{1}{n_1} + \frac{1}{n_2}} = \frac{2}{\frac{1}{3} + \frac{1}{2}} = 2.4

• Standard error calculation using $n_h$:

s_{M_1 - M_2} = \sqrt{\frac{2 MSE}{n_h}} = \sqrt{\frac{2 \times 1.333}{2.4}} = 1.054

• Test statistic and probability:

t = \frac{4 - 3}{1.054} = 0.949

* Two-tailed $p = 0.413$

All Pairwise Comparisons Among Means

• Problem of Multiple Comparisons and Inflated Type I Error:

  • Performing individual $t$-tests across all possible pairs of means inflates the familywise Type I error rate beyond $\alpha$

  • The number of pairwise comparisons for $k$ means is given by $\frac{k(k-1)}{2}$.

* For $k = 2$ means: $1$ comparison.
* For $k = 12$ means: $66$ comparisons.

• Probability of Type I error increases dramatically with number of means:

* For $12$ means, the probability of at least one false positive among $66$ comparisons is approximately $0.70$

• Tukey's Honestly Significant Difference (HSD) Test:

  • Maintains the familywise Type I error rate using the studentized range distribution.

  • Data from the Smiles and Leniency case study:

* Sample size $n = 34$ per condition (Total $N = 136$).

• Compute $MSE$ (mean of group variances):

MSE = \frac{3.34 + 2.83 + 2.11 + 2.32}{4} = 2.65

• Compute Tukey $Q$ statistic for each pair:

Q = \frac{M_i - M_j}{\sqrt{\frac{MSE}{n}}}

* Standard error term denominator: $\sqrt{\frac{2.65}{34}} = 0.279$
* Error degrees of freedom: $df = N - k = 136 - 4 = 132$

• Results of Six Pairwise Comparisons:

* False vs. Felt: $M_i - M_j = 0.46$, $Q = 1.65$, $p = 0.649$
* False vs. Miserable: $M_i - M_j = 0.46$, $Q = 1.65$, $p = 0.649$
* False vs. Neutral: $M_i - M_j = 1.25$, $Q = 4.48$, $p = 0.010$ (Statistically Significant)
* Felt vs. Miserable: $M_i - M_j = 0.00$, $Q = 0.00$, $p = 1.000$
* Felt vs. Neutral: $M_i - M_j = 0.79$, $Q = 2.83$, $p = 0.193$
* Miserable vs. Neutral: $M_i - M_j = 0.79$, $Q = 2.83$, $p = 0.193$

• Interpretation of Pairwise Comparisons:

  • The apparent contradiction (False equals Miserable, Miserable equals Neutral, but False differs from Neutral) is resolved by avoiding accepting $H_0$ on non-significant tests. Failing to reject $H_0$ indicates insufficient evidence of a difference, not proof of equality. The false smile yields higher leniency than neutral control, while miserable smile is either equal to false smile, equal to control, or intermediate.

• Assumptions and Computer Integration:

  • Assumptions: Normality, homogeneity of variance, and independent observations. Robust to normality violations, but sensitive to variance heterogeneity and non-independence.

  • ANOVA Summary Table for Smiles and Leniency Data:

* Mean Square Error ($MSE$) equals $2.6489$ ($MS_{Error}$).

• Tukey's test does not require a prior significant ANOVA $F$-test to be valid (Wilkinson et al., 1999).

  • Unequal Sample Sizes in Tukey Test (Optional):

• Calculate $SSE$ and $df_e = N - k$, obtain $MSE = \frac{SSE}{df_e}$, and replace $n$ in the $Q$ formula with the harmonic mean ($n_h$) for each specific pair being compared.

Specific Comparisons (Independent Groups)

• Linear Combinations and Contrast Coefficients:

  • Planned comparisons test specific hypotheses formulated prior to data collection.

  • Data from a $2 \times 2$ factorial experiment ($N = 24$, $n = 6$ per cell) assessing attribution of outcome to self based on Self-Esteem (High=1, Low=2) and Task Outcome (Success=1, Failure=2):

  • Means: Success/High = $7.333$, Success/Low = $5.500$, Failure/High = $4.833$, Failure/Low = $7.833$

  • Main effect contrast for Outcome (Success vs. Failure):

  • Mean Success = $\frac{7.3333 + 5.5000}{2} = 6.4167$

  • Mean Failure = $\frac{4.8333 + 7.8333}{2} = 6.3333$

  • Mean Difference = $6.4167 - 6.3333 = 0.083$

  • Coefficients ($c_i$): Success/High ($+0.5$), Success/Low ($+0.5$), Failure/High ($-0.5$), Failure/Low ($-0.5$

  • Linear Combination ($L$):

L = \sum c_i M_i = 0.5(7.333) + 0.5(5.500) - 0.5(4.833) - 0.5(7.833) = 0.083

• Significance test formula for $L$:

t = \frac{L}{\sqrt{\frac{\sum c_i^2 MSE}{n}}}

* Cell variances: Success/High = $1.867$, Success/Low = $1.100$, Failure/High = $2.167$, Failure/Low = $1.367$
* Pooled Variance ($MSE$) = \frac{1.867 + 1.100 + 2.167 + 1.367}{4} = 1.625\n    * Sum of squared coefficients: \sum c_i^2 = (0.5)^2 + (0.5)^2 + (-0.5)^2 + (-0.5)^2 = 1.0\n    * Standard error of L$:$\sqrt{\frac{1.0 \times 1.625}{6}} = 0.5204\n    * Test statistic: t = \frac{0.083}{0.5204} = 0.160\n    * Degrees of freedom: df = N - k = 24 - 4 = 20\n    * Two-tailed p = 0.874 (Non-significant main effect).\n\n* **Interaction Contrasts (Difference Between Differences):**\n  * Evaluating whether the effect of outcome differs across self-esteem levels:\n    * High self-esteem difference (Success - Failure): 7.333 - 4.833 = 2.500\n    * Low self-esteem difference (Success - Failure): 5.500 - 7.833 = -2.333\n    * Difference between differences: 2.500 - (-2.333) = 4.833\n    * Coefficients (c_i$): Success/High ($+1$), Failure/High ($-1$), Success/Low ($-1$), Failure/Low ($+1\n    * Linear combination: L = 1(7.333) - 1(4.833) - 1(5.500) + 1(7.833) = 4.833\n    * Sum of squared coefficients: \sum c_i^2 = 1^2 + (-1)^2 + (-1)^2 + 1^2 = 4\n    * Standard error of L$:$\sqrt{\frac{4 \times 1.625}{6}} = 1.0408\n    * Test statistic: t = \frac{4.833}{1.0408} = 4.64\n    * Degrees of freedom df = 20\n    * Two-tailed p = 0.0002 (Statistically significant interaction).\n\n* **Multiple Comparisons and Familywise Error Control:**\n  * **Per-comparison Error Rate (\alpha):** Probability of Type I error for a single comparison.\n  * **Familywise Error Rate (FW):** Probability of making one or more Type I errors across a family of comparisons.\n  * **Bonferroni Inequality:**\n\nFW \le c \alpha\n\n    * Where c is the number of comparisons.\n  * **Bonferroni Correction:** Set per-comparison significance level to \alpha / c$to keep familywise error rate below$\alpha.\n  * Controlling familywise error reduces statistical power. Familywise control is unnecessary when comparisons test independent hypotheses.\n\n* **Orthogonal Comparisons:**\n  * Comparisons are independent (orthogonal) if the sum of the products of their coefficients equals zero:\n\n\sum c_{1i} c_{2i} = 0\n\n  * Example of Orthogonal Contrasts:\n    * C_1 = (0.5, 0.5, -0.5, -0.5)\n    * C_2 = (1, -1, -1, 1)\n    * Product sum: 0.5(1) + 0.5(-1) + (-0.5)(-1) + (-0.5)(1) = 0.5 - 0.5 + 0.5 - 0.5 = 0\n  * Example of Non-Orthogonal Contrasts:\n    * C_1 = (0.5, -0.5, 0.5, -0.5)\n    * C_2 = (0.5, -0.5, 0.0, 0.0)\n    * Product sum: 0.25 + 0.25 + 0.0 + 0.0 = 0.5 

e 0\n\n\n# Difference Between Two Means (Correlated Pairs)\n\n* **Rationale for Repeated Measures / Correlated Pairs Designs:**\n * In within-subjects designs, scores across conditions originate from the same subjects, creating dependent observations.\n * ADHD Treatment case study (N = 24$children tested in both D0 placebo and D60$0.6\,mg conditions):\n * High correlation between conditions (r = 0.80\n * An independent-groups t$-test on these data yields$t = 1.42$,$df = 46$,$p = 0.15 (non-significant).\n * Analyzing as correlated pairs yields M_{Diff} = 4.96$,$t = 3.22$,$df = 23$,$p = 0.0038 (significant).\n * Correlated t-tests possess greater statistical power because subject variance is removed, reducing standard error.\n\n* **Variance Sum Law and Standard Error Reduction:**\n * Variance of difference scores (\sigma_{X - Y}^2):\n\n\sigma_{X - Y}^2 = \sigma_X^2 + \sigma_Y^2 - 2 r \sigma_X \sigma_Y\n\n * For D0 (s^2 = 128.02$,$s = 11.31$) and D60 ($s^2 = 151.78$,$s = 12.32$) with$r = 0.80:\n\ns_{Diff}^2 = 128.02 + 151.78 - 2(0.80)(11.31)(12.32) = 56.82\n\n * Higher positive correlation r$decreases difference score variance, reducing standard error and increasing$t.\n\n\n# Specific Comparisons (Correlated Observations)\n\n* **Experimental Framework (Weapons and Aggression Study):**\n * N = 32 subjects read aggressive or control words primed by weapon or non-weapon words:\n * `aw`: aggressive word, weapon prime\n * `an`: aggressive word, non-weapon prime\n * `cw`: control word, weapon prime\n * `cn`: control word, non-weapon prime\n * **Main Effect Comparison (L_1$):** Priming effect across all words ($L_1 = (an + cn) - (aw + cw)).\n * Sample mean of L_1 = 5.875\,ms$,$s_M = 4.2646\n * Test statistic: t = \frac{5.875}{4.2646} = 1.378$,$df = 31$, two-tailed$p = 0.178\n * **Interaction Comparison (L_2$):** Differential priming for aggressive vs. control words ($L_2 = (an - aw) - (cn - cw)).\n * Sample mean of L_2 = 8.4375\,ms$,$s_M = 3.9128\n * Test statistic: t = \frac{8.4375}{3.9128} = 2.156$,$df = 31$, two-tailed$p = 0.039 (Statistically Significant).\n\n* **Orthogonal Comparisons in Correlated Designs:**\n * Correlation between contrast scores L_1$and$L_2$across subjects is$r = 0.24. True orthogonal comparisons with correlated observations are rare.\n\n\n# Pairwise Comparisons (Correlated Observations)\n\n* **Limitations of Tukey HSD for Correlated Observations:**\n * Tukey HSD assumes equal variance of difference scores across all pairwise condition pairs, an assumption usually violated in repeated measures.\n\n* **Methodology (Pairwise Correlated t-Tests with Bonferroni Correction):**\n * Stroop Interference study (N = 47 subjects across Words [W], Color [C], and Interference [I] conditions):\n\n| Comparison | Mean Difference | Standard Deviation (s$) | Standard Error ($s_M$) |$t$|$p |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| W - C | -4.15$|$2.99$|$0.44$|$-9.53$|$< 0.001 |\n| W - I | -20.51$|$7.84$|$1.14$|$-17.93$|$< 0.001 |\n| C - I | -16.36$|$7.47$|$1.09$|$-15.02$|$< 0.001 |\n\n * Bonferroni adjusted alpha threshold for 3$comparisons:$\alpha / 3 = 0.05 / 3 = 0.0167.\n * All three pairwise differences are statistically significant (p < 0.001).\n\n\n# Review Exercises and Solutions Framework\n\n* **Question 5 (Sampling Distribution Visualization):**\n\n\n\n * For H_0: \mu = 50$,$M = 60$falls far in the tail; values exceeding approximately$\pm 3.27$standard errors are required for significance at$\alpha = 0.05.\n\n* **Question 19 (Homogeneity of Variance Violation):**\n\n\n\n * Population 1 (variance \sigma^2 = 1.0$) and Population 2 (variance$\sigma^2 = 4.0$$) depict a clear violation of the equal variance assumption.