Hypothesis Testing Principles and Methods

Foundations of Hypothesis Testing

  • Purpose: Evaluates sample data to assess the relative plausibility of competing claims regarding unknown population parameters, determining whether observed differences reflect true effects or random chance.

  • Null Hypothesis (H0H_0): Represents the default claim of no effect, no difference, or status quo; always contains an equality condition (e.g., H0:μ=μ0H_0: \mu = \mu_0, H0:p=p0H_0: p = p_0) and is presumed true until sufficient sample evidence indicates otherwise.

  • Alternative Hypothesis (HAH_A): Represents the operational research question asserting an effect, difference, or parameter shift.

    • Two-Sided (Two-Tailed): Detects any non-directional difference (HA:μ≠μ0H_A: \mu \neq \mu_0).

  • Right-Tailed (Upper-Tailed): Detects an increase or value higher than the null parameter (HA:μ>μ0H_A: \mu > \mu_0).

  • Left-Tailed (Lower-Tailed): Detects a decrease or value lower than the null parameter (HA:μ<μ0H_A: \mu < \mu_0).

Hypothesis Testing Procedure

Flowchart of Steps in Hypothesis Testing
  • Step 1: State the Hypotheses: Specify H0H_0 and HAH_A, identifying directionality (left-, right-, or two-tailed).

  • Step 2: Select Significance Level (α\alpha): Predetermine the acceptable probability of committing a Type I error (standard values: α=0.05\alpha = 0.05, α=0.01\alpha = 0.01).

  • Step 3: Calculate Test Statistic: Standardize sample findings relative to null expectations using ZZ or tt.

  • Step 4: Determine the pp-Value or Rejection Region:

    • Rejection Region (Critical Value) Method: Compare the test statistic directly to critical threshold values established by α\alpha and sample size.

    • pp-Value Method: Calculate the probability of obtaining sample results as extreme as, or more extreme than, those observed given that H0H_0 is true.

  • Step 5: Make a Statistical Decision:

    • If the test statistic falls into the critical rejection region (or p≤αp \le \alpha), reject H0H_0.

    • If the test statistic falls in the region of no significance (or p>αp > \alpha), fail to reject H0H_0. Correct terminology requires stating "fail to reject H0H_0" or "do not reject H0H_0"; never conclude "accept H0H_0" or that H0H_0 is proven.

  • Step 6: State Conclusion in Context: Express findings in relation to the original research scenario, explicitly reporting the test statistic, α\alpha, and pp-value.

Decision Errors and Statistical Power

Type I and Type II Error Matrix
  • Type I Error (α\alpha): Occurs when a true null hypothesis is rejected (false-positive conclusion). The probability is set by the investigator prior to sampling: α=P(Reject H0∣H0 is True)\alpha = P(\text{Reject } H_0 \mid H_0 \text{ is True}).

  • Type II Error (β\beta): Occurs when a false null hypothesis is not rejected (false-negative conclusion): β=P(Fail to Reject H0∣HA is True)\beta = P(\text{Fail to Reject } H_0 \mid H_A \text{ is True}).

  • Statistical Power (1−β1 - \beta): The probability of correctly rejecting a false null hypothesis, confirming the validity of a true alternative hypothesis.

  • Error Trade-off: Lowering α\alpha increases β\beta, and lowering β\beta increases α\alpha.

  • pp-Value Interpretation Guidelines:

    • p>0.05p > 0.05: Results not statistically significant.

    • 0.05≤p≤0.100.05 \le p \le 0.10: Trend toward statistical significance.

    • 0.01≤p≤0.050.01 \le p \le 0.05: Statistically significant.

    • 0.001≤p≤0.010.001 \le p \le 0.01: Highly significant.

    • p<0.001p < 0.001: Very highly significant.

One-Sample Tests for Continuous Outcomes

  • Large Sample (n>30n > 30) — ZZ-Test:

    • Formula:     Z=Xˉ−μ0snZ = \frac{\bar{X} - \mu_0}{\frac{s}{\sqrt{n}}}

    • Two-sided critical value at α=0.05\alpha = 0.05 is ±1.960\pm 1.960.

    • Framingham Heart Study Total Cholesterol Example: H0:μ=203H_0: \mu = 203, HA:μ≠203H_A: \mu \neq 203, n=3310n = 3310, Xˉ=200.3\bar{X} = 200.3, s=36.8s = 36.8.     Z=200.3−20336.83310=−4.22Z = \frac{200.3 - 203}{\frac{36.8}{\sqrt{3310}}} = -4.22     Because −4.22<−1.96-4.22 < -1.96, reject H0H_0 (p<0.0001p < 0.0001); participants differ significantly from the national average.

  • Small Sample (n≤30n \le 30) — tt-Test:

    • Formula:     t=Xˉ−μ0snt = \frac{\bar{X} - \mu_0}{\frac{s}{\sqrt{n}}}

    • Degrees of freedom: df=n−1df = n - 1.

    • Diabetic Diastolic BP Example: H0:μ=74.4 mm HgH_0: \mu = 74.4\,\text{mm Hg}, HA:μ≠74.4 mm HgH_A: \mu \neq 74.4\,\text{mm Hg}, n=10n = 10, Xˉ=84 mm Hg\bar{X} = 84\,\text{mm Hg}, s=9.1 mm Hgs = 9.1\,\text{mm Hg}, df=9df = 9. Two-sided critical value at α=0.05\alpha = 0.05 is ±2.262\pm 2.262.     t=84−74.49.110=3.34t = \frac{84 - 74.4}{\frac{9.1}{\sqrt{10}}} = 3.34     Because 3.34>2.2623.34 > 2.262, reject H0H_0; mean diastolic BP differs significantly from the general population.

One-Sample Tests for Dichotomous Outcomes

  • Sample Proportion:   p=xnp = \frac{x}{n}   where xx is positive occurrences and nn is total subjects.

  • Test Statistic (ZZ-Test for Proportions):   Z=p−p0p0(1−p0)nZ = \frac{p - p_0}{\sqrt{\frac{p_0(1 - p_0)}{n}}}

  • Directional Critical Values (α=0.05\alpha = 0.05):

    • Lower-tailed: Zcrit=−1.645Z_{\text{crit}} = -1.645.

    • Upper-tailed: Zcrit=+1.645Z_{\text{crit}} = +1.645.

  • Framingham Heart Study Smoking Example (Lower-Tailed):

    • H0:p=0.211H_0: p = 0.211, HA:p<0.211H_A: p < 0.211, n=3536n = 3536, x=482x = 482, p=0.136p = 0.136, α=0.05\alpha = 0.05.

    • Calculated test statistic: Z=−10.93Z = -10.93.

    • Decision: −10.93<−1.645-10.93 < -1.645; reject H0H_0 (p<0.0001p < 0.0001). Smoking prevalence is significantly lower.

  • Newborn Gender Example (Upper-Tailed):

    • H0:p=0.5H_0: p = 0.5, HA:p>0.5H_A: p > 0.5, n=25468n = 25468, x=13173x = 13173, p=0.5172p = 0.5172, α=0.05\alpha = 0.05.

    • Calculated test statistic: Z=5.49Z = 5.49.

    • Decision: 5.49>+1.6455.49 > +1.645; reject H0H_0. Male birth prevalence is significantly higher than female birth prevalence.

Two Matched Samples for Continuous Outcomes

  • Design: Paired pre- and post-intervention evaluations on single subjects (nn pairs), yielding mean difference XˉD\bar{X}_D and standard deviation of difference sds_d.

  • Hypotheses: H0:μd=0H_0: \mu_d = 0 versus HA:μd≠0H_A: \mu_d \neq 0 (or directional: μd>0\mu_d > 0, μd<0\mu_d < 0).

  • Test Statistic:

    • For n>30n > 30:     Z=XˉD−μdsdnZ = \frac{\bar{X}_D - \mu_d}{\frac{s_d}{\sqrt{n}}}

    • For n≤30n \le 30 (df=n−1df = n - 1):     t=XˉD−μdsdnt = \frac{\bar{X}_D - \mu_d}{\frac{s_d}{\sqrt{n}}}

  • Framingham Offspring Study Systolic BP Example:

    • H0:μd=0H_0: \mu_d = 0, HA:μd≠0H_A: \mu_d \neq 0, n=15n = 15, XˉD=−5.3\bar{X}_D = -5.3, sd=12.8s_d = 12.8, α=0.05\alpha = 0.05, df=14df = 14.

  • Critical values: ±2.145\pm 2.145.

  • Calculated statistic:   t=−5.3−012.815=−1.60t = \frac{-5.3 - 0}{\frac{12.8}{\sqrt{15}}} = -1.60

  • Decision: −2.145<−1.60<2.145-2.145 < -1.60 < 2.145; fail to reject H0H_0. There is no statistically significant difference in systolic blood pressures over time.