Comprehensive Guide to Confidence Intervals and Hypothesis Testing Inversion

Python Environment and Initial Data Preparation

• The core Python computational stack for hypothesis testing and confidence interval estimation comprises three primary libraries:

  • pandas (imported as pd) for tabular data ingestion, filtering, and summary statistics.
  • numpy (imported as np) for vectorized array manipulation and numerical operations such as square roots.
  • scipy.stats (imported from scipy) for computing standard normal probability density functions, cumulative distribution functions, and percent point functions.

• Python import setup:

import pandas as pd
import numpy as np
from scipy import stats

• The High Time Resolution Universe (HTRU) Pulsar Survey dataset is utilized to analyze the physical properties of pulsars, specifically treating confirmed pulsars as a random sample of all pulsars in the universe.

  • Remote dataset location: "https://github.com/UM-Data-Science-101/data/raw/refs/heads/main/Old/HTRU_2.csv"
  • Attribute column names assigned during loading:
  1. "intensity"
  2. "profile_sd"
  3. "profile_excess_kurtosis"
  4. "profile_skewness"
  5. "snr_mean"
  6. "snr_sd"
  7. "snr_excess_kurtosis"
  8. "snr_skewness"
  9. "pulsar"

• Loading and filtering data in Python:

htru_url = ("https://github.com/UM-Data-Science-101/data/raw/" "refs/heads/main/Old/HTRU_2.csv")
htru = pd.read_csv(htru_url, names=["intensity", "profile_sd", "profile_excess_kurtosis", "profile_skewness", "snr_mean", "snr_sd", "snr_excess_kurtosis", "snr_skewness", "pulsar"])
pulsars = htru.loc[htru["pulsar"] == 1]
n = len(pulsars)

• The resulting sample size of confirmed pulsars is $n = 1639$.

Two-Sided Hypothesis Testing for Population Mean Intensity

• Two-sided hypothesis testing evaluates whether an unknown population expected value $E(X)$ equals a specific hypothesized constant $c$, against the alternative that it differs from $c$:

  • Null hypothesis: $H_0: E(X) = c$
  • Alternative hypothesis: $H_1: E(X) \neq c$

• Computing sample statistics for pulsar intensity:

  • Sample mean (observed statistic): observed_statistic = pulsars["intensity"].mean() $\approx 56.690608$
  • Sample variance: s2 = pulsars["intensity"].var()
  • Standard error of the mean: standard_error = np.sqrt(s2 / n) $\approx 0.741214$

• Standard error computation formula: $\text{SE} = \sqrt{\frac{s^2}{n}} = \frac{s}{\sqrt{n}}$

• Decision rule for a significance level $\alpha = 0.05$ (5% probability of committing a Type I error):

  • Under the standard normal approximation, reject $H_0$ if the observed sample mean lies more than 2 standard errors away from the hypothesized value $c$.
  • Rejection threshold: reject if $|z| > 2$, where the test statistic is defined by: $z = \frac{\text{observed\_statistic} - c}{\text{standard\_error}}$

• Test 1: Hypothesized mean $c = 60$

  • Hypotheses: $H_0: E(X) = 60$ vs $H_1: E(X) \neq 60$

• Computed test statistic: $z = \frac{56.690608 - 60}{0.741214} \approx -4.464829725202467$

  • Conclusion: Because $|-4.4648| > 2$, the test statistic falls deep within the rejection region. Reject the null hypothesis $E(X) = 60$.

• Test 2: Hypothesized mean $c = 57$

  • Hypotheses: $H_0: E(X) = 57$ vs $H_1: E(X) \neq 57$

• Computed test statistic: $z = \frac{56.690608 - 57}{0.741214} \approx -0.4174130007366421$

  • Conclusion: Because $|-0.4174| < 2$, the test statistic does not exceed the critical boundary. Fail to reject the null hypothesis $E(X) = 57$.

• Test 3: Hypothesized mean $c = 55$

  • Hypotheses: $H_0: E(X) = 55$ vs $H_1: E(X) \neq 55$

• Computed test statistic: $z = \frac{56.690608 - 55}{0.741214} \approx 2.2808648155739077$

• Conclusion: Because $|2.2809| > 2$, the test statistic falls into the rejection region. Reject the null hypothesis $E(X) = 55$.

• Mathematical implication: Any hypothesized mean $c < 55$ will yield a $z$-score greater than $2.2809$, and any $c > 60$ will yield a $z$-score less than $-4.4648$. Therefore, all values $c \le 55$ and $c \ge 60$ are rejected at $\alpha = 0.05$.

Theoretical Foundations and Derivation of Confidence Intervals

• Point estimation principles:

  • For independent and identically distributed (I.I.D.) data, the sample mean $\bar{X}$ is an unbiased estimator for $E(X)$, meaning $E[\bar{X}] = E[X]$.
  • By the Law of Large Numbers (LLN), as the sample size increases, $\bar{X}$ converges in probability to the true population mean $E(X)$.
  • Because a single point estimate provides no measure of uncertainty, an interval of plausible parameter values is required.

• Inversion of hypothesis tests:

  • A confidence interval (CI) is formally defined as the set of all null hypothesized values $c$ that fail to be rejected at significance level $\alpha$.
  • Testing the family of hypotheses $H_0: \theta = c$ versus $H_1: \theta \neq c$ across all real numbers $c$ yields the $100(1 - \alpha)\%$ confidence interval.
  • Setting $\alpha = 0.05$ corresponds directly to a 95% confidence interval.

• Algebraic derivation of acceptance regions:

  • Fix $\alpha$ such that the two-tailed rejection region is $(-\infty, -a) \cup (a, \infty)$, where $a$ is the critical value.
  • The acceptance region (region of non-rejection) for a test statistic $T$ with standard error $\text{SE}(T)$ is defined by: $-a \le Z \le a$

• Substituting the $Z$-score formula: $-a \le \frac{T - c}{\text{SE}(T)} \le a$

• Defining $V = \text{SE}(T)$ for algebraic convenience: $-a \le \frac{T - c}{V} \le a$

• Multiplying across all sides by $V$ (where $V > 0$): $-aV \le T - c \le aV$

• Subtracting $T$ from all sides: $-aV - T \le -c \le aV - T$

• Multiplying through by $-1$ (reversing inequalities): $T - aV \le c \le T + aV$

• In interval notation, the set of non-rejected values is: $c \in [T - aV, T + aV] \quad \text{or} \quad T \pm aV$

• For $\alpha = 0.05$, the standard critical value is $a = 2$ (via the empirical rule approximation), yielding the classic 95% confidence interval formula: $T \pm 2V$

Calculating Confidence Intervals for the Mean

• 95% Confidence Interval for pulsar mean intensity ($\alpha = 0.05$, $a = 2$):

(round(observed_statistic - 2 * standard_error, 2), round(observed_statistic + 2 * standard_error, 2))
# Output: (55.21, 58.17)

• Lower bound: $56.690608 - 2(0.741214) = 55.20818 \approx 55.21$

• Upper bound: $56.690608 + 2(0.741214) = 58.173036 \approx 58.17$

• Interpretation: At an error tolerance of $\alpha = 0.05$, any hypothesis smaller than $55.2$ or larger than $58.2$ is rejected. The plausible range for $E(X)$ is $(55.2, 58.2)$.

  • 99.7% Confidence Interval for pulsar mean intensity ($\alpha = 0.003$, $a = 3$):

• Formulated to limit Type I error to no more than 0.3% of tests.

(round(observed_statistic - 3 * standard_error, 2), round(observed_statistic + 3 * standard_error, 2))
# Output: (54.47, 58.91)

• Lower bound: $56.690608 - 3(0.741214) = 54.466966 \approx 54.47$
• Upper bound: $56.690608 + 3(0.741214) = 58.91425 \approx 58.91$
• Precision vs. Confidence Trade-off: The 99.7% interval $(54.47, 58.91)$ contains substantially more values than the 95% interval $(55.21, 58.17)$. Decreasing the permissible Type I error rate widens the interval, reducing precision.

Confidence Intervals for Population Proportions

• Standard error differences between single-hypothesis testing and interval estimation:

  • Under an isolated null hypothesis $H_0: p = p_0$ (e.g., $p_0 = 0.5$), the standard error is evaluated directly at the hypothesized null parameter: $\text{SE}_0 = \sqrt{\frac{p_0(1 - p_0)}{n}}$

• When testing an entire continuum of hypotheses simultaneously to construct a confidence interval, $p$ is estimated from data using the sample proportion $\hat{p}$: $\text{SE} = \sqrt{\frac{\hat{p}(1 - \hat{p})}{n}}$

• Case Study: Proportion of Michigan owner-occupied homes with a mortgage:

  • Source data: American Community Survey (ACS) Public Use Microdata Sample (PUMS) dataset:
  • URL: "https://github.com/UM-Data-Science-101/data/raw/refs/heads/main/Old/pums_short.csv.gz"
  • Filtering criteria:
  • State identifier: ST == 26 (Michigan)
  • Housing tenure: TEN.isin([1, 2]) (owner-occupied housing)
  • Mortgage status: TEN == 1 indicates owned with a mortgage or loan; TEN == 2 indicates owned free and clear

• Python implementation for ACS mortgage data:

pums_url = ("https://github.com/UM-Data-Science-101/data/raw/" "refs/heads/main/Old/pums_short.csv.gz")
homes = pd.read_csv(pums_url)
owned = homes.loc[(homes["ST"] == 26) & homes["TEN"].isin([1, 2])]
mortgage = owned["TEN"] == 1
phat = mortgage.mean()
se = np.sqrt(phat * (1 - phat) / len(owned))
ci_95 = (round(phat - 2 * se, 3), round(phat + 2 * se, 3))
# Output: (0.554, 0.578)

• The 95% confidence interval for the proportion of Michigan owner-occupied homes with an active mortgage is $[0.554, 0.578]$.

Critical Values for Arbitrary Significance Levels

• Origin of standard multipliers:

  • Multipliers $a = 2$ and $a = 3$ are approximations based on the Gaussian empirical rule (68-95-99.7 rule).
  • For arbitrary confidence levels $100(1 - \alpha)\%$, the critical value $a$ must satisfy: $P(|Z| \le a) = 1 - \alpha$

• Due to the symmetry of the normal distribution, the probability in each tail is $\frac{\alpha}{2}$. Therefore, the cumulative probability up to $a$ is: $P(Z \le a) = 1 - \frac{\alpha}{2}$

• Finding critical values in Python:

  • The function scipy.stats.norm.ppf(q) computes the quantile (percent point function) corresponding to a lower-tail cumulative probability $q$.
  • For a 90% confidence interval ($\alpha = 0.10$):
  • Area in each tail: $\frac{0.10}{2} = 0.05$
  • Target quantile: $q = 1 - 0.05 = 0.95$
  • Computation: a = stats.norm.ppf(0.95) $\approx 1.6448536 \approx 1.645$

• Computing the 90% confidence interval for the mortgage proportion:

a = stats.norm.ppf(0.95)
ci_90 = (round(a, 3), round(phat - a * se, 3), round(phat + a * se, 3))
# Output: (1.645, 0.556, 0.576)

• The critical value is $a = 1.645$.
• The 90% confidence interval is $[0.556, 0.576]$, which is narrower than the 95% interval $[0.554, 0.578]$.

Empirical Verification of Confidence Interval Coverage Rates

• Definition of coverage rate:

  • The coverage rate is the long-run proportion of repeated independent random samples from the same population in which the computed confidence interval contains the true population parameter.
  • A correctly calibrated $100(1 - \alpha)\%$ confidence interval must achieve a coverage rate equal to $1 - \alpha$.

• Simulation design:

  • Population: The entire set of $N = 6489$ Michigan owner-occupied homes from the ACS dataset is treated as the true finite population.
  • True population parameter: p_true = mortgage.mean().
  • Sample size per trial: $n = 200$.
  • Number of Monte Carlo iterations: $10000$.

• Python implementation of coverage simulation:

p_true = mortgage.mean()

def ci_covers(n, confidence):
    sample = mortgage.sample(n)
    phat = sample.mean()
    se = np.sqrt(phat * (1 - phat) / n)
    a = stats.norm.ppf(1 - (1 - confidence) / 2)
    return (phat - a * se <= p_true) and (p_true <= phat + a * se)

coverage_90 = pd.Series([ci_covers(200, 0.90) for _ in range(0, 10000)])
coverage_99 = pd.Series([ci_covers(200, 0.99) for _ in range(0, 10000)])

• Simulation results for 90% confidence level:

  • First 5 sampled outcomes (coverage_90.head()):
  • Index 0: True
  • Index 1: False
  • Index 2: True
  • Index 3: True
  • Index 4: True
  • Empirical coverage rate: coverage_90.mean() = $0.8991$ (approximately 89.91%, matching the theoretical 90% target).

• Simulation results for 99% confidence level:

  • First 5 sampled outcomes (coverage_99.head()):
  • Index 0: True
  • Index 1: True
  • Index 2: True
  • Index 3: True
  • Index 4: True
  • Empirical coverage rate: coverage_99.mean() = $0.9892$ (approximately 98.92%, matching the theoretical 99% target).

Structural Characteristics and Methodological Notes

• Interval estimators:

  • Confidence intervals serve as interval estimators, outputting a set of plausible values for an unknown parameter rather than a single point estimate.

• Central Limit Theorem approximations:

  • The use of standard normal critical values relies on the Central Limit Theorem (CLT) to approximate the distribution of the test statistic under the null hypothesis.
  • Consequently, confidence intervals constructed via the CLT have approximate coverage rather than exact nominal coverage.
  • The accuracy of this approximation depends directly on the sample size $n$; smaller samples exhibit greater deviation from nominal coverage.

• Duality between confidence intervals and p-values:

  • There is a direct mathematical equivalence between hypothesis testing, p-values, and confidence intervals:
  • Any hypothesized value $c$ that lies inside a $(1 - \alpha)100\%$ confidence interval has a p-value strictly greater than $\alpha$ ($p > \alpha$).
  • Any hypothesized value $c$ that falls outside the $(1 - \alpha)100\%$ confidence interval has a p-value less than or equal to $\alpha$ ($p \le \alpha$).