Statistical Hypothesis Testing, Error Types, and Normality Tests

Review of Descriptive Statistics and Theoretical Probability Distributions

• Standard Error of the Mean ($\text{SE}_{\bar{x}}$) and Estimation Uncertainty:

  • The sample mean ($\bar{x}$) represents a point estimate of the true population mean ($\mu$).

  • To quantify the precision and certainty of this estimate, the standard error of the mean is calculated.

  • For normally distributed data, constructing an interval of $\bar{x} \pm 1\,\text{SE}_{\bar{x}}$ indicates that the true population mean lies within this bounded range with a defined probability.

• Quantiles and Symmetic Descriptions of Data:

  • Quantiles serve as non-parametric measures to characterize the dispersion, spread, and symmetry of empirical data distributions.

  • They identify cutoff boundaries for specified proportions of observations (for instance, determining the threshold below which the $10\%$ smallest values lie).

• Summary Visualizations and Subsetting:

  • Boxplots: Provide a compact graphical summary of the distribution, highlighting the median, quartiles, interquartile range, and potential outliers.

  • Subsets: Extraction of specific sub-datasets from a larger primary dataset to isolate targeted variables or cohorts for comparative analysis.

• Sample Frequencies versus Population Probability Distributions:

  • Empirical Samples: Characterized by observed frequencies, reflecting the reality that certain values occur more frequently than others within a finite sample.

  • Underlying Populations: Governed by continuous or discrete theoretical probability distributions specifying how probability mass or density is distributed across all possible outcomes of a random process.

  • Total Area Under the Curve: For all valid probability density functions, the integral across the entire parameter space is strictly equal to $1$ ($\int_{-\infty}^{\infty} f(x)\,dx = 1$). This guarantees that any measured subject (such as measuring the circumference of an ash tree) will fall somewhere within the total defined domain.

• Probability Density Function (PDF) versus Cumulative Distribution Function (CDF):

  • Probability Density Function: A mathematical auxiliary function used to construct continuous probability distributions via integration.

  • Discrete Random Variables: Probabilities can be read directly at any specific integer point. The probability of observing a composite event (such as obtaining a $4$, $5$, or $6$) is calculated by direct summation of the individual probabilities: $P(X \in \{4, 5, 6\}) = P(X=4) + P(X=5) + P(X=6)$.

  • Continuous Random Variables: The probability of observing any single exact real-valued number is infinitesimally small ($P(X = x) = 0$) because there are infinitely many decimal values between any two numbers. Probabilities can only be evaluated across intervals by calculating the definite integral under the curve of the probability density function.

  • Cumulative Distribution Function: Represents the definite integral of the density function over a given interval up to a defined point $x$, representing the cumulative probability ($\int_{-\infty}^{x} f(t)\,dt$) of observing a value less than or equal to $x$.

• Key Theoretical Distributions and Their Properties:

  • Normal Distribution: A continuous, strictly symmetric distribution characterized by a single central peak (maximum). It accommodates any real decimal value.

  • Uniform Distribution (Gleichverteilung): Can be discrete (e.g., a balanced six-sided die where each face from $1$ to $6$ has an equal probability of occurrence) or continuous (e.g., an idealized spinning wheel where the probability of stopping at any continuous angular point along the circumference is equally distributed). It is fully symmetric across its domain.

  • Poisson Distribution: A discrete distribution designed for count data consisting strictly of non-negative integers ($0, 1, 2, \dots$). A defining property is that the mean is equal to the variance ($\mu = \sigma^2$). Consequently, distributions with small means have correspondingly small variances, whereas large means feature wide dispersion. It is intrinsically asymmetric (right-skewed), with the skewness being pronounced at small means and diminishing as the mean increases.

  • Binomial Distribution: Models discrete proportion data based on a fixed, known number of independent trials ($n$) and a constant probability of success ($p$) per trial. The expected mean is given by $\mu = n \times p$, with variance given by $\sigma^2 = n \times p \times (1 - p)$.

Foundations of Statistical Hypothesis Testing

• Definition of a Hypothesis:

  • A hypothesis is an assertion or proposition whose validity is provisionally assumed, but which has not been empirically proven or verified.

  • In inferential statistics, hypotheses represent assumptions regarding relationships, differences, or effects within populations, which are systematically evaluated using empirical data obtained from representative samples.

• The Asymmetry of Verification and Falsification:

  • Within mathematical and empirical logic, it is virtually impossible to conclusively prove or verify a universal hypothesis using finite empirical data.

  • It is far more rigorous and mathematically tractable to falsify or refute a hypothesis.

  • Consequently, statistical testing sets up a complementary pair of competing hypotheses: an alternative/working hypothesis and a null hypothesis.

• Alternative Hypothesis ($H_1$) versus Null Hypothesis ($H_0$):

  • Working / Alternative Hypothesis ($H_1$): The substantive hypothesis representing the effect, difference, or relationship that the investigation seeks to support (e.g., that hares run faster than snails).

  • Null Hypothesis ($H_0$): The structurally simpler, complementary benchmark hypothesis stating that no effect, no difference, or no systematic relationship exists.

  • Symmetry Rule in Formulation: The null hypothesis is never formulated as the inverse directional claim (e.g., $H_0$ is never formulated as "snails run faster than hares"). Instead, it states neutrality: there is no difference in running speed between hares and snails.

• Representative Examples of Hypothesis Formulation:

  • Locomotion Velocity (Hares vs. Snails):

  • Working Hypothesis ($H_1$): Hares run faster than snails (a significant velocity difference exists).

  • Null Hypothesis ($H_0$): Hares and snails exhibit the exact same locomotion speed (difference equals zero).

  • Agricultural Plant Growth and Fertilization:

  • Working Hypothesis ($H_1$): Fertilizer application increases plant growth (Factor $A$ exerts a positive causal effect on Variable $B$).

  • Null Hypothesis ($H_0$): Fertilizer application has no effect on plant growth (Factor $A$ does not influence Variable $B$; response values remain baseline-invariant).

  • Sexual Dimorphism in Human Body Height:

  • Working Hypothesis ($H_1$): Mean body height differs significantly between adult men and adult women (Group $X$ differs from Group $Y$ with respect to metric $M$).

  • Null Hypothesis ($H_0$): Mean body height does not differ between men and women (Group $X$ and Group $Y$ are identical with respect to metric $M$).

• The p-Value and the Significance Level ($\alpha$):

  • p-Value: Measures the exact conditional probability of observing sample data as extreme as, or more extreme than, the empirical dataset obtained, assuming that the null hypothesis ($H_0$) is strictly true.

  • Significance Level ($\alpha$): An arbitrary, a priori threshold established before conducting the test, defining the maximum risk of rejecting a true null hypothesis. The universal standard convention in scientific disciplines is $\alpha = 0.05$ ($5\%$).

  • Decision Rule:

  • If $p \le \alpha$: The observed data are considered too improbable under the null model. The null hypothesis ($H_0$) is formally rejected, and the working hypothesis ($H_1$) is accepted.

  • If $p > \alpha$: The observed data are consistent with random sampling variability under $H_0$. The null hypothesis cannot be rejected.

  • Probabilistic Limitation: Statistical hypothesis tests never yield absolute, deterministic certainty. Inferences remain strictly probabilistic statements regarding the likelihood of the observed patterns.

Decision Errors and Statistical Power

• The Statistical Decision Matrix:

  • Evaluating an empirical sample against a population null hypothesis leads to four mutually exclusive operational outcomes:

  • $H_0$ is True in Nature and Retained by Test: Correct decision ($P = 1 - \alpha = 0.95$ or $95\%$).

  • $H_0$ is True in Nature but Rejected by Test: Type I Error (Fehler 1. Art / False Positive; $P = \alpha = 0.05$ or $5\%$).

  • $H_0$ is False in Nature and Retained by Test: Type II Error (Fehler 2. Art / False Negative; $P = \beta$).

  • $H_0$ is False in Nature and Rejected by Test: Correct decision / Statistical Power ($P = 1 - \beta$).

• Detailed Characterization of Error Types:

  • Type I Error (Type 1 / False Positive / $\alpha$):

  • Occurs when a statistical test identifies a significant effect or group difference when, in reality, no such difference exists.

  • The probability of committing this error is directly controlled by the experimenter through the selection of $\alpha$ (typically $0.05$).

  • Type II Error (Type 2 / False Negative / $\beta$):

  • Occurs when a statistical test fails to reject the null hypothesis despite the existence of a real, systematic effect or difference in the population.

  • The probability $\beta$ depends on the chosen alpha level, the magnitude of the true effect, data variability, and the total sample size ($n$).

  • Statistical Power ($1 - \beta$):

  • The probability that a test correctly identifies and rejects a false null hypothesis.

  • Structural Mnemonic Device:

  • Type 1 Error corresponds to False Positive (the numeral $1$ graphically aligns with the letter P in Positive).

  • Type 2 Error corresponds to False Negative (the numeral $2$ conceptually pairs with the letter N in Negative).

• Taxonomic Case Studies in Ursid Morphology (Bear Shoulder Heights):

  • Case 1: Kodiak Bear (Ursus arctos middendorffi) versus European Brown Bear (Ursus arctos arctos):

  • Underlying Reality: True shoulder heights are physiologically very similar; $H_0$ is true.

  • Hypotheses: $H_0$ states mean shoulder heights do not differ; $H_1$ states mean shoulder heights differ.

  • Correct Decision: Retaining $H_0$ ($P = 95\%$ under $\alpha = 0.05$).

  • Potential Error: Rejecting $H_0$ due to unrepresentative sampling constitutes a Type I error ($\alpha = 5\%$).

  • Case 2: Kodiak Bear (Ursus arctos middendorffi) versus American Black Bear (Ursus americanus):

  • Underlying Reality: American black bears are substantially smaller than Kodiak bears; $H_1$ is true ($H_0$ is false).

  • Hypotheses: $H_0$ states mean shoulder heights do not differ; $H_1$ states mean shoulder heights differ.

  • Correct Decision: Rejecting $H_0$ in favor of $H_1$ occurs with probability $1 - \beta$ (Statistical Power).

  • Potential Error: Failing to reject $H_0$ (e.g., claiming they share identical shoulder heights) constitutes a Type II error ($\beta$).

• Error Trade-offs, Variance, and Sample Size Calibration:

  • Errors cannot be eliminated entirely in statistical testing; they can only be mathematically managed.

  • The Trade-off Dilemma: Artificially lowering $\alpha$ (e.g., from $0.05$ to $0.01$) reduces the risk of a Type I error to $1\%$, but simultaneously inflates the probability of committing a Type II error ($\beta$), diminishing test sensitivity.

  • The Function of Sample Size ($n$): The primary robust method to simultaneously reduce $\beta$ and increase power without compromising $\alpha$ is to expand sample size ($n$).

  • Impact of Variance: Higher sample dispersion/variance directly inflates the error rate $\beta$.

  • Pilot Studies (Vorstudien): When high variance is suspected, researchers should execute preliminary pilot studies using smaller cohorts to accurately estimate parameter variance, which informs power calculations to determine the minimal necessary sample size ($n$) for the main study.

Testing Normality: Context, Principles, and Hypotheses

• The Methodological Prerequisite Hierarchy:

  • In comparative experimental designs (such as measuring the biomass of greenhouse plants divided into fertilized and unfertilized pots), the primary analytical objective is typically testing for differences between group means.

  • Student's t-Test: The most statistically powerful parametric procedure for testing differences between two continuous group means.

  • Parametric Assumptions of the t-Test:

  • Homogeneity of variances between groups (homoscedasticity).

  • Normal distribution of the response data within populations.

  • Methodological Routing: If the assumption of normality is violated, the parametric t-test is invalid, necessitating non-parametric alternatives such as the Wilcoxon rank-sum test (Wilcoxon Rangsummentest), which does not assume a normal distribution.

• Hypothesis Formulation for Normality Testing:

  • Unlike comparative experimental testing where researchers typically seek significant differences, testing for normality evaluates the following structure:

  • Null Hypothesis ($H_0$): The empirical data distribution does not differ significantly from a theoretical normal distribution ($H_0$: data are normally distributed).

  • Working Hypothesis ($H_1$): The empirical data distribution differs significantly from a theoretical normal distribution ($H_1$: data are not normally distributed).

  • Decision Consequences for Downstream Analysis:

  • Large p-Value ($p > 0.05$): Retain $H_0$. Data are assumed to be normally distributed; the parametric t-test is permitted. Any decision error here is strictly a Type II error ($\beta$).

  • Small p-Value ($p \le 0.05$): Reject $H_0$. Data deviate significantly from normality; parametric t-tests cannot be used. Any decision error here is strictly a Type I error ($\alpha$).

• Standardized Scientific Significance Reporting:

  • Academic publications frequently denote significance levels using standard asterisk notation rather than printing full decimal values:

  • $p > 0.05$: Not significant ($\text{n.s.}$).

  • $0.05 < p \le 0.10$: Occasionally denoted with a dot ($\cdot$) to describe a "trend"; however, this practice is explicitly omitted and discouraged in standard rigorous testing frameworks.

  • $p \le 0.05$: Significant (denoted by $*$).

  • $p \le 0.01$: Very significant (denoted by $**$).

  • $p \le 0.001$: Highly significant (denoted by $***$).

  • Complete Formal Reporting Standard: A rigorous scientific test report must state: the specific test procedure employed, the empirical test statistic value, and the corresponding p-value relative to $\alpha$ (e.g., "KS test, $d = 0.024$, $p > 0.05$").

Methods for Assessing Normality: KS-Test, Q-Q Plot, and Shapiro-Wilk Test

• The Kolmogorov-Smirnov Test (KS-Test):

  • Underlying Mechanism: Evaluates the maximum absolute vertical divergence between the empirical cumulative distribution function ($S(x_i)$) of the sample and the theoretical cumulative distribution function ($F(x_i)$) of a normal distribution sharing the identical sample mean ($\bar{x}$) and sample standard deviation ($s$).

  • Test Statistic ($d$): The maximum distance between the two cumulative curves:     $d = \max |S(x_i) - F(x_i)|$

  • Critical Value ($d_\alpha$): Derived historically from mathematical reference tables based on sample size ($n$) and significance level ($\alpha = 0.05$). If empirical $d > d_\alpha$, the deviation is statistically significant ($p \le 0.05$).

  • Implementation in R:

  • Executed via ks.test(x, y = "pnorm", mean = mean(x), sd = sd(x)).

  • Note: Straight quotes must be ensured when entering "pnorm" to prevent syntax execution errors.

  • By default, computes a two-sided test (alternative = "two-sided"), evaluating positive and negative curve deviations.

  • Technical Warning Handling:

  • R outputs a console warning message if the dataset contains identical/tied numerical values (e.g., duplicated velocity entries). This warning can be safely ignored.

  • Methodological Strengths:

  • Highly robust against extreme outliers due to its reliance on cumulative integration.

  • Tolerates smaller sample sizes.

  • Exceptionally versatile: applicable to other continuous or discrete distributions (e.g., Poisson) and can evaluate whether two distinct empirical samples originate from the same underlying distribution.

  • Critical Methodological Weaknesses:

  • Features very low statistical power (geringe Teststärke).

  • Overly conservative and excessively forgiving; it frequently fails to reject $H_0$ even when datasets exhibit substantial, visible skewness (high Type II error rate).

  • Parametric Adaptation: The Lilliefors test represents a specialized modification of the Kolmogorov-Smirnov test calibrated specifically to improve performance when estimating normal distribution parameters from the sample.

• Visual Diagnostic Testing: The Quantile-Quantile Plot (Q-Q Plot):

  • Underlying Mechanism: A bivariate graphical method that plots the empirical quantiles calculated from the observed sample directly against the corresponding theoretical quantiles derived from an ideal standard normal distribution.

  • Interpretation of Alignment:

  • If sample data are strictly normally distributed, all plotted quantile points fall along a straight diagonal reference line.

  • Peripheral Tail Deviations: Minor deviations at the extreme lower and upper ends of the diagonal are normal and typically negligible because very few empirical data points populate the extreme tails.

  • Core Concordance: High adherence across the central quantile domain provides strong qualitative evidence of normality.

  • Systematic Distortions: Outliers manifest as isolated points detached from the line; continuous systematic curves indicate non-normal distributional shape (such as skewness or heavy tails).

  • Implementation in R:

  • qqnorm(data): Generates the scatterplot of empirical versus theoretical quantiles.

  • qqline(data): Superimposes the theoretical linear normal trajectory.

  • Limitations: Visual evaluation is inherently subjective; it lacks an objective numerical cutoff threshold (such as a p-value against $\alpha = 0.05$), requiring analytical experience to judge acceptable variance.

• The Shapiro-Wilk Test:

  • Underlying Mechanism: Developed specifically as a dedicated, high-power test for univariate normality. It evaluates the ratio of the sample's empirical variance to the theoretical variance expected under an ideal normal distribution.

  • Test Statistic ($W$): Quantifies the ratio of variances. A $W$ statistic value approaching $1.0$ indicates near-perfect concordance with a theoretical normal distribution.

  • Implementation in R:

  • Executed simply via shapiro.test(data).

  • Does not require manual entry of mean or standard deviation parameters; R estimates these internal parameters automatically.

  • Console Output Interpretation: Displays the formal test name, dataset identifier, calculated test statistic ($W$), and the resulting p-value.

  • Methodological Superiority: Possesses significantly higher statistical power than the Kolmogorov-Smirnov test. It reliably detects minor to moderate non-normality and maintains a substantially lower Type II error probability.

• Comparative Empirical Case Study: Log-Normal Skewed Data:

  • Scenario: Diagnostic evaluation of an artificially generated right-skewed dataset generated from a log-normal distribution (logarithmierte Normalverteilung).

  • Kolmogorov-Smirnov Test Performance: Produces a non-significant result ($p > 0.05$), erroneously retaining $H_0$ and classifying the skewed data as normal (Type II error attributable to low power).

  • Shapiro-Wilk Test Performance: Yields an empirical p-value of $p = 1.598 \times 10^{-8}$ (scientific notation: 1.598e-08, indicating that the decimal is shifted $8$ positions to the left: $0.00000001598$). This decisively rejects $H_0$ at $p \le 0.001$ (highly significant, $***$), correctly identifying strong non-normality.

  • Q-Q Plot Confirmation: Plotted quantiles display a severe, systematic upward curvature away from the reference line, corroborating the Shapiro-Wilk test result.

  • Analytical Recommendations:

  • Always plot and visually inspect the data distribution first via a histogram.

  • When selecting a formal numerical test for normality, the Shapiro-Wilk test is substantially superior to the standard Kolmogorov-Smirnov test and represents the preferred analytical standard across biological research disciplines.