PS1007 #3Inferential Statistics, Hypothesis Testing, and Distributional Analysis
Foundations of Inferential Statistics and Sampling Theory
Core Purpose of Inferential Statistics:
Inferential statistics provides mathematical procedures to determine whether patterns, differences, or associations observed in a sample can be generalized to an entire population.
It tests whether sample findings represent genuine population effects or are merely artifacts of random sampling variability.
Population versus Sample:
Population: The entire collection of individuals, events, or measurements of interest (e.g., all adults in the United States, all UK psychology undergraduates, or all breast cancer patients in the UK). Population parameters (such as the true population mean, , or population standard deviation, ) are typically fixed but unknown.
Sample: A smaller, representative subset selected from the target population (e.g., adults randomly selected in the US, or psychology students). Sample statistics (such as the sample mean, , or sample standard deviation, ) are computed directly from observed data to estimate population parameters.
Sampling Error and Random Variation:
Sampling error is the natural discrepancy between a sample statistic and the true population parameter resulting from taking a sample rather than testing the entire population.
Common synonymous terms used across scientific literature include:
Noise
Chance
Luck
Randomness
Random variation
Random error
Sampling error
Random fluctuation
Random sampling variability
The Fundamental Sampling Challenge: When drawing an empirical sample, it is impossible to know directly whether that specific sample closely resembles the population or deviates dramatically from it.
Foundational Sampling Theorems:
The Law of Large Numbers (LLN): States that as the sample size () increases (), the sample mean () converges closer and closer to the true population mean (). Larger samples consistently yield more accurate and reliable estimates of population parameters.
The Glivenko–Cantelli Theorem (Fundamental Theorem of Statistics): Asserts that as the sample size increases (), the empirical sample distribution function converges uniformly to the true population cumulative distribution function, regardless of its shape.
The Central Limit Theorem (CLT):
Asserts that as sample size () increases, the sampling distribution of the sample mean approaches a normal distribution, regardless of the shape of the underlying population distribution (whether uniform, exponential, skewed, or bimodal).
Simulation Demonstration: Taking repeated samples of increasing sizes (, , , , and ) from an extremely positively skewed population of scores demonstrates that the resulting distribution of sample means transforms from highly skewed into a bell-shaped, normal distribution.
Empirical Rule of Thumb: A sample size of at least () is traditionally considered safe for invoking the CLT, though distributions with moderate variation often approximate normality at sample sizes as small as .
The Mathematical Caveat: The CLT requires that the underlying distribution has a defined mean and variance. Pathological distributions that lack a defined mean (such as the Cauchy distribution) are exceptions, though they are virtually never encountered in empirical applied sciences.
Standard Deviation () versus Standard Error of the Mean ():
Standard Deviation ( or ):
Quantifies the degree of dispersion or variability of individual raw scores around the sample mean.
Bakery Analogy: If a sample of bakery muffins yields an average weight of with an of , this indicates that an individual muffin randomly selected from the batch will typically deviate from the sample mean by approximately .
Standard Error of the Mean ( or ):
Measures the precision or accuracy of the sample mean as an estimate of the true population mean.
Reflects the standard deviation of the sampling distribution of sample means across infinite hypothetical resamples of size .
Formula for SEM: Where is the sample standard deviation and is the sample size.
Worked Bakery Calculation: With and : This indicates that if batches of muffins are repeatedly sampled, the calculated sample means will typically vary by approximately from the true population mean.
Relationship with Sample Size: As increases, decreases in proportion to . Small samples produce unstable, "shaky" means with high , whereas large samples yield stable estimates with low .
Standard errors can also be calculated for other sample statistics (such as the median, standard deviation, or -th quantile), though the term universally refers to the mean unless stated otherwise.
The Framework of Null Hypothesis Significance Testing (NHST)
Core Conceptual Structure of NHST:
NHST is the dominant inferential paradigm across psychology, medicine, and the broader social sciences.
It evaluates whether an empirical observation (e.g., an observed mean difference between two experimental groups) is sufficiently extreme to justify rejecting a default model of "pure chance."

Formulation of Hypotheses:
Null Hypothesis ():
The default, conservative baseline claim stating that there is no effect of an intervention, no difference between groups, or no association between variables in the population.
Expressed mathematically as a precise, single point parameter: , , or .
Examples across research domains:
Differences between groups: There is no difference in depression scores between a psychotherapy group and a control group.
Relationships between variables: There is no association between daily physical activity and telomere length.
Interventions: There is no difference in recovery time following myocardial infarction between patients receiving a new medication and those receiving standard care.
Predictive outcomes: The presence of a father during the first years of life has no association with violent crime conviction rates by age .
Alternative Hypothesis ( or ):
The operational research hypothesis proposing that a real effect, difference, or association exists in the population.
Non-Directional (Two-Tailed): Postulates an effect in either direction (; e.g., "There is a difference in depression scores between the therapy group and the control group"). Used when there is no firm theoretical or empirical justification to predict polarity.
Directional (One-Tailed): Postulates an effect in a specific direction ( or ; e.g., "The therapy group will have lower depression scores than the control group"). Requires rigorous a priori justification and cannot be changed post-hoc once data are collected.
Why Science Tests the Null Rather than the Alternative:
Scientific Parsimony and Legal Conservatism: Science defaults to the simplest model (no effect) unless empirical evidence forces a more complex explanation. This directly mirrors the legal doctrine where a defendant is presumed innocent until proven guilty beyond reasonable doubt.
Mathematical Tractability: The null hypothesis specifies an exact numerical value (e.g., difference ), allowing the exact construction of a probability distribution (the sampling distribution under ). In contrast, the alternative hypothesis encompasses an infinite range of non-zero values, making it impossible to construct a single sampling distribution without assuming an arbitrary effect size.
Error Control: Testing allows researchers to directly set and limit the probability of false positive conclusions ().
The Four-Step NHST Procedure:
Step 1: Formulate clear null () and alternative () hypotheses prior to data collection.
Step 2: Collect empirical sample data and compute an appropriate test statistic (e.g., , , , or ).
Step 3: Derive the -value associated with the observed test statistic under the sampling distribution assuming is true.
Step 4: Compare the -value against the a priori significance threshold ():
If : Reject ; the result is declared statistically significant.
If : Fail to reject ; the result is declared statistically non-significant.
Understanding the -Value:
Verbatim Definition: The probability of obtaining a test statistic or sample result at least as extreme as the one observed, assuming that the null hypothesis is completely true.
Range: Mathematically bounded between and ().
: Observed sample result is completely identical to the null hypothesis expectation.
: Observed result would be mathematically impossible if were true.
Smaller -values indicate increasing incompatibility between the collected sample data and the null hypothesis.
Significance Level ():
The pre-set threshold determining the tolerable risk of committing a Type I error.
Conventional standard is (, or a chance).
Stricter thresholds (e.g., or ) are adopted in high-stakes fields like clinical pharmacology or particle physics to reduce false positives to .
More lenient thresholds (e.g., ) might be tolerated in non-critical scenarios (such as predicting ice cream truck arrival times), accepting a false-positive risk.
Critical Interpretive Caveats:
Never "Accept" the Null Hypothesis: A non-significant result () only indicates insufficient evidence to reject . It does not demonstrate that the true difference is zero. Retaining may stem from low statistical power, excessive measurement noise, small sample size, or poor design.
Rejecting Does Not "Prove" : A statistically significant result indicates that the data are improbable under , not that the research theory is conclusively confirmed.
Statistical Significance versus Effect Size:
The -value conveys nothing about the magnitude or real-world importance of an effect.
Empirical Illustration:
Experiment 1 (Small Sample): A difference between Drug A and Drug B yields a non-significant .
Experiment 2 (Large Sample): A tiny difference between Drug A and Drug B yields a statistically significant .
With a sufficiently large sample, trivial and clinically meaningless differences will reliably achieve statistical significance ().
Sampling Distributions and Hypothesis Testing in Practice
The Nature of Sampling Distributions:
A sampling distribution is the theoretical probability distribution of a specific statistic (e.g., mean, difference between means, median, , ) that would emerge from drawing an infinite number of random samples of size from a specified population under .
It maps the area under the curve directly to probability, establishing which sample outcomes are probable under chance and which are extreme.
The Standard Normal Distribution and Empirical Probabilities:
A standard normal distribution is symmetric, unimodal, and bell-shaped, characterized by specific fixed areas under the curve:
contains (or ) of all observations ( on each side of the mean).
contains (or ) of observations ( between and ).
contains (or ) of observations ( between and , with in each tail beyond ).
Exactly encapsulates of the total area under the curve, leaving exactly in each tail.

Worked Empirical Case Study: Height Differences by Pet Preference:
Research Question: Are individuals who prefer dogs as pets taller than individuals who prefer cats as pets?
Hypotheses:
(no height difference, ).
(two-tailed non-directional difference).
Sample Data (, per group):
Dog owners (): Mean height , standard deviation .
Cat owners (): Mean height , standard deviation .
Observed difference between sample means:
Standard Error of the Mean Difference (SEMD):
The standard deviation of the sampling distribution of the difference between two independent sample means.
Formula: Where and are the sample standard deviations, and and are the sample sizes.
Calculation with Actual Study Data:
Mathematical Determinants of SEMD Spread:
Effect of Sample Size () holding SD constant (, ):
Small sample ( per group):
Baseline sample ( per group): .
Large sample ( per group):
Principle: As sample size increases, standard error decreases, producing a narrower, taller sampling distribution.
Effect of Underlying Sample Variance () holding sample size constant ():
Lower variance (, ):
Baseline variance (, ): .
Higher variance (, ):
Principle: As underlying population/sample standard deviation increases, standard error increases, widening the sampling distribution.

Evaluating Significance against Critical Thresholds:
Under , the sampling distribution of the mean difference is centered at .
The acceptance region is bounded by .
For , the critical thresholds are:
Because the observed sample difference of lies far beyond (exceeding ), it is exceedingly unlikely to have occurred through random sampling error alone. is rejected ().
Thresholds Across Alpha Levels (Two-Tailed):
: Critical ( area under the curve; in each tail).
: Critical ( area under the curve; in each tail).
: Critical ( area under the curve; in each tail).
Decision Errors and Statistical Power
The Four Potential Outcomes in Decision Theory:
Crossing the true state of reality ( is true vs. is false) with the statistical decision (reject vs. fail to reject ) creates a decision matrix:

True Negative (Correct Decision): Null hypothesis is true in reality, and the test fails to reject . Probability .
False Positive (Type I Error, ):
The null hypothesis is true in reality, but the test incorrectly rejects .
Concluding there is an effect or difference when none actually exists.
Equal to the pre-set significance level (; typically ).
Examples: Convicting an innocent person in court; concluding a completely ineffective drug lowers blood pressure because a random sample showed a lower mean purely by chance.
False Negative (Type II Error, ):
The null hypothesis is false in reality (a real difference exists), but the test fails to reject .
Concluding there is no effect when an effect actually exists.
Probability designated as .
Examples: Acquitting a guilty criminal due to insufficient evidence; concluding an effective medication does not work because sampling noise obscured the effect.
True Positive (Statistical Power, ):
The null hypothesis is false in reality, and the test correctly rejects .
The sensitivity or ability of a test to detect a genuine effect of a given magnitude.
Mathematical Row Probabilities:
When is true:
When is false:
Probability Theory and Chance Extremes:
The Infinite Monkey Theorem: An infinite number of monkeys randomly striking typewriter keys for an infinite duration will eventually produce every finite text, including the complete works of William Shakespeare, word-for-word, purely by random permutation.
Dice Rolling Example: Rolling consecutive sixes on a fair six-sided die has an exact probability of: This is roughly a chance. Although extraordinarily rare, it is entirely possible under pure randomness. NHST accounts for this by accepting an explicit error rate () of rare sampling fluctuations.
Interrelationships Governing Statistical Power ():
In a continuous sampling model comparing a null distribution (centered at ) against an alternative distribution (centered at the true effect size ):
The Type I error () defines the critical cutoff line on the null curve.
The area under the alternative curve that falls to the left of this cutoff is the Type II error ().
The remaining area under the alternative curve to the right of the cutoff is Statistical Power ().
Three Factors Dictating Power:
Sample Size (): Increasing reduces standard error, making both the null and alternative distributions skinnier and narrower. This reduces the distribution overlap, shrinking and substantially increasing power ().
Sample/Population Variability (): Increasing underlying noise or standard deviation widens and flattens both curves, increasing their overlap and diminishing statistical power.
Effect Size (): A larger true difference pushes the alternative distribution further away from the null distribution along the horizontal axis, minimizing overlap and increasing power.
One-Tailed versus Two-Tailed Significance Tests:
Two-Tailed Test (Non-Directional):
Evaluates the possibility of an effect in either direction ().
The critical alpha region () is evenly divided into two symmetric tails ( in each tail).
Critical .
For an SEMD of , the critical difference threshold is .

One-Tailed Test (Directional):
Evaluates an effect in only one specific, predicted direction ( or ).
The entire alpha region is placed in one single tail ().
Critical (or ).
For an SEMD of , the critical difference threshold is .

Methodological Comparison:
A two-tailed test at has the exact same overall Type I error rate as a one-tailed test at , but has less power to detect an effect in the targeted direction because the critical threshold is set further out ( vs. ).
Directional hypotheses must be fixed before data collection. Changing from a two-tailed to a one-tailed test post-hoc after seeing sample results is methodologically invalid because it doubles the effective Type I error rate.
Distributional Characteristics and Assessing Normality
Anatomy of the Normal Distribution:
A perfectly normal distribution exhibits three distinct mathematical properties:
Modality: Unimodal (possessing a single peak where mean, median, and mode coincide exactly).
Skewness: Perfectly symmetric with zero skew ().
Kurtosis: Mesokurtic ( in excess kurtosis notation, or in raw moment notation).
Distributional Departures from Normality:
Modality Variations: Distributions can be bimodal (two distinct peaks, such as eruption intervals for the Old Faithful geyser) or uniform (completely flat, where every value across a interval has equal probability).
Skewness (Asymmetry):
Positive (Right) Skew: The right tail is elongated with excess scores clustering at the lower end (e.g., student statistics anxiety scores, household wealth). Mean exceeds median ().
Negative (Left) Skew: The left tail is elongated with excess scores clustering at the higher end (e.g., high student screen time, easy exam scores). Mean is less than median ().

Kurtosis (Peakedness and Tail-Fatness):
Leptokurtic ( excess kurtosis, or raw): Sharply pointed central peak with heavy, fat tails containing excess outliers relative to a normal distribution.
Platykurtic ( excess kurtosis, or raw): Broad, flattened central peak with light, thin tails and fewer outliers.
Mesokurtic ( excess kurtosis, or raw): Standard bell-shaped normal curve.

Three-Pronged Approach to Assessing Normality:
Method 1: Descriptive Statistics (Skewness and Kurtosis):
Rules of Thumb for Absolute Values:
Value : Mild skewness/kurtosis (close to normal).
Value between and : Moderate skewness/kurtosis.
Value : Extreme deviation from normality.
Z-Score Significance Transformation: To evaluate whether observed skewness or kurtosis is statistically significantly different from zero, divide the statistic by its respective standard error:
If the calculated -score falls outside the range of to , the deviation from normality is statistically significant at .
Method 2: Formal Statistical Tests of Normality:
Hypotheses Tested:
: The sample data are drawn from a normally distributed population.
: The sample data are not drawn from a normally distributed population.
Decision Rule:
If : Fail to reject ; normality can be safely assumed.
If : Reject ; data depart significantly from a normal distribution.
Available Tests and Sample Size Sensitivity:
Shapiro-Wilk Test: Recommended as the most powerful test for small to moderate samples (). Not suitable for very large datasets ().
Kolmogorov-Smirnov Test (with Lilliefors significance correction): Suited for larger samples (), but lacks statistical power in small samples.
The Large-Sample Fallacy: Both tests are overly sensitive when sample sizes exceed (), frequently flagging trivial, clinically inconsequential deviations as statistically significant (), producing false positives. Methodologists (e.g., Howell, 2010, Statistical Methods for Psychology) caution against relying solely on formal tests without visual plots.
Method 3: Visual Diagnostic Inspection:
Quantile-Quantile (Q-Q) Plots: Plots observed sample quantiles against quantiles expected from a theoretical normal distribution. If the sample is normal, the observed points adhere tightly to the diagonal reference line. Systematic curved departures reveal skewness, while S-shaped deviations reveal kurtosis.
Box Plots: Displays median, interquartile range ( to percentiles), minimum, maximum, and directly flags isolated individual outliers.
SPSS Worked Diagnostic Example:
Evaluating two continuous variables (
VAR00001andVAR00002), each with :

Descriptive Analysis for
VAR00001:Mean (); Median ; Variance ; .
Skewness (): Because falls within , skewness is mild and non-significant.
Kurtosis (): Because falls within , kurtosis is non-significant.
Formal Tests: Shapiro-Wilk statistic , , ; Kolmogorov-Smirnov statistic , , . Because , normality is retained.
Visual Diagnostics: The Q-Q plot points closely follow the diagonal line. However, the box plot flags case number () as an isolated low outlier.


Descriptive Analysis for
VAR00002:Mean (); Median ; Variance ; .
Substantial discrepancy between mean () and median () indicates pronounced asymmetry.
Skewness (); Kurtosis ().
Formal Tests: Shapiro-Wilk statistic , , ; Kolmogorov-Smirnov statistic , , . Because , the null hypothesis of normality is rejected.
Visual Diagnostics: The Q-Q plot shows systematic non-linear bowing away from the diagonal line, confirming non-normality.

SPSS Step-by-Step Procedure for Normality Testing:
Step 1: Navigate through the top toolbar via
AnalyzeDescriptive StatisticsExplore....Step 2: Transfer target continuous variables into the
Dependent Listbox. (If comparing groups, place the categorical grouping variable into theFactor Listto check distributions separately per group).Step 3: Click the
Plots...button on the right panel.Step 4: Under the Descriptive menu, check
Histogram. Under Boxplots, chooseFactor levels together.Step 5: Check the box labeled
Normality plots with tests.Step 6: Click
Continue, then clickOKto generate full descriptive tables, Kolmogorov-Smirnov, Shapiro-Wilk, Q-Q plots, and box plots.