3- Sampling and Sampling Distributions

Foundations of Statistical Inference and Parameter Estimation

• Core Purpose of Statistical Inference:

  • The primary objective in inferential statistics is to draw valid conclusions and make probabilistic generalizations about unknown population parameters based on known sample statistics.

  • Population Parameters: Fixed, descriptive numerical values summarizing an entire population of scores on a random variable $X$. Standard parameters include:

    • $\mu$: Population mean.

    • $\sigma^2$: Population variance.

    • $\sigma$: Population standard deviation.

  • Sample Statistics: Descriptive numerical values computed directly from an observed subset (sample) of scores drawn from the parent population. Standard statistics include:

    • $\bar{X}$: Sample mean.

    • $s^2$: Sample variance.

    • $s$: Sample standard deviation.

• Conceptual Pitfalls in Statistical Reporting:

  • Statistical figures can be framed in ways that produce distorted impressions while maintaining technical truth. Darrell Huff, in How to Lie with Statistics (illustrated by Irving Geis), observed:

    • "A well-wrapped statistic is better than Hitler’s 'big lie'; it misleads, yet it cannot be pinned on you."

  • Rigorous sampling designs and understanding theoretical sampling distributions are the critical defenses against erroneous or deceptive statistical inferences.

Sampling Methodologies: Random versus Opportunity Sampling

• Simple Random Sampling:

  • Definition: A sampling procedure wherein every single individual, object, or event in the defined population possesses an equal and consistent probability of being selected at any point during the sampling process.

  • Independence: Random sampling ensures complete probabilistic independence; the selection of any one individual or event has zero influence on the likelihood of selecting any other individual or event.

  • Sampling Mechanics and Contexts (Gonick & Smith, 2005):

    • Sampling with Replacement: Each drawn unit is recorded and immediately returned to the population before the next unit is chosen. This procedure maintains strictly identical probabilities across all sequential draws.

    • Sampling without Replacement: Selected units are removed and not returned. In finite populations, this causes selection probabilities to shift incrementally with each successive draw; in infinite or very large populations, the effect is negligible.

    • Finite vs. Infinite Populations: Sampling design must account for population bounds; an infinite population inherently supports constant selection probabilities even without replacement.

• Opportunity (Convenience) Sampling:

  • Definition: Selecting the first $N$ available population units that happen to present themselves, rather than executing a structured random selection scheme.

  • Systematic Bias and Volunteer Bias:

    • Opportunity samples introduce substantial systematic bias because participants who volunteer or are easily accessible typically differ on key psychological, demographic, or behavioral traits from the broader population.

    • Units do not have an equal or known probability of selection, violating the foundational assumptions required for inferential statistical models.

The Sampling Problem and Empirical Sample Variability

• The Sampling Problem Defined:

  • Even when drawn from the exact same population, different individual random samples will almost always yield different sample statistics due to random sampling variability (chance fluctuation).

  • No single sample mean ($\bar{X}$) can be presumed to match the true population mean ($\mu$) exactly.

• Discrete Numerical Demonstration:

  • Consider a finite population of $11$ discrete scores on random variable $X$:         $\{1, 2, 2, 2, 3, 4, 4, 5, 5, 6, 7\}$

  • True Population Mean:         $\mu = 4.1$

  • Sample 1 ($N = 3$): Scores drawn are $1, 2, 3$.         $\bar{X}_1 = \frac{1 + 2 + 3}{3} = 2.00$

  • Sample 2 ($N = 3$): Scores drawn are $5, 6, 7$.         $\bar{X}_2 = \frac{5 + 6 + 7}{3} = 6.00$

  • Observation: Both samples were selected randomly from the exact same population ($\mu = 4.1$), yet their sample means are separated by $4.00$ full scale points ($2.00$ vs. $6.00$), clearly illustrating the magnitude of error inherent in isolated small samples.

• Activity Case Study: Pre-Teen Anxiety Assessment:

  • Three independent researchers study pre-teen anxiety, each drawing from the identical theoretical population of all possible scores on a $1$-to-$5$ Likert assessment.

  • Each researcher collects an independent sample of size $N = 7$:

    • Study 1 Raw Scores: $3, 1, 3, 2, 2, 1, 2$             $\bar{X}_1 = \frac{3 + 1 + 3 + 2 + 2 + 1 + 2}{7} = \frac{14}{7} = 2.00$

    • Study 2 Raw Scores: $4, 4, 5, 3, 5, 5, 4$             $\bar{X}_2 = \frac{4 + 4 + 5 + 3 + 5 + 5 + 4}{7} = \frac{30}{7} \approx 4.29$

    • Study 3 Raw Scores: $3, 3, 4, 2, 3, 3, 4$             $\bar{X}_3 = \frac{3 + 3 + 4 + 2 + 3 + 3 + 4}{7} = \frac{22}{7} \approx 3.14$

  • Analytical Takeaway: Isolated sample means ($2.00$, $4.29$, and $3.14$) vary widely around the underlying population mean. Relying on any single sample provides an incomplete picture unless the statistical properties of all possible sample means are modeled systematically.

Nature and Structure of Sampling Distributions

• Formal Definition of a Sampling Distribution:

  • A sampling distribution is a distribution of values taken by a statistic in all possible samples of the exact same size ($N$) drawn repeatedly and infinitely from the same parent population.

  • It is fundamentally a theoretical, mathematical model rather than an empirically gathered distribution.

  • The abscissa (horizontal axis / x-axis) represents the calculated values of the sample statistic (e.g., $\bar{X}$), while the ordinate represents the probability density or theoretical frequency.

• Step-by-Step Construction of the Sampling Distribution of the Mean:

  • Step 1: Identify a parent population of scores characterized by a mean $\mu$ and standard deviation $\sigma$.

  • Step 2: Draw an infinite number of random samples, each containing an identical sample size $N$.

  • Step 3: Calculate the individual sample mean ($\bar{X}_1, \bar{X}_2, \dots, \bar{X}_J$) and sample standard deviation ($s_1, s_2, \dots, s_J$) for every single sample $J$.

  • Step 4: Plot the resulting infinite set of sample means onto a new coordinate plane. The resulting curve is the sampling distribution of the mean.

• Key Parameters of the Sampling Distribution of the Mean:

  • Expected Value ($\mu_{\bar{X}}$): The arithmetic mean of the entire sampling distribution of the mean.

  • Standard Error ($\sigma_{\bar{X}}$): The standard deviation of the sampling distribution of the mean, quantifying the expected average distance between an observed sample mean $\bar{X}$ and the true population mean $\mu$.

The Central Limit Theorem (CLT)

• Core Scope of the Theorem:

  • Given any parent population with a finite mean $\mu$ and finite standard deviation $\sigma$, regardless of the parent population's distributional shape, three foundational mathematical principles describe the sampling distribution of the mean for any specified sample size $N$.

• Principle 1: Expected Value and Unbiasedness:     $\mu_{\bar{X}} = \mu$

  • The mean of the sampling distribution of the mean (the expected value) is identically equal to the mean of the parent population from which the sample means were drawn.

  • Property of Unbiased Estimation: Because $\mu_{\bar{X}} = \mu$, the sample mean $\bar{X}$ is formally designated an unbiased estimator of the population mean $\mu$. On average, across infinite replications, the sample mean neither overestimates nor underestimates the true parameter.

• Principle 2: Variance and Standard Error:

  • Variance of the Sampling Distribution:         $\sigma_{\bar{X}}^2 = \frac{\sigma^2}{N}$

    • The variance of the sampling distribution of the mean for a given sample size $N$ equals the variance of the parent population divided by the sample size.

  • Standard Error of the Mean (Standard Deviation of the Sampling Distribution):         $\sigma_{\bar{X}} = \frac{\sigma}{\sqrt{N}}$

    • The standard error equals the standard deviation of the parent population divided by the square root of the sample size.

• Principle 3: Distributional Shape and Convergence toward Normality:

  • Parent Population Normal: If the parent population is itself normally distributed, the sampling distribution of the mean will be perfectly normal for any sample size $N$:         $\text{If } X \sim N(\mu, \sigma^2), \text{ then } \bar{X}_N \sim N\left(\mu_{\bar{X}}, \frac{\sigma^2}{N}\right)$

  • Non-Normal Parent Populations: As sample size $N$ approaches infinity ($N \to \infty$), the sampling distribution of the mean increasingly approximates a normal distribution, regardless of the skewness, kurtosis, or multimodality of the parent population.

  • Convergence Across Different Shapes:

    • Uniform Population: Flat across its range. By $n = 5$, the sampling distribution forms a peaked, triangular shape; by $n = 30$, it is a smooth, narrow, bell-shaped normal curve.

    • Bimodal Population: Features two distinct peaks. By $n = 5$, the valley between the modes is filled in, producing a unimodal slightly asymmetric distribution; by $n = 30$, the distribution is fully symmetric and normal.

    • Skewed Population: Heavy tail to the right or left. By $n = 5$, skewness is noticeably attenuated; by $n = 30$, skewness is practically eliminated, yielding a symmetric normal distribution.

Computational Practice: Calculating Standard Error

• Problem Scenario:

  • Let random variable $X$ follow a normal distribution:         $X \sim N(10, 81)$

  • Extracted Population Parameters:         $\mu = 10$         $\sigma^2 = 81$         $\sigma = \sqrt{81} = 9$

• Sample Size Evaluations:

  • Condition 1: Standard error for sample size $N = 10$:         $\sigma_{\bar{X}} = \frac{\sigma}{\sqrt{N}} = \frac{9}{\sqrt{10}} = \frac{9}{3.16} = 2.85$

  • Condition 2: Standard error for sample size $N = 20$:         $\sigma_{\bar{X}} = \frac{\sigma}{\sqrt{N}} = \frac{9}{\sqrt{20}} = \frac{9}{4.47} = 2.01$

  • Condition 3: Standard error for sample size $N = 30$:         $\sigma_{\bar{X}} = \frac{\sigma}{\sqrt{N}} = \frac{9}{\sqrt{30}} = \frac{9}{5.48} = 1.64$

Impact of Sample Size on Standard Error and Precision

• General Structural Rules:

  • Sampling distributions are strictly defined relative to a specific sample size ($N$).

  • The standard error ($\sigma_{\bar{X}}$) is inversely proportional to the square root of $N$:         $N \uparrow \implies \sigma_{\bar{X}} \downarrow$

  • As $N \to \infty$, $\sigma_{\bar{X}} \to 0$.

  • A smaller standard error produces less dispersion across the sampling distribution, meaning sample means cluster tightly around $\mu_{\bar{X}}$.

  • As the standard error decreases, the statistical precision of any single sample mean ($\bar{X}$) as an estimate of the population mean ($\mu$) substantially increases.

• Comparative Benchmark (Gravetter & Wallnau, 2013, 2017):

  • Given parent population $X \sim N(80, 20)$, where $\mu = 80$ and $\sigma = 20$:

    • For $N = 1$:             $\sigma_{\bar{X}} = \frac{20}{\sqrt{1}} = 20$             The sampling distribution is identical in spread to the individual parent population scores.

    • For $N = 4$:             $\sigma_{\bar{X}} = \frac{20}{\sqrt{4}} = \frac{20}{2} = 10$             The standard error is halved, causing the distribution to narrow substantially.

    • For $N = 100$:             $\sigma_{\bar{X}} = \frac{20}{\sqrt{100}} = \frac{20}{10} = 2$             The standard error contracts to one-tenth of its initial value, creating a sharp, tall, highly concentrated distribution centered at $80$.

• External Exploratory References:

  • HyperStat sampling concepts reference: http://davidmlane.com/hyperstat/A13660.html

  • Interactive sampling distribution simulator: http://onlinestatbook.com/stat_sim/sampling_dist/