Sampling Distributions of Statistics

Foundations of Statistical Inference and Statistics

• Statistics concerns itself primarily with conclusions and predictions resulting from chance outcomes that occur in carefully planned experiments or investigations.

• The inferential process involves taking sample observations from a specified population and using the results of that sample to make inferences about the population itself, including population parameters such as its mean and variance.

• Drawing sound inferences requires determining the probability distributions of specific functions of the observable random variables comprising the sample. These functions are termed statistics.

• The mathematical properties of these distributions allow for the formulation of probability statements regarding inferences drawn from a sample about the population.

• A statistic is formally defined as a function of observable random variables; because it depends on random variables, a statistic is itself a random variable possessed of its own probability distribution.

• Let $X_1, X_2, \dots, X_n$ denote a random sample drawn from a random variable $X$, and let $x_1, x_2, \dots, x_n$ denote the specific numerical values assumed by the sample.

• A real-valued function $T = F(X_1, X_2, \dots, X_n)$ is a statistic which assumes the observed realization $t = F(x_1, x_2, \dots, x_n)$.

• Common examples of statistics include:

  • The sample mean:     $\bar{X} = \frac{1}{n}\sum_{i=1}^n X_i$
  • The sample variance:     $S^2 = \frac{1}{n}\sum_{i=1}^n (X_i - \bar{X})^2$
  • The corrected (unbiased) sample variance:     $\hat{S}^2 = \frac{1}{n - 1}\sum_{i=1}^n (X_i - \bar{X})^2$
  • The minimum of the sample:     $L = \min(X_1, X_2, \dots, X_n)$
  • The maximum of the sample:     $H = \max(X_1, X_2, \dots, X_n)$
  • The sample range:     $R = H - L$

Sampling Distributions: Principles and Construction

• The probability distribution of a statistic $T$ is designated as the sampling distribution of $T$.

• The term "sampling distribution" derives from the fact that the statistic $T$ depends directly on a random sample, and therefore its distribution is derived from the underlying distribution of the sample.

• Formally, the sampling distribution of a statistic is defined as the distribution of all possible values that can be assumed by that statistic, computed from all possible samples of the same size randomly drawn from the population.

• Prior to observing empirical data, the sampling distribution indicates what values the statistic is likely to assume and how likely it is to assume those values.

• In analyzing any sampling distribution, three fundamental characteristics are required:

  • The mean of the sampling distribution.
  • The variance of the sampling distribution.
  • The functional form (shape) of the distribution.

• While empirical sampling distributions are rarely constructed manually in practical applications, the theoretical procedure for constructing the distribution from a discrete, finite population of size $N$ with samples of size $n$ is conceptualized as follows:

  • Step (a): Randomly select all possible samples of size $n$ from the finite population of size $N$.
  • Step (b): Calculate the numerical value of the statistic of interest for every selected sample.
  • Step (c): Tabulate the distinct observed values assumed by the statistic along with the absolute frequency of occurrence for each distinct value.
  • Step (d): Compute the relative frequencies (empirical probabilities) associated with each distinct value of the statistic to establish the probability function.

Sampling Distribution of the Mean for Infinite Populations

• The probability distribution of the sample mean, $\bar{X}$, is designated as the sampling distribution of the sample mean (or the sampling distribution of means).

• Theorem for Infinite Populations: If the random variables $X_1, X_2, \dots, X_n$ constitute a random sample from an infinite population having mean $\mu$ and variance $\sigma^2$, then:

  • The expectation of the sample mean is:     $E(\bar{X}) = \mu$
  • The variance of the sample mean is:     $\text{var}(\bar{X}) = \frac{\sigma^2}{n}$

• Proof of Expected Value:

  • By definition, the sample mean is:     $\bar{X} = \frac{1}{n}(X_1 + X_2 + \dots + X_n)$
  • Applying the expectation operator:     $E(\bar{X}) = E\left(\frac{X_1 + X_2 + \dots + X_n}{n}\right)$$E(\bar{X}) = \frac{1}{n}[E(X_1) + E(X_2) + \dots + E(X_n)]$
  • Because each sample observation is drawn from a population with mean $\mu$, $E(X_i) = \mu$ for all $i = 1, 2, \dots, n$:     $E(\bar{X}) = \frac{1}{n}(\mu + \mu + \dots + \mu)$$E(\bar{X}) = \frac{1}{n}(n\mu) = \mu$

• Proof of Variance:

  • Applying the variance operator to the linear combination:     $\text{var}(\bar{X}) = \text{var}\left(\frac{X_1 + X_2 + \dots + X_n}{n}\right)$$\text{var}(\bar{X}) = \frac{1}{n^2}[\text{var}(X_1) + \text{var}(X_2) + \dots + \text{var}(X_n)]$
  • Since the random variables are mutually independent with common variance $\sigma^2$:     $\text{var}(\bar{X}) = \frac{1}{n^2}(\sigma^2 + \sigma^2 + \dots + \sigma^2)$$\text{var}(\bar{X}) = \frac{1}{n^2}(n\sigma^2) = \frac{\sigma^2}{n}$

• Properties of the Distribution of the Mean:

  • The expected value of $\bar{X}$ is identical to the population mean $\mu$.
  • The variance of $\bar{X}$ equals $\frac{1}{n}$ times the population variance $\sigma^2$.
  • The distribution of $\bar{X}$ is centered about $\mu$, but its spread contracts progressively as the sample size $n$ increases.
  • The mean of the sampling distribution of $\bar{X}$ is customarily denoted as $\mu_{\bar{X}} = E(\bar{X})$, and its variance is denoted as $\sigma_{\bar{X}}^2 = \text{var}(\bar{X})$.
  • The standard deviation of the sampling distribution of $\bar{X}$ is designated as the standard error of the mean:     $\sigma_{\bar{X}} = \frac{\sigma}{\sqrt{n}}$

• The formula establishes that the dispersion of the sample mean diminishes inversely with the square root of the sample size $n$.

Demonstration: Sampling with Replacement from a Finite Population

• Consider a population consisting of four numerical values: $4, 5, 7, 8$, where population size $N = 4$.

• Population Parameters:

  • Population mean:     $\mu = \frac{1}{N}\sum_{i=1}^N X_i = \frac{4 + 5 + 7 + 8}{4} = \frac{24}{4} = 6.0$
  • Population variance:     $\sigma^2 = \frac{1}{N}\sum_{i=1}^N (X_i - \mu)^2$$\sigma^2 = \frac{(4 - 6)^2 + (5 - 6)^2 + (7 - 6)^2 + (8 - 6)^2}{4}$$\sigma^2 = \frac{(-2)^2 + (-1)^2 + 1^2 + 2^2}{4} = \frac{4 + 1 + 1 + 4}{4} = \frac{10}{4} = 2.5$

• Complete Enumeration of Samples of Size $n = 2$ Drawn with Replacement:

  • The total number of possible samples is $N^n = 4^2 = 16$.
  • Sample 1: Indices $(1,1)$, Values $(4,4)$, Mean $\bar{X}_1 = 4.0$
  • Sample 2: Indices $(1,2)$, Values $(4,5)$, Mean $\bar{X}_2 = 4.5$
  • Sample 3: Indices $(1,3)$, Values $(4,7)$, Mean $\bar{X}_3 = 5.5$
  • Sample 4: Indices $(1,4)$, Values $(4,8)$, Mean $\bar{X}_4 = 6.0$
  • Sample 5: Indices $(2,1)$, Values $(5,4)$, Mean $\bar{X}_5 = 4.5$
  • Sample 6: Indices $(2,2)$, Values $(5,5)$, Mean $\bar{X}_6 = 5.0$
  • Sample 7: Indices $(2,3)$, Values $(5,7)$, Mean $\bar{X}_7 = 6.0$
  • Sample 8: Indices $(2,4)$, Values $(5,8)$, Mean $\bar{X}_8 = 6.5$
  • Sample 9: Indices $(3,1)$, Values $(7,4)$, Mean $\bar{X}_9 = 5.5$
  • Sample 10: Indices $(3,2)$, Values $(7,5)$, Mean $\bar{X}_{10} = 6.0$
  • Sample 11: Indices $(3,3)$, Values $(7,7)$, Mean $\bar{X}_{11} = 7.0$
  • Sample 12: Indices $(3,4)$, Values $(7,8)$, Mean $\bar{X}_{12} = 7.5$
  • Sample 13: Indices $(4,1)$, Values $(8,4)$, Mean $\bar{X}_{13} = 6.0$
  • Sample 14: Indices $(4,2)$, Values $(8,5)$, Mean $\bar{X}_{14} = 6.5$
  • Sample 15: Indices $(4,3)$, Values $(8,7)$, Mean $\bar{X}_{15} = 7.5$
  • Sample 16: Indices $(4,4)$, Values $(8,8)$, Mean $\bar{X}_{16} = 8.0$

• Sampling Distribution Parameters from Empirical Values:

  • Mean of the distribution of means:     $\mu_{\bar{X}} = E(\bar{X}) = \frac{1}{16}\sum_{i=1}^{16} \bar{X}_i$$\mu_{\bar{X}} = \frac{4.0 + 4.5 + 5.5 + 6.0 + 4.5 + 5.0 + 6.0 + 6.5 + 5.5 + 6.0 + 7.0 + 7.5 + 6.0 + 6.5 + 7.5 + 8.0}{16} = \frac{96.0}{16} = 6.0$
  • Variance of the distribution of means:     $\sigma_{\bar{X}}^2 = \frac{1}{16}\sum_{i=1}^{16} (\bar{X}_i - \mu_{\bar{X}})^2$$\sigma_{\bar{X}}^2 = \frac{(4.0 - 6)^2 + (4.5 - 6)^2 + (5.5 - 6)^2 + \dots + (8.0 - 6)^2}{16}$$\sigma_{\bar{X}}^2 = \frac{(-2.0)^2 + (-1.5)^2 + (-0.5)^2 + 0^2 + (-1.5)^2 + (-1.0)^2 + 0^2 + 0.5^2 + (-0.5)^2 + 0^2 + 1.0^2 + 1.5^2 + 0^2 + 0.5^2 + 1.5^2 + 2.0^2}{16}$$\sigma_{\bar{X}}^2 = \frac{4.0 + 2.25 + 0.25 + 0 + 2.25 + 1.0 + 0 + 0.25 + 0.25 + 0 + 1.0 + 2.25 + 0 + 0.25 + 2.25 + 4.0}{16} = \frac{20.0}{16} = 1.25$

• Verification of Theoretical Predictions:

  • For the mean:     $\mu_{\bar{X}} = 6.0 = \mu$
  • For the variance:     $\frac{\sigma^2}{n} = \frac{2.5}{2} = 1.25$$\sigma_{\bar{X}}^2 = \frac{\sigma^2}{n} = 1.25$

Sampling Distribution of the Mean for Finite Populations

• When sampling without replacement from a finite population of size $N$, observations are statistically dependent.

• Theorem for Finite Populations: If $\bar{X}$ is the mean of a random sample of size $n$ drawn without replacement from a finite population of size $N$ with mean $\mu$ and variance $\sigma^2$, then:

  • Expected value:     $E(\bar{X}) = \mu$
  • Variance:     $\text{var}(\bar{X}) = \frac{\sigma^2}{n}\left(\frac{N - n}{N - 1}\right)$

• Proof of Variance for Finite Populations:

  • Express the sample mean as:     $\bar{X} = \frac{1}{n}\sum_{i=1}^n X_i$
  • Applying variance:     $\text{var}(\bar{X}) = \text{var}\left(\frac{1}{n}\sum_{i=1}^n X_i\right) = \frac{1}{n^2}\text{var}\left(\sum_{i=1}^n X_i\right)$
  • Expanding the variance of dependent random variables:     $\text{var}\left(\sum_{i=1}^n X_i\right) = \sum_{i=1}^n \text{var}(X_i) + 2\sum_{1 \le i < j \le n} \text{cov}(X_i, X_j)$$\text{var}\left(\sum_{i=1}^n X_i\right) = n\sigma^2 + 2\binom{n}{2}\text{cov}(X_i, X_j) = n\sigma^2 + n(n - 1)\text{cov}(X_i, X_j)$
  • To calculate $\text{cov}(X_i, X_j)$, where $i \neq j$:     $\text{cov}(X_i, X_j) = E[(X_i - \mu)(X_j - \mu)] = \sum_{k=1}^N \sum_{l=1, l \neq k}^N \frac{(X_k - \mu)(X_l - \mu)}{N(N - 1)}$
  • Using the algebraic identity for the sum of products:     $\left(\sum_{k=1}^N (X_k - \mu)\right)^2 = \sum_{k=1}^N (X_k - \mu)^2 + \sum_{k=1}^N \sum_{l \neq k} (X_k - \mu)(X_l - \mu)$
  • Because deviations about the mean sum to zero, $\sum_{k=1}^N (X_k - \mu) = 0$:     $0 = \sum_{k=1}^N (X_k - \mu)^2 + \sum_{k=1}^N \sum_{l \neq k} (X_k - \mu)(X_l - \mu)$$\sum_{k=1}^N \sum_{l \neq k} (X_k - \mu)(X_l - \mu) = -\sum_{k=1}^N (X_k - \mu)^2 = -N\sigma^2$
  • Substituting this back into the covariance formula:     $\text{cov}(X_i, X_j) = \frac{-N\sigma^2}{N(N - 1)} = -\frac{\sigma^2}{N - 1}$
  • Substituting the covariance into the total variance:     $\text{var}(\bar{X}) = \frac{1}{n^2}\left[n\sigma^2 + n(n - 1)\left(-\frac{\sigma^2}{N - 1}\right)\right]$$\text{var}(\bar{X}) = \frac{\sigma^2}{n} - \frac{(n - 1)\sigma^2}{n(N - 1)} = \frac{\sigma^2}{n}\left(1 - \frac{n - 1}{N - 1}\right)$$\text{var}(\bar{X}) = \frac{\sigma^2}{n}\left(\frac{(N - 1) - (n - 1)}{N - 1}\right) = \frac{\sigma^2}{n}\left(\frac{N - n}{N - 1}\right)$

• The Finite Population Correction Factor:

  • The expression $\frac{N - n}{N - 1}$ is designated as the finite population correction (fpc) factor.
  • When $N$ is exceptionally large relative to the sample size $n$, the correction factor approaches unity:     $\frac{N - n}{N - 1} \approx 1$

• Example: Sampling Without Replacement:

  • For the population $\{4, 5, 7, 8\}$ ($N = 4$, $\mu = 6.0$, $\sigma^2 = 2.5$), sampling $n = 2$ elements without replacement yields $\binom{4}{2} = 6$ distinct sample combinations:
  • Sample 1: Values $\{4, 5\}$, Mean $\bar{X} = 4.5$
  • Sample 2: Values $\{4, 7\}$, Mean $\bar{X} = 5.5$
  • Sample 3: Values $\{4, 8\}$, Mean $\bar{X} = 6.0$
  • Sample 4: Values $\{5, 7\}$, Mean $\bar{X} = 6.0$
  • Sample 5: Values $\{5, 8\}$, Mean $\bar{X} = 6.5$
  • Sample 6: Values $\{7, 8\}$, Mean $\bar{X} = 7.5$
  • The mean of the distribution of means is:     $\mu_{\bar{X}} = \frac{4.5 + 5.5 + 6.0 + 6.0 + 6.5 + 7.5}{6} = \frac{36.0}{6} = 6.0$
  • The variance of the distribution of means is:     $\sigma_{\bar{X}}^2 = \frac{(4.5 - 6)^2 + (5.5 - 6)^2 + (6.0 - 6)^2 + (6.0 - 6)^2 + (6.5 - 6)^2 + (7.5 - 6)^2}{6}$$\sigma_{\bar{X}}^2 = \frac{2.25 + 0.25 + 0.0 + 0.0 + 0.25 + 2.25}{6} = \frac{5.0}{6} \approx 0.8333$
  • Theoretical verification:     $\sigma_{\bar{X}}^2 = \frac{\sigma^2}{n}\left(\frac{N - n}{N - 1}\right) = \frac{2.5}{2}\left(\frac{4 - 2}{4 - 1}\right) = 1.25 \times \frac{2}{3} = \frac{2.5}{3} = \frac{5}{6} \approx 0.8333$

Sampling Distribution of Proportions

• Parameter Definitions:

  • The population proportion $P$ is defined as:     $P = \frac{X}{N}$     where $X$ represents the total number of elements in the population possessing a specific trait, and $N$ is the total population size.
  • The sample proportion $\hat{p}$ is defined as:     $\hat{p} = \frac{x}{n}$     where $x$ represents the observed number of elements in the sample possessing the trait, and $n$ is the sample size.

• Sampling with Replacement:

  • A sample proportion $\hat{p}$ represents the proportion of successes obtained by dividing the count of successes $x$ by the sample size $n$.
  • When a sample of size $n$ is drawn with replacement, the count of successes obeys the Binomial probability law:     $x \sim \text{Binomial}(n, P)$
  • Theorem for Proportions with Replacement:
  • Expected value:       $E(\hat{p}) = P$
  • Variance:       $\text{var}(\hat{p}) = \frac{P(1 - P)}{n}$
  • Example: Defective Bulbs with Replacement:
  • A consignment contains $20\%$ defective bulbs ($P = 0.20$), and a random sample of size $n = 250$ is drawn with replacement.
  • Expectation of the sample proportion:       $E(\hat{p}) = P = 0.20$
  • Variance of the sample proportion:       $\text{var}(\hat{p}) = \frac{P(1 - P)}{n} = \frac{0.20(1 - 0.20)}{250} = \frac{0.20 \times 0.80}{250} = \frac{0.16}{250} = 0.00064$
  • Standard error of the sample proportion:       $\sigma_{\hat{p}} = \sqrt{\text{var}(\hat{p})} = \sqrt{0.00064} \approx 0.0253$

• Sampling Without Replacement:

  • When sampling is conducted without replacement from a finite population, the count of successes $x$ obeys the Hypergeometric probability law.
  • Theorem for Proportions Without Replacement:
  • Expected value:       $E(\hat{p}) = P$
  • Variance:       $\text{var}(\hat{p}) = \frac{P(1 - P)}{n}\left(\frac{N - n}{N - 1}\right)$
  • Example: Male Students Without Replacement:
  • In a class of $N = 50$ students, $20$ are female and $30$ are male. A random sample of $n = 12$ students is drawn without replacement.
  • The population proportion of males is:       $P = \frac{30}{50} = 0.60$
  • Variance of the sample proportion:       $\text{var}(\hat{p}) = \frac{P(1 - P)}{n}\left(\frac{N - n}{N - 1}\right)$$\text{var}(\hat{p}) = \frac{0.60(1 - 0.60)}{12}\left(\frac{50 - 12}{50 - 1}\right) = \frac{0.24}{12}\left(\frac{38}{49}\right)$$\text{var}(\hat{p}) = 0.02 \times \frac{38}{49} = \frac{0.76}{49} \approx 0.01551$
  • Standard error of the sample proportion:       $\sigma_{\hat{p}} = \sqrt{\frac{0.76}{49}} = \frac{\sqrt{0.76}}{7} \approx \frac{0.8718}{7} \approx 0.1245$

Sampling Distribution of the Variance

• Definitions and Sample Variance:

  • Let $X_1, X_2, \dots, X_n$ denote a random sample of size $n$ drawn from a distribution with mean $\mu$ and variance $\sigma^2$.
  • The standard sample variance is defined as:     $S^2 = \frac{1}{n}\sum_{i=1}^n (X_i - \bar{X})^2$

• Theorem for Infinite Populations or Sampling with Replacement:

  • The expected value of the sample variance $S^2$ is:     $E(S^2) = \left(\frac{n - 1}{n}\right)\sigma^2$

• Proof of Expected Sample Variance:

  • Express the deviations about the sample mean using the identity:     $X_i - \bar{X} = (X_i - \mu) - (\bar{X} - \mu)$
  • Squaring and summing both sides over all $i$:     $\sum_{i=1}^n (X_i - \bar{X})^2 = \sum_{i=1}^n [((X_i - \mu) - (\bar{X} - \mu))^2]$$\sum_{i=1}^n (X_i - \bar{X})^2 = \sum_{i=1}^n (X_i - \mu)^2 - 2(\bar{X} - \mu)\sum_{i=1}^n (X_i - \mu) + n(\bar{X} - \mu)^2$
  • Because $\sum_{i=1}^n (X_i - \mu) = n(\bar{X} - \mu)$, the cross-term simplifies to:     $-2n(\bar{X} - \mu)^2$
  • Combining the expressions:     $\sum_{i=1}^n (X_i - \bar{X})^2 = \sum_{i=1}^n (X_i - \mu)^2 - n(\bar{X} - \mu)^2$
  • Dividing both sides by $n$ yields the formula for $S^2$:     $S^2 = \frac{1}{n}\sum_{i=1}^n (X_i - \mu)^2 - (\bar{X} - \mu)^2$
  • Taking expectations of both sides:     $E(S^2) = \frac{1}{n}\sum_{i=1}^n E[(X_i - \mu)^2] - E[(\bar{X} - \mu)^2]$
  • Recognizing that $E[(X_i - \mu)^2] = \text{var}(X_i) = \sigma^2$ and $E[(\bar{X} - \mu)^2] = \text{var}(\bar{X}) = \frac{\sigma^2}{n}$:     $E(S^2) = \frac{1}{n}(n\sigma^2) - \frac{\sigma^2}{n} = \sigma^2 - \frac{\sigma^2}{n} = \left(\frac{n - 1}{n}\right)\sigma^2$

• Corrected Sample Variance and Bias Correction:

  • Because $E(S^2) = \frac{n - 1}{n}\sigma^2 < \sigma^2$, the sample variance $S^2$ is a biased estimator of $\sigma^2$, systematically underestimating the population variance.
  • For large sample sizes (such as $n \ge 30$), $\frac{n - 1}{n} \approx 1$, meaning $E(S^2)$ is approximately equal to $\sigma^2$.
  • To eliminate the negative bias entirely, the corrected sample variance (unbiased estimator of variance), denoted $\hat{S}^2$, is defined:     $\hat{S}^2 = \frac{n}{n - 1}S^2 = \frac{1}{n - 1}\sum_{i=1}^n (X_i - \bar{X})^2$
  • Expectation of the corrected sample variance:     $E(\hat{S}^2) = E\left(\frac{n}{n - 1}S^2\right) = \frac{n}{n - 1}E(S^2) = \frac{n}{n - 1}\left(\frac{n - 1}{n}\sigma^2\right) = \sigma^2$
  • The computational formula for the corrected sample variance is:     $\hat{S}^2 = \frac{1}{n - 1}\left[\sum_{i=1}^n X_i^2 - \frac{1}{n}\left(\sum_{i=1}^n X_i\right)^2\right]$

• Practical Application Guidelines:

  • When the sample size is small ($n < 30$), the corrected sample variance $\hat{S}^2$ must be employed.
  • When the sample size is large ($n \ge 30$), the uncorrected sample variance $S^2$ is conventionally used, as the numerical disparity between $n$ and $n - 1$ is practically negligible.

• Finite Population Theorem for Variances:

  • Theorem: If sampling is conducted without replacement from a finite population of size $N$, the mean of the sampling distribution of variances is:     $E(S^2) = \left(\frac{N}{N - 1}\right)\left(\frac{n - 1}{n}\right)\sigma^2$
  • As $N \to \infty$, the correction ratio approaches unity:     $\lim_{N \to \infty} \left(\frac{N}{N - 1}\right) = 1$$\lim_{N \to \infty} E(S^2) = \left(\frac{n - 1}{n}\right)\sigma^2$

• Numerical Verification (Preceding Example):

  • For the population $\{4, 5, 7, 8\}$ where $N = 4$, $n = 2$, and $\sigma^2 = 2.5$:
  • The expected value of the sample variance is:       $E(S^2) = \left(\frac{2 - 1}{2}\right)(2.5) = 1.25$
  • The expected value of the corrected sample variance is:       $E(\hat{S}^2) = \sigma^2 = 2.5$