Sampling Distributions and Point Estimation Comprehensive Study Notes

Foundations of Population and Sample Statistics

To understand sampling distributions, one must distinguish between a population and a sample space. A population refers to the entire group of interest, while a sample is a subset of that population. Within this framework, a population parameter is a fixed numerical value describing a characteristic of the population, whereas a sample statistic is a value calculated from sample data used to estimate that parameter. For instance, in parameter estimation, statistics are utilized to gauge the effectiveness of pilot training or psychotherapy programs. A core objective involves using the sample standard deviation ($s$) to estimate the population standard deviation ($\sigma$).

Several fundamental questions guide the study of these distributions: What is a sampling distribution? What is the standard error? What is the principle of maximum likelihood? What is bias in a statistical sense? What is a confidence interval? What is the central limit theorem? Additionally, why is the number $1.96$ considered a significant value in this field?

The Mechanics of Sampling Distributions

A sampling distribution is a probability distribution of a statistic obtained through all possible samples of a fixed size ($N$). Because every sample drawn from a population contains different observed values, the resulting sample means will vary. To generate a sampling distribution, the following process is followed: First, a sample of size $N$ (such as $5$, $10$, or $1000$) is taken from the population. Second, a statistic, such as the mean ($\bar{x}$), is computed and recorded. Third, this process is repeated infinitely for large populations. Finally, the resulting distribution of the statistic over these repeated samples is plotted.

Consider an example involving a supervisor with six employees whose years of experience are $2$, $4$, $6$, $6$, $7$, and $8$. The population mean for these six elements is $\mu = 5.5$. If two employees are chosen randomly ($n = 2$) without replacement, the first observation has a probability of $1/6$ and the second has a probability of $1/5$. There are $15$ possible combinations, such as ($2,4$) with a mean of $3$, ($2,6$) with a mean of $4$, and ($7,8$) with a mean of $7.5$. The probabilities for each sample mean are as follows: $P(3) = 1/15$, $P(4) = 2/15$, $P(4.5) = 1/15$, $P(5) = 3/15$, $P(5.5) = 1/15$, $P(6) = 2/15$, $P(6.5) = 2/15$, $P(7) = 2/15$, and $P(7.5) = 1/15$.

In a second example, consider a population of six elements ($1, 2, 3, 4, 5, 6$) where samples of size $2$ are selected with replacement. This implies each element has a $1/6$ chance of being selected, resulting in $36$ possible outcomes ranging from ($1,1$) with a mean of $1$ to ($6,6$) with a mean of $6$. While there is only one way to achieve a mean of $1$, there are six different combinations that result in a mean of $3.5$. The sampling distribution illustrates the specific relationship between the probability of a statistic and the statistic's numerical value for all possible samples of size $N$.

Quantitative Properties of the Sample Mean and Standard Error

The mean of the sampling distribution is defined by the expected value of the sample mean. For a set of independent observations, the expected value of the sample mean is equal to the population mean:

$E(\bar{X}) = E\left(\frac{1}{n}(X_1 + X_2 + ... + X_n)\right) = \mu$

The variance of the sampling distribution is calculated as:

$Var(\bar{X}) = Var\left(\frac{X_1 + X_2 + ... + X_n}{n}\right) = \frac{\sigma^2}{n}$

The standard deviation of the sampling distribution is specifically referred to as the Standard Error ($\sigma_{\bar{x}}$). When individuals are selected independently, it is expressed as:

$\sigma_{\bar{x}} = \frac{\sigma}{\sqrt{n}}$

If observations are not selected independently, a finite population correction must be applied to the variance:

$Var(\bar{X}) = \frac{\sigma^2}{n} \times \frac{N - n}{N - 1}$

The Central Limit Theorem and Standard Normal Applications

The Central Limit Theorem (CLT) states that the sampling distribution of means becomes normal as the sample size ($N$) increases, regardless of the shape of the original distribution. This principle also applies to other statistics, such as variance, and dictates that binomial distributions become normal as $N$ increases.

Whenever a sampling distribution of the mean follows a normal distribution, we can compute the standardized normal variable $Z$ using the formula:

$Z = \frac{\bar{x} - \mu}{\sigma_{\bar{x}}} = \frac{\bar{x} - \mu}{\sigma / \sqrt{n}}$

Example Case Study: A spark plug manufacturer claims spark plug lives are normally distributed with a mean of $60,000\,miles$ and a standard deviation of $4000\,miles$. A random sample of $16$ plugs yields an average life of $58,500\,miles$. To find the probability of finding a sample mean of this value or less, we first calculate the standard error:

$\sigma_{\bar{x}} = \frac{4000}{\sqrt{16}} = 1000$

Then we calculate the $Z$-score:

$P(\bar{x} < 58,500) = P\left(Z < \frac{58,500 - 60,000}{1000}\right)$

$P(Z < -1.50) = 0.0668$

Acceptance Intervals and Sample Proportions

An acceptance interval is a range within which a sample mean has a high probability of occurring, given a known population mean and variance. If a sample mean falls within this interval, we accept the conclusion that the sample originated from that population. The interval is calculated as:

$\mu \pm Z_{\alpha/2} \sigma_{\bar{x}}$

In the context of Bernoulli and binomial distributions, the probability of success is denoted as $P$. The sample proportion ($p$) is used to make inferences about the population proportion. Let $X$ be the number of successes in a sample of $n$ observations; the sample proportion is $p = X/n$. Because $p$ is the mean of a set of independent random variables, the Central Limit Theorem allows us to model it as a normally distributed random variable for large sample sizes.

For a binomial distribution, the mean and variance of the sample proportion are:

$E(p) = P$

$Var(p) = \frac{P(1 - P)}{n}$

The standard normal variable for proportions is:

$Z = \frac{p - P}{\sigma_p}$

Example Case Study: A sample of $270$ homes is taken to estimate the proportion with unsafe wiring. If the true population proportion is $P = 0.2$, the probability that the sample proportion falls between $16\%$ and $24\%$ is calculated by first finding the standard error of the proportion:

$\sigma_p = \sqrt{\frac{0.2(0.8)}{270}} = 0.024$

$P(0.16 < p < 0.24) = P\left(\frac{0.16 - 0.2}{0.024} < Z < \frac{0.24 - 0.2}{0.024}\right) = 0.9050$

The Chi-Square Distribution and Sample Variance

Consider a random sample of $n$ observations from a population with an unknown mean ($\mu$) and unknown variance ($\sigma^2$). While the population variance is $\sigma^2 = E(X - \mu)^2$, the sample variance is calculated as:

$s^2 = \frac{\sum_{i=1}^{n} (x_i - \bar{x})^2}{n - 1}$

The expected value of the sample variance is equal to the population variance: $E(s^2) = \sigma^2$. For a normally distributed population, the ratio of the sample and population variance follows a chi-square distribution with $n - 1$ degrees of freedom ($v$):

$\chi_{(n-1)}^2 = \frac{(n-1)s^2}{\sigma^2} = \frac{\sum_{i=1}^n (x_i - \bar{x})^2}{\sigma^2}$

The chi-square distribution is defined only for positive values. Its mean and variance are linked to the degrees of freedom:

$E(\chi_v^2) = v$

$Var(\chi_v^2) = 2v$

Further proof shows that because $E[(n - 1)s^2 / \sigma^2] = n - 1$, then $Var(s^2) = 2\sigma^4 / (n - 1)$. When reading chi-square tables, specific values are denoted as $\chi_{\alpha, v}^2$. For example, with $v = 39$ degrees of freedom, a value corresponding to a $0.05$ upper tail would be written as $P(\chi_{39}^2 > K) = 0.05$.

Example Case Study: Shirley Mendez, a quality assurance manager at Green Valley Foods, wants to ensure the variation in package weights is limited. For a sample of $n = 20$, she seeks an upper limit for the ratio $s^2 / \sigma^2$ such that the probability of being above the limit is $0.025$. Using the chi-square table for $v = 19$ and $\alpha = 0.025$, the value is $32.852$. The limit ($k_u$) is found via:

$P(\chi_{19}^2 > 19 \times k_u) = 0.025$

$19 \times k_u = 32.852$

$k_u = \frac{32.852}{19} = 1.729$

This indicates that with $97.5\%$ probability, the ratio of sample variance to population variance will be below $1.729$.