Comprehensive Study Notes: Sampling Distributions, Central Limit Theorem, and Estimation

Core Principles of Sampling Distributions and Estimators

• Key Statistical Definitions:

  • Estimator: A sample statistic derived from a random sample to infer the value of an unknown population parameter.

  • Estimate: The specific numerical value that an estimator takes in a given sample.

  • Population Parameter: A numerical measurement describing a characteristic of a population, typically denoted by Greek letters (e.g., population mean $\mu$, population proportion $\pi$, population standard deviation $\sigma$).

  • Sample Statistic: A numerical measurement describing a characteristic of a sample, typically denoted by Roman letters (e.g., sample mean $\bar{X}$, sample proportion $p$, sample standard deviation $S$).

• Parameter and Estimator Pairings:

  • The sample mean $\bar{X}$ serves as the estimator for the population mean $\mu$.

  • The sample proportion $p$ serves as the estimator for the population proportion $\pi$.

  • The sample standard deviation $S$ serves as the estimator for the population standard deviation $\sigma$.

• Sampling Variation:

  • A sample statistic is a random variable whose observed value depends entirely on which specific population items are selected into the sample.

  • Due to sampling variation, sample statistics vary from sample to sample.

• Sampling Error and Bias:

  • Sampling Error: The difference between a sample estimate and the corresponding population parameter caused by random chance in sample selection:         $\text{Sampling Error} = \bar{X} - \mu$

  • Bias: The difference between the expected value of the estimator and the true parameter value:         $\text{Bias} = E(\bar{X}) - \mu$

  • Unbiased Estimator: An estimator is unbiased if its expected value equals the parameter being estimated ($E(\bar{X}) = \mu$). On average, an unbiased estimator neither overstates nor understates the true population parameter.

• Standard Error of the Mean:

  • The Standard Error of the Mean ($\sigma_{\bar{X}}$) measures the variability of sample means across all theoretical samples of size $n$ drawn from a population:         $\sigma_{\bar{X}} = \frac{\sigma}{\sqrt{n}}$

  • This formula assumes sampling with replacement or sampling without replacement from an infinite population.

  • As the sample size $n$ increases, the standard error of the mean decreases, causing the sampling distribution to concentrate more tightly around the population mean $\mu$.

• Sampling Distribution for Normal Populations:

  • If a population is normally distributed with mean $\mu$ and standard deviation $\sigma$, the sampling distribution of $\bar{X}$ is exactly normally distributed for any sample size $n$:         $\mu_{\bar{X}} = \mu$         $\sigma_{\bar{X}} = \frac{\sigma}{\sqrt{n}}$

  • The standardized $Z$-score for a sample mean is:         $Z = \frac{\bar{X} - \mu}{\sigma_{\bar{X}}} = \frac{\bar{X} - \mu}{\frac{\sigma}{\sqrt{n}}}$

• Empirical Demonstration: GMAT Scores Example:

  • Consider eight independent random samples of size $n = 5$ drawn from a large population of MBA applicant GMAT scores with population mean $\mu = 520.28$:

    | Sample Number | Score 1 | Score 2 | Score 3 | Score 4 | Score 5 | Sample Mean ($\bar{X}$) |     | :--- | :--- | :--- | :--- | :--- | :--- | :--- |     | 1 | 490 | 580 | 440 | 580 | 430 | 504 |     | 2 | 310 | 590 | 730 | 710 | 540 | 576 |     | 3 | 500 | 450 | 510 | 570 | 610 | 528 |     | 4 | 450 | 590 | 710 | 240 | 510 | 500 |     | 5 | 420 | 640 | 470 | 530 | 640 | 540 |     | 6 | 450 | 670 | 390 | 500 | 470 | 496 |     | 7 | 490 | 450 | 590 | 640 | 650 | 564 |     | 8 | 670 | 610 | 550 | 540 | 540 | 582 |

*   The individual test scores exhibit wide scatter (ranging from 240 to 730), whereas the sample means exhibit substantially reduced dispersion tightly grouped around the population mean $\mu = 520.28$

Statistical Error, Mean Squared Error, and Estimator Selection

• Evaluating Estimators:

  • Consider measuring a line's length using five different rulers, yielding measurements of $971$, $972$, $973$, $979$, and $980\,\text{cm}$.

  • Candidate estimates include the median ($973\,\text{cm}$) and the mean ($975\,\text{cm}$).

• Mean Squared Error (MSE):

  • The sample mean minimizes the total Mean Squared Error relative to all observed data points, making it the mathematically optimal point estimate.

  • Evaluating MSE across candidate values relative to the raw dataset ($971, 972, 973, 979, 980$):

    • Value $971$: $\text{MSE} = 30$

    • Value $972$: $\text{MSE} = 23$

    • Value $973$ (Median): $\text{MSE} = 18$

    • Value $974$: $\text{MSE} = 15$

    • Value $975$ (Mean): $\text{MSE} = 14$

    • Value $976$: $\text{MSE} = 15$

    • Value $977$: $\text{MSE} = 18$

    • Value $978$: $\text{MSE} = 23$

    • Value $979$: $\text{MSE} = 30$

    • Value $980$: $\text{MSE} = 39$

The Central Limit Theorem (CLT)

• Formal Statement of the Central Limit Theorem:

  • If random samples of size $n$ are drawn from any population with mean $\mu$ and standard deviation $\sigma$, the sampling distribution of the sample mean $\bar{X}$ approaches a normal distribution with mean $\mu_{\bar{X}} = \mu$ and standard deviation $\sigma_{\bar{X}} = \frac{\sigma}{\sqrt{n}}$ as sample size $n$ increases.

  • If the underlying population is normal, $\bar{X}$ is normally distributed for any sample size $n$.

  • If the underlying population is non-normal, $\bar{X}$ is approximately normal provided $n \ge 30$

• Discrete Uniform Population Illustration ($n = 2$):

  • Consider a discrete uniform population consisting of integers $\left\{0, 1, 2, 3\right\}$.

  • Population Parameters: $\mu = 1.5$, $\sigma = 1.118$

  • All possible samples of size $n = 2$ drawn with replacement:

    | Row \ Col | a | b | c | d |     | :--- | :--- | :--- | :--- | :--- |     | 1 | (0,0) | (1,0) | (2,0) | (3,0) |     | 2 | (0,1) | (1,1) | (2,1) | (3,1) |     | 3 | (0,2) | (1,2) | (2,2) | (3,2) |     | 4 | (0,3) | (1,3) | (2,3) | (3,3) |

*   Means of all possible samples of size $n = 2$:

    | Row \ Col | a | b | c | d |     | :--- | :--- | :--- | :--- | :--- |     | 1 | 0.0 | 0.5 | 1.0 | 1.5 |     | 2 | 0.5 | 1.0 | 1.5 | 2.0 |     | 3 | 1.0 | 1.5 | 2.0 | 2.5 |     | 4 | 1.5 | 2.0 | 2.5 | 3.0 |

*   While the original population distribution is perfectly uniform (each value $\left\{0, 1, 2, 3\right\}$ occurs with count 4), the distribution of sample means ($n=2$) forms a symmetric triangular distribution centered at $\bar{X} = 1.5$ (counts: mean 0.0: 1; mean 0.5: 2; mean 1.0: 3; mean 1.5: 4; mean 2.0: 3; mean 2.5: 2; mean 3.0: 1).

• Practical Utility of the CLT:

  1. Given sample data from a valid random sample, accurate assumptions can be made regarding the population parameter.

  2. Given known population parameters, accurate probabilities can be calculated for sample statistics.

  3. Given population parameters and sample statistics, formal inferences can establish whether a sample was likely drawn from that population.

  4. Given two independent samples, inferences can determine if both samples originated from the same population.

• CLT Probability Calculations and Applications:

  • Expected Range Formula:         $\mu \pm Z \frac{\sigma}{\sqrt{n}}$

  • Example 1: General Population Inference

    • Given: Population mean $\mu = 8$, standard deviation $\sigma = 3$, sample size $n = 36$.

    • Find: $P(7.8 < \bar{X} < 8.2)$

    • Standard Error:             $\sigma_{\bar{X}} = \frac{3}{\sqrt{36}} = 0.5$

    • $Z$-transformation:             $Z_{\text{lower}} = \frac{7.8 - 8}{0.5} = -0.4$             $Z_{\text{upper}} = \frac{8.2 - 8}{0.5} = 0.4$

    • Probability:             $P(-0.4 < Z < 0.4) = 0.6554 - 0.3446 = 0.3108$

  • Example 2: Plumber Salaries

    • Given: Population mean annual salary $\mu = \$46,700$, $\sigma = \$5,600$, sample size $n = 42$

    • Find: Probability that sample mean salary exceeds $\$44,000$

    • Standard Error:             $\sigma_{\bar{X}} = \frac{5600}{\sqrt{42}} = 864.10$

    • $Z$-score:             $Z = \frac{44000 - 46700}{864.10} = -3.125$

    • Probability calculation yields $P(\bar{X} > 44000) \approx 0.9991$

  • Example 3: Air Conditioner Maintenance Time

    • Given: Severely right-skewed population distribution with mean $\mu = 1.0\,\text{hour}$, standard deviation $\sigma = 0.1\,\text{hour}$, sample size $n = 70$ air conditioners. Budgeted time is $1.1\,\text{hours}$ per unit.

    • Sampling Distribution: Approximately $N\left(1.0, \frac{0.1}{\sqrt{70}}\right) = N(1.0, 0.012)$

    • $Z$-transformation:             $Z = \frac{1.1 - 1.0}{0.12} = 0.83$

    • Exceedance Probability:             $P(\bar{X} > 1.1) = P(Z > 0.83) = 1 - 0.7967 = 0.2033$

    • There is a $20.33\%$ probability that technicians will exceed budgeted hours.

Confidence Intervals for Means (Known and Unknown Population Standard Deviation)

• General Interval Structure:     $\text{Point Estimate} \pm (\text{Critical Value})(\text{Standard Error})$

  • The sample statistic serves as the point estimate and center of the interval.

  • The margin of error is the product of the critical value and standard error.

• Confidence Level $(1 - \alpha)$ and Critical Values:

  • A relative frequency interpretation: $95\%$ of all confidence intervals constructed across repeated sampling contain the true population parameter.

  • A specific calculated interval either contains or does not contain the true parameter (no probability statement attaches to an calculated interval).

    | Confidence Level | Confidence Coefficient ($1 - \alpha$) | Critical Value ($Z_{\alpha/2}$) |     | :--- | :--- | :--- |     | 80.0% | 0.800 | 1.280 |     | 90.0% | 0.900 | 1.645 |     | 95.0% | 0.950 | 1.960 |     | 98.0% | 0.980 | 2.330 |     | 99.0% | 0.990 | 2.580 |     | 99.8% | 0.998 | 3.080 |     | 99.9% | 0.999 | 3.270 |

• Confidence Interval for $\mu$ ($\sigma$ Known):

  • Assumptions: $\sigma$ is known, population is normally distributed (or $n > 30$).

  • Formula:         $\bar{X} \pm Z_{\alpha/2} \frac{\sigma}{\sqrt{n}}$

  • Cereal Fill Simulation Example:

    • Given: $\mu = 368$, $\sigma = 15$, sample size $n = 25$. Standard error $\sigma_{\bar{X}} = \frac{15}{\sqrt{25}} = 3.0$

    • Theoretical $95\%$ range centered on $\mu$: $368 \pm 1.96(3.0) = (362.12, 373.88)$

    • Evaluating constructed intervals across independent sample draws ($n=25$):

        | Sample # | Sample Mean ($\bar{X}$) | Lower Limit | Upper Limit | Contains $\mu=368$? |         | :--- | :--- | :--- | :--- | :--- |         | 1 | 362.30 | 356.42 | 368.18 | Yes |         | 2 | 369.50 | 363.62 | 375.38 | Yes |         | 3 | 360.00 | 354.12 | 365.88 | No |         | 4 | 362.12 | 356.24 | 368.00 | Yes |         | 5 | 373.88 | 368.00 | 379.76 | Yes |

*   *Circuit Resistance Example*:
    *   Given: Sample size $n = 11$, $\bar{X} = 2.22\,\Omega$, population $\sigma = 0.35\,\Omega$
    *   $95\%$ Confidence Interval Calculation:

            $2.22 \pm 1.96 \frac{0.35}{\sqrt{11}} = 2.22 \pm 0.2068 = (2.0132, 2.4268)\,\Omega$

• Confidence Interval for $\mu$ ($\sigma$ Unknown) & Student's $t$ Distribution:

  • In real-world applications, $\sigma$ is almost universally unknown (calculating $\sigma$ requires prior knowledge of $\mu$).

  • When $\sigma$ is unknown, sample standard deviation $S$ replaces $\sigma$, introducing extra variability. Consequently, Student's $t$ distribution replaces the normal distribution.

  • Assumptions: $\sigma$ is unknown, population is normal (or $n > 30$).

  • Formula:         $\bar{X} \pm t_{\alpha/2} \frac{S}{\sqrt{n}}$

  • Degrees of Freedom ($d.f.$):

    • Defined as the number of observations free to vary after calculating the sample statistic.

    • $d.f. = n - 1$

    • Degrees of Freedom Example: If three numbers have a mean of $8.0$, and $X_1 = 7, X_2 = 8$, $X_3$ is constrained to be $9$. Thus, $d.f. = 3 - 1 = 2$

  • Critical Value Comparison Table ($t$ vs. $Z$):

    | Confidence Level | $t\,(d.f.=10)$ | $t\,(d.f.=20)$ | $t\,(d.f.=30)$ | $Z\,(d.f.=\infty)$ |     | :--- | :--- | :--- | :--- | :--- |     | 0.80 | 1.372 | 1.325 | 1.310 | 1.280 |     | 0.90 | 1.812 | 1.725 | 1.697 | 1.645 |     | 0.95 | 2.228 | 2.086 | 2.042 | 1.960 |     | 0.99 | 3.169 | 2.845 | 2.750 | 2.580 |

*   The $t$ distribution family features heavier ("fattest") tails than the Standard Normal distribution, generating wider, more conservative confidence intervals for small samples. As $d.f. \to \infty$, the $t$ distribution converges to the standard normal $Z$ distribution.

*   *Example Calculation ($t$ Distribution)*:
    *   Given: Sample $n = 25$, $\bar{X} = 50$, $S = 8$.
    *   Degrees of freedom: $d.f. = 25 - 1 = 24$
    *   Critical value for $95\%$ confidence ($\alpha/2 = 0.025$): $t_{0.025, 24} = 2.064$
    *   Interval calculation:

            $50 \pm 2.064 \frac{8}{\sqrt{25}} = 50 \pm 3.302 = (46.698, 53.302)$ * Normality of the population for small samples ($n < 30$) should be verified using boxplots or normal probability plots.

• Sample Size Determination for the Mean:

  • Margin of Error equation:         $e = Z_{\alpha/2} \frac{\sigma}{\sqrt{n}}$

  • Solving for required sample size $n$:         $n = \frac{Z_{\alpha/2}^2 \sigma^2}{e^2}$

  • Requirements: Desired confidence level ($Z_{\alpha/2}$), allowable margin of error ($e$), and standard deviation ($\sigma$).

  • Sample Size Calculation Example:

    • Given: $\sigma = 45$, desired margin of error $e = \pm 5$, $90\%$ confidence level ($Z = 1.645$).

    • Calculation:             $n = \frac{(1.645)^2 (45)^2}{5^2} = \frac{(2.706025)(2025)}{25} = 219.19$

    • Rounding rule: Always round up to the next integer ($n = 220$).

  • If $\sigma$ is unknown during planning, estimate $\sigma$ using pilot sample standard deviation $S$ or a conservative upper bound estimate of $\sigma$

Confidence Intervals and Sample Size Determination for Population Proportions

• Population Proportion Characteristics:

  • Population proportion parameter: $\pi$

  • Variance of a proportion: $\sigma^2 = \pi(1 - \pi)$

  • Standard deviation of a proportion: $\sigma = \sqrt{\pi(1 - \pi)}$

  • Sample proportion estimator: $p = \frac{X}{n}$

  • Standard Error of the sample proportion:         $\sigma_{\bar{p}} = \sqrt{\frac{p(1 - p)}{n}}$

• Sampling Distribution Normality Condition:

  • The sample proportion distribution is approximately normal if sample size $n$ satisfies:         $n\pi > 5 \quad \text{and} \quad n(1 - \pi) > 5$         (or $np > 5$ and $n(1 - p) > 5$ in sample applications).

• Confidence Interval Formula for Proportions:     $p \pm Z_{\alpha/2} \sqrt{\frac{p(1 - p)}{n}}$

• Proportion Probability Example:

  • Given: Voter support $\pi = 0.40$, sample size $n = 200$

  • Find: $P(0.40 \le p \le 0.45)$

  • Standard Error Calculation:         $\sigma_p = \sqrt{\frac{0.40(1 - 0.40)}{200}} = \sqrt{\frac{0.24}{200}} = 0.03464$

  • $Z$-score for upper bound $0.45$:         $Z = \frac{0.45 - 0.40}{0.03464} = 1.44$

  • Cumulative Normal Area:         $P(0 \le Z \le 1.44) = 0.9251 - 0.5000 = 0.4251 \quad (42.51\%\text{ or } 43\%$

• Confidence Interval Worked Example: Left-Handedness:

  • Given: Random sample $n = 100$, $X = 25$ left-handed individuals ($p = 0.25$).

  • Checking sample conditions: $np = 25 > 5$ and $n(1-p) = 75 > 5$

  • $95\%$ Confidence Interval Calculation:         $0.25 \pm 1.96 \sqrt{\frac{(0.25)(0.75)}{100}} = 0.25 \pm 1.96 (0.0433) = 0.25 \pm 0.0848$

  • Interval endpoints: $0.1651 \le \pi \le 0.3349$ ($16.51\%$ to $33.49\%$).

• Conservative Sample Size Proportion Mechanics ($n = 30$ Reference):

  • When $\pi$ is unknown during survey planning, using $p = 0.50$ maximizes $p(1-p) = 0.25$, yielding the most conservative (largest required) sample size.

    | $p$ | $p(1-p)$ | $\sqrt{\frac{p(1-p)}{30}}$ |     | :--- | :--- | :--- |     | 0.01 | 0.0099 | 0.002 |     | 0.05 | 0.0475 | 0.009 |     | 0.10 | 0.0900 | 0.018 |     | 0.20 | 0.1600 | 0.037 |     | 0.50 | 0.2500 | 0.091 |

• Sample Size Determination Formula for Proportions:     $e = Z_{\alpha/2} \sqrt{\frac{\pi(1 - \pi)}{n}} \implies n = \frac{Z_{\alpha/2}^2 \pi(1 - \pi)}{e^2}$

  • Sample Size Calculation Example:

    • Given: Estimate population proportion defective within margin of error $e = \pm 0.03$ with $95\%$ confidence ($Z = 1.96$). A pilot sample indicates $p = 0.12$

    • Calculation:             $n = \frac{(1.96)^2 (0.12)(0.88)}{(0.03)^2} = \frac{(3.8416)(0.1056)}{0.0009} = \frac{0.405673}{0.0009} = 450.74$

    • Rounded up required sample size: $n = 451$

Practical Applications, Biases, and Empirical Examples

• WWII Aircraft Bullet Holes (Survivorship Bias):

  • Analyzing bullet hole distributions per square meter across returning combat aircraft:

    | Section of Plane | Average Holes / sq meter |     | :--- | :--- |     | Engine | 1.11 |     | Fuselage | 1.73 |     | Fuel System | 1.55 |     | Rest of Plane | 1.80 |

*   *Survivorship Bias Analysis*: Armor should be added to areas with *fewer* damage marks on returning planes (Engine and Fuel System). Planes hit in those critical sections crashed and were unavailable for analysis; observed data reflected surviving aircraft only.

• Mutual Funds Analysis and Survivorship Bias:

  • Morningstar Large Blend Category (1995–2004): $178.4\%$ Growth ($10.8\%$ annual return) - biased due to measuring only surviving funds.

  • Savant Study (1995–2004, including failed/merged funds): $134.5\%$ Growth ($8.9\%$ annual return).

  • Benchmark S&P 500 Index (1995–2004): $212.5\%$ Growth.

  • 2011 Review of Finance study evaluating nearly 5,000 mutual funds demonstrated that surviving funds ($2,641$) overstated average returns by $20\%$ relative to the complete population of funds.

• The Flaw of Averages: Film Portfolio Analysis:

  • Single Film Return Dataset ($28$ Films): Probability of loss = $25\%$, Mean return = $\$20.23\,\text{Million}$, Standard Deviation = $\$36.98\,\text{Million}$

  • Bootstrapped Data ($5,000$ Runs): Mean return = $\$20.47\,\text{Million}$, Standard Error = $\$6.93\,\text{Million}$

  • Effect of Diversification Across Film Portfolios (Simulating 1,000 runs):

    | Portfolio Size | Mean Return | Standard Deviation | Probability of Loss (Mean < 0) |     | :--- | :--- | :--- | :--- |     | 2 Films | $20.02 Million | 25 | 21.4% |     | 4 Films | $21.13 Million | 18 | 8.5% |     | 8 Films | $20.53 Million | 13 | 1.3% |     | 12 Films | $19.66 Million | 10 | 0.5% |

• Bureau of Labor Statistics (BLS) Employment Situation Summary (December 2025):

  • Reported Statistics: Nonfarm payroll employment changed by $+50,000$; Unemployment rate stood at $4.4\%$.

  • Confidence Interval Construction: BLS conducts analyses at a $90\%$ confidence level ($1.6$ standard errors).

  • Nonfarm Establishment Survey Change: $90\%$ Confidence Interval margin of error is $\pm 130,000$.

    • For the $+50,000$ job point estimate, the $90\%$ confidence interval spans $-80,000$ to $+180,000$.

    • Because the interval includes zero, BLS cannot confirm with confidence that employment actually increased.

    • A reported increase of $+250,000$ yields an interval ($+120,000$ to $+380,000$) strictly greater than zero, statistical confirmation of net job growth.

  • Household Survey Statistics: At a $6.0\%$ unemployment rate, the $90\%$ confidence interval for monthly change in unemployment level is $\pm 300,000$, and for the unemployment rate is $\pm 0.2$ percentage points.

• Uncertainty Reduction Curves (How to Measure Anything):

  • Relative interval width shrinks exponentially as sample size increases from $n = 2$ to $n = 30$, with diminishing marginal uncertainty reduction beyond $n = 30$ for both Standard Normal and Student-t distributions.

Methodological, Reporting, and Ethical Standards

• Non-Sampling Errors:

  • Non-sampling errors stem from operational/systemic factors rather than standard sample variation.

  • Categories include:

    • Coverage Errors: Failure to sample a representative segment of the population.

    • Non-Response Errors: Inability to collect data from specific respondents.

    • Interview/Measurement Errors: Untruthful or inaccurate responses from survey subjects.

    • Processing Errors: Mistakes made during data collection, entry, or computational synthesis.

• Ethical Reporting Requirements:

  1. Point estimates must always be reported accompanied by confidence interval estimates or standard margins of error.

  2. The exact level of confidence (e.g., $90\%$, $95\%$, 99\%$) must be explicitly stated.\n 3. The sample size (n) must be reported.\n 4. A clear contextual interpretation of the confidence interval must accompany all technical output.\n\n* **Summary of Margin of Error Determinants**:\n * Sampling error / margin of error decreases when:\n * Sample size (n) increases.\n * The estimated percentage approaches extreme bounds (0\%$or$100\%$$).

     • The required level of confidence decreases.