Exhaustive Study Notes: Inferential Statistics, Sampling Distributions, Central Limit Theorem, and the t-Distribution

Core Principles of Statistical Inference

• Definition and Purpose of Statistical Inference:

  • Inferential statistics involves drawing a sample from a broader population, taking a measurement (statistic) on that sample, and using that measurement to guide conclusions about the unknown population parameter without directly measuring the entire population.
  • Direct measurement of an entire population is known as a census, which is generally unfeasible or impossible due to practical constraints.
  • A survey captures data from a subset of the population to infer details about the parameter.
  • The inferential process operates as a logical flow:     $\text{Population (Parameter } \mu \text{ or } p) \longrightarrow \text{Sample} \longrightarrow \text{Statistic (} \bar{x} \text{ or } \hat{p} \text{)} \longrightarrow \text{Inference back to Population}$

• Data Collection and Types of Bias:

  • Bias: Any systematic distortion in sampling or measurement that produces inaccurate estimates.
  • Sampling Bias / Selection Bias:
  • Occurs when the sampling method systematically favors certain outcomes or excludes portions of the population.
  • Voluntary response surveys introduce significant selection bias because individuals with strong feelings self-select into participation.
  • Measurement Bias:
  • Occurs when the measuring tool or instrument systematically understates or overstates true values (such as an uncalibrated atmospheric sensor).
  • Response Bias:
  • Occurs when participants provide inaccurate responses or self-select into answering specific survey questions.

• Accuracy vs. Precision:

  • Accuracy (Unbiasedness):
  • Indicates that sample statistics are centered around the true parameter value.
  • Achieved through Simple Random Sampling (SRS), which mathematically guarantees that measurements hover around the true parameter value, even when the exact parameter value is unknown.
  • Precision (Variability):
  • Indicates how close repeated sample measurements are to one another.
  • Controlled directly by sample size ($n$). Larger sample sizes reduce variability among sample statistics.
  • Ideal Sampling Design:
  • Combines Simple Random Sampling (ensuring accuracy) with a large sample size $n$ (ensuring high precision/low standard error).
  • A large sample size without Simple Random Sampling produces precise but systematically inaccurate (biased) results.

Measures of Center and Variability: Means vs. Proportions

• Categorizing Quantitative vs. Categorical Measures:

  • Every statistical parameter falls into one of two primary categories depending on the nature of the variable: a mean or a proportion.

• Population and Sample Means:

  • Applied when analyzing continuous quantitative metrics or monetary amounts (e.g., spending per customer).
  • Population Mean ($\mu$): The true average across all $N$ items in the population.
  • Sample Mean ($\bar{x}$): The calculated average from $n$ sampled observations, serving as the point estimate for $\mu$.
  • Population Standard Deviation ($\sigma$): The true variability among individual units in the population.
  • Sample Standard Deviation ($s$): The calculated variability from $n$ sampled observations, serving as the point estimate for $\sigma$.

• Population and Sample Proportions:

  • Applied when measuring categorical outcomes or counting specific attributes within a group (e.g., proportion of satisfied customers, percentage of left-handed students).
  • Counts alone are insufficient without context; expressing the count relative to the total group size yields a proportion or percentage.
  • Population Proportion ($p$): The true proportion of successes across the entire population.
  • Sample Proportion ($\hat{p}$ or $\bar{p}$): The observed proportion of successes in a sample of size $n$, serving as the point estimate for $p$.
  • Conceptual treatment for proportions matches that for means, though the operational formulas differ slightly.

Sampling Variability, Point Estimates, and Plausible Ranges

• Statistics as Random Variables:

  • Every sample statistic (such as $\bar{x}$ or $\hat{p}$) is a random variable because its value depends on the specific observations drawn into the sample.
  • Sampling Variability: The natural, expected variation among statistics calculated from different samples drawn from the same population. It represents natural variation rather than experimental error.

• Empirical Example: Customer Spending:

  • In a cafe population of $N = 4000$ customers:
  • Sample 1 ($n = 100$): yields a point estimate of $\bar{x}_1 = \$9.40$.
  • Sample 2 ($n = 100$): yields a point estimate of $\bar{x}_2 = \$9.18$.
  • Sample 3 ($n = 100$): yields a point estimate of $\bar{x}_3 = \$9.63$.
  • All calculated sample means are correct; differences arise solely from sampling variability across distinct groups of $100$ customers.

• Point Estimates vs. Confidence Intervals (Ranges of Plausible Values):

  • Point Estimate:
  • A single summary number derived from a sample (e.g., $\bar{x} = \$9.40$) offering the best guess for the parameter.
  • Fails to report uncertainty, sampling variability, or degree of precision.
  • Range of Plausible Values (Confidence Interval):
  • Bounded by lower and upper limits (e.g., $\$9.01$ to $\$9.79$).
  • Midpoint Property: The point estimate sits at the center of the range:       $\text{Midpoint} = \frac{\$9.01 + \$9.79}{2} = \$9.40$
  • Margin of Error: The distance from the point estimate to either limit:       $\text{Margin of Error} = \$9.40 - \$9.01 = \$9.79 - \$9.40 = \$0.39$
  • Confidence Interpretation: Expresses the level of certainty that the true parameter $\mu$ lies within the interval bounds.
  • Width vs. Confidence Trade-off:
    • Seeking higher confidence requires a wider interval range.
    • Accepting lower confidence yields a narrower interval range.

Population, Sample, and Sampling Distributions

• Distinguishing the Three Types of Distributions:

  • Population Distribution:
  • The distribution of individual values across all members of the population.
  • Example: Individual spending values of all $N = 4000$ customers, characterized by population mean $\mu$ and standard deviation $\sigma$
  • Sample Distribution:
  • The distribution of individual values within a single sample of size $n$
  • Example: Spending values of the $n = 100$ customers in Sample 1, characterized by sample mean $\bar{x}$ and standard deviation $s$
  • Sampling Distribution:
  • The probability distribution of a sample statistic (e.g., all possible values of $\bar{x}$) calculated from all possible simple random samples of size $n$ drawn from the population.
  • It represents a distribution of statistics, not individual data points.

• Combinatorial Scale of Sampling Distributions:

  • The number of distinct simple random samples of size $n$ drawn from a population of size $N$ is exceptionally large.
  • Example comparison: In a lottery game choosing $6$ numbers from $49$, there are over $10,000,000,000$ ($10^{10}$) distinct combinations.
  • Selecting $n = 100$ individuals from a population of $N = 4000$ produces trillions of distinct sample combinations.

• Process of Constructing a Sampling Distribution (Height Example):

  • Consider a population of $N = 10000$ people with true mean height $\mu = 5\text{ ft } 4\text{ in}$ ($5.33\text{ ft}$ or $5.4\text{ ft}$).
  • Step 1: Draw an SRS of size $n = 5$
  • Step 2: Measure heights and compute the sample mean $\bar{x}_1$ (e.g., $5.3\text{ ft}$).
  • Step 3: Plot $\bar{x}_1$ on a frequency histogram.
  • Step 4: Draw a second SRS of size $n = 5$, compute $\bar{x}_2$ (e.g., $5.7\text{ ft}$), and plot it.
  • Step 5: Repeat this process thousands of times.
  • Outcome: The histogram of stacked sample means ($\bar{x}$ values) forms the sampling distribution curve.

The Sampling Distribution of the Sample Mean

• Center of the Sampling Distribution:

  • Under simple random sampling, the expected value of the sampling distribution of $\bar{x}$ equals the true population mean $\mu$:     $\mu_{\bar{x}} = E(\bar{x}) = \mu$
  • This shows that the sample mean $\bar{x}$ is an unbiased estimator of $\mu$.

• Spread and Standard Error ($\sigma_{\bar{x}}$):

  • The standard deviation of a sampling distribution is named the Standard Error ($SE$) to distinguish it from individual observation standard deviations.
  • Mathematical Formula:     $\sigma_{\bar{x}} = \frac{\sigma}{\sqrt{n}}$
  • Effect of Population Variability ($\sigma$): Larger population standard deviation increases standard error.
  • Effect of Sample Size ($n$): Increasing sample size $n$ increases the denominator $\sqrt{n}$, decreasing the standard error $\sigma_{\bar{x}}$.
  • Diminishing Returns Rule: Cutting standard error in half requires quadrupling the sample size ($4n$).

• Averaging Effect on Variability:

  • Taking averages reduces variability because individual high values offset low values within a sample.
  • In customer spending where population standard deviation is $\sigma = \$4.00$:
  • Individual transaction values deviate from the mean by an average of $\$4.00$
  • For samples of size $n = 100$, standard error drops to:       \sigma_{\bar{x}} = \frac{\4.00}{\sqrt{100}} = \frac{\4.00}{10} = \$0.40
  • Standard error progression across sample sizes ($n$) when $\sigma = \$4.00$:
  • n = 16 \implies \sigma_{\bar{x}} = \frac{\4.00}{\sqrt{16}} = \frac{\4.00}{4} = \$1.00
  • n = 25 \implies \sigma_{\bar{x}} = \frac{\4.00}{\sqrt{25}} = \frac{\4.00}{5} = \$0.80
  • n = 64 \implies \sigma_{\bar{x}} = \frac{\4.00}{\sqrt{64}} = \frac{\4.00}{8} = \$0.50
  • n = 100 \implies \sigma_{\bar{x}} = \frac{\4.00}{\sqrt{100}} = \frac{\4.00}{10} = \$0.40
  • n = 400 \implies \sigma_{\bar{x}} = \frac{\4.00}{\sqrt{400}} = \frac{\4.00}{20} = \$0.20

The Central Limit Theorem (CLT)

• Formal Statement of the Central Limit Theorem:

  • Regardless of the underlying population distribution shape (skewed right, skewed left, uniform, exponential), the sampling distribution of the sample mean $\bar{x}$ approaches a normal distribution as sample size $n$ becomes sufficiently large:     $\bar{x} \sim N\left(\mu, \frac{\sigma}{\sqrt{n}}\right)$

• Critical Misconceptions Clarified:

  • The CLT does not state that the original population distribution becomes normal.
  • The CLT does not state that individual data points in a single sample become normally distributed.
  • The CLT does not correct for sampling bias or non-random sampling methods.

• Required Conditions for Applying the CLT for Means:

  1. Random Sampling and Independence: Observations must be gathered via Simple Random Sampling.
  2. 10% Condition: Sample size $n$ must be no larger than $10\%$ of population size ($n \le 0.10 N$) when sampling without replacement.
  3. Sample Size Requirements ($n$):
  • Normal Population: If the population is normally distributed, the sampling distribution of $\bar{x}$ is normal for any sample size $n$ (e.g., $n = 2, 5, 12$).
  • Unknown or Moderately Skewed Population: Standard rule of thumb requires $n \ge 30$
  • Strongly Skewed Population or Extreme Outliers: Requires $n \ge 50$

The Sampling Distribution of Sample Proportions

• Defining Categorical Metrics and Point Estimates:

  • When evaluating binary categorical variables (e.g., customer satisfaction: yes/no), data analysis relies on proportions.
  • Example: In a sample of $n = 100$ customers drawn from $N = 4000$, $82$ express satisfaction.
  • Point estimate for population proportion $p$:     $\hat{p} = \frac{82}{100} = 0.82 \quad (82\%)$
  • Range representation: An observed sample proportion of $\hat{p} = 0.82$ with a margin of error of $\pm 7.5\%$ yields a range of $74.5\%$ to $89.5\%$ ($0.745$ to $0.895$).

• Properties of the Sampling Distribution of $\hat{p}$:

  • Center: Expected value of sample proportions under Simple Random Sampling equals the true population proportion $p$:     $E(\hat{p}) = p$
  • Standard Error of Proportion:     $\sigma_{\hat{p}} = SE(\hat{p}) = \sqrt{\frac{p(1-p)}{n}}$
  • Sample Size Effect: Standard error decreases as sample size $n$ increases following an inverse square-root relationship.

• Success-Failure Condition for Proportions:

  • The sampling distribution of $\hat{p}$ is approximately normal if the sample size is large enough to expect at least $10$ successes and $10$ failures:     $n p \ge 10 \quad \text{and} \quad n(1-p) \ge 10$
  • When conditions are met, the sampling model is expressed as:     $\hat{p} \sim N\left(p, \sqrt{\frac{p(1-p)}{n}}\right)$

Practical Application: z-Distribution vs. t-Distribution

• Theoretical Formulas vs. Practical Substitutions:

  • Population parameters $\mu$, $\sigma$, and $p$ are typically unknown in practical scenarios.
  • Consequently, sample statistics must be substituted into theoretical standard error equations.

• Proportions: Single Guess and the $z$ Statistic:

  • Parameter $p$ is unknown in standard error formula $\sqrt{\frac{p(1-p)}{n}}$.
  • Practical substitution replaces $p$ with sample statistic $\hat{p}$:     $SE(\hat{p}) = \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$
  • Because only one parameter substitution is made ($\hat{p}$ for $p$), the Standard Normal Distribution ($z$-distribution) remains valid for probability calculations and confidence intervals.

• Means: Double Guess and the Need for the $t$ Statistic:

  • Parameters $\mu$ and $\sigma$ are both unknown in theoretical model $N\left(\mu, \frac{\sigma}{\sqrt{n}}\right)$.
  • Practical substitution requires two concurrent parameter estimates:
  1. Substituting sample mean $\bar{x}$ for population mean $\mu$
  2. Substituting sample standard deviation $s$ for population standard deviation $\sigma$
  • Practical Standard Error formula for means:     $SE(\bar{x}) = \frac{s}{\sqrt{n}}$
  • Estimating $\sigma$ using sample standard deviation $s$ introduces additional uncertainty.
  • Because of this additional variability from making two guesses, the standard normal $z$-distribution fails to provide accurate probabilities for sample means. Inference for sample means requires the $t$ Statistic ($t$-distribution).

• Properties of the $t$-Distribution:

  • Symmetric and unimodal bell shape centered at $0$
  • Total area under the probability curve equals $1.0$
  • Possesses heavier (fatter) tails than the standard normal $z$-distribution, reflecting added variability from estimating $\sigma$ with $s$
  • Controlled by Degrees of Freedom ($df$):     $df = n - 1$
  • As sample size $n$ increases ($df \to \infty$), tail weight decreases and the $t$-distribution approaches the Standard Normal $z$-distribution.