Statistical Inference for Data Science

Statistical Inference and Representative Samples

• Statistical inference uses representative samples to learn about population parameters, such as the binomial probability ($p$) or normal distribution mean ($\mu$) and variance ($\sigma^2$).
• A representative sample must be randomly selected to ensure every individual has an equal chance of selection, improving the reliability of conclusions.
• Biased samples do not generalize to the population, leading to misleading conclusions.

Defining Random Samples and Statistics

• A collection $X_1, X_2, \dots, X_n$ is a random sample from a population with distribution $f(x)$ if the joint distribution is the product of marginals:

$f(x_1, x_2, \dots, x_n) = \prod_{i=1}^{n} f(x_i) = f(x_1)f(x_2) \dots f(x_n)$

• This implies that the random variables are independent.
• In the frequentist paradigm, population parameters ($p$, $\mu$, $\sigma^2$) are fixed but unknown constants.
• A statistic is a function of a random sample; because it depends on random data, the statistic itself is a random variable.

Fundamental Sample Statistics

• Sample Proportion ($\hat{P}$): Used for binary random variables to estimate the population proportion ($p$).

$\hat{P} = \frac{1}{n} \sum_{i=1}^{n} X_i$

• Sample Mean ($\bar{X}$): Used to estimate the population mean ($\mu$).

$\bar{X} = \frac{1}{n} \sum_{i=1}^{n} X_i$

• Sample Variance ($S^2$): Used to estimate the population variance ($\sigma^2$).

$S^2 = \frac{1}{n-1} \sum_{i=1}^{n} (X_i - \bar{X})^2$

Sampling Distributions and the Central Limit Theorem

• Sampling variation occurs because different samples from the same population yield different values for a statistic.
• The probability distribution of a statistic is called its sampling distribution.
• For any sample mean ($\bar{X}$), the expected value is $E(\bar{X}) = \mu$ and the variance is $Var(\bar{X}) = \frac{\sigma^2}{n}$.
• Normal Population: If the population is normal, the sampling distribution of the mean is exactly normal:

$\bar{X} \sim N(\mu, \frac{\sigma^2}{n})$

• Central Limit Theorem (CLT): For non-normal populations with large sample sizes ($n$), the distribution of the sample mean is approximately normal:

$\bar{X} \sim N(\mu, \frac{\sigma^2}{n})$

• Small sample sizes (e.g., $n = 2$ in highly skewed exponential distributions) are insufficient for the CLT to apply.

Parametric Statistical Models

• A parametric model assumes the population distribution $f_Y(y)$ has a known functional form depending on a vector of parameters $\theta = (\theta_1, \dots, \theta_p)$.
• Normal Model: Parameterized by mean ($\mu$) and standard deviation ($\sigma$).

$f_Y(y; \mu, \sigma) = \frac{1}{\sigma \sqrt{2\pi}} \exp(-\frac{(y-\mu)^2}{2\sigma^2})$

• Simple Linear Model: Parameterized by intercept ($\beta_0$) and slope ($\beta_1$).

$Y = \beta_0 + \beta_1 x + \epsilon, \quad \epsilon \sim N(0, \sigma^2)$

Point Estimation and Evaluative Properties

• A point estimator ($\hat{\Theta}$) is a function used to provide a single "best guess" for a parameter $\theta$.
• Bias: The difference between the expected value of the estimator and the true parameter.

$Bias(\hat{\Theta}) = E(\hat{\Theta}) - \theta$

• An estimator is unbiased if $E(\hat{\Theta}) = \theta$.
• Standard Error (SE): The standard deviation of the estimator, quantifying uncertainty. Smaller SE indicates higher precision.

$SE(\hat{\Theta}_j) = \sqrt{Var(\hat{\Theta}_j)}$

• Mean Squared Error (MSE): Measures the total error by combining squared bias (systematic error) and variance (random error).

$MSE(\hat{\Theta}) = Bias(\hat{\Theta})^2 + Var(\hat{\Theta})$

• While the sample mean typically has lower MSE for normal data, the sample median may outperform it in small samples from skewed distributions (e.g., Gamma Distribution) or data with outliers.