Data C102 Midterm 2 Study Guide: Parametric Modeling, Inference, Sampling, and GLMs

Comprehensive vocabulary flashcards covering the core concepts, philosophies, estimators, algorithms, and models from Lectures 8 through 17 of Data C102.

Frequentist Philosophy (Concept of Randomness)

A statistical philosophy assuming the parameter $\theta$ is an unknown fixed constant and observed data $X$ is a random sample from likelihood $p(X|\theta)$, where probability represents long-run relative frequency across repeated trials.

Bayesian Philosophy (Concept of Randomness)

A statistical philosophy assuming both parameter $\Theta$ and data $X$ are jointly random variables drawn from $p(\theta, x) = p(x|\theta)p(\theta)$, where probability quantifies epistemic belief or uncertainty conditional on available evidence.

Prior Distribution ($p(\theta)$)

A probability distribution that mathematically specifies prior beliefs or uncertainty regarding parameter $\Theta$ before observing the dataset $X$.

Likelihood Function ($p(x|\theta)$)

A function that computes the probability density or probability mass of observing dataset $x$ given a specific fixed parameter value $\theta$.

Posterior Distribution ($p(\theta|x)$)

The conditional probability distribution over parameter $\Theta$ after conditioning on observed sample data $X = x$, calculated via Bayes' Rule as $p(\theta|x) = \frac{p(x|\theta)p(\theta)}{p(x)}$.

Bayesian Evidence (Marginal Likelihood)

The total marginal probability of observing dataset $x$ integrated over all parameter values, defined as $p(x) = \int p(x|\theta)p(\theta)\,d\theta$, serving as the normalizing constant in Bayes' rule.

Conjugate Prior

A family of prior distributions $\mathcal{P}$ where, for a given likelihood model, the resulting posterior $p(\theta|x)$ belongs to the exact same parametric family $\mathcal{P}$ for all possible data outcomes, bypassing numerical integration.

Exponential Family (Canonical Density Form)

A distribution family defined by $p(x|\eta) = h(x)\exp\left(\eta^T T(x) - A(\eta)\right)$, where $\eta$ is the canonical parameter vector, $T(x)$ is the sufficient statistic vector, $A(\eta)$ is the log-partition function, and $h(x)$ is the base measure.

Log-Partition Function ($A(\eta)$) Properties

The normalizing cumulant function in an exponential family whose derivatives generate the moments of the sufficient statistics: first derivative $A'(\eta) = \mathbb{E}[T(X)]$ and second derivative $A''(\eta) = \text{Var}(T(X))$.

Point Estimator vs. Interval Estimator

A point estimator $\hat{\theta}(X)$ maps an observed sample to a single coordinate in parameter space, whereas an interval estimator maps data to a set or range $[L(X), U(X)]$ expressing parameter uncertainty.

Maximum Likelihood Estimator (MLE) and Minimized Risk

The estimator defined by $\hat{\theta}_{\text{MLE}}(x) = \arg\max_\theta \log(p(x|\theta))$, which selects the parameter maximizing data likelihood and asymptotically minimizes Frequentist risk under 0-1 loss.

Maximum A Posteriori (MAP) Estimator and Minimized Risk

The estimator defined by $\hat{\theta}_{\text{MAP}}(x) = \arg\max_\theta [\log(p(x|\theta)) + \log(p(\theta))]$, which yields the posterior mode and minimizes Bayes risk under 0-1 loss $L(\theta, \hat{\theta}) = I(\theta \ne \hat{\theta})$.

Posterior Mean Estimator and Minimized Risk

The Bayesian estimator defined by $\hat{\theta}_{\text{mean}}(x) = \mathbb{E}[\Theta|X = x] = \int \theta p(\theta|x)\,d\theta$, which minimizes Bayes risk under squared error loss ($L_2$ loss) $L(\theta, \hat{\theta}) = (\theta - \hat{\theta})^2$.

Posterior Median Estimator and Minimized Risk

The Bayesian estimator defined by $P(\Theta \le \hat{\theta}_{\text{median}}|X = x) = 0.5$, which minimizes Bayes risk under absolute error loss ($L_1$ loss) $L(\theta, \hat{\theta}) = |\theta - \hat{\theta}|$.

Conditional Frequentist Bias of the Posterior Mean

In the Normal-Normal model, the estimator conditional bias is $\mathbb{E}_{X|\theta}[\hat{\theta}_{\text{mean}} - \theta] = \lambda(\mu_\theta - \theta)$, showing systematic shrinkage toward prior mean $\mu_\theta$ with shrinkage factor $\lambda$.

Joint Bayesian Bias of the Posterior Mean

When evaluated across the joint distribution of parameters and data, the expected error is $\mathbb{E}_{\Theta, X}[\hat{\theta}_{\text{mean}} - \Theta] = 0$, proving the posterior mean is strictly unbiased in the Bayesian framework.

Bayes Risk Trade-Off of Posterior Mean vs. MLE

The posterior mean trades off conditional bias to drastically reduce estimation variance, achieving strictly lower overall Bayes risk than the MLE ($R_B(\hat{\theta}_{\text{mean}}) < R_B(\hat{\theta}_{\text{MLE}})$) for any finite prior variance.

Empirical Bayes (EB) Optimization Target

A hybrid framework where prior hyperparameters $\beta = (a, b)$ are learned directly from observed data by maximizing marginal likelihood: $\hat{\beta}_{\text{EB}} = \arg\max_\beta \int p(X|\theta)p(\theta|\beta)\,d\theta$.

Empirical Bayes Kidney Cancer Application

Low-population counties display extreme false-positive death rates under naive MLE; Empirical Bayes fits a shared prior across all counties, shrinking high-variance small county estimates toward national averages while preserving large county data.

Prior Predictive Distribution Formula

The marginal distribution of unobserved future data before observing any current training dataset, defined as $p(x_{\text{new}}) = \int p(x_{\text{new}}|\theta)p(\theta)\,d\theta$.

Posterior Predictive Distribution (PPD) Formula

The distribution of future unobserved data conditioned on an observed training set, defined as $p(x_{\text{new}}|x_{\text{obs}}) = \int p(x_{\text{new}}|\theta)p(\theta|x_{\text{obs}})\,d\theta$.

Two-Step Simulation from the PPD

To sample future observations $x_{\text{new}} \sim \text{PPD}$, first sample parameter values from the posterior $\theta^{(s)} \sim p(\theta|x_{\text{obs}})$, then sample replicated data from the likelihood $x_{\text{new}}^{(s)} \sim p(x|\theta^{(s)})$.

Prior Predictive Check vs. Posterior Predictive Check

A prior predictive check simulates data from the prior predictive to see if modeling assumptions generate domain-plausible values before fitting; a PPC simulates synthetic datasets from the PPD to check if the fitted model recovers observed sample features.

Posterior Predictive Check (PPC) Procedure

Draw $S$ posterior samples $\theta^{(s)} \sim p(\theta|X_{\text{obs}})$, generate synthetic datasets $X_{\text{sim}}^{(s)} \sim p(x|\theta^{(s)})$, compute test statistic values $T_{\text{sim}}^{(s)} = T(X_{\text{sim}}^{(s)})$, and compare to observed test statistic $T_{\text{obs}} = T(X_{\text{obs}})$.

Posterior Predictive p-Value Interpretation

Calculated as $p_{\text{PPC}} = \frac{1}{S}\sum_{s=1}^S I(T_{\text{sim}}^{(s)} \ge T_{\text{obs}})$, where values near $0.5$ indicate good fit, while extreme values ($< 0.025$ or $> 0.975$) indicate severe model misspecification.

Conditional Independence in Directed Graphs

The factorization condition $p(A, B|C) = p(A|C)p(B|C)$, which holds if all directional paths between $A$ and $B$ are blocked by conditioning set $C$ via d-separation rules.

Plate Notation Conventions

A graphical representation where unshaded circles represent unobserved/latent random variables, shaded circles represent observed data, solid dots denote fixed hyperparameters, and rectangular boxes (plates) denote conditionally independent replications.

Joint Factorization Rule for Directed Graphical Models

The theorem stating the joint density of all graph nodes decomposes into the product of each variable conditioned on its direct parent nodes: $p(v_1, \dots, v_K) = \prod_{k=1}^K p(v_k|\text{parents}(v_k))$.

Ancestral Sampling (Graphical Models)

A forward generative process that simulates samples by following the directed acyclic graph in topological order, sampling parent nodes from their marginals before sampling child nodes conditional on parent draws.

Complete, No, and Partial Pooling

Complete pooling forces a single global parameter onto all groups (high bias); no pooling estimates independent parameters per group (high variance for small groups); partial pooling places a shared hierarchical prior over group parameters (optimal balance).

Monte Carlo Integration

A computational technique approximating intractable continuous expectations $\mathbb{E}_p[f(X)] = \int f(x)p(x)\,dx$ with sample mean $\frac{1}{S}\sum_{s=1}^S f(x^{(s)})$ using independent draws $x^{(s)} \sim p(x)$.

Inverse CDF Sampling Algorithm

An exact sampling method that draws standard uniform variates $u \sim \text{Uniform}(0, 1)$ and applies the target's continuous inverse cumulative distribution function to yield $x = F^{-1}(u) \sim F$.

Rejection Sampling Envelope Condition

The strict criterion requiring a proposal distribution $q(x)$ and finite constant $M$ such that $M q(x) \ge p(x)$ (or $M q(x) \ge \tilde{p}(x)$ for unnormalized target $\tilde{p}(x)$) across all points in the support.

Rejection Sampling for Unnormalized Targets

Iteratively generate candidate $Y \sim q(x)$ and draw $u \sim \text{Uniform}(0, 1)$; accept candidate $Y$ if $u \le \frac{\tilde{p}(Y)}{M q(Y)}$, repeating until $S$ accepted draws are collected.

Stationary Distribution of a Markov Chain

A probability distribution $\pi$ satisfying $\pi(x) = \sum_y \pi(y) T(y \rightarrow x)$ (or $\pi^T T = \pi^T$) that is preserved asymptotically under the transition kernel $T$.

Markov Chain Mixing Time and Burn-In

Mixing time is the number of transition steps necessary for the chain to approach its target stationary distribution; burn-in denotes the initial non-stationary iterations discarded to prevent initial-state bias.

Detailed Balance Condition

The reversibility condition $\pi(x) T(x \rightarrow y) = \pi(y) T(y \rightarrow x)$ for all pairs $x, y$, which provides a sufficient guarantee that $\pi$ is the unique stationary distribution of kernel $T$.

Metropolis Acceptance Probability (Symmetric Proposal)

For symmetric proposals where $q(x \rightarrow y) = q(y \rightarrow x)$, candidate $y$ is accepted from state $x$ with probability $\alpha(x, y) = \min\left(1, \frac{\tilde{p}(y)}{\tilde{p}(x)}\right)$.

Metropolis-Hastings Acceptance Probability (Asymmetric)

For asymmetric proposals, candidate $y$ is accepted with probability $\alpha(x, y) = \min\left(1, \frac{\tilde{p}(y)q(x|y)}{\tilde{p}(x)q(y|x)}\right)$, correcting proposal asymmetry to satisfy detailed balance with unnormalized target $\tilde{p}$.

Gibbs Sampling Algorithm

An MCMC algorithm that samples a vector $\theta = (\theta_1, \dots, \theta_D)$ by iteratively updating each scalar dimension from its full conditional distribution $\theta_d \sim p(\theta_d|\theta_{-d}, X)$ with acceptance probability $1$.

Prediction Task vs. Estimation Task

Estimation infers unobservable latent parameters $\theta$ and evaluates quality through parameter loss, whereas prediction forecasts new observable outcomes $Y_{\text{new}}$ and evaluates quality via predictive loss on test data.

Bayesian Point Prediction under Squared Error Loss

Under loss $L(y, \hat{y}) = (y - \hat{y})^2$, the optimal Bayesian point prediction is the posterior predictive mean $\hat{y}^* = \mathbb{E}[Y_{\text{new}}|x_{\text{new}}, D]$.

Bayesian Point Prediction under Absolute Error Loss

Under loss $L(y, \hat{y}) = |y - \hat{y}|$, the optimal Bayesian point prediction is the posterior predictive median $\hat{y}^* = \text{Median}(Y_{\text{new}}|x_{\text{new}}, D)$.

Bayesian Point Prediction under Asymmetric Weighted Loss

When penalizing overestimation and underestimation at different linear rates, the loss-minimizing prediction is a specified percentile or quantile of the posterior predictive distribution.

Six Core Components of a Generalized Linear Model

The six components are feature vector $x$, regression weights $\beta$, linear predictor $\eta = \beta^T x$, link function $g(\mu)$ with inverse $g^{-1}(\eta)$, prediction parameter $\mu$, and exponential family likelihood $Y \sim \text{Distribution}(\mu, \theta)$.

Linearity in Parameters Requirement (GLMs)

A GLM requires that the linear predictor must be strictly linear in the coefficients $\beta$ ($\eta = \beta^T \phi(x)$), which fully permits non-linear transformations of input features $\phi(x)$ while disallowing non-linear parameter forms.

Logit Link Function in Logistic Regression

The link function mapping success probability $p \in (0, 1)$ to the unconstrained real line via log-odds: $g(p) = \text{logit}(p) = \log\left(\frac{p}{1 - p}\right) = \beta^T x$.

Logistic Inverse Link Function

The inverse link function mapping linear predictor $\eta \in \mathbb{R}$ back onto the valid probability interval $(0, 1)$: $g^{-1}(\eta) = \text{logistic}(\eta) = \frac{1}{1 + \exp(-\eta)}$.

Softmax Inverse Link (Multi-Class Logistic Regression)

The inverse link function mapping class linear predictor vectors to valid categorical probability distributions across $K$ classes: $\pi_k(x) = \frac{\exp(\beta_k^T x)}{\sum_{j=1}^K \exp(\beta_j^T x)}$.