Introduction to Machine Learning (BUSN 5000E)

This set of vocabulary flashcards covers key concepts in Introduction to Machine Learning, including prediction metrics, overfitting, regularization methods like LASSO, and classification via logistic regression.

Mean-square prediction error (MSPE)

A measure used to evaluate prediction accuracy that captures the tradeoff between bias and variance, defined as $MSPE(\hat{e}_0) = E[(y^0 - \hat{y}^0)^2] = \text{var}(u^0) + \text{var}(\hat{y}^0) + [\text{bias}(\hat{y}^0)]^2$.

Overfitting

A problem occurring when a model is too complex relative to the sample size $N$, resulting in low bias but high variance, and coefficients that reflect noise more than the underlying relationship.

Adjusted R2

A goodness-of-fit measure that penalizes the inclusion of variables with little explanatory power, calculated as $1 - \frac{SSE / (N - K - 1)}{SST / (N - 1)}$.

Shrinkage estimators

Machine learning algorithms that insert a tradeoff between bias and variance directly into the OLS minimization problem through a penalty that taxes model complexity and shrinks coefficient estimates toward zero.

Supervised learning

A category of machine learning involving data with both an outcome and many predictors, used for regression (continuous outcomes) and classification (categorical outcomes) problems.

Unsupervised learning

A category of machine learning characterized by having no outcome variable, focusing instead on cluster analysis and pattern recognition.

LASSO

An acronym for Least Absolute Shrinkage and Selection Operator; a regularized regression form that adds a penalty to the OLS objective function: $SSE + \lambda \sum_{k=1}^K |\beta_k|$.

Tuning parameter ($\lambda$)

The parameter in LASSO that determines the strength of the penalty; a large value filters out both noise and signal, while a small value allows noise to overwhelm the signal.

Sparse models

Models where only a small share of the variables are expected to have non-zero coefficients; LASSO is particularly appealing for these because it can force some coefficients to be exactly zero.

Cross-validation (CV)

A process to find optimal model complexity by randomly dividing the training sample into $M$ folds, iteratively holding one out for validation while estimating on the others, and averaging the MSPEs.

Standardization

The required process of putting all covariates on the same scale before using shrinkage estimators, translating $x_{ik}$ into $\tilde{x}_{ik} = \frac{x_{ik} - \bar{x}_k}{\hat{\sigma}_k}$.

Deviance

Technically defined as $-2 \ln(L)$, where $L$ is the likelihood function; in the context of R's glmnet, percent deviance explained acts as a measure similar to $R^2$.

Linear Probability Model (LPM)

The use of OLS regression when the dependent variable is binary, which may result in predicted response probabilities that fall outside the unit interval (0,1).

Logistic regression (Logit)

A probability model preferred for classification problems where predictions are bounded by the (0,1) interval using the logistic cumulative distribution function $G(z) = \frac{\exp(z)}{1 + \exp(z)}$.

Maximum Likelihood Estimators (MLEs)

Values of $\beta_k$ computed numerically to maximize the log-likelihood function, used for estimating nonlinear models like logistic regression.

Average Partial Effects (APEs)

Also listed as Average Marginal Effects (AMEs); the interpreted impact of a predictor on the response probability in a nonlinear model, which is not equivalent to the estimated coefficient $\hat{\beta}_k$.