Introduction to Machine Learning and Statistical Learning

Historical and conceptual overview of machine learning and statistical learning, including definitions for regression, classification, bias-variance tradeoff, and data types.

Drag Force Equation

The formula used to compute drag as a function of velocity: $F_D = \frac{1}{2} \rho A C_D v^2$, where $\rho$ is air density, $A$ is frontal area, $C_D$ is the drag coefficient, and $v$ is velocity.

Statistical Learning Problem Equation

The relationship defined as Y = f(X) + \text{\epsilon}, where $f$ is the relationship between target $Y$ and input $X$, and \text{\epsilon} is independent error noise.

Target Variable

Also known as the response, dependent variable, or $Y$; it is what the model wishes to predict.

Predictors

Also known as inputs, features, independent variables, or $X$; these are used to predict the target.

Categorical Data

A type of data limited to a fixed set of qualitative values, such as True vs False or specific labels like a favorite Formula 1 team.

Ordinal Data

A form of data consisting of a scale of increasing values.

n

In tabular data representation, this represents the number of observations, examples, or instances.

p

In tabular data representation, this represents the number of inputs, predictors, or features.

Inference

One of the main uses of $\hat{f}(X)$, focused on understanding how the features of $X$ affect the target $Y$.

Loss Function

A function that quantifies the prediction error of a model and depends on the specific problem being solved.

Mean Squared Error (MSE)

The most common loss function for regression problems, defined for all observations as: \mathcal{L}_{MSE} = \frac{1}{n} \text{\sum}_{i=1}^{n} (y_i - \text{\hat{f}}(x_i))^2.

Regression Function

The ideal relationship defined as the average value of $Y$ for datapoints with $X = x$, expressed as: $f(x) = E(Y | X = x)$.

Nearest Neighbor Approach

A method that relaxes the regression function definition by averaging $Y$ values in a neighborhood $\mathcal{N}(x)$ of $x$, expressed as $\hat{f}(x) = \text{Ave} \{ Y | X \in \mathcal{N}(x) \}$.

Curse of Dimensionality

A phenomenon where nearest neighbor methods perform poorly when $p$ is large because neighbors tend to be far away in high dimensions.

George Box Quote

"Essentially, all models are wrong, but some are useful."

Parametric Models

Models that assume $f$ follows a fixed functional form with parameters, such as linear regression: $Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \text{...} + \beta_p X_p$.

Non-parametric Models

Models that do not make assumptions about the functional form of $f$, deriving predictions directly from the data (e.g., nearest-neighbors).

Generalization

The ability of a machine learning model to predict accurately on unseen data.

Data Splitting Ratio

The typical division of data into $70\text{-}80\%$ for training sets and $20\text{-}30\%$ for test sets.

Bias-Variance Tradeoff Decomposition

The mathematical breakdown of expected prediction error: E(y_0 - \text{\hat{f}}(x_0))^2 = \text{Var}(\text{\hat{f}}(x_0)) + \text{Bias}(\text{\hat{f}}(x_0))^2 + \text{Var}(\text{\epsilon}).

Variance (Bias-Variance Tradeoff)

Reflects how much the estimated model $\hat{f}$ changes depending on the training data used.

Bias (Bias-Variance Tradeoff)

Reflects the difference between the estimated model $\hat{f}$ and the true relationship $f$.

Irreducible Error

The error represented by $\text{Var}(\epsilon)$ which cannot be estimated or removed even if the model is perfect.

Binary Classification

The simplest classification problem where the target $Y$ has exactly $K = 2$ classes.

Misclassification Rate

The most common error metric for classification, defined as: \text{Ave}(I(y_0 \neq \text{\hat{y}}_0)), where $I$ is an indicator variable.

Unsupervised Learning

A type of learning where there is no outcome variable $Y$, and the objective is to find groups or combinations of features involving the predictors.