DATA 303 Quiz 1

G

Statistical Learning

Using data to estimate relationships between variables and make predictions or understand those relationships.

Statistical Learning Goal

Estimate an unknown relationship between predictors X and a response Y.

Response Variable (Y)

The outcome or target variable that we want to predict or understand.

Predictor (X)

A variable or feature used to predict or explain the response variable Y.

Feature

Another term for a predictor variable.

Training Data

Data used to fit or train a statistical learning model.

Test Data

Data not used to train the model and used to evaluate how well it performs on new observations.

f(X)

The true but unknown relationship between the predictors X and response Y.

f̂(X)

An estimate of the true function f(X) obtained from the data.

Predicted Value (Ŷ)

The value predicted by the estimated function f̂(X).

Y = f(X) + ε

The basic statistical learning model, where Y is the response, f(X) is the underlying relationship, and ε is random noise.

ε (Epsilon)

The random error or noise term in the statistical learning model.

E[ε] = 0

The assumption that the error term has an average value of zero.

Reducible Error

Error caused by an imperfect estimate of the true relationship f(X); it can potentially be reduced by improving the model.

Irreducible Error

Random noise in the data that cannot be eliminated, even with a perfect estimate of f(X).

Prediction

Using a model to predict the response Y for new observations.

Inference

Using a model to understand the relationship between predictors X and the response Y.

Prediction vs Inference

Prediction focuses on accurately predicting Y, while inference focuses on understanding how X and Y are related.

Regression

A statistical learning problem where the response variable Y is quantitative/numerical.

Classification

A statistical learning problem where the response variable Y is qualitative/categorical.

Quantitative Variable

A numerical response variable, such as income, height, blood pressure, or house price.

Qualitative Variable

A categorical response variable, such as yes/no, disease/no disease, or different disease types.

Supervised Learning

Learning where the training data contains both predictors X and known response values Y.

Unsupervised Learning

Learning where predictors X are available but there is no response variable Y or labels.

Semi

Supervised Learning

Clustering

An unsupervised learning method that groups observations based on similarities in their predictors.

Parametric Model

A model that assumes a specific functional form for f(X) and estimates a fixed number of parameters.

Nonparametric Model

A model that does not assume a specific functional form for f(X) and can adapt to more complicated relationships.

Parameter

A fixed numerical quantity in a statistical model that must be estimated from the data.

Hyperparameter

A value chosen to control how a learning method operates rather than being estimated as a model parameter.

Linear Regression

A parametric method that assumes the response is a linear function of the predictors.

Linear Regression Equation

f(X) = β₀ + β₁X₁ + β₂X₂ + ... + βₚXₚ.

β₀

The intercept parameter in a linear regression model.

β₁, β₂, ..., βₚ

Coefficients representing the relationship between each predictor and the response.

Interpretability

How easily the relationship learned by a model can be understood or explained.

Flexibility

The ability of a statistical learning method to fit complicated or nonlinear relationships.

Flexibility vs Interpretability

As model flexibility generally increases, interpretability generally decreases.

Simple Model

A relatively inflexible model that is easier to interpret but may fail to capture complicated relationships.

Flexible Model

A model capable of capturing complicated relationships but generally more difficult to interpret.

Neural Network

A highly flexible statistical learning method that can model complex nonlinear relationships but is often difficult to interpret.

Logistic Regression

A parametric classification method that estimates probabilities for categorical outcomes.

K

Nearest Neighbors (KNN)

K in KNN

The number of nearest observations used to make a KNN prediction.

KNN Classification

Assigns a new observation to the class represented by the majority of its K nearest neighbors.

KNN Regression

Uses nearby observations to estimate a numerical response.

Small K

Produces a more flexible KNN model with lower bias and higher variance.

Large K

Produces a less flexible KNN model with higher bias and lower variance.

K = 1

An extremely flexible KNN model that can have very low training error but high variance and overfitting.

KNN Flexibility

KNN becomes more flexible as K decreases.

Euclidean Distance

A common distance measure used to determine which observations are nearest in KNN.

Euclidean Distance Formula

d = √[(x₁

Bias

Error caused by an overly simplistic model or incorrect assumptions about the relationship between X and Y.

Variance

The amount by which a model's estimated function would change if it were trained on a different data set.

High Bias

Usually associated with models that are too simple and underfit the data.

Low Bias

Usually associated with more flexible models that can better capture complicated relationships.

High Variance

Usually associated with highly flexible models that are sensitive to the particular training data.

Low Variance

Usually associated with less flexible models whose predictions are more stable across different training sets.

Bias

Variance Tradeoff

Bias

Variance Formula

Bias²

The squared bias component of prediction error caused by an incorrect or overly simple model.

Expected Test MSE

The expected prediction error on new observations, consisting of variance, squared bias, and irreducible error.

Overfitting

When a model fits the training data extremely closely, including random noise, and therefore performs poorly on new data.

Underfitting

When a model is too simple to capture the important relationship in the data.

Training Error

The error made by a model when predicting observations that were used to train the model.

Test Error

The error made when predicting observations that were not used to train the model.

Training Error vs Test Error

Training error generally decreases as flexibility increases, while test error can decrease and then increase because of overfitting.

U

Shaped Test Error

Mean Squared Error (MSE)

The average squared difference between observed and predicted numerical responses.

MSE Formula

MSE = (1/n) Σ(yᵢ

Regression Error Measure

Mean squared error is commonly used to evaluate regression predictions.

Classification Error Rate

The proportion of observations that are incorrectly classified.

Classification Accuracy

The proportion of observations that are correctly classified.

Accuracy Formula

Accuracy = Number of Correct Predictions / Total Number of Predictions.

Error Rate Formula

Error Rate = Number of Incorrect Predictions / Total Number of Predictions.

Lower MSE

Indicates better regression prediction performance, assuming the same data and evaluation setting.

Lower Error Rate

Indicates better classification performance.

Irreducible Error and Model Flexibility

Increasing model flexibility cannot eliminate irreducible error.

What Happens to Bias as Flexibility Increases?

Bias generally decreases.

What Happens to Variance as Flexibility Increases?

Variance generally increases.

What Happens to Training Error as Flexibility Increases?

Training error generally decreases.

What Happens to Test Error as Flexibility Increases?

Test error generally decreases at first but can increase after the model becomes too flexible.

Why Can a More Flexible Model Be Worse?

It can overfit the training data and have high variance, causing poor performance on new data.

Why Is a Low Training Error Not Enough?

A model can have low training error because it overfits the training data while still having high test error.

Why Is Irreducible Error Important?

It represents noise that cannot be removed and therefore places a lower limit on prediction error.

Parametric Advantage

Parametric models are generally simpler, easier to interpret, and require estimation of only a limited number of parameters.

Parametric Disadvantage

If the assumed functional form is wrong, the model may have high bias.

Nonparametric Advantage

Nonparametric models are flexible and can capture complicated relationships without assuming a particular functional form.

Nonparametric Disadvantage

They generally require more data and are less interpretable.

When Is a Simple Model Useful?

When interpretability is important or when the true relationship is relatively simple.

When Is a Flexible Model Useful?

When the true relationship is complicated or nonlinear and sufficient data are available.

Prediction Example

Predicting the price of a house from square footage, number of bathrooms, location, and other features.

Inference Example

Determining how house price changes as square footage increases while holding other predictors fixed.

Regression Example

Predicting someone's income or blood pressure.

Classification Example

Predicting whether someone has a disease or whether the stock market goes up or down.

Supervised Learning Example

Predicting house prices when each house in the training data has a known price.

Unsupervised Learning Example

Grouping observations into clusters when no response labels are provided.

Clustering vs Classification

Clustering is unsupervised and discovers groups without labels, while classification is supervised and predicts known classes.

Linear Regression vs KNN

Linear regression is parametric and relatively inflexible, while KNN is nonparametric and can be much more flexible.

Linear Regression vs Neural Network

Linear regression is highly interpretable and relatively inflexible, while neural networks are highly flexible and difficult to interpret.

KNN and Parameters

KNN does not estimate a fixed set of model parameters; K primarily acts as a hyperparameter controlling flexibility.