Assessing Model Accuracy

Mean Squared Error (MSE)

A commonly-used measure to evaluate the performance of a statistical learning method by quantifying the accuracy of predictions.

Training MSE

The mean squared error calculated on the training data; of limited interest for evaluating future prediction accuracy.

Test MSE

The mean squared error calculated on unseen test data; reflects the prediction accuracy for future observations.

Overfitting

Occurs when a model captures noise in the training data, leading to a low training MSE but a high test MSE.

Bias-Variance Trade-Off

The balance between bias and variance in a model, where low bias is preferred at the expense of high variance and vice versa.

Flexibility of a model

Refers to the model's ability to fit the training data; greater flexibility can reduce bias but might increase variance.

Irreducible error

The lowest possible test MSE theoretically achievable, regardless of the model used.

U-Shape in Test MSE Curve

A pattern observed when increasing model flexibility initially decreases test MSE, then causes it to increase again.

Bias

The error introduced by approximating a real-life problem with a simpler model.

Variance

The amount by which the estimate of a function changes if a different training dataset is used.