Model Complexity and Generalization

These flashcards cover key terminology and concepts related to model complexity and generalization in machine learning, helping to prepare for exams on the subject.

Model Complexity

Flexibility of a model in capturing predictor-response relationships.

Underfitting

When a model is too simple, leading to high bias and low variance; it misses important structures in the data.

Overfitting

When a model is too complex, resulting in low bias and high variance; it fits noise and performs poorly on unseen data.

Train-Test Split

A procedure to evaluate generalization by splitting data into a training set (for training the model) and a test set (for assessing model performance).

Bootstrapping

A resampling method used to approximate the sampling distribution of estimators by drawing samples with replacement.

Cross-Validation

A method for assessing the reliability of a model by partitioning data into subsets, training on some subsets, and validating on others.

Bias-Variance Tradeoff

The balance between the error due to bias (over-simplification) and variance (sensitivity to data), which impacts prediction accuracy.

Performance Metrics

Measures such as R², Adjusted R², RMSE, and MAE that evaluate the effectiveness of a model in predictions.

K-Fold Cross-Validation

A technique where data is divided into k folds; the model is trained on k-1 folds and tested on the remaining fold, repeating until all folds are tested.

Leave-One-Out Cross-Validation (LOOCV)

A special case of k-fold where k equals the number of observations; trains on n-1 samples and tests on 1, leading to low bias but high variance.