Lecture Notes on Ensemble Learning and Regularization

These flashcards cover key concepts related to ensemble learning, regularization, and evaluation metrics in machine learning.

Ensemble Learning

A machine learning paradigm that combines multiple models to create a stronger overall model.

Bagging

A technique that reduces variance by training multiple models on random subsets of the training data with replacement.

Boosting

A technique that reduces bias by sequentially training models, where each new model focuses on correcting the errors made by previous models.

Bias

Error due to overly simplistic assumptions in the learning algorithm, which can lead to underfitting.

Variance

Error introduced by too much complexity in the learning algorithm, which can lead to overfitting.

Overfitting

A modeling error that occurs when the model is too complex and captures noise in the training data.

Underfitting

A modeling error that occurs when the model is too simple to capture the underlying structure of the data.

Bootstrapping

A resampling technique used in bagging to create multiple training datasets by sampling with replacement.

Random Forest

An ensemble method that utilizes multiple decision trees to improve predictive accuracy and control overfitting.

Precision

The proportion of true positive predictions among all positive predictions made by the model.

Recall

The proportion of true positive predictions among all actual positive instances in the dataset.

F1 Score

The harmonic mean of precision and recall, used as a single metric to evaluate a model's accuracy in classification tasks.

Confusion Matrix

A table used to describe the performance of a classification model, showing true positives, false positives, true negatives, and false negatives.

Imbalanced Dataset

A dataset in which the classes are not represented equally, leading to potential bias in the model's predictions.

Regularization

A technique used to reduce overfitting by adding a penalty term to the loss function, constraining the model's complexity.

Ridge Regression

A type of linear regression that incorporates L2 regularization to penalize large coefficients.

Lasso Regression

A type of linear regression that incorporates L1 regularization to penalize large coefficients and potentially eliminate irrelevant features.

Hyperparameter

A parameter of a machine learning model that is set before the training process and governs the learning process itself.