11 - Ensemble Models

Ensemble Models

Using multiple models that ‘work’ together

Bootstrap Aggregation

Also known as bagging

Bootstrap Aggregation

Aims to address the problem of overfitting on the training data

Bootstrap Aggregation

Often applied to decision trees (notorious for overfitting)

Bootstrap Sampling

Step 1

Parallel Training

Step 2

Aggregation

Step 3

Bootstrap Sampling

Generate multiple random subsets of the training data

Parallel Training

Train an independent base model on each bootstrap sample

Aggregation

Combine predictions of the individual models through average (regression) or majority voting (classification)

Random Forest

Bagging model for decision trees

Overfit

Random Forest: Individual decision trees tend to _____

Regularization effect

Random Forest: Averaging multiple overfit trees has a _____

Boosting

Ensemble learning technique where many ‘weak’ models are trained sequentially

Boosting

Start with a simple, weak model, then make another model to learn from the previous model’s mistakes, and so on

AdaBoost

Boosting algorithm for decision trees

AdaBoost

Uses many ‘weak’ decision trees trained sequentially to learn from the previous model’s mistakes

Decision Stump

A decision tree with only one question (i.e., depth= 1)

Decision Stump

Because it can only ask one question, it is not capable of learning complex patterns

AdaBoost

Trains several decision stumps in sequence, where each stump focuses on the weakness of the previous stump

AdaBoost

Originally designed for classification, but there are variants for regression

Consensus

AdaBoost: Final prediction is based on the _____ of the different stumps

Influence

Some stumps have a bigger _____ on the prediction than others

Equal

AdaBoost General Algorithm:

1. Assign _____ weights to all training instances

Best

AdaBoost General Algorithm:

2. Choose the _____ stump according to the weights

Amount of say

AdaBoost General Algorithm:

3. Compute the _____ of the stump

Weights

AdaBoost General Algorithm:

4. Adjust the _____

Step 2

AdaBoost General Algorithm:

5. Repeat _____ for the next stump

Weights

Represents the “attention” that should be given to each instances

Misclassified

Weights: Instances that are _____ should be given more attention

Amount of say

Represents the “voting power” of each stump, based on how well it distinguishes between the classes

Gradient Boosting

Boosting algorithm for regression where a sequence of shallow trees are trained on the combined residuals (errors) of all previous trees

Total prediction

Gradient Boosting: The final prediction is the _____ of all the trees

Stacked Models

Ensemble learning technique where you train multiple diverse models (called base learners), then feed their predictions as inputs to a meta-learner that learns how to best combine them

Stacking

Ensemble Learning: Creating multiple models

Stacking

Ensemble Learning: Create another model that accepts their predictions and improves of them

Bagging

Ensemble Learning: Use high variance models

Bagging

Ensemble Learning: Lower the variance by getting the average

Boosting

Ensemble Learning: Use high bias models

Boosting

Ensemble Learning: Lower the bias by stringing them together and learning from past mistakes