Chap 12: Study Notes on Ensemble Models

Ensemble Models

12.1 Introduction to Ensemble Models

• Learning Objectives:

  • Define ensemble models.

  • Identify suitable base models for regression or classification.

  • Compare sequential and parallel ensemble models.

  • Use predictions from base models to make an ensemble prediction.

Overview

• Data scientists balance model complexity with model accuracy.

• Models that are too complex tend to overfit, while simpler models may have higher error rates.

• Ensemble modeling is a solution to improve model performance by combining predictions from various base models.

• Base Models: The individual models used within an ensemble model.

Participation Activity

12.1.1: Visualizing an Ensemble Model

• Visual representation of base models and their predictions against an ensemble model.

  • Base Model 1

  • Base Model 2

  • Ensemble Model

Types of Ensemble Models

12.1.2: Ensemble Visualization

• Input Features: Albumin and Cholesterol

• Predictions from base models for diagnosing hepatitis C based on the two features.

  • Base Model 1:

  • Threshold at 35 for Albumin.

  • Patients with Albumin ≤ 35 predicted to have hepatitis C.

  • Base Model 2:

  • Threshold at 5 for Cholesterol.

  • Patients with Cholesterol ≤ 5 predicted to have hepatitis C.

  • Ensemble Model:

  • Combines predictions where patients with (Albumin < 35 AND Cholesterol < 5) are predicted to have hepatitis C.

  • Patients with (Albumin > 36 AND Cholesterol > 5) do not have hepatitis C.

  • Conflict zones where models disagree are noted.

Selection of Base Models

12.1.3: Types of Ensemble Models

• Ensemble models combine predictions from multiple base models and can be fitted in different sequences.

  • Sequential Ensemble Models:

  • Fit base models one at a time.

  • Update dataset and predictions iteratively.

  • Focuses on regions of data that had low accuracy in previous models.

  • Example: Boosting

  • Parallel Ensemble Models:

  • Fit base models simultaneously.

  • Combine predictions at the end.

  • Example: Bagging and Stacking

  • Efficiency: Sequential models take longer whereas parallel models are more time-efficient.

Methods of Making Predictions

12.1.4: Using Predictions from Base Models

• Ensemble models make final predictions by combining outputs through different techniques:

  • Classification Models:

  • Use majority voting; the most frequently predicted class is chosen as the ensemble prediction.

  • Regression Models:

  • Use averaging of predictions to get the final output.

Boosting

12.2 Bootstrapping in Boosting

• Boosting: A sequential ensemble technique where the performance of base models is iteratively improved by adjusting weights.

• Weighing Mechanism:

  • Start with equal weights for all instances.

  • After fitting a base model, misclassified instances receive higher weights.

  • Goal: Focus future models on addressing previously misclassified cases.

Advantages and Disadvantages of Boosting

• Advantages:

  • Reduces bias.

  • Improves model performance on weak areas.

• Disadvantages:

  • Only one model fitted at a time can lead to slower processing.

  • High bias models may distort weight adjustments.

12.2.1: Adaptive Boosting

• AdaBoost: A specific boosting algorithm with no fixed number of base models; stops fitting once optimal performance is achieved.

• Algorithm Steps:

  1. Assign equal weights to all instances.

  2. Fit the first model.

  3. Compute error.

  4. Update weights based on error.

  5. Repeat until error meets threshold or max models are reached.

Bagging

12.3 Overview of Bagging

• Bagging: Short for bootstrap aggregating.

• Bagging fits multiple models independently, typically of the same structure, and aggregates their predictions.

• Bootstrapping: Each model is trained on a random sample drawn with replacement.

• Advantages include faster execution and reduced variance in predictions.

Participation Activity

12.3.2: Bagging Application

• Predicting outcomes for patients with hepatitis C based on their Albumin and Cholesterol levels.

Stacking

12.4 Overview of Stacking

• Stacking: Combines multiple types of models instead of a single-type model.

• Predictions from base models are fed into a meta model for final predictions.

• Variability arises because predictions from different types of models are combined.

• Commonly, classification tasks use logistic regression as the meta model, while regressions use linear regression.

Participation Activity

12.4.1: Stacking Application

• Predict patient outcomes using various models and observe the benefits of varying predictions against a their meta-model.

Case Study: Bob Ross

12.5 Application of Ensemble Models

• Dataset: Contains information from the Bob Ross show. Each episode has features such as elements present in paintings.

• Analyzed for predicting the presence of specific features (like trees) based on binary indicators within the dataset.

Performance Comparison

• Analyze accuracy of boosted ^ and bagged models using confusion matrices to determine the better model for prediction based on performance metrics across training and testing datasets.