3.3 Ensemble Methods

Section 3.3 of Exam MAS-II

Bagging (3 Steps)

1. Create b bootstrap samples from the original training dataset.

2. Construct a decision tree for each bootstrap sample using recursive binary splitting.

3. Predict the response of a new observation by averaging the predictions (regression trees) or by using the most frequent category (classification trees) across all trees.

Bagging Properties:

• Does Increasing b cause overfitting?

• Bagging reduces ____.

• Out-of-bag error is a valid estimate of ______.

• Increasing b does not cause overfitting.

• Bagging reduces variance.

• Out-of-bag error is a valid estimate of test error.

Random Forests (3 Steps)

1. Create b bootstrap samples from the original training dataset.

2. Construct a decision tree for each bootstrap sample using recursive binary splitting. At each split, a random subset of variables are considered.

3. Predict the response of a new observation by averaging the predictions (regression trees) or by using the most frequent category (classification trees) across all trees.

Random Forest Properties:

• _______ is a special case of random forests.

• Increasing __ does not cause overfitting.

• Decreasing __ reduces the correlation between ________.

• Bagging is a special case of random forests.

• Increasing b does not cause overfitting.

• Decreasing k reduces the correlation between predictions.

Boosting Process

Boosting Properties:

• Does Increasing b cause overfitting?

• Boosting reduces ____.

• __ controls the complexity of the boosted model.

• __ controls the rate at which boosting learns.

• Increasing b can cause overfitting.

• Boosting reduces bias.

• d controls the complexity of the boosted model.

• lambda controls the rate at which boosting learns.

Bayesian Additive Regression Trees (BART) Process

BART Properties:

• Like bagging and random forests, BART incorporates ________ into its modeling process.

• Like boosting, BART _______ _____ ______ to capture information not captured by previous trees.

• Like bagging and random forests, BART incorporates randomness into its modeling process.

• Like boosting, BART sequentially builds trees to capture information not captured by previous trees.