EC 4308: Tree-Based Ensemble Methods - Bagging, Random Forests, and Boosting

Comprehensive fill-in-the-blank practice flashcards covering tree-based ensemble methods: the bootstrap, bagging, out-of-bag error estimation, random forests, boosting, gradient boosting, classification metrics, and tree interpretability tools.

The bootstrap resampling technique was originally introduced in a seminal 1979 paper by __________.

Bradley Efron

The core mechanism of the bootstrap is to emulate sampling from the population by repeatedly resampling observations from the original dataset __________.

with replacement

When dividing wealth between two assets with returns $X$ and $Y$, the value of $\alpha$ that minimizes total portfolio variance is given by __________.

$$\frac{\sigma_Y^2 - \sigma_{XY}}{\sigma_X^2 + \sigma_Y^2 - 2\sigma_{XY}}$$

In the bootstrap resampling diagram shown, each bootstrap dataset $Z^{*b}$ is created by drawing observations from original dataset $Z$ of size $N$ __________.

with replacement

A common time-series modification used when standard bootstrap sampling is invalid due to temporal dependence is the __________ bootstrap.

block

The term "bagging" was introduced by Leo Breiman in 1996 and is short for __________.

bootstrap aggregating

Decision trees have low bias but suffer from high __________, which makes them ideal base learners for bagging.

variance

The sample mean of $N$ i.i.d. random variables with variance $\sigma^2$ has variance equal to __________.

$$\frac{\sigma^2}{N}$$

Because bootstrap samples are sampled with replacement, on average approximately __________ of the original observations are omitted from a given bootstrap sample and labeled out-of-bag (OOB).

1/3

As the number of bootstrap replications $B \rightarrow \infty$, out-of-bag (OOB) mean squared error approaches __________ error.

LOOCV (Leave-One-Out Cross-Validation)

When applying bagging to classification trees, predictions across the $B$ bootstrap trees are aggregated using __________.

majority vote

Random forests improve upon bagging by decorrelating the trees, considering only a random subset of $m$ predictors at each split where the standard recommendation for classification is $m =$ __________.

$$\sqrt{P}$$

For regression problems, the standard rule of thumb for the number of randomly chosen candidate predictors $m$ at each split in a random forest is $m =$ __________.

$$\frac{P}{3}$$

Standard bagging can be viewed as an extreme case of random forests where the parameter $m$ equals __________.

$$P$$

The concept of boosting as an ensemble learning method was introduced in 1990 by __________ in "The strength of weak learnability".

Robert Schapire

Unlike bagging which trains deep independent trees in parallel, boosting fits a series of __________ sequentially and aggregates them slowly.

weak learners

In $L_2$ boosting for regression trees, each successive tree $h_m(x)$ is fit directly to the current iteration's __________.

residuals

In the boosting update step $f_m(x) = f_{m-1}(x) + \lambda h_m(x)$, the parameter $\lambda$ controls the rate of learning and is referred to as the __________.

shrinkage parameter

Gradient boosting, developed by Friedman (1999), generalizes boosting to arbitrary differentiable loss functions by fitting base learners to the __________ of the loss.

negative gradient (or pseudo-residuals)

Stochastic gradient boosting adds regularization by growing each tree using only a random subsample of the data, typically chosen to be of size __________.

$$0.5N$$

In classification performance evaluation, sensitivity (also termed recall) is defined as the formula __________.

$$\frac{TP}{TP + FN}$$

In binary classification evaluation, specificity measures the proportion of actual negatives that are correctly identified and is calculated as __________.

$$\frac{TN}{TN + FP}$$

The Receiver Operating Characteristic (ROC) curve evaluates a classification model across various probability thresholds by plotting the true positive rate against the __________.

false positive rate (FPR)

In this relative importance plot for the boosted tree predicting 401(k) eligibility, the single predictor with dominant influence (relative importance near 80) is __________.

inc (income)

To understand the marginal effect of an individual feature in complex tree ensembles, __________ plots display the estimated function after averaging out the effects of all other predictors.

partial dependence

Among popular specialized gradient boosting libraries, __________ is specifically designed for handling categorical features effectively.

CatBoost