Lecture 9: Decision Trees and Random Forest 2

Q: How do you pick a split in a decision tree?

Choose the variable and cutoff that best separates the data, usually using Gini impurity for classification.

Q: When do you stop growing a decision tree?

When nodes are pure (one label) or have too few data points.

Q: What is bootstrap aggregation (bagging)?

Resampling data with replacement, training models on each sample, and aggregating their results to reduce overfitting.

Q: Why does a fully grown single decision tree overfit?

It perfectly memorizes the training data, losing generalization.

Q: How do random forests improve over single decision trees?

They grow many trees on bootstrapped samples and average or vote across them to reduce variance.

Q: What extra randomness is added in random forests?

Each split considers a random subset of predictor variables instead of all variables.

Q: In Machine Learning, what are the three key steps?

Train on past data, predict on new data, evaluate performance.

Q: Why use ensemble models?

Because no single model is perfect; combining models can improve accuracy.

Q: What is stacking in ensemble modeling?

Using outputs from different models as new features for a final model.

Q: How does linear weighted stacking work?

1. Split training data into two parts (train1 and train2).
   2. Train several models (e.g., Random Forest, GLM, GBM, SVM) on train1.
   3. Score each model on train2, using the scores as new features.
   4. Combine these new features with original features on train2.
   5. Train a final GLM model (including interaction terms) on this combined data.
   6. Apply the stacked pipeline to test data