02 Pruned Trees

Understanding Trees in Predictive Modeling

  • Trees are commonly used in predictive modeling but suffer in terms of prediction accuracy.

  • One major issue is overfitting, where the model performs extremely well on training data but fails on unseen data.

Reasons for Overfitting

  • Leaf Node Definitions: Trees are often grown deep until each leaf node has a minimal number of observations (e.g., less than 5). This leads to detailed fits specific to the training data.

  • Complex Trees: Trees can become complex, using input variables multiple times for splits, which can lead to intricate structures that fit noise rather than true patterns in the data.

  • High Variance: Overfitted trees display high variance, meaning their predictions can significantly change with slight alterations in input data.

Strategies to Combat Overfitting

Prune Trees to Reduce Complexity

  • Grow Smaller Trees: Stop growing trees earlier when nodes have a higher minimal count of observations (e.g., 50 or 500). This introduces bias but reduces variance, adhering to the bias-variance trade-off.

  • Thresholds on Split Quality: Set thresholds for the decrease in residual sum of squares (RSS) between splits to limit growth and enhance robustness.

Complete Tree Growth Followed by Pruning

  • A better approach might be to grow an entire tree to observe the structure, then selectively prune branches that do not improve predictions.

  • Pruning Mechanism: Identify branches to remove based on their contribution to prediction accuracy using RSS assessments.

Cost Complexity Pruning

  • Regularization Concept: Similar to LASSO and Ridge regression, cost complexity pruning introduces a penalty for excessive leaves, encouraging simpler, more generalizable models.

  • RSS Formula: The test error can be expressed taking RSS into account, and the aim is to minimize this while also penalizing tree size (number of leaves).

  • Alpha Parameter for Tuning: Introduces flexibility in adjusting the penalty applied during pruning, ensuring models are appropriately complex for the dataset used.

The Process of Pruning Trees

Step 1: Initial Tree Fitting

  • Fit a complete, overfitting-prone tree to the complete training dataset

Step 2: Generate Subtrees through Pruning

  • Use the full tree to produce a sequence of potential subtrees by trimming branches based on RSS and alpha constraints.

Step 3: Cross Validation for Hyperparameter Tuning

  • Perform cross-validation on the trimmed trees to identify the alpha value that yields the best predictive accuracy. Multiple folds can be created from the training dataset for this process.

Model Selection and Testing

  • Once the best alpha is determined, fit the tree to the entirety of the training data and evaluate it against a reserved test dataset.

  • Compare the performance against other models (e.g., linear regression, LASSO) by analyzing their respective test errors.

  • The ultimate goal is identifying the best performing and most robust model tailored to the specific dataset.