04 Trees vs Linear

Comparison of Trees and Linear Models

Regression Trees

  • Predicts values through step functions based on the regions of the input space.

  • Steps in prediction:

    • If the observation falls within region ( r_m ), the prediction is based on the classification error rate ( c_m ).

    • Uses an indicator function to determine if the observation is in ( r_m ).

  • More flexible in modeling nonlinear and complex relationships than linear regression.

Visual Examples

  1. **Linear Structure:

    • In 2D input space, a linear model can fit a decision boundary well, represented as a plane in 3D space.

    • Decision trees struggle with non-axis aligned boundaries, leading to poor fits.

  2. **Nonlinear Structure:

    • Nonlinear boundaries that don’t align with polynomials are difficult for linear regression but manageable for decision trees due to their stepwise nature.

Model Selection

  • Choosing between regression trees and linear models often requires empirical testing via cross-validation to determine the best fit for the specific dataset.

Advantages of Regression Trees

  • Interpretability:

    • Trees are easier to explain and understand, closely mirroring human decision-making processes.

  • Handling Qualitative Predictors:

    • They don’t require complex transformations for categorical variables, unlike linear models which need dummy variables.

Disadvantages of Regression Trees

  • Predictive Accuracy:

    • Typically, trees do not achieve as high predictive accuracy compared to linear or polynomial models unless the data fits the tree structure.

  • Robustness Issues:

    • Decision trees can be sensitive to changes in training data; small variations can significantly alter the model, reducing reliability.

Enhancements through Ensemble Methods

  • Ensemble methods can significantly improve decision tree performance, increasing both robustness and predictive accuracy.

  • Combining multiple trees, such as in Random Forests or Gradient Boosting, allows for better generalization and more reliable predictions.