3.2 Decision Trees

Section 3.2 of Exam MAS-II

Root node

Terminal nodes or leaves

Internal nodes

Parent nodes

Child nodes

Branches

Stump

Terminal nodes

Internal nodes

The node before any splits occur. It contains all observations.

The partitions of the predictor space.

Points along the tree where splits occur.

Nodes where splits occur.

Nodes produced by a split.

Lines that connect any two nodes.

A decision tree with only one internal node.

Do not have child nodes.

Have child nodes.

Decision Tree Algorithm

1. Construct a large tree using recursive binary splitting.

2. Obtain a sequence of best subtrees, as a function of , using cost complexity pruning.

3. Choose by applying -fold cross-validation. The that results in the lowest cross-validation error is selected.

4. The best subtree is the subtree created in step 2 with the selected value.

Tree Construction:

Recursive binary splitting partitions the predictor space into ______

The resulting regions are _____

• Recursive binary splitting partitions the predictor space into as many regions as possible while subject to stopping criteria.

• The resulting regions are high-dimensional rectangles, or boxes.

Tree Construction:

For regression, the best splits are _____. (describe equation)

For classification, the best split is the split that _____. (describe equation)

Classification Tree Construction: Impurity Measures and Equations

Classification Tree Construction:

1. If a node is relatively pure, then it contains observations that are

2. The Gini index and cross entropy are sensitive to _____. They are favored in _____.

3. The classification error rate is not sensitive to _____. It is favored in _____.

• If a node is relatively pure, then it contains observations that are mainly from the same class.

• The Gini index and cross entropy are sensitive to node purity. They are favored in tree-growing.

• The classification error rate is not sensitive to node purity. It is favored in tree-pruning.

Tree Pruning: What’s it called and what does it minimize for regression and classification

Cross Validation

1. The number of terminal nodes is directly related to ______.

2. Decision trees will outperform linear regression when the chosen functional form _____ approximates the true relationship between the response and explanatory variables.

• The number of terminal nodes is directly related to flexibility.

• Decision trees will outperform linear regression when the chosen functional form poorly approximates the true relationship between the response and explanatory variables.

Advantages of Trees (4)

Disadvantages of Trees (2)

Advantages of Trees

• Easy to interpret and explain

• Can be presented visually

• Manage categorical variables without the need of dummy variables

• Mimic human decision-making

Disadvantages of Trees

• Not robust

• Do not have the same degree of predictive accuracy as other statistical methods