BMGT 394H #6: Ensemble Learning + Interpretability

0.0(0)
Studied by 0 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/29

encourage image

There's no tags or description

Looks like no tags are added yet.

30 Terms

1
New cards

ensemble

combines predictions of many base models into one; 2 types: bagging and boosting

2
New cards

t/f: averaging a set of observations increases variance

false

3
New cards

bagging

  1. bootstrap: draw many random samples from training data

  2. train separate decision tree on each sample independently

  3. aggregate: average predictions (regression) or take majroity vote (classification)


4
New cards

random forest

bagging, but each time a tree considers a split, it can only choose from a random sample of predictors


5
New cards

t/f: when m = all the predcitors, random forest is bagging

true

6
New cards

boosting

grows trees slowly by considering the misclassified or residuals

  1. fit a small simple tree

  2. find what it got wrong, increase weight of misclassified points (or fit next tree to remaining erros/residuals)

  3. fit the next tree which focuses on the error cases

  4. repeat many times; the final model is a weighted combination of all trees


7
New cards

learning rate

controls how big a step each new tree takes in boosting

8
New cards

bagging mainly reduces

variance

9
New cards

boosting mainly reduces

bias

10
New cards

bagging combines

simple average or majority vote

11
New cards

boosting combines

weighted sum

12
New cards

t/f: bagging has a high overfitting risk

false; adding trees doesn’t hurt

13
New cards

t/f: boosting has a high overfitting risk

boosting can overfit with too many trees or high learning rate

14
New cards

ensemble learnign works better because of

diversity; models make different errors, which cancel out when combined

15
New cards

t/f: decision trees are very unstable

true; small changes in data can produce a very different tree

16
New cards

why are most ensemble methods based on decision trees?

decision trees are unstable → diversity

17
New cards

pro of ensemble

better predictive performance than a single tree

18
New cards

con of ensemble

not interpretable; can’t follow hundreds of trees the way you follow one

19
New cards

global interpretability

how does the model behave overall? which variables matter the most in general and how?

20
New cards

local interpetability

why did the model make this rpedictino for this one record?

21
New cards

global is better for __

understanding the model as a whole: ranking variables, seeing general direction of effects, business strategy, auditing overall fairness

global results are averages and can hide individual differences

22
New cards

local is better for __

explaining an individual decision, no generalizing

23
New cards

variable (feature) importance

ranks how much each variable contributes to the model’s predictions

24
New cards

tree-based importance

how much each variable reduces error across all splits that use it, summed over all trees

25
New cards

permutation importance

shuffles one variable’s values and measure how much accuracy drops

26
New cards

what is a limitation of variable importance?

does not tell you which direction or what shape the effect of a variable is

27
New cards

ICE plot

shows how a model’s prediction changes for each individual observation as you vary one predictor, while keeping that observation’s other predictors fixed

28
New cards

PDP

average of all the ICE lines

29
New cards

shapley value

determines the contribution of each player participating in a collaborative work

30
New cards

LIME

explains one prediction from a black-box model by fitting a simple, easy-to-read model just around that one record