1/29
Looks like no tags are added yet.
ensemble
combines predictions of many base models into one; 2 types: bagging and boosting
t/f: averaging a set of observations increases variance
false
bagging
bootstrap: draw many random samples from training data
train separate decision tree on each sample independently
aggregate: average predictions (regression) or take majroity vote (classification)

random forest
bagging, but each time a tree considers a split, it can only choose from a random sample of predictors

t/f: when m = all the predcitors, random forest is bagging
true
boosting
grows trees slowly by considering the misclassified or residuals
fit a small simple tree
find what it got wrong, increase weight of misclassified points (or fit next tree to remaining erros/residuals)
fit the next tree which focuses on the error cases
repeat many times; the final model is a weighted combination of all trees
learning rate
controls how big a step each new tree takes in boosting
bagging mainly reduces
variance
boosting mainly reduces
bias
bagging combines
simple average or majority vote
boosting combines
weighted sum
t/f: bagging has a high overfitting risk
false; adding trees doesn’t hurt
t/f: boosting has a high overfitting risk
boosting can overfit with too many trees or high learning rate
ensemble learnign works better because of
diversity; models make different errors, which cancel out when combined
t/f: decision trees are very unstable
true; small changes in data can produce a very different tree
why are most ensemble methods based on decision trees?
decision trees are unstable → diversity
pro of ensemble
better predictive performance than a single tree
con of ensemble
not interpretable; can’t follow hundreds of trees the way you follow one
global interpretability
how does the model behave overall? which variables matter the most in general and how?
local interpetability
why did the model make this rpedictino for this one record?
global is better for __
understanding the model as a whole: ranking variables, seeing general direction of effects, business strategy, auditing overall fairness
global results are averages and can hide individual differences
local is better for __
explaining an individual decision, no generalizing
variable (feature) importance
ranks how much each variable contributes to the model’s predictions
tree-based importance
how much each variable reduces error across all splits that use it, summed over all trees
permutation importance
shuffles one variable’s values and measure how much accuracy drops
what is a limitation of variable importance?
does not tell you which direction or what shape the effect of a variable is
ICE plot
shows how a model’s prediction changes for each individual observation as you vary one predictor, while keeping that observation’s other predictors fixed
PDP
average of all the ICE lines
shapley value
determines the contribution of each player participating in a collaborative work
LIME
explains one prediction from a black-box model by fitting a simple, easy-to-read model just around that one record