mod 08: overfitting, regularization

overfitting occurs when a model fits the — data very wells but performs poorly on — data

training, test

underfitting occurs when a model is too — and cannot capture the — in the data

simple, pattern

a model that performs well on new data is said to — well

generalize

a linear model like f(x) = w0 + w1x can lead to — if the true relationship is more complex

underfitting

a high degree polynomial model may lead to — because it fits noise in the data

overfitting

an overfit regression model typically has very — training error and — test error

low, high

underfitting is associated with high —

bias

overfitting is associated with high —

variance

variance refers to how sensitive a model is to — — in the — data

small changes, training

bias refers to error caused by overly — assumptions

simple

increasing model complexity TOO MUCH leads to —

overfitting

decreasing model complexity TOO MUCH leads to —

underfitting

FIXING OVERFITTING

• collect more —

• selecting only important input variables: — —

data, feature selection

feature selection helps reduce — by — model complexity

overfitting, lowering

regularization helps prevent —

it works by making — —

overfitting, weights small

Regularized Cost Function

the first term is the — —

the second term is the — —

the second term penalizes — —

data term, regularization term, large weights

Role of lambda

lambda controls the tradeoff between — the data and keeping weights —

fitting, small

• If λ = 0:

  • No —

  • Model may —

• If λ is very large:

  • Weights go to —

  • Model becomes too —

When λ is extremely large:

• Model becomes: f(x) = —

• This predicts the — of y values

regularization, overfit, 0, simple, w0, average

If we have many features (x₁ … x₁₀₀), regularization:

• forces weights to be —

• reduces risk of —

small, overfitting

what is this?

regularized linear/logistic regression