04 - Regularized Linear Regression

Higher Order Features

Gives the model flexibility to capture more complex trends

Bias and variance

Models can be generally described in terms of _____

High Bias

Model is too simple, too “stiff”

High Bias

No matter how much you try to adjust the parameters, it cannot capture certain kinds of patterns

High Variance

Model is too complex, too “flexible”

High Variance

Sometimes, it’s too flexible that it fits the data too much

Underfitting

Model did not fit the training data well

Underfitting

Model is too high bias to capture actual patterns

Underfitting

Model not trained properly

Overfitting

Model fits the training data too well, but performs poorly on unseen (test) data

Underfitting

High training error, high test error

Overfitting

Low training error, high test error

Degree of the polynomial

Typical Overfitting Plot: The training error decreases as the _____ increases

Increasing

Typical Overfitting Plot: The testing error, measured on independent data, decreases at first, then starts _____

Regularization

Refers to methods to reduce overfitting in models

Listen

Data Quantity Effect: More data means the models are more likely to “_____” to the general trend

Magnitude

When fitted, parameters tend to increase in _____ as the order increases

Large magnitude

Wild swings are caused by _____ coefficients

LR with Regularization

Forces the model to minimize both the training error and the magnitude of the parameters

Large

Regularization Constant (𝝀): If 𝜆 is _____, training error has little impact on loss

0

Regularization Constant (𝝀): If 𝜆 is _____, no regularization

Negative

Regularization Constant (𝝀): If 𝜆 is _____, the higher the weights, the better

Small

Regularization Constant (𝝀): If 𝜆 is _____, regularization considered in the loss

Lasso Regression

Has a larger tendency to produce 0 coefficients

Ridge Regression

Weights approach 0, but almost always never reaches 0

Ridge Regression

Not good for datasets with a large number of features

Lasso Regression

Least Absolute Shrinkage and Selection Operator

Lasso Regression

Weights approach 0, this makes the feature useless

Lasso Regression

Good for removing features from dataset with a large number of features