Module III – Linear Regression, Regularization, and Cross Validation

Machine learning taxonomy

Regression

• kind of supervised learning

• y = f(x,ϕ)

y

• target variable

• output variable, dependent variable, labels

• scalar (i.e., a number)

• can be continuous or discrete

x

• input variable

• aka descriptors, features, independent variables

• can be continuous or discrete

• vector (i.e., can be many values for each instance

f

model → typically choose linear model

ϕ

• model parameters

• often also called w or β

• vector (i.e., can be many)

Types of Regression Models

Assumptions of Linear Regression

Linearity

• relationship between independent and dependent variables is linear

• change between each is proportional

Homoscedasticity

• spread of errors should be relatively uniform, regardless of predictor value

• residuals maintain constant variance across all independent variables

Linear Regression Model Form ****

Human learning

• guessing

• trying diff combos of phi0 and phi1 to find line of fit

Loss Function

• optimal parameters phi are those that minimize L(phi)

• best model has minimal loss

• sum of square error loss

gradient descent

• optimization algorithm used to train machine learning models by minimizing errors by playing w phi0 and phi1 to map out loss surface

• We ask the model to search for the values of phi that minimize L(phi)

• One way to do this is by gradient descent on the loss surface.

linear regression w/ nonlinear features

choosing what features to use

• Select features based on the data or the physics (e.g., a harmonic spring needs a quadratic term)

• Try many features and pare down to those that decrease the loss the most

• Trade-off model accuracy (low loss) and complexity (more features)

• Check performance on new data to see how model performs? Use regularization or cross validation!

regularization

• encourages a complex model to avoids over-fitting and find simpler solutions in the Goldilocks Zone

• popular method is adding penalty term to cost function to punish large or non-zero coefficients

under-fitting

• too simple to capture underlying pattern in data

optimal-fitting

• just right number of parameters needed

over-fitting

• model is too complex relative to training data available

• high accuracy on training data but low accuracy on test data

• train loss (MSE) low but validation loss (MSE) high!

regularized linear regression loss function

• lamda (hyperparameter) is the strength of regularization (or punish the model)

• view the regularized linear regression loss function as a generalization of our standard linear regression loss function

• lamda controls the trade-off between fitting the data well (first term) and minimizing the size of the coefficients (second term)

• For lamda =0 we recover the standard linear regression least squares loss. For lamda > 0 we apply a penalty that tends to shrink the coefficients to zero.