Linear Regression v2 (Notes)

Linear Regression

A method to model the relationship between a dependent variable (y) and one or more independent variables (x1, x2,…, xn).

Dependent Variable

The outcome variable that is being predicted or explained in a regression model, typically denoted as y.

Independent Variable

The input features used to predict the dependent variable in a regression model, typically denoted as x1, x2,…, xn.

Multiple Linear Regression

An extension of linear regression that uses multiple features to predict the dependent variable.

Mean Squared Error (MSE)

loss function that measures how well a regression model predicts target values, defined as the average of the squared differences between actual and predicted values.

Residuals

The differences between observed values and predicted values in a regression model

Gradient Descent

An optimization algorithm used to minimize a loss function by iteratively adjusting model parameters.

Objective Function

minimized in optimization problems, in the context of regression, referring to minimizing the sum of squared errors.

Ridge Regression

A linear regression technique that adds an L2 penalty to the loss function to prevent overfitting by shrinking coefficients.

Lasso Regression

A linear regression technique that adds an L1 penalty to the loss function, allowing for automatic feature selection by shrinking some coefficients to zero.

Elastic Net

A linear regression technique that combines both L1 and L2 penalties, useful for datasets with correlated predictors.

Regularization Parameter (λ)

Controls the strength of the penalty added to the loss function in Ridge and Lasso regressions.

Intercept (β0)

The expected value of the dependent variable when all independent variables are zero, represented as β0.

Coefficients (β1, β2, ... , βn)

The values that represent the influence of each independent variable on the dependent variable in a regression model.

Cross Validation

estimate the skill of machine learning models by dividing the dataset into training and test sets multiple times.

ensures model generalizes well and avoids overfitting

Feature Selection

The process of selecting a subset of relevant features for a regression model to improve its performance and interpretability.

Optimization Algorithm

A method or process used to minimize or maximize an objective function by adjusting parameters.

Learning Rate (α)

A hyperparameter that controls how much to change the model in response to the estimated error each time the model weights are updated.

Step-by-Step Workflow for Gradient Descent

initializing coefficients, computing predictions, calculating loss, computing gradients, and updating coefficients.

Sum of Squared Errors (SSE)

A measure of the total deviation of the predicted values from the actual values, which is minimized in regression.

Kink at Zero

An important feature of the L1 penalty in Lasso Regression that can lead to feature selection by forcing some coefficients to be exactly zero.

Impact of Alpha (α)

Refers to how changes in the regularization parameter affect the strength of penalization and model performance.

Sharp Corner in Lasso Regression

The feature of the L1 penalty that enables it to reduce some coefficients to zero, effectively performing feature selection.