Linear Regression v2 (Notes)

Linear Regression

A method to model the relationship between a dependent variable and one or more independent variables.

Multiple Linear Regression

An extension of simple linear regression that uses multiple features to predict a dependent variable.

Dependent variable (y)

The variable that we are trying to predict in regression analysis.

Independent variables (x1, x2, ..., xn)

The predictors or features used to predict the dependent variable in regression.

Residuals

The difference between the observed values and the predicted values in a regression model.

Gradient Descent

An optimization algorithm used to minimize a loss function by iteratively adjusting model parameters.

Mean Squared Error (MSE)

A loss function that measures the average of the squares of the errors or deviations.

Sum of Squared Errors (SSE)

Measures the total deviation of the predicted values from the actual values.

Ridge Regression

A technique in linear regression that adds an L2 penalty to prevent overfitting by shrinking coefficients.

Lasso Regression

A regression method that adds an L1 penalty, allowing for feature selection by shrinking some coefficients to zero.

Elastic Net

A regression method that combines L1 and L2 penalties to improve performance in models with correlated features.

Learning Rate (α)

A parameter that controls the step size during the adjustment of coefficients in gradient descent.

Intercept (β0)

The expected mean value of the dependent variable when all independent variables are equal to zero.

Coefficient (βj)

The value that represents the degree of change in the dependent variable for each unit of change in an independent variable.

Loss Function

A method of evaluating how well specific algorithm models the data; commonly used in regression.

Feature Selection

The process of selecting a subset of relevant features for model construction.

Training set

The portion of the dataset used to train the model.

Test set

The portion of the dataset used to evaluate the performance of the model.

Cross Validation

A technique for assessing how the results of a statistical analysis will generalize to an independent data set.

Regularization

A technique used to prevent overfitting by adding a penalty to the loss function.

Bias-Variance Tradeoff

A key concept in machine learning that describes the trade-off between the error due to bias and the error due to variance.

When to use Ridge regression?

when you have many predictors that are correlated, and you want to keep all of them in the model but reduce their influence

When to use Lasso Regression?

when you suspect that only a few predictors are important, and you want to automatically select the most relevant ones

When to use Elastic net?

when you have a mix of correlated and uncorrelated predictors, or when you want the benefits of both Ridge and Lasso

Overfitting

Low training error, high test error

undercutting

high training error, high test error