Gradient Descent and Model Optimization

Flashcards covering key concepts of gradient descent, model optimization, feature selection, and related statistical measures.

Gradient Descent

An optimization algorithm that iteratively adjusts parameters to minimize a cost function.

Cost Function

A function that measures the error or loss in predictions of a model; it needs to be minimized.

Learning Rate (Alpha)

A hyperparameter that determines the size of the steps taken in the direction of the gradient in gradient descent.

Gradient

The vector of partial derivatives indicating the direction of the steepest increase in a function, used to find minima.

Convex Function

A function where a line segment between any two points on the graph of the function lies above the graph; ensures a single global minimum.

Global Minimum

The lowest point on a cost function graph, representing the best possible prediction accuracy for a model.

Local Minimum

A point on a cost function where the function has a lower value than its neighbors but may not be the lowest overall.

Mean Squared Error (MSE)

A common loss function used in regression that squares the errors between predicted and actual values.

P-Value

A statistical measure that indicates the probability of obtaining the observed results assuming that the null hypothesis is true.

Backward Elimination

A feature selection technique that removes the least significant features based on p-values from the model until all remaining features are statistically significant.

Overfitting

A modeling error that occurs when a model is too complex and captures noise rather than the underlying pattern.

Feature Scaling

A method used to standardize the range of independent variables or features of data, essential for many machine learning algorithms.