Linear Regression, Linear discriminant & Logistic Regression

Bias parameter

Allows the model to fit data that does not naturally pass through the origin by shifting the prediction. It is included in the parameter ector w by the addition of a dummy variable with constant value 1.

MLE method

Find the likelihood, take the log, differentiate w.r.t theta, set to zero and solve for theta

Assumptions for parameter learning

(1) data for each class have a Gaussian distribution
(2) the 2 Gaussian distributions have the same covariance matric
We can then find parameters through MLE

Logistic Regression

Uses the sigmoid function to map linear combinations of features to probabilities. No assumptions about feature distribution. Only assumes a linear relationship between the log odds of the class and the features.

MLE for Logistic Regression

Calculate via gradient descent

Gradient descent

Start with a random weight values and adjust each w to minimise the negative log likelihood