4-1 algorithms in ML, linear regression, and logistic regression

model

a mathematical representation of a real world process; a predictive model forecast a future outcome based on past behaviors

algorithm

a set of rules used to make a calculation or solve a problem

regression

a prediction method whose output is a real number, a value that represents a quantity along a line

classification

a prediction method that assigns each data point to a predefined category, a type f operating system

target

dependent variable, is the output of the model or the variable you wish to predict

1. supervised learning

2. classification

3. regression

trade off between accuracy and interpretability

model interpretability

• similar models like linear regression and decision trees ar easily interpretable but might underperform

• crucial for trust, regulatory compliance, and clinical adoption

model performance

• complex models like deep neural networks often deliver higher performance but were less interpretable

• choosing between interpretability and performance involves balancing accuracy with the need for understandable predictions

computational efficiency

• training time

• interference time

• memory usage

• scalability

bias variance tradeoff

complex models

• capture intricate patterns

• risk overfitting and may lack generalization

simple models

• less detailed

• more robust and generalize better to new data

balancing act

complex models

• low bias

• high variance

simple models

• high bias

• low variance

*find balance

linear regression

• estimates the linear relationship between a scalar response and one or more variables

• predicts continuous outcome based on one or more predictors/variables

• relationship sare modeled using linear predictor functions whose unknown model parameters are estimated from the data

ordinary least squares

• using gradient descent to minimize the sum of the squares of the residuals

• the better the linear regression fits the data in comparison to the simple average, the closer the value of R² is to 1

4 common regression metrics for linear regression

linearity

the change in the outcome variable should be proportional to the change in the predictor variables

independence

the observations should be independent from each other

homoscedasticity

the spread of residuals should be roughly the same across all values of the predicted outcome

normality

the errors should follow a normal distribution

advantages of linear regression

• simple model

• computationally efficient

• interpretability of the output

disadvantages of linear regression

• overly simplistic — cannot capture real world complexity

• linearity assumption

• severely affected by outliers

• independence of variables

advantages of logistic regression

• easier to implement, interpret, and very efficient to train

• easily extend to multiple classes

• good accuracy for many simple data sets and it performs well when the dataset is linearly separable

disadvantages of logistic regression

• if the number of observations is less than the number of features, it may lead to overfitting

• the major limitation is the assumption of linearity between the dependent variable and the independent variables

when to use linear regression

• outcome is continuous and unbounded

• assume linear relationship between predictors and the outcome

• want to quantify the effect of each predictor on the outcome

when to use logistic regression

• the outcome is binary (yes/no, 1/0)

• want to estimate probabilities of an outcome

• interested in the odds ratios for various predictors