Data Science Quiz 4

Linear Regression

• powerful tool for estimating slope coefficients and intercept values

Linear Regression Use Cases

• statics and dynamics for friction analysis

• physics

• business

Basis Function

• you have one or a small number of inputs to a system

• the input is subject to multiple distinct transforms before multiplying against regression coefficients

Polynomial Regression

• scikit-learn

• making a model that has transforms

Generalization

• informative guess-making

• can be robust to noise

• transferability to knowledge

• smooth model

Particularization

• case-by-case basis

• emphasis on smaller details

• inflections, not smooth

• worse than generalization

Overfitting

the tendency of data mining procedures to tailor models to the training data, at the expense of previously unseen data points

• basically builds a model using pure memorization

• closed form solutions cannot overfit

Irreducible Error

• min/max test dataset performance

• closed form linear regression solution

• minimizes the MSE

What causes the irreducible error?

• noise

• bias

• variance

Bias

induced when the model is too simple, causes underfitting

Variance

induced by complex model’s sensitives to small changes in the training data

Accepted Irreducible Error Equation

error = bias² + variance + Epsilon

SVM Kernel

pre-processing on the data

SVM class_weight

if you have more of one class

SVM max_iter

how long the model can look for a solution

Naive Bayes

assume conditional independence

SVM

linear solver, can use kernel

Decision Tree

arbitrary depth

good at non-lineraity

entropy-based

Logistic Regression

• probability-based

• weight optimization

• sigmoid function

Multi-Layer Perceptron

• arbitrary size

• backpropagation and gradient descent

• customization

Train Data

fit models directly

Validation Data

judge how to adjust model complexity

Test Data

final evaluation of model performance

Validation Set

• tune the model hyperparameters

• either separate from training set or a slice of training set

Test Set

• unbiased gauge of model performance on completely unseen data

• final determination of model performance

• only one

k-Fold Cross Validation (CV)

• split training set into K distinct validation sets

Stratification

preserving the ratio/distribution of data across splits

Decision Tree Strategies

• limit tree growth

• prune the tree

• random forest

Taming Complexity works with _

• decision trees

• logistic regression

• MLP

What is an indicator of overfitting / problems

large weights because of high variance

Regularization

the idea that we can prevent more complex parametric models from overfitting by constraining the weights

• guards against overfitting, giving us a superior test set MSE/MAE. does not find the best loss minima (in ridge loss)

Objective Function

expansion of the loss function that can include additional terms that have nothing to do with the ground-truth labels or outputs

• such as regularization penalties

Ridge Regression

• Tikhonov regularization

• penalize sum of squares

• push the weights towards lower numerical values

• sub/superscript (exp & root)

Lasso Regression

• used less than ridge

• penalize absolute values

• tends to engender sparsity where a subset of weights are pushed to be zero

• only subscript (just abs())

lambda

• hyperparameter

• controls how important the regularization term is compared to the loss term

What is the closed form solution with regularization?

pairing MSE/L2 loss with ridge/L2 regularization

Ensembling

exploiting the overfitting tendency of some models

Random Trees

ensemble supervised machine learning algorithm that builds many individual decision trees to make more accurate and stable predictions than a single tree

Ensembling + Cross Validation

1. use k-fold CV to pick some optimal hyperparameters

2. make a set of K learner models, trees, regressors, that train on the K training splits we used for CV

Cross Validation

1. use k-fold CV to pick some optimal hyperparameters

2. re-train optimal model design on the full training dataset

3. evaluate on test set

What is the first step to learning the weights?

• picking your loss

• to make better weights, you must quantify how they’re wrong

QR Decomposition

• used for full-rank matrices

• less intense than original calculation but still yields same result

Singular Value Decomposition

• used for low-rank & ill-conditioned matrices

• scikit-learn will check

Gradient Descent

• simplest form of iterative function optimization

• solves linear regression w/ MAE loss

• solves logistic regression

The Learning Rate n (long n)

determines how far we move with each update