Finding Good Models

hyperparameters

Often, one considers not only a single individual model, but a whole family of models.

- This results in additional parameters, which are not optimized during training, and

which are called hyperparameters

hypothesis space

The space of functions the model is allowed to pick from

model capacity

the capacity of a model means its ability to fit to a wide variety of functions

the effective capacity (what we actually can fit in practice) is smaller than the representational capacity (what we could fit in theory)

optimizing hyperparams

usually, something is a hyperparameter because it is either very hard to optimize for (e.g., no gradient for integers), or because optimization leads to a useless result

underfitting and overfitting

Models with insufficient capacity are unable to solve complex tasks.

- Models with too high capacity will learn the noise in the training data, leading to bad

predictions.

Why you cannot select hyperparameters on the test set

optimising for this test set – introducing an optimistic bias

test set error is random variable

=> use a validationset from training set fpor hyperparam tuning

k fold cross validation

Each fold uses a different validation set. Validation error is computed for every fold,

and average validation error over all folds is used to assess performance.

k choice for k fold

Too small k: each training fold contains little data, which may cause underfitting within

each fold and gives an inaccurate estimate of the validation error.

- Larger k: training folds are larger and closer to the full dataset, giving a

lower-variance and generally better-calibrated error estimate.

- The extreme case k = L trains on all but one sample at a time and is called

leave-one-out cross-validation (LOOCV). It gives the lowest variance but is

computationally the most expensive.

- Practical recommendation: k = 5 or k = 10 is standard and works well in most

settings. Prefer larger k when data is very limited.

bias variance tradeoff

Bias decreases with capacity, while variance increases. At ideal capacity, we have the optimal trade-off

need to keep both bias and variance in check for a good estimator

bayes optimal predictor

the true function is the same as the Bayes-optimal predictor. In other words, for every x, the mean E[y | x] is equal to f (x)

bias var decomp

if we choose a model based on minimization of expected risk, we in fact minimize MSE!

minimising expected risk is equivalent to minimising MSE, which controls both bias and variance of the estimator simultaneously, just as desired.

regularization

Trade (a little bit of) bias for a (potentially dramatic) reduction in variance

ridge regression

to prevent extreme parameter values is to add another term to the lossfunction which penalizes the norm of the parameter vector,

regularizer weight

The regularizer weight λ ≥ 0 is a hyperparameter - optimization together with theloss does not make sense, as the optimizer would just choose λ = 0. Note that λ controls regularization strength independently of model capacity – they are two separate tools for controlling overfitting.

weight decay

it adds another term to the gradient descent which decreases (“decays”)parameter values in each descent step, proportional to how large they already are. Inparticular, overly large values of individual parameters become less attractive for the optimizer, reducing overfitting.

Lasso

-Unlike ridge regression, the Laplace prior places relatively more mass near zero. This encourages sparse solutions: many components of ˆθ are driven to exactly zero

-automatic feature seletcion

-has no closed-form solution and requires specialisedconvex optimisation algorithms.

early stopping

During training, the training error will gradually decrease. However, what often happens is that validation error first decreases, but then increases again (since we start to overfit).