CPSC 330 Lecture 8

Hyperparameter Optimization

Manual hyperparameter optimization

pros: we may have some intuition about what might work

cons:

• it takes a lot of work

• not reproducible

• our intuition might be worse than a data-driven approach in very complicated cases

Automated hyperparameter optimization

• Formulate the hyperparameter optimization as one big search problem

Optimization bias of parameter learning

• overfitting of the training error

Optimization bias of hyper-parameter learning

• Overfitting of the validation error

• One out of 100 might score high due to random noise rather than true generalization capability.

• The validation accuracy or score reported will be higher than the model's actual performance on completely unseen test data.