8 Validation overfitting

What is underfitting?

A model is too restricted to capture relevant structure, so both training and held-out error can be high.

What is overfitting?

A flexible model captures training-specific variation that does not transfer. Training error can improve while held-out error worsens.

What does the familiar error-versus-model-complexity diagram illustrate?

Training error tends to decrease with flexibility; held-out error may first improve and later worsen. The smooth U shape is illustrative, not guaranteed for every measured split.

What does increasing model complexity correspond to in KNN versus K-means?

KNN generally becomes more locally flexible as K decreases. In K-means, increasing K asks for more groups. The same letter does not mean the same modeling choice.

What is the bias-variance tradeoff?

Restrictive procedures can miss structure through systematic bias; flexible procedures can vary strongly across training samples. Good prediction balances them with irreducible noise.

How do parameters and hyperparameters differ?

Parameters are fitted quantities such as coefficients or mixture means. Hyperparameters control modeling/training choices, such as K, a penalty or a maximum interaction degree.

What are training, validation and test data used for?

Training fits parameters; validation guides model/hyperparameter choices; an untouched test set assesses the selected procedure after choices are complete.

Can a dataset remain an untouched test set after selecting the best K on it?

No. Using its results to choose K makes it validation data, regardless of its variable name. A separate assessment is needed for an untouched estimate.

What does cross-validation rotate?

It rotates held-out validation folds within the development data. Every fold’s preprocessing and supervised selection must use only that fold’s training observations.

Why can feature selection on the full dataset invalidate later cross-validation?

The validation labels already influenced the selected representation. Repeat supervised selection inside each training fold to evaluate the full procedure honestly.

What is nested evaluation intended to separate?

Inner model/hyperparameter selection from outer assessment of the selection procedure. Reporting the best inner score alone can be optimistic.

Does stratified splitting solve class imbalance?

No. It approximately preserves class proportions across splits; it does not create equal class sizes or guarantee strong positive detection.

Why should forecasting evaluation usually respect chronological order?

Future predictions must use information available at the forecast origin. Random splits can permit future patterns or information to influence evaluation of earlier cases.

Why can K = 1 give a perfect KNN fit on the training observations?

If a training query is allowed to match itself, its own label/response is available as its nearest neighbor. This is resubstitution, not evidence of perfect generalization.

Does changing K from 1 to 2 remove self-neighbor contamination?

No. The query’s own response can still contribute. Hold the query out or use a procedure that explicitly excludes it.

Does a slightly better test score than training score automatically prove an error?

No. Finite-sample variation or different sample difficulty can produce it. Inspect the evaluation design rather than treating one expected trend as an absolute law.