Overfitting and Regularization in Neural Networks

Vocabulary flashcards covering core topics on neural network overfitting, early stopping, data augmentation, inductive bias, L1/L2 regularization, weight decay, and dropout.

Early Stopping

A regularization technique in neural network training where training is interrupted as soon as the validation loss stops decreasing, preventing the model from fitting input noise and entering the overfitting region.

Training-Validation Curves

Plots tracking training error and validation error across training epochs, used to monitor the learning state and identify underfitting, optimal performance, or overfitting.

Data Augmentation

The process of generating artificial data points ($N' > N$) from an existing dataset by applying label-preserving transformations, making overfitting significantly harder for the model.

SpecAugment

A simple data augmentation method designed for automatic speech recognition that operates directly on speech spectrograms using time warping, frequency masking, and time masking.

Inverse Problem (in Machine Learning)

An intrinsically ill-posed task of inferring a complete probability distribution from a finite set of samples, as infinitely many distributions could produce the observed target values.

Inductive Bias

Prior background knowledge or deliberate algorithmic preferences built into a learning system to constrain the search space of candidate functions and select a single solution.

No Free Lunch Theorem

A theoretical result published by Wolpert (1996) stating that all algorithms perform equally well when averaged over all possible problems, proving that learning purely from data without inductive bias is impossible.

L2 Regularization

A regularization technique that adds a penalty proportional to the sum of squared weights ($\frac{1}{2} \rho \frac{\beta}{1} \times \text{loss} + \frac{1}{2} \rho \times \text{norm}$ or $\frac{\rho}{2} \times \text{loss} + \frac{1}{2} \times \text{norm}$ or $\frac{1}{2} \times \text{norm}$) to the objective function, typically combined with minibatch SGD.

Weight Decay

The steady reduction of neural network weights at each SGD iteration resulting from L2 regularization, causing weights that receive small gradient updates to progressively shrink toward zero.

L1 Regularization

A regularization method adding the sum of absolute weight values to the loss function, which enforces sparsity in the model parameters and acts as a mechanism for feature selection.

Model Ensembling

A technique to reduce overfitting by combining predictions from multiple distinct models (e.g., via weighted averaging), which is computationally demanding for large neural networks.

Co-adaptation

An undesirable phenomenon during overfitting where individual network units adapt specifically to compensate for mistakes made by other units, creating complex, brittle interdependencies.

Dropout

A regularization method that randomly drops hidden and input units along with their connections during training with probability $1 - p$ at each epoch, preventing co-adaptation and approximating model ensembling.