Neural Networks & How to Train Them - Vocabulary

Vocabulary terms and definitions from the introductory guide on neural networks and model training.

Neural Network

A parameterized function that transforms an input into an output through connected layers of computation.

Generalization

The central goal of neural network training: achieving useful predictions on new examples from the intended deployment setting, rather than simply memorizing the training set.

Weight

A parameter $w_j$ that controls an input connection to a neuron.

Bias

A parameter $b$ that shifts a neuron's pre-activation independently of the input.

Hyperparameters

Configurations for the model or training process (such as layer width, number of layers, learning rate, batch size, and regularization strength) that are selected using validation results rather than updated by the learning algorithm.

Forward Propagation

The process of evaluating neural network layer operations in sequential order to produce a prediction, while retaining intermediate values for backpropagation.

ReLU

An activation function defined as $\text{ReLU}(z) = \text{max}(0, z)$, with a derivative of $1$ for $z > 0$ and $0$ for $z < 0$.

Sigmoid

An activation function defined as $\text{sigmoid}(z) = \frac{1}{1 + \text{exp}(-z)}$, which produces outputs in the range $(0, 1)$.

Tanh

An activation function with an output range of $(-1, 1)$ and a derivative of $1 - \text{tanh}(z)^2$.

Mean Squared Error (MSE)

A regression loss function defined as $\text{MSE} = \frac{1}{N} \times \text{sum}_i((\text{y}\_hat_i - y_i)^2)$ that penalizes large residuals especially strongly.

Logit

An unrestricted model score produced before conversion into a probability.

Softmax

A function that converts $K$ logits to probabilities summing to $1$ for mutually exclusive classes using $p_k = \frac{\text{exp}(z_k)}{\text{sum}_j(\text{exp}(z_j))}$.

Partial Derivative

A measure of local loss sensitivity when a specific parameter changes while all other parameters are held fixed.

Backpropagation

An algorithm that applies the chain rule from the loss back toward earlier layers to compute gradients by reusing intermediate derivatives.

Vanishing Gradients

A training phenomenon where repeated derivative products become very small, limiting learning in earlier layers.

Exploding Gradients

A training phenomenon where repeated derivative products become very large, destabilizing parameter updates.

Central Finite-Difference Check

A debugging technique that approximates $\frac{dL}{dw}$ using $\frac{L(w + \text{eps}) - L(w - \text{eps})}{2 \times \text{eps}}$ while holding other parameters fixed.

Mini-batch

The subset of training examples used to estimate a gradient for one parameter update step.

Epoch

One complete pass through the entire training dataset.

Adam

An adaptive optimization algorithm that tracks an exponential average of gradients $m_t$ and squared gradients $s_t$ to adjust parameter updates individually.

AdamW

An optimization algorithm variant that decouples weight decay from Adam's adaptive gradient calculation.

Data Leakage

A flaw occurring when training or model selection utilizes information that will be unavailable in the intended deployment prediction setting.

Underfitting

A condition where a model fails to capture useful patterns in the data, producing high training and validation error.

Overfitting

A condition where a model fits training-specific details that fail to transfer to new data, resulting in low training error but significantly worse held-out error.

Patience

In early stopping, the specified number of evaluations allowed without validation improvement before training is halted.

Inverted Dropout

A regularization layer where activations are multiplied by a random keep mask $r \thicksim \text{Bernoulli}(1-q)$ and scaled by $\frac{1}{1-q}$ during training, and left unmasked during evaluation.

Precision

A classification metric defined as $\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}$, measuring the fraction of positive predictions that are correct.

Recall

A classification metric defined as $\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}$, measuring the fraction of actual positive cases successfully identified.

F1 Score

A performance metric calculated as $F1 = \frac{2 \times \text{TP}}{2 \times \text{TP} + \text{FP} + \text{FN}}$, balancing precision and recall via their harmonic mean.

Calibration

The property assessing whether predicted probabilities correspond to real observed event frequencies in the target dataset.