1/24
Vocabulary practice flashcards covering multi-layer feed-forward neural networks, activation functions, the forward pass, and backpropagation based on Module 7.
Neuron (or Unit)
A fundamental computational block that computes a weighted sum of its inputs plus a bias and passes the result through an activation function: a=ϕ(∑jwjxj+b).
Feed-Forward Neural Network
A stack of layers of units where each unit computes its output from the outputs of the previous layer, and its output becomes an input to the next layer.
Input Layer
The layer holding the raw feature values fed into the network, which performs no computation of its own.
Hidden Layer
Any layer between the input and output layers that computes learned, non-linear combinations of the inputs to bend the feature space.
Output Layer
The final layer of a neural network that produces the prediction, configured with an activation function suited to the specific machine learning task.
Activation Function
The non-linear mathematical operation applied to a unit's weighted sum (z), enabling stacked layers to bend decision boundaries and preventing them from collapsing into a single linear map.
Sigmoid
An S-shaped activation function defined as ϕ(z)=1+e−z1 that maps any real number to an output probability between 0 and 1.
Sigmoid Derivative (Slope)
The derivative of the sigmoid function, given by σ′(z)=σ(z)(1−σ(z)), which attains a maximum value of 0.25 at z=0 and approaches 0 at the tails.
ReLU (Rectified Linear Unit)
A non-linear activation function defined as max(0,z) that outputs the input if it is positive and 0 otherwise, commonly used for hidden layers in deep networks.
tanh (Hyperbolic Tangent)
An activation function defined as ϕ(z)=ez+e−zez−e−z that maps real-valued inputs to an output range from −1 to 1, with a derivative of 1−tanh2(z) (at most 1).
Linear Collapse
The mathematical principle where adding successive layers without non-linear activations adds no expressive power, collapsing algebraically into a single linear transformation: W2(W1x+b1)+b2=(W2W1)x+(W2b1+b2).
Forward Pass
The process of computing network predictions from input to output by sequentially evaluating the weighted sum z(l)=W(l)a(l−1)+b(l) and activation a(l)=ϕ(z(l)) at each layer.
Weight Grid
A matrix representation of layer weights where multiplying the input vector by the grid computes every weighted sum in the layer simultaneously.
Backpropagation
An algorithm that computes the gradient ∂w∂L of the loss with respect to every weight in a single backward sweep from output to input using the chain rule.
Delta (δ)
A unit's calculated share of the blame for the loss, defined mathematically as the partial derivative of the loss with respect to its pre-activation weighted sum: δj=∂zj∂L.
Weight Gradient (∂w∂L)
The rate at which the overall loss changes with respect to a specific weight, computed during backpropagation as the receiving unit's delta multiplied by the incoming activation: ∂wji∂L=δjai.
Chain Rule
The calculus rule stating that if y depends on u and u depends on x, the overall rate of change is the product of the intermediate slopes: dxdy=dudy⋅dxdu.
Softmax
An output layer activation function defined as yk=∑mezmezk used in multi-class classification to normalize raw outputs into probabilities that sum to 1.
Multi-Label Classification Output Layer
An output layer architecture using independent sigmoid activations (yk=σ(zk)) on multiple units to predict separate probabilities for non-mutually exclusive classes.
Binary Classification Output Layer
An output layer consisting of a single unit with a sigmoid activation (y=σ(z)) trained with binary cross-entropy loss to output a probability between 0 and 1.
Regression Output Layer
An output layer configuration consisting of a single unit with no activation function (y=z) trained with squared error loss to output any real number.
Automatic Differentiation
A framework capability that records the forward computation and automatically calculates the gradients ∂w∂L for all parameters during backpropagation.
XOR Problem
A non-linearly separable classification problem where the output is 1 only when exactly one of two binary inputs is 1, which a single linear unit cannot solve without a hidden layer.
Layer Parameter Count Formula
The formula P=nout(nin+1) used to calculate the total number of trainable weights and biases in a dense layer receiving nin inputs and containing nout units.
Learning Rate (η)
A hyperparameter that determines the step size taken in the direction of the negative gradient during parameter updates: w←w−η∂w∂L.