Module 7: Neural Networks and Backpropagation

0.0(0)
Studied by 0 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/24

flashcard set

Earn XP

Description and Tags

Vocabulary practice flashcards covering multi-layer feed-forward neural networks, activation functions, the forward pass, and backpropagation based on Module 7.

25 Terms

1
New cards

Neuron (or Unit)

A fundamental computational block that computes a weighted sum of its inputs plus a bias and passes the result through an activation function: a=ϕ(∑jwjxj+b)a = \phi\left(\sum_j w_j x_j + b\right).

2
New cards

Feed-Forward Neural Network

A stack of layers of units where each unit computes its output from the outputs of the previous layer, and its output becomes an input to the next layer.

3
New cards

Input Layer

The layer holding the raw feature values fed into the network, which performs no computation of its own.

4
New cards

Hidden Layer

Any layer between the input and output layers that computes learned, non-linear combinations of the inputs to bend the feature space.

5
New cards

Output Layer

The final layer of a neural network that produces the prediction, configured with an activation function suited to the specific machine learning task.

6
New cards

Activation Function

The non-linear mathematical operation applied to a unit's weighted sum (zz), enabling stacked layers to bend decision boundaries and preventing them from collapsing into a single linear map.

7
New cards

Sigmoid

An S-shaped activation function defined as ϕ(z)=11+e−z\phi(z) = \frac{1}{1 + e^{-z}} that maps any real number to an output probability between 00 and 11.

8
New cards

Sigmoid Derivative (Slope)

The derivative of the sigmoid function, given by σ′(z)=σ(z)(1−σ(z))\sigma'(z) = \sigma(z)(1 - \sigma(z)), which attains a maximum value of 0.250.25 at z=0z = 0 and approaches 00 at the tails.

9
New cards

ReLU (Rectified Linear Unit)

A non-linear activation function defined as max⁡(0,z)\max(0, z) that outputs the input if it is positive and 00 otherwise, commonly used for hidden layers in deep networks.

10
New cards

tanh (Hyperbolic Tangent)

An activation function defined as ϕ(z)=ez−e−zez+e−z\phi(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}} that maps real-valued inputs to an output range from −1-1 to 11, with a derivative of 1−tanh⁡2(z)1 - \tanh^2(z) (at most 11).

11
New cards

Linear Collapse

The mathematical principle where adding successive layers without non-linear activations adds no expressive power, collapsing algebraically into a single linear transformation: W2(W1x+b1)+b2=(W2W1)x+(W2b1+b2)W_2(W_1\mathbf{x} + \mathbf{b}_1) + \mathbf{b}_2 = (W_2 W_1)\mathbf{x} + (W_2 \mathbf{b}_1 + \mathbf{b}_2).

12
New cards

Forward Pass

The process of computing network predictions from input to output by sequentially evaluating the weighted sum z(l)=W(l)a(l−1)+b(l)\mathbf{z}^{(l)} = W^{(l)}\mathbf{a}^{(l-1)} + \mathbf{b}^{(l)} and activation a(l)=ϕ(z(l))\mathbf{a}^{(l)} = \phi\left(\mathbf{z}^{(l)}\right) at each layer.

13
New cards

Weight Grid

A matrix representation of layer weights where multiplying the input vector by the grid computes every weighted sum in the layer simultaneously.

14
New cards

Backpropagation

An algorithm that computes the gradient ∂L∂w\frac{\partial L}{\partial w} of the loss with respect to every weight in a single backward sweep from output to input using the chain rule.

15
New cards

Delta (δ\delta)

A unit's calculated share of the blame for the loss, defined mathematically as the partial derivative of the loss with respect to its pre-activation weighted sum: δj=∂L∂zj\delta_j = \frac{\partial L}{\partial z_j}.

16
New cards

Weight Gradient (∂L∂w\frac{\partial L}{\partial w})

The rate at which the overall loss changes with respect to a specific weight, computed during backpropagation as the receiving unit's delta multiplied by the incoming activation: ∂L∂wji=δjai\frac{\partial L}{\partial w_{ji}} = \delta_j a_i.

17
New cards

Chain Rule

The calculus rule stating that if yy depends on uu and uu depends on xx, the overall rate of change is the product of the intermediate slopes: dydx=dydu⋅dudx\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}.

18
New cards

Softmax

An output layer activation function defined as yk=ezk∑mezmy_k = \frac{e^{z_k}}{\sum_m e^{z_m}} used in multi-class classification to normalize raw outputs into probabilities that sum to 11.

19
New cards

Multi-Label Classification Output Layer

An output layer architecture using independent sigmoid activations (yk=σ(zk)y_k = \sigma(z_k)) on multiple units to predict separate probabilities for non-mutually exclusive classes.

20
New cards

Binary Classification Output Layer

An output layer consisting of a single unit with a sigmoid activation (y=σ(z)y = \sigma(z)) trained with binary cross-entropy loss to output a probability between 00 and 11.

21
New cards

Regression Output Layer

An output layer configuration consisting of a single unit with no activation function (y=zy = z) trained with squared error loss to output any real number.

22
New cards

Automatic Differentiation

A framework capability that records the forward computation and automatically calculates the gradients ∂L∂w\frac{\partial L}{\partial w} for all parameters during backpropagation.

23
New cards

XOR Problem

A non-linearly separable classification problem where the output is 11 only when exactly one of two binary inputs is 11, which a single linear unit cannot solve without a hidden layer.

24
New cards

Layer Parameter Count Formula

The formula P=nout(nin+1)P = n_{\text{out}}(n_{\text{in}} + 1) used to calculate the total number of trainable weights and biases in a dense layer receiving ninn_{\text{in}} inputs and containing noutn_{\text{out}} units.

25
New cards

Learning Rate (η\eta)

A hyperparameter that determines the step size taken in the direction of the negative gradient during parameter updates: w←w−η∂L∂ww \leftarrow w - \eta \frac{\partial L}{\partial w}.