Deep Learning: Fundamentals, CNNs, and RNN Architectures

0.0(0)
Studied by 0 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/34

flashcard set

Earn XP

Description and Tags

Vocabulary practice flashcards reviewing core deep learning concepts including model capacity, bias-variance trade-off, optimization via SGD, feedforward nets, activation and loss functions, CNN operations, LSTM gates, and RNN design patterns.

35 Terms

1
New cards

Capacity

The range of functions that a model can approximate, reflecting its ability to learn different patterns from data.

2
New cards

Width

The number of nodes contained within a single layer of a neural network model.

3
New cards

Depth

The total number of layers present in a neural network model.

4
New cards

Overfitting

A condition where a model performs very well on training data but poorly on unseen test data by memorizing training data rather than generalizing patterns.

5
New cards

Underfitting

A condition where a model fails to learn patterns in the training data properly and cannot generalize well to new data, resulting from an overly simple model.

6
New cards

Reducible Errors

Errors in machine learning that can be minimized to improve model accuracy; they are categorized into bias and variance.

7
New cards

Irreducible Errors

Inherent model errors that cannot be reduced because they stem from unknown, uncontrollable variables.

8
New cards

Bias

The difference between a model's predicted values and actual values, representing an algorithm's inability to capture true underlying data relationships due to excessive assumptions.

9
New cards

Variance

The amount of variation observed in a model's predictions when trained across different training datasets.

10
New cards

Bias-Variance Trade-off

The process of balancing bias and variance errors to find an optimal sweet spot that minimizes both underfitting and overfitting in supervised learning.

11
New cards

Stochastic Gradient Descent (SGD)

An optimization algorithm that minimizes the loss function by iteratively updating model parameters using one randomly selected training example at a time.

12
New cards

Deep Feed Forward Network

An artificial neural network (also called a Multilayer Perceptron) where data strictly moves forward from the input layer through one or more hidden layers to the output layer without looping backwards.

13
New cards

Rectified Linear Unit (ReLU)

A non-linear activation function defined as f(x)=max⁡(0,x)f(x) = \max(0, x), which outputs the input directly if positive and outputs zero if negative.

14
New cards

Leaky ReLU

A non-linear activation function variant that allows a small, non-zero negative value when the input is negative instead of returning zero.

15
New cards

Parametric ReLU (PReLU)

An activation function that learns the slope parameter for negative input values during training rather than keeping it fixed.

16
New cards

Exponential Linear Unit (ELU)

A non-linear activation function that generates smooth, continuous negative values for inputs below zero.

17
New cards

Scaled Exponential Linear Unit (SELU)

A non-linear activation function designed to induce self-normalizing properties and maintain stable values across deep network layers.

18
New cards

Loss Function

A function that measures the prediction error for a single individual data point.

19
New cards

Cost Function

A function that calculates the aggregate prediction error across an entire dataset by combining all individual loss values.

20
New cards

Mean Squared Error (MSE)

A regression loss function that squares the differences between actual and predicted values and averages them, making it sensitive to outliers.

21
New cards

Mean Absolute Error (MAE)

A regression loss function that calculates the average of the absolute differences between actual and predicted values, offering greater robustness to outliers than MSE.

22
New cards

Huber Loss

A regression loss function combining properties of MSE and MAE to achieve outlier robustness while maintaining smoothness.

23
New cards

Binary Cross-Entropy

A classification loss function used for binary (two-class) problems, typically used in conjunction with a Sigmoid activation in the final neuron.

24
New cards

Categorical Cross-Entropy

A multi-class classification loss function used with one-hot encoded target vectors, typically paired with a Softmax activation in the output layer.

25
New cards

Sparse Categorical Cross-Entropy

A multi-class classification loss function that accepts integer labels instead of one-hot encoded vectors, providing faster computation.

26
New cards

Convolutional Neural Network (CNN)

A specialized deep learning architecture designed for image processing that automatically extracts spatial features such as edges, textures, and objects.

27
New cards

Transposed Convolution

A convolutional operation used for up-sampling that converts smaller feature maps into larger ones to reconstruct or generate images.

28
New cards

Dilated Convolution

Also called Atrous Convolution; an operation that expands the receptive field by skipping pixels between filter elements without increasing parameter count.

29
New cards

Long Short-Term Memory (LSTM)

A recurrent neural network variant engineered with gating mechanisms to retain long-term dependencies and resolve the vanishing gradient problem.

30
New cards

Cell State

The memory pathway in an LSTM network that carries and preserves long-term information across time steps.

31
New cards

Forget Gate

An LSTM gating component that employs a Sigmoid function to output values between 00 and 11, determining which old information to discard from the cell state.

32
New cards

Input Gate

An LSTM gating component utilizing Sigmoid and Tanh activations to decide which new information should be stored in the cell state.

33
New cards

Output Gate

An LSTM gating component that regulates which parts of the cell state are emitted as the final hidden state or output.

34
New cards

Bidirectional RNN

A recurrent architecture that concurrently deploys forward and backward RNN layers to process sequential data using both past and future contextual information.

35
New cards

Sequence-to-Sequence (Seq2Seq) RNN

An RNN framework featuring an encoder that compresses an input sequence into a context vector and a decoder that generates a new sequence from that vector.