Ch5: Sequential Models

What is a sequence model?

A machine learning model that inputs or outputs sequences of data.

Give 3 examples of sequential data.

Text streams, audio clips, time-series data, video clips, weather data.

How is RNN different from a traditional neural network?

It output depends on prior elements in the sequence (has memory). Traditional NN inputs/outputs are independent.

Does RNN share parameters across layers?

Yes, parameters are reused/looped across each time step.

Name the 4 types of RNN architectures.

One to One, One to Many, Many to One, Many to Many

What type of RNN is used for image captioning?

One to Many (single image input → many words output)

What type of RNN is used for sentiment analysis?

Many to One (many words input → single sentiment output)

What type of RNN is used for machine translation?

Many to Many (many input words → many output words)

What type is used for music generation?

One to Many

What are the 3 main drawbacks of RNN?

Slow computation, short-term memory only, vanishing/exploding gradients

What is the vanishing gradient problem?

Gradients become very small during backpropagation through time, causing the model to stop learning.

What causes vanishing gradients?

Gradients < 1 multiplied repeatedly during chain rule → shrink exponentially.

What is the exploding gradient problem?

Gradients become very large and crash the model.

What happens when gradients are > 1?

They get exponentially larger and eventually blow up the model.

why does LSTM exist?

to prevent vanishing and exploding gradient problems.

What is the key difference between RNN and LSTM?

LSTM has a memory cell that can hold information for extended periods.

How many interacting layers does LSTM have?

4

Name the 3 gates in an LSTM cell.

Input gate, Forget gate, Output gate

What does the forget gate do?

Controls what information is removed from the memory cell.

What does the input gate do?

Controls what information is added to the memory cell.

What does the output gate do?

Controls what information is output from the memory cell.

Name 3 applications of LSTM.

Language translation, speech recognition, time series prediction, music generation, sentiment analysis.

How many gates does GRU have and what are they?

2 gates; update gate and reset gate.

Does GRU have a separate cell state (Cₜ)?

No. GRU only has hidden state (hₜ).

What is the advantage of GRU over LSTM?

Simpler architecture → faster training time.

Does GRU solve the vanishing gradient problem?

Yes.

How does a bidirectional RNN differ from a standard RNN?

Standard RNN reads left to right; bidirectional reads left to right AND right to left.

When is bidirectional RNN useful?

When context from both directions is needed (e.g., sentiment analysis, speech recognition).

What two types of neural networks are used in image captioning?

CNN (encoder for images) + RNN/LSTM/Transformer (decoder for text)

What is the role of the encoder in image captioning?

Extracts visual features from the input image (using CNN like ResNet).

What is the role of the decoder in image captioning?

Generates caption text word by word (using RNN/LSTM/Transformer).

How does an LSTM generate new text?

Takes a seed sequence, predicts next character/word, appends it, repeats.

What loss function is commonly used for text generation training?

Categorical cross-entropy.

After training, what is fed back into the model during generation?

The predicted character/word becomes part of the input for the next prediction.