Lecture 4: Sequence Modeling - Recurrent and Recursive Nets (RNNs)

Overview of Sequence Modeling and RNNs

• Context and Progression:

  • Previously, the course covered Multi-Layer Perceptrons (MLP), which are standard feedforward neural networks consisting of an input layer, one or more hidden layers, and an output layer.

  • CNNs (Convolutional Neural Networks) were discussed as tools to learn from grid-structured data, specifically images.

  • The current focus shifts to sequential data, where the order of arrivals is critical.

• Examples of Sequential Data:

  • Time Series: Sales figures, temperature readings, and sensor streams.

  • Text: Characters, words, and sentences.

  • Audio: Phonemes distributed over time.

  • Video: Frames presented in a specific temporal order.

• The Core Question: How can a neural network remember useful information from the past while processing new, incoming inputs?

Understanding Sequential Data

• The Importance of Order: In sequential problems, the interpretation of data depends on the sequence.

  • Fixed Input Setting: Denoted as $x = [x_1, x_2, x_3, x_4]$. The model observes all features simultaneously.

  • Sequential Setting: Denoted as $x^{(1)}, x^{(2)}, x^{(3)}, \dots, x^{(\tau)}$. The model receives data as a stream over discrete steps.

• Practical Implications of Order:

  • Text: In the phrase "the movie was not good," the final word "good" combined with "not" changes the entire sentiment interpretation.

  • Finance: Today's stock price is dependent on the recent history of prices.

  • Video: Identifying an action depends on observing motion across multiple consecutive frames.

• Requirements for a Good Sequence Model:

  • Memory: The ability to store past information.

  • Shared Logic: Using consistent logic across different time steps.

  • Flexibility: The ability to handle variable-length inputs.

Limitations of Plain Feedforward Networks

• Variable Input Length: Standard ANNs struggle because sentence lengths vary. Forcing a fixed length requires padding or truncation.

• Position-Specific Weights: ANNs learn separate rules for "word 1," "word 2," etc. This leads to poor generalization for longer sequences than those seen in training.

• The RNN Solution: Recurrent Neural Networks use parameter sharing across time, reusing the same transition rule at every step.

Core Architecture of Recurrent Neural Networks (RNNs)

• The Hidden State: An RNN maintains a hidden state ($h_t$) that acts as the model's memory.

• Typical Mathematical Form:

  • The state transition function: $h_t = f(h_{t-1}, x_t; \theta)$

  • The internal activation: $a_t = Wh_{t-1} + Ux_t + b$

  • The hidden state calculation: $h_t = \tanh(a_t)$

  • The output calculation: $o_t = Vh_t + c$

• Parameter Definitions:

  • $x_t$: Current input at time step $t$.

  • $h_t$: A summary of the past up to time step $t$.

  • $o_t$: The output generated at time step $t$.

  • $U, W, V$: Shared weight matrices reused for all time steps.

  • $b, c$: Bias vectors.

Visualizing and Unfolding RNNs

• Circuit View: A visualization where the hidden unit feeds back into itself, delayed by one time step. This creates a loop representing memory, allowing the model to process sequences of any length.

• Unfolding Through Time: The recurrent graph can be unfolded into a deep chain. Once unfolded, the RNN resembles a deep network where the depth corresponds directly to the sequence length.

The Power of Parameter Sharing

• Mechanism: The same matrices ($U, W, V$) are applied at every time step.

• Benefits:

  • Fewer Parameters: Significant reduction compared to having unique weights for every position.

  • Generalization: The model can process sequences of lengths it did not encounter during training.

  • Position Invariance: The model can detect a pattern (e.g., the year "2009") regardless of where it appears in a sentence (e.g., "I went to Japan in 2009" vs. "In 2009, I went to Japan").

Input–Output Patterns in RNNs

Numerical Examples of RNN Forward Passes

Example 1: Scalar Time Series

• Parameters: $h_t = \tanh(0.5 h_{t-1} + 1.0 x_t)$, $y_t = h_t$.

• Initial Conditions: $h_0 = 0$, $x_1 = 1, x_2 = 2, x_3 = 1$.

• Step 1: $h_1 = \tanh(0.5 \times 0 + 1 \times 1) = \tanh(1) \approx 0.762$

• Step 2: $h_2 = \tanh(0.5 \times 0.762 + 2) = \tanh(2.381) \approx 0.983$

• Step 3: $h_3 = \tanh(0.5 \times 0.983 + 1) = \tanh(1.492) \approx 0.904$

• Observation: The hidden state $h_t$ acts as a smoother version of the raw input because it accumulates history.

Example 2: Text Data (Words)

• Input Sequence: "I love AI"

• Embeddings: $x_1 = \begin{pmatrix} 1 \\ 0 \end{pmatrix}$, $x_2 = \begin{pmatrix} 0 \\ 1 \end{pmatrix}$, $x_3 = \begin{pmatrix} 1 \\ 1 \end{pmatrix}$.

• Weights: $W = \begin{pmatrix} 0.5 & 0 \\ 0 & 0.5 \end{pmatrix}$, $U = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix}$, $h_0 = \begin{pmatrix} 0 \\ 0 \end{pmatrix}$.

• Formula: $h_t = \tanh(Wh_{t-1} + Ux_t)$.

• Step 1 ("I"): $h_1 = \tanh(\begin{pmatrix} 1 \\ 0 \end{pmatrix}) \approx \begin{pmatrix} 0.762 \\ 0 \end{pmatrix}$.

• Step 2 ("love"): $h_2 = \tanh(\begin{pmatrix} 0.381 \\ 0 \end{pmatrix} + \begin{pmatrix} 0 \\ 1 \end{pmatrix}) = \tanh(\begin{pmatrix} 0.381 \\ 1 \end{pmatrix}) \approx \begin{pmatrix} 0.364 \\ 0.762 \end{pmatrix}$.

• Step 3 ("AI"): $h_3 = \tanh(W h_2 + U x_3) = \tanh(\begin{pmatrix} 0.182 \\ 0.381 \end{pmatrix} + \begin{pmatrix} 1 \\ 1 \end{pmatrix}) = \tanh(\begin{pmatrix} 1.182 \\ 1.381 \end{pmatrix}) \approx \begin{pmatrix} 0.828 \\ 0.881 \end{pmatrix}$.

• Final Utility: $h_3$ serves as the semantic summary of the entire sentence.

Example 3: Video Frames

• Process: CNN extracts features from frames; RNN processes the sequence of features.

• Input: $x_1 = 0.2, x_2 = 0.6, x_3 = 0.9$.

• Parameters: $h_t = \tanh(0.7h_{t-1} + 0.8x_t)$, $h_0 = 0$.

• Step 1: $h_1 = \tanh(0.16) \approx 0.159$

• Step 2: $h_2 = \tanh(0.7 \times 0.159 + 0.8 \times 0.6) = \tanh(0.591) \approx 0.531$

• Step 3: $h_3 = \tanh(0.7 \times 0.531 + 0.8 \times 0.9) = \tanh(1.092) \approx 0.798$

• Interpretation: Evidence of motion accumulates across the frames to recognize an action (e.g., waving).

RNN as a Probabilistic Sequence Model

• Modeling Sequences: RNNs model the joint probability $P(y_1, y_2, \dots, y_T)$.

• Chain Rule of Probability: $P(y_1, \dots, y_T) = \prod_{t=1}^{T} P(y_t | y_1, \dots, y_{t-1})$.

• Language Modeling: At each step, the model predicts the next word based on all previous words.

• Output Mechanism: $o_t = Vh_t + c$, and $\hat{y}_t = \text{softmax}(o_t)$, providing a probability distribution over the possible next tokens.

Training RNNs: Loss and Backpropagation

• Total Loss: Generally calculated as the sum of losses over all time steps: $L = \sum_{t=1}^{T} L^{(t)}$.

• Specific Loss: For classification tasks (like language modeling), the loss is typically the negative log-likelihood: $L^{(t)} = -\log p(y_t | x_1, \dots, x_t)$.

• Backpropagation Through Time (BPTT):

  • The RNN is unfolded into a computational graph.

  • Standard backpropagation is run on the unfolded graph.

  • Cost: Forward pass is sequential; memory requirements grow linearly with the sequence length.

Vanishing and Exploding Gradients

• The Problem: During training, gradients involve products of multiple Jacobians: $\frac{\partial h_t}{\partial h_{t-k}} = \prod_{i=t-k+1}^{t} \frac{\partial h_i}{\partial h_{i-1}}$.

• Vanishing Gradients: Occur if the product factors are mostly less than 1. This prevents the model from learning long-term dependencies.

• Exploding Gradients: Occur if factors are mostly greater than 1, leading to numerical instability.

• Simple Intuition:

  • If $h_t \approx 0.5 h_{t-1}$, then after 5 steps, influence scales by $0.5^{5} = 0.03125$ (only 3% remains).

  • If $h_t \approx 1.5 h_{t-1}$, then after 5 steps, influence scales by $1.5^{5} \approx 7.59$ (blows up).

Advanced Training Techniques and Variants

• Teacher Forcing: Used when previous outputs are fed as future inputs.

  • Training: The true "ground truth" token is fed as the previous input at each step to stabilize training.

  • Testing: True future tokens are unavailable; the model must use its own predicted output.

  • Issue: This leads to a train/test mismatch where errors can accumulate during testing.

• Bidirectional RNNs: Address the limitation that standard RNNs only see past context. Uses two RNNs—one left-to-right and one right-to-left. Their hidden states are combined to provide both past and future context.

• Encoder–Decoder (Sequence-to-Sequence): Handles cases where input and output lengths differ (e.g., Machine Translation).

  • Encoder: Reads the input sequence and compresses it into a fixed-size context vector ($c$).

  • Decoder: Generates the output sequence starting from the context vector ($c$).

Recursive Nets vs. Recurrent Nets

• Recurrent Nets: Process sequences. The hidden state evolves over linear time. Best for chains (text, speech, time series).

• Recursive Nets: Process tree structures. They combine child nodes into parent representations. Best for hierarchical structures like parse trees.

Domain Applications Summary

• Time Series: Sensor values, price forecasting, anomaly detection.

• Text: Next-word prediction, tagging, sentiment analysis.

• Speech: Transcription of acoustic frames.

• Video: Action recognition and captioning.

• Biology: DNA/Protein sequence classification and motif detection.

Limitations of Vanilla RNNs

• Difficulty learning long-term dependencies.

• Sequential computation makes it hard to parallelize processing.

• Unstable training on very long sequences.

• The context vector in encoder-decoder structures acts as a memory bottleneck.

• Note: These limitations directly motivated the development of LSTM, GRU, and eventually Transformers.