GRU, Attention, and Transformer Fundamentals

Housekeeping & Course Logistics

• Mid-term is “coming up” (later fixed to next week, 6:30 – 9:10 pm, 3-hour in-class exam).
  • Coverage: CNNs, RNN, GRU, LSTM, linear regression, residual connections, soft-max+cross-entropy, forward/backward propagation, receptive field, same-padding, max-pooling, Transformer intro.
  • Students should derive soft-max & cross-entropy gradients, know gate counts, be able to hand-compute forward/ backward passes for small nets.

LSTM vs. GRU (Motivation)

• LSTM: three gates (Forget, Input, Output) + separate cell state → powerful but computationally heavy.

• GRU: authors felt LSTM “too complicated”; merge/ remove gates to reduce parameter count & time complexity.

• Common interview Q:
  • LSTM = 1 input, 2 outputs (hidden state & cell state).
  • GRU = 1 input, 1 output (hidden state only).

• Activation functions:
  • $\sigma$ (sigmoid) → gate/forget decisions.
  • $\tanh$ → squeeze range to $[-1,1]$, mitigate vanishing/ exploding gradients.

Detailed GRU Walk-Through

High-level structure

• Inputs: current symbol $x<em>t$ & previous hidden state $h</em>{t-1}$.

• Outputs: new hidden state $h_t$.

• Two gates: Update (zt) & Reset (rt).
  • Update replaces combined role of LSTM’s Forget+Input gates.
  • Reset controls how much past information to drop when computing candidate state.

Mathematical Form (PyTorch style)

• Reset gate:
  $r<em>t=\sigma\bigl(W</em>{xr}x<em>t + b</em>{xr} + W<em>{hr} h</em>{t-1}+b_{hr}\bigr)$

• Update gate:
  $z<em>t=\sigma\bigl(W</em>{xz}x<em>t + b</em>{xz} + W<em>{hz} h</em>{t-1}+b_{hz}\bigr)$

• Candidate hidden:
  $\tilde h<em>t = \tanh\bigl(W</em>{xh}x<em>t + b</em>{xh} + r<em>t \odot (W</em>{hh}h<em>{t-1}+b</em>{hh})\bigr)$
  (Hadamard/element-wise product $\odot$ = “Hammard product” in lecture wording).

• New hidden:
  $h<em>t = (1-z</em>t) \odot h<em>{t-1} + z</em>t \odot \tilde h_t$
  (Intuition: blend old memory & new candidate according to update gate.)

• Fewer heavy $\tanh$/$\sigma$ calls vs LSTM → fewer tensor ops ⇒ faster.

RNN Family Limitations

• Sequential dependency ⇒ little parallelism; long sequences strain GPU memory & lengthen training.

• Bidirectional RNNs exist but rarely used in industry (“bidirectional RNN is dead” per instructor).

Attention & Transformer Revolution

Paper: “Attention Is All You Need” (Vaswani et al., 2017)

• Citation count: 188,409 (as quoted).
  • Authored by Google Research / Google Brain; influenced by Geoffrey Hinton.

• Goal: discard recurrence & convolution; rely solely on self-attention → massive parallelism on GPUs.

• Achievements:
  • WMT 2014 En→Fr BLEU = 41.0; En→De = 28.4.
  • Trained in 3.5 days on 8× P100 vs weeks for RNN baselines.

Encoder–Decoder Overview

[Input sentence] → Embedding + Positional Encoding
               → N stacked Encoder blocks
Encoder output Z --------------→ consumed by Decoder
[GO] token → shifted-right outputs → N stacked Decoder blocks
                               → Soft-max → next-word probs

• Positional encoding needed because attention is content-based; author used sinusoidal $\sin,\cos$ scheme.

• Auto-regressive decoding: model consumes its own previous outputs; training uses teacher forcing with right-shift + causal mask.

Single Transformer Block Internals

1. Multi-Head Self-Attention
   • Each head computes $\text{Attention}(Q,K,V)=\text{Softmax}\bigl((QK^T)/\sqrt{d<em>k}\bigr)V$ • Uses Query/Key/Value linear projections. • Scaling factor $1/\sqrt{d</em>k}$ stabilizes gradients.

2. Add & Norm (residual + layer norm).

3. Position-wise Feed-Forward network (two dense layers, ReLU/GeLU).

4. Add & Norm again.

• Stacked N = 6 encoders & N = 6 decoders in original paper.

Self-Attention Intuition (class whiteboard example)

• Sentence: “The girl kicked the ball.” Focus word = “kicked” (query).

• Keys = {the, girl, kicked, the, ball}; compute similarity → soft-max weights.
  • High with “girl” & “ball”, low with stop-words.

• Weighted sum (Value vectors) gives contextual representation for “kicked”.

Masked Multi-Head Attention in Decoder

• Causal mask prevents “cheating” by hiding future positions during training.

• Enables parallel computation over sequence while preserving auto-regressive semantics.

Positional Encoding Details

• Original Transformer uses deterministic sinusoid:
  $\text{PE}<em>{(pos,2i)}=\sin(pos/10000^{2i/d</em>{model}})$
  $\text{PE}<em>{(pos,2i+1)}=\cos(pos/10000^{2i/d</em>{model}})$

• Alternative methods (learned embeddings, rotary, etc.) introduced later (BERT, GPT-Neo, RoPE).

Practical Engineering Notes

• Word embeddings: word2vec, GloVe, fastText common; modern models learn embeddings end-to-end.

• Masking, shifting, teacher-forcing implemented by feeding token and padding future tokens with −∞ before soft-max.

• Soft-max acts as multi-class classifier; outputs probability distribution over vocabulary.

Large-Language-Model Ecosystem

• GPT-1 (paper available) → GPT-2/3 architecture unreleased; companies expose via API.

• Fine-tuning tricks: LoRA / QLoRA (low-rank adaptation) factorize weight updates to cut GPU memory; mandatory reading for NLP interviews.

Research & Industry Context

• Google maintains strong AI research (Transformer, PageRank, Gemini 2.5, Colab, Gmail 4 GB origin story).
  • Baidu once led search efficiency but censored data hampered progress.

• GPUs became standard after 2012 AlexNet breakthrough; parallel computation central to Transformer training.

Exam Study Checklist (as per instructor)

• Derive forward/backward pass for:
  • Simple dense layer $y=Wx+b$
  • Soft-max + cross-entropy
  • CNN conv+pool; compute receptive field & same-padding output size.
  • Residual connections gradient flow.

• Know gate counts & equations for RNN/LSTM/GRU.

• Explain vanishing/exploding gradients & $\tanh$ role.

• Sketch Transformer encoder/decoder; define Query/Key/Value.

• Explain positional encodings & causal mask.

• Outline LoRA factorization idea.