E077 Huyen Ch2 — Understanding Foundation Models

Huyen: three design decisions that explain most differences between foundation models

Training data; model architecture and size; and post-training (how the model is aligned with human preferences). Sampling, often overlooked, also strongly shapes outputs. (pp. 49-50)

Common Crawl and C4

Common Crawl is a nonprofit's sporadic web crawl (about 2-3 billion pages a month in 2022-2023); C4 is Google's cleaned subset. Their quality is questionable (clickbait, misinformation, fake news), yet variants are used by most models that disclose sources, e.g., GPT-3 and Gemini. (pp. 50-51)

"Use what we have, not what we want": the risk, and why not just train on all data

Models do well on tasks present in available data but not necessarily on yours, so curate data for your needs. More data needs more compute and doesn't always help: a 1.3B model trained on 7B tokens of high-quality code beat much larger models (Gunasekar et al., 2023). (p. 51)

Low-resource languages and the world-to-Common Crawl ratio (Tables 2-1, 2-2)

English is about 45.88% of Common Crawl, 8x the next language (Russian, 5.97%). Languages with limited training data are low-resource. The ratio of world population share to Common Crawl share should ideally be 1; higher means more under-represented (Punjabi ≈ 231.56; English ≈ 0.40). (pp. 51-52)

Why isn't "translate everything to English and back" ideal?

1) You still need a model that understands the under-represented language well enough to translate it. | 2) Translation loses information, e.g., Vietnamese pronouns that encode the speakers' relationship collapse to "I" and "you". (p. 54)

Why can non-English use be slower and more expensive?

Latency and cost scale with tokens, and tokenization is less efficient for some languages. On MASSIVE, median token length was 7 for English, 32 for Hindi, and 72 for Burmese, so Burmese takes about 10x longer and costs about 10x more per token-billed API. (p. 55)

Domain-specific models: why needed, with examples

General models rarely do well on tasks they never saw in training. Drug discovery (protein, DNA, RNA data) and cancer screening (X-ray, fMRI; privacy-restricted) need curated datasets. Examples: AlphaFold (~100,000 proteins), NVIDIA BioNeMo, Google Med-PaLM2. (pp. 56-57)

Seq2seq and the two problems the transformer solved

Seq2seq (2014) uses an RNN encoder and decoder. Problems: (1) the decoder sees only the input's final hidden state (like answering from a book summary); (2) RNNs process input and output sequentially, which is slow for long sequences. (pp. 58-59)

Attention mechanism (Huyen's book analogy)

It lets the model weigh the importance of different input tokens when generating each output token, like answering by referencing any page in the book instead of only the summary. Attention predates the transformer (2014) but took off once the transformer showed it works without RNNs. (pp. 59-60)

Prefill vs decode (transformer inference)

Prefill: input tokens are processed in parallel, building the intermediate state (key and value vectors) needed for the first output token. Decode: output tokens are generated one at a time, the remaining sequential bottleneck. (p. 60) | Connection: prefill is like a parallel datapath while decode is a serial loop, the same split you face when pipelining an FPGA design.

Query, key, and value vectors (and multi-head attention)

Q is the decoder's current state (the person looking for info); K represents each previous token (the page number); V is that token's learned content (the page's content). Attention scores come from the Q·K dot product. Multi-head attention splits Q, K, V across heads, e.g., Llama 2-7B: 4096 / 32 heads = 128 dims each. (pp. 60-62)

Why is extending context length hard for transformers?

Every previous token has its own key and value vectors, so longer sequences mean more K and V vectors to compute and store. (p. 61) | Connection: same pressure as on-chip memory in your FPGA work: state that grows with sequence length must live somewhere, and memory bandwidth becomes the bottleneck.

Transformer block: what's inside, and the role of activation functions

An attention module (query, key, value, and output projection matrices) plus an MLP module (linear/feedforward layers separated by nonlinear activations such as ReLU = max(0, x) or GELU). Simple activations work because the model only needs nonlinearity, and they're cheaper to compute. (pp. 62-63)

Modules around the transformer blocks, and what determines model size

Before the blocks: an embedding module (token and positional embeddings; position count naively sets max context length). After: an output layer ("unembedding" or model head) mapping vectors to token probabilities. Size depends on model dimension, number of blocks (layers), feedforward dimension, and vocabulary size. Context length affects memory, not parameter count. (pp. 63-64)

Alternatives to the transformer, and why it's hard to replace

RWKV: an RNN that can be trained in parallel. SSMs (state space models), evolving through S4 → H3 → Mamba (linear vs the transformer's quadratic scaling in sequence length; Mamba-3B matches transformers twice its size) → Jamba (hybrid transformer + Mamba MoE). Any replacement must perform at the scale and on the hardware people care about. (pp. 65-66) | Connection: SSMs and RNNs carry a running state step to step, much like the membrane-potential state in your SNNs.

Sparse models and mixture-of-experts (MoE)

A sparse model has many zero-value parameters (7B at 90% sparse = 700M non-zero), so a large sparse model can need less compute than a small dense one. MoE splits parameters into experts, and only some are active per token. Mixtral 8x7B: 46.7B total, but 2 experts per token = 12.9B active, so it has the cost and speed of a 12.9B model. (p. 68) | Connection: similar to event-driven SNN sparsity, where cost tracks active work rather than total size.

Estimating inference memory from parameter count

Parameters × bytes per parameter. A 7B model stored at 2 bytes (16 bits) per parameter needs at least 14 GB of GPU memory for inference; the real figure is higher (Ch. 7). (p. 68)

Three numbers that signal a model's scale

Parameters: a proxy for learning capacity. Training tokens: a proxy for how much it learned (tokens, not samples, are the better dataset measure; training tokens = dataset tokens × epochs). FLOPs: a proxy for training cost. (pp. 69, 71)

FLOPs vs FLOP/s, and utilization

FLOPs count the floating point operations a task needs (GPT-3-175B: 3.14 × 10^23). FLOP/s is a machine's peak rate (H100 NVL: 60 TFLOP/s). Utilization is the share of peak you actually use: 50% is okay, above 70% is great. On 256 H100s at 70% and $2/h, GPT-3-175B would cost over $4M. (pp. 70-71)

Inverse scaling

Cases where bigger models do worse. Anthropic (2022) found more alignment training could make models align less with human preference. The Inverse Scaling Prize found larger models are sometimes worse on memorization and strong-prior tasks, but awarded no first or second prizes because no real-world failures were shown. (p. 71)

Compute-optimal and the Chinchilla scaling law

A compute-optimal model gets the best performance for a fixed compute budget. Chinchilla (DeepMind, 2022): training tokens ≈ 20x parameters (a 3B model needs ~60B tokens), and when you double model size you double tokens. It assumes data is much cheaper than compute and was derived for dense models on mostly human data. (pp. 72-73)

Why might a team pick a model smaller than compute-optimal? And the cost of the last few %

Production cares about usability, not only quality: Llama chose smaller models that are easier and cheaper to run, which aided adoption. Improvements get pricier: going from 90% to 95% costs more than 85% to 90%, and a 2% error rate may need 10x the data or compute of 3%. (pp. 73-74)

Parameter vs hyperparameter, and scaling extrapolation

Parameters are learned in training; hyperparameters are set by users (layers, model dimension, vocab size, batch size, epochs, learning rate). Large models often get one shot, so scaling extrapolation (hyperparameter transferring) studies small models and extrapolates (e.g., 40M → 6.7B). Emergent abilities make this less accurate. (pp. 74-75)

Two visible scaling bottlenecks

Training data: dataset growth outpaces new data, data restrictions now block ~45% of C4, AI-generated web content keeps growing, and proprietary data becomes a competitive advantage. Electricity: data centers use 1-2% of global power, possibly 4-20% by 2030, so they can grow at most ~50x. (pp. 75-77)

Why post-train? The two issues of a pre-trained model

1) Self-supervision optimizes for text completion, not conversation ("talks like a web page, not a human"). | 2) Indiscriminate internet data can make outputs racist, sexist, rude, or wrong. Post-training aligns the model with human preferences to fix both. (p. 78)

The two steps of post-training

1) Supervised finetuning (SFT) on high-quality instruction data, optimizing for conversation instead of completion. | 2) Preference finetuning, typically reinforcement learning: RLHF (GPT-3.5, Llama 2), DPO (Llama 3), RLAIF (potentially Claude). In the Shoggoth meme: pre-training makes the monster, SFT makes it socially acceptable, preference finetuning adds the smiley face. (pp. 78-79)

Pre-training vs post-training: what each optimizes

Pre-training optimizes token-level quality (predicting the next token); post-training optimizes whole responses users prefer. Post-training used only ~2% of InstructGPT's compute (98% went to pre-training) and unlocks capabilities that are hard to reach by prompting alone. (pp. 78-79)

Demonstration data and behavior cloning

(prompt, response) examples showing how the model should respond, covering the range of tasks (QA, summarization, translation...). The model clones the behavior. It needs skilled labelers (~90% of InstructGPT labelers had a college degree) and up to 30 min per pair: at $10/pair, 13,000 pairs ≈ $130,000. (pp. 81-82)

RLHF: the two parts

1) Train a reward model that scores the foundation model's outputs. | 2) Optimize the foundation (SFT) model to generate responses that maximize reward-model scores, often with PPO (proximal policy optimization). DPO is simpler; Huyen features RLHF because it is more flexible. (pp. 84, 87)

Reward model training data: pointwise vs comparison data

Pointwise scoring (rating each response independently) is inconsistent across and even within labelers. Comparison data (prompt, winning_response, losing_response) is easier: it takes 3-5 minutes and about $3.50 per comparison vs $25 to write a response. InstructGPT labelers agreed about 73% of the time. (pp. 84-85)

Best of N strategy

Generate multiple outputs and pick the one(s) a reward model scores highest, with no RL step. Stitch Fix and Grab found the reward model alone good enough for their applications. (p. 88)

Sampling, and greedy sampling

Sampling is how a model chooses an output from all possible options; Huyen calls it one of the most underrated concepts, explaining hallucinations and inconsistency. Greedy sampling always picks the most likely outcome: fine for classification, but it makes language model output boring. (pp. 50, 88-89)

Logits, softmax, and logprobs

A model outputs a logit vector, one logit per vocabulary token. Logits can be negative and don't sum to 1; softmax converts them to probabilities. Logprobs (log-scale probabilities) reduce underflow with huge vocabularies and are useful for classification and evaluation, but many providers limit them. (pp. 89-93)

Temperature

Logits are divided by temperature T before softmax. Higher T means rarer tokens are more likely (more creative, possibly less coherent); lower T is more consistent and boring. Example: logits [1, 2] → B 73% at T = 1, 88% at T = 0.5. T = 0 in practice means argmax. Providers usually allow 0-2, and ~0.7 is often suggested for creative use. (pp. 90-92)

Top-k sampling

Take the top-k logits and run softmax only over those (k typically 50-500), which reduces softmax computation over a huge vocabulary. Smaller k is more predictable but less interesting. (p. 94)

Top-p (nucleus) sampling, and min-p

Sum the probabilities of the most likely tokens in descending order until reaching p (typically 0.9-0.95), then sample only from that set. The candidate count adapts to context; it doesn't necessarily cut compute but works well in practice. Min-p sets a minimum probability a token must have to be considered. (pp. 94-95)

Stopping conditions

Stop after a fixed number of tokens (may cut off mid-sentence) or at stop tokens/words such as end-of-sequence. They keep latency and cost down, but stopping early can produce malformed output, e.g., JSON missing closing brackets. (p. 95)

Test time compute

Generate multiple responses per query to raise the chance of a good one: best of N, or beam search (keep the most promising candidates at each step). Varying sampling variables increases diversity. Cost scales roughly with outputs (2 outputs ≈ 2x cost). (p. 96)

Picking the best of several outputs

Highest average logprob (sequence logprob = sum of token logprobs, divided by length to avoid favoring short outputs; the OpenAI API uses this). Or a reward model/verifier: OpenAI's math verifiers gave a boost similar to a 30x larger model. Or app heuristics (shortest, first valid SQL, first finished). (pp. 96-99)

Self-consistency (most common output) and robustness

For exact-answer tasks, sample several times and pick the most frequent answer; Google sampled 32 outputs per MMLU question for Gemini. A model is robust if small input changes don't drastically change outputs; the less robust, the more multiple sampling helps (though swapping a brittle model is better). (p. 99)

Structured outputs: the two scenarios

1) Tasks that require structure, e.g., semantic parsing such as text-to-SQL or text-to-regex, or classification into valid classes. | 2) Outputs consumed by downstream apps or tools, e.g., an email returned as JSON with set keys; especially important for agentic workflows. (pp. 99-100)

Scenario: Your chatbot for Telugu speakers performs much worse than in English. What does Ch2 say explains this?

Under-representation in training data: Telugu is among the most under-represented languages in Common Crawl and among GPT-4's worst on MMLU. Language structure and culture can also matter. Options include language-specific models; translating via English risks information loss. (pp. 53-55)

Scenario: A hospital wants a general-purpose LLM to screen X-rays for cancer. What's Huyen's caution?

General models rarely do well on domain tasks they never saw. Medical scans are scarce online because of privacy, so you'd likely need a curated, domain-specific dataset or model. (p. 57)

Scenario: You must serve a 7B-parameter model in 16-bit precision on a GPU with 8 GB of memory. Will it fit?

No. 7B × 2 bytes = at least 14 GB just for the weights, and the actual memory needed is higher. (p. 68) | Connection: same budget arithmetic as fitting SNN weights into FPGA on-chip memory, where bit width × weight count must fit.

Scenario: You have a compute budget for a 3B-parameter model. Per Chinchilla, about how many training tokens should you use?

About 60B tokens (≈20 tokens per parameter) for compute-optimal training. (p. 72)

Scenario: Your creative-writing assistant keeps producing safe, boring phrases. Which sampling change helps?

Raise the temperature (around 0.7 is often suggested for creative use) and avoid greedy sampling, so lower-probability tokens can surface. Experiment, since higher T can hurt coherence. (pp. 89-91)

Scenario: A classifier app needs the same answer every time for the same input. Which temperature setting?

Set temperature to 0 (in practice, pick the highest-logit token via argmax) or very low, so outputs are consistent. (p. 92)

Scenario: For a yes/no prompt only 2 tokens matter, but for "What's the meaning of life?" many do. Which sampling strategy adapts to this?

Top-p (nucleus) sampling: the candidate set grows or shrinks until the cumulative probability reaches p, unlike top-k's fixed k. (p. 94)

Scenario: Your model reads text from a product image correctly only about half the time. What cheap fix does Huyen describe?

Sample multiple times per image (they tried three) and keep the valid or most common result; the less robust the model, the more this helps. Swapping out a brittle model is the better long-term fix. (p. 99)

Scenario: Your app asks for JSON, but responses sometimes come back missing closing brackets. What's a likely cause from Ch2?

An early stopping condition (e.g., a max-token cap) cut generation off before the format finished. Stopping conditions save latency and cost but can malform structured outputs. (p. 95)