LLM Architecture, Fine-Tuning, and Reinforcement Learning Vocabulary

Vocabulary flashcards derived from lecture content on LLM pretraining, supervised fine-tuning, knowledge representation, reinforcement learning, and RLHF.

Base Model

A token-level internet document simulator whose parameters store world knowledge obtained during pretraining.

Supervised Finetuning (SFT)

A post-training process where a base model is trained on conversation datasets consisting of problem statements and demonstrated solutions for imitation.

Vague Recollection

Knowledge stored directly within the parameters of a language model, analogous to information read a month ago.

Working Memory

Knowledge contained directly within the tokens of the model's active context window.

System Message

A prompt placed at the beginning of every conversation that reminds the model about its identity to establish a sense of self.

Swiss Cheese Model

A concept describing current LLM capabilities where some tasks work exceptionally well while others display random brittleness.

Reinforcement Learning (RL)

A post-training process where a model practices problem statements through trial and error to internalize cognitive strategies and reasoning.

DeepSeek-R1

A research work demonstrating the incentivization of reasoning capabilities in LLMs via reinforcement learning.

Aha Moment

An emergent intermediate behavior in models like DeepSeek-R1-Zero where the model spontaneously learns to rethink its problem-solving steps using an anthropomorphic tone.

RLHF

Reinforcement Learning from Human Feedback; a method used in un-verifiable domains where human rankings of model rollouts are used to train a simulator of human preferences.

Discriminator Generator Gap

The principle that it is empirically easier to discriminate or evaluate quality (e.g., choosing the best poem) than it is to generate high-quality content.

Reward Model

A neural network simulator trained on human preference orderings to automatically score model outputs during RLHF.

Reward Model Gaming

A failure mode in RLHF where the model discovers adversarial examples or nonsensical token sequences that exploit flaws in the lossy reward model simulator.