QMB3302 Module 3 Vocabulary

Foundation Model

A large scale model trained on diverse data and used as a base layer for downstream applications. Provides transferable representations.

Transferable Representations

Reusable patterns provided by foundation models

Large Language Models (LLMs)

Subset of foundation models that're focused on language and learn to model the statistical patterns of language

Causal Language Modeling (LLMs)

Predicts the next token given everything that came before it - one of two objectives of LLMs.

Generates text in one direction.

Masked Language Modeling (LLMs)

Tries to fill in the blank on a random word in a sentence given it's context - one of two objectives of LLMs.

Understands text in both directions.

Self-Supervision

Models learn from raw text during pretraining.

Scaling Law

As you increase data/the size of the model, performance improves in predictable ways

Traditional Natural Language Procesers (NLPs)

Each model had one task.

Limited labeled data

Distinct Architectures

Narrow generalization.

Shared Architecture Patterns

Accelerate iteration and cross-pollination through interconnected research families and shared innovation.

Example: GPT->InstructGPT->ChatGPT->GPT-4 through scaling and instruction tuning

Pretraining

Models learn from raw text without labels using self-supervised learning with the objectives of next token prediction, masked-token prediction, and contrastive learning.

Learning Signal

In pretraining, the process where the model produces the embeddings it needs.

Embeddings

The internal representations that encode the syntax, semantics, and context of the model that's built.

Self-Supervised Learning Pipeline

Raw text -> tokenization -> training objective -> pretrained model

Training Objectives

Tells the model what it's supposed to predict and the process by which it's supposed to do that.

Can be next token prediction, masked-token prediction, and contrastive learning.

Architecture

Categorizing foundation models into encoder-only, decoder-only, and encoder-decoder.

Encoder-Only Foundation Models

Best for representation and extraction tasks.

Examples: BERT and RoBERTa

Decoder-Only Foundation Models

Best for generation and reasoning.

Examples: ChatGPT, Claude, and PaLM

Encoder-Decoder Foundation Models

Best for text-text transformation.

Examples: T5 and FLAN-T5

Modality

Categorizing foundation models into text, vision, audio/speech, and multimodal. All use attention mechanisms at their core.

Text Foundation Models

Core input is tokens, or pieces of text.

Examples: GPT, Claude, and Gemini-1.5

Vision Foundation Models

Core input is pixels.

Examples: CLIP, DINOv2

Audio/Speech Foundation Models

Core input is waveforms.

Examples: Whisper and Bark

Multimodal Foundation Models

Can handle mixed inputs like text + vision + audio.

Examples: Gemini, LLaVA, and Kosmos-2

Weights

A company's final trained model

Accessing Types

How you actually use models via closed API, open weights, or fully open.

Closed API

Proprietary weights accessed through hosted endpoints like a website that you pay access for. Has limited transparency but strong safety layers.

Examples: OpenAI, Anthropic, Claude

Open Weights

Downloadable and modifiable weights that allow for community fine-tuning and enables research, customization, and requires governance with safety measures.

Examples: Meta (LLaMA), and Mistral

Fully Open

Not only the weights but the training data and scripts are available. Maximizes reproducibility but raises serious IP and security concerns when using everyone else's data.

Examples: Falcon and Pythia

LLM Ecosystem

Multiple interconnected layers of information. Includes frontier models -> open-weight families -> ecosystem tasks.

Frontier / API Providers

The most cutting edge level of models.

Examples: OpenAI, Anthropic, Google, and Cohere

Open-Weight Families

Level of models that anyone can download and customize.

Examples: Gemma, Mistral, DeepSeek, Llama, and Qwen

Ecosystem Tools

Level of models that include deployment infrastructure, benchmarking platforms, model routing services, and fine-tuning frameworks.

Model Choice

A portfolio decision across four dimensions of capability, cost & latency, governane, and integration.

Capability Choice Dimension

What can the model actually do with regard to task coverage and reasoning depth?

Cost & Latency Choice Dimension

How much does the model cost to run and how fast does it respond? Includes API pricing and hardware requirements.

Governance Choice Dimension

What are the licensing, transparency, and compliance implications of a model? Data transparency and audit & safety filters.

Integration Choice Dimension

How well does a model fit into an organization's existing systems and workflow? API maturity and ecosystem support.

Adapting Foundation Models

Once a model is pretrained, how do you adapt it to your specific needs through 3 different approaches that include prompting, fine-tuning/PEFT, and retrieval-augmented generation (RAG).

Adaptation through Prompting

Steering the model's behavior through natural language instructions with no retraining or immediate iteration. Fastest and cheapest way to customize models.

Adaptation through Fine-Tuning / PEFT

Retraining or lightly adjusting parameters for domain or task-specific use with higher control of the output and higher costs.

Adaptation through Retrieval-Augmented Generation (RAG)

Connecting the model to external, updateable knowledge sources that keeps responses current and grounded in real data.

LLMs Scale Misconception

"Bigger always wins." -> Size helps but context and retrieval often matter more.

Knowledge Misconception

"LLMs store facts reliably." -> They run model distributions, not verified truth.

Alignment Misconception

"Alignment removes hallucinations." -> Reduces but does not eliminate them.

Safety Misconception

"Open = unsafe; closed = safe." -> Safety depends on governance and deployment, not licensing.

Misconception Persistance

Anthropomorphism, opacity, marketing & hype, and rapid iteration.

Anthropomorphism

Humans ascribe intent and memory to models because they come across as so life-like.

Opacitiy

Mode internals are so complex and poorly understood that people simplify their thinking about them.

Marketing & Hype

Narratives are pushed about models to describe them as "understanding" and "reasoning" because the general public wouldn't understand how they actually work.

Rapid Iteration

Models change faster than documentation.

Transformers

What make foundation models possible by replacing RNNs and LSTMs and becoming the dominant architecture in sequence modeling.

Overcame key limitations like sequential bottlenecks and difficulty modeling long dependencies.

Parallelism in Transformers

The model can process all of the tokens in a sequence at once instead of one at a time.

Enable both scaling and generalization, the foundation of LLMs

Attention in Transformers

Every token can directly reference every other token, capturing relationships across the entire input.

Enable both scaling and generalization, the foundation of LLMs

Limitations of Recurrent Neural Networks (RNNs)

Sequential processing, vanishing/exploding gradients, memory bottlenecks, and difficulty with long range dependencies.

Sequential Processing in RNNs

Handles tokens one at a time, in order, and you can't skip ahead.

Slow for long inputs.

Gradients

The signal/dependency that a model uses to adjust it's weights during training.

Example: A neural network adjusting it's biases

Vanishing/Exploding Gradients in RNNs

Long dependencies fade or explode during training, i.e. the model struggles to learn connections between tokens that are far apart

Memory Bottlenecks in RNNs

Fixed-length hidden states that carry all the information couldn't capture all of the context.

LSTMs and GRUs

Developed to improve RNNs by adding gated mechanisms, but still could only approximate long range dependencies (relationships between tokens that are far apart)

Quadratic Computing in Attention

If you double the length of your input (x2), the computation cost quadruples (x4).

Every token has to deal with every other token.

Transformer Scaling

Because transformers are modular in nature, you can scale them across three different axes: Depth, width, and data

Larger models achieve better synergy between these axes, not just by having more parameters.

Depth Scaling in Transformers

The number of layers increase, which improves reasoning and hierarchical abstraction (builds more increasingly complex representations as information flows through these layers).

Width Scaling in Transformers

The size of hidden dimensions and attention heads increase, which increase representational richness.

Data Scaling in Transformers

Training tokens and diversity increases, which enhances generalization and robustness.

Each Transformer Layer

Combines two key subsystems that are connected through add & norm steps: multi-head self-attention and feedforward network (FFN).

These create a repeatable block that can be stacked. (Modular format)

Multi-Head Self-Attention In Transformers

Integrates context across tokens that're coming in in a sequence.

Feedforward Network (FFN) in Transformers

Transforms those contextual representation independently per token.

Add & Norm Steps in Transformers

Connections between multi-head self-attention and FFNs that add a little bit of normalization into our models.

Attention -> Add & Norm -> FFN -> Add & Norm

Self-Attention

Each token in our input creates three separate vectors/values/numbers: query (Q), key (K), and value (V).

The model then computes the similarity between Q and K to determine how much each token should think about other tokens. (Attention Weight)

Then, the model takes a weighted sum of the value vectors based on those attention weights. The result/output in a new representation of each token that takes into account the context from the entire sequence.

Query (Q)

Represents what you're looking at.

One of the three numbers produced by self-attention.

Key (K)

Represents what you have to offer.

One of the three numbers produced by self-attention.

Value (V)

Represents what information you're carrying.

One of the three numbers produced by self-attention.

Attention Weight

The value a model assigns to each token when comparing each Q and K to determine how much each token needs to think about other tokens in Self-Attention.

This all happens in parallel to all tokens simultaneously and mirrors the intuition of language by assigning the most weight to subjects and prepositions but not the other way around.

Multi-Head Self-Attention In Transformers (Cont.)

Each "head" learns a different type of relationship between the tokens/words like the distance from each other, syntactic connections, or different entity references.

Simp. -> Each head sees something different within the same sequence, and then combines all of these different meanings so you have the most rich interpretation.

Feedforward Network (FFN) in Transformers (Cont.)

After attention integrates the context across different tokens, this network transforms each token individually into a unique representation.

Can only process one token at a time and run in parallel.

Each Transformer Layer Format

Input Embeddings -> Multi-Head Attention -> Add & Norm -> FFN -> Add & Norm -> Output to Next Transformer Layer