LLM Training and Fine-Tuning

Large Language Model Training and Fine-Tuning

Overview

• This lecture focuses on how to train and fine-tune large language models.

Schedule:

• Pre-training

• Supervised Fine-tuning

• Parameter-Efficient Fine-tuning

• Instruction Tuning

• Self-Destruct Adapter

• LoRA (Low-Rank Adaptation)

• Cura

• Prompt Tuning

• Prefix Tuning

Recap of Language Models

• Language Model: Assigns a probability to a sequence of words or tokens.

  • Example: A valid sentence in a human corpus has a higher probability than a random sequence of words.

• Conditional Language Model: Assigns a probability to a single word given a context (next word prediction).

  • Example: "During the weekend, I usually go…" with different words having different probabilities based on context.

Fundamentals of Machine Learning

• Machine Learning: A function with parameters that adapt to model real-world data distributions.

• Training: Tuning parameters to resemble training data samples.

• Observations: Known examples seen during training.

• Inference/Testing: Predicting new, unseen data using learned patterns.

• Example: Dog vs. Cat classification using image recognition.

  • A function with parameters is trained to categorize images.

  • After training, the model can predict whether a new image is a cat or dog.

Types of Machine Learning

• Supervised Learning: Learning process supervised by human annotations (labeled data).

• Unsupervised Learning: No human supervision (e.g., clustering, dimension reduction).

• Self-Supervised Learning: Learns about the data itself using pseudo-labels.

• Semi-Supervised Learning: Uses both labeled and unlabeled data.

• Reinforcement Learning: Learns a policy from interacting with an environment to maximize rewards.

Large Language Models

• Learned functions with billions of parameters.

• Model the distribution of human text (e.g., documents).

• Often based on Transformers (neural network-based language modeling).

• Trained on numerous question-answer pairs to generate or predict outputs given inputs.

GPT Development Lineage

• GPT-1 -> GPT-2 (large-scale pre-training) -> Code GPT-3 (trained on more code) -> Instructor GPT-3 (instruction tuning) -> GPT-3.5 (combined) -> Text DaVinci 002 (more instruction tuning) -> ChatGPT/Text DaVinci 003 (reinforcement learning with human preference).

• Key processes: pre-training, instruction tuning, reinforcement learning with human preference.

Training Instructor GPT-3

• Three phases:

  • Pre-training: Learn general word knowledge from a large corpus.

  • Supervised Fine-tuning: Tailor to tasks and follow user instructions.

  • Reinforcement Learning: Tailor to human preference.

Analogy to Human Education

• Recognizing words (tokenization) -> Reading textbooks (pre-training) -> Doing exercises (instruction and fine-tuning) -> Getting feedback (reinforcement learning from human feedback).

Model Development

• Training starts with broad word knowledge (pre-training).

• Fine-tuning for professional skills.

• Adapting to the real world via feedback alignment.

• Learning to use tools effectively.

• Da Vinci (trained on rapid text) -> Da Vinci 002 (incorporated more code) -> Da Vinci 003 (human annotation and instruction tuning) -> GPT-3.5 Turbo (trained on chat data).

Pre-Training

• Base of all following processes and the starting point for large model development.

Size of Training Data

• Rapid increase in data size:

  • BERT: 3 million tokens.

  • GPT-3: 300 billion tokens.

  • Chinchilla (2022): 1.4 trillion tokens.

Data Requirements

• Large amounts of data are needed to learn true knowledge behind the text.

• Need more data for word knowledge than for semantics and syntactic knowledge.

  • 1-10 million tokens to map semantics.

  • Billions of tokens to gain true knowledge / common sense.

Increase in Data and Model Size

• GPT-1 (2018): 117 million parameters (trained on several thousand books).

• GPT-2: 10x larger (trained on 40GB of text).

• GPT-3: 175 billion parameters (trained on 580GB of text, 300 billion tokens, 300,000 Harry Potter books).

Data Acquisition

• Models mainly use cloned data from public web text.

• Controversies over property issues of larger models.

• Debate whether it's just statistics or replication of raw text.

• Scaling law: larger models generally perform better.

Self-Supervised Learning

• A form of unsupervised learning using pseudo-labels (labels are part of the data itself).

• Examples: predicting the next word or the missing part of an image.

• No specific goal like classification or translation; the goal is to predict about the data itself.

• Predict a part of the data from the rest of the data (future from past, etc.).

• Masked language modeling (BERT): Mask a token and predict it.

Illustration: Supervised vs. Unsupervised vs. Semi-Supervised vs. Self-Supervised

• Supervised: Human-annotated images of grapes to classify what is a grape.

• Unsupervised: Clustering images into different categories without labels.

• Semi-Supervised: Learning from labeled and unlabeled data.

• Self-Supervised: Adapting to a dataset to predict about the data itself (a pretext task).

Pre-training with Self-Supervised Objectives

• Masked language modeling (BERT style): predict a masked token

• Causal language modeling (GPT style): predict the next token

Causal Language Modeling (GPT Style)

• Predict the next token given the context.

• Input is provided, and the model continues from there.

• Training involves predicting the next token and comparing it with the ground truth to update model parameters.

Shortcomings of Masked Language Modeling (BERT Style)

• Not designed for generation.

• Not scalable to large language models due to the limited objective of filling in the blank.

• GPT style (decoder-only architecture) is more widely used for large language models.

Training GPT Style Models

• Use text data from the internet.

• Model predicts the next token each time.

• Predicted token is compared with the ground truth to get a loss, and model parameters are updated.

• Tokens themselves are the labels (pseudo-labels); no manual annotations are needed.

Training Data

• Text data is abundant on the internet (Wikipedia, Reddit, etc.).

• GPT-3 is trained on public web text (60% from common crawl data, 22% from web text, 16% from books, 3% from Wikipedia).

• Any valid human text can technically be used, but considerations include data quality and duplicate data.

Tricks to Improve Pre-Training

• Filter the data: Use data quality filtering and make high-quality data seen more often.

• Reduce Duplication: Check for and reduce duplication danger.

• Think about Domains: Sample more difficult texts from different domains.

• Mask Trash Tokens: Mask unused information like exact time and username.

• Large Language Models Teach Large Language Models: Use a larger teacher model to generate outputs and guide the training of student models (GPT-3).

Data Quality

• Improving data quality greatly helps larger models of the same size.

• GPT-3 performs better than LLaMA of the same scale due to training on better quality data.

Practical Considerations

• New York Times sued OpenAI in 2023 with ongoing updates.

• Reddit ready to charge for using common data to train larger models.

• Anthropic uses data from Crowd.

Cost and Necessity of Pre-Training

• Pre-training is very expensive and non-trivial, but necessary for now.

• Allows learning rich text representation, modeling distribution structures of human language, and initializing weights for downstream models.

Thoughts on Pre-Training

• Reasoning problems are hard for large language models because good answers might not exist online.

• AI-generated text might flood communities, leading to less human-written text in the future.

Fine-Tuning

• Pre-training is not the end but the beginning.

• Models learn knowledge but are not inherently useful without fine-tuning.

• Fine-tuning adapts models to new datasets, achieves design goals, adapts to new domains, and follows specific behavior patterns.

Understanding User's Intent

• Pre-trained language models often fail to understand a user's intent.

• For example, GPT-3 might complete a prompt rather than answering it.

• Instruction tuning allows the language model to understand and answer questions, distinguishing it from simple text completion.

Process of Fine-Tuning

• Knowledge transfer from pre-trained models to target tasks through fine-tuning on specific datasets.

• Knowledge is encoded in the model's parameters, and transferring parameters transfers the knowledge.

The Move to Generalist Models

• Current large models are generalists rather than specialists.

• A single large model can perform a wide range of tasks surprisingly well.

• Including all tasks in the training dataset allows models to generalize to unseen tasks.

Instruction Tuning

• Larger models can generalize to unseen tasks after instruction tuning.

• Fine-tuned models generalize better than few-shot prompting on unseen tasks.

Strategies for Fine-Tuning

• Based on updated parameters: full fine-tuning, parameter-efficient fine-tuning (adapters, LoRA, Cura, prompt tuning, prefix tuning).

• Based on supervision: supervised fine-tuning, self-supervised fine-tuning, and reinforcement fine-tuning.

Self-Supervised Fine-Tuning

• Continuing with domain adaptation pre-training.

• Incorporates new knowledge into the old one.

Reinforcement Fine-Tuning

• Will be discussed in separate sections, focusing on making models align with human preference and use tools.

Types of Fine-Tuning

Full Fine-Tuning

• Every single parameter in the model is updated.

• Shortcomings: high computational cost, catastrophic forgetting, and overfitting.

• Challenging due to the GPU memory requirements and disk storage.

• However, it can be the most versatile and powerful, especially with a large domain gap.

Parameter-Efficient Fine-Tuning (PEFT)

• Keeps most of the parameters frozen, adjusting and updating only a small portion.

• Approaches: selective, adapters, and additive.

Advantages

• Less prone to catastrophic forgetting because all the pre-trained weights are frozen, retaining all the knowledge.

Selective Fine-Tuning

• Only updates some of the model parameters.

• Example: BitFit updates biases in a feed-forward layer, freezing the weights and attention matrix.

• Larger models may only need biases, which are usually enough for new tasks.

LoRA (Low-Rank Adaptation) - Adapters

• Add new parameters to the models while keeping the original model weights frozen, and trains only the newly added weights.

• Used to approximate a big merger by the product of two more matrices, and attach these two small matrices as a plugin to each key layer's weights.

• Process: Plugin added in each key layer, the final weights are the sum of the two matrices, training happens only on the yellow square, unique set of plugins for every single downstream task.

Composition

• Lowers the number of parameters.

• Decomposes the original weight matrix into a product of two smaller matrices.

Example with All for this Layer

• Original size of the weights is $512 \times 64$ (over 30,000 trainable parameters).

• Decompose the weight size into two smaller matrices, choose a low rank of eight, have two matrices: w1 is $8 \times 64$, w2 is $512 \times 8$ (final parameters will be like 5000, this is an 86% reduction in the parameters).

Q-LoRA

• Original model quantized during fine-tuning; later requires dequantization to original precision.

• Lora- not quantized, all original precision- helps with memory GPU memory.

Comparison

• Full Fine-tuning: No adapters, all weights optimizing the parameters are just the same as the base model.

• Lora: Has plugin inserted into each key layer, and update happens only for the plugin.

• Q-Lora: Quantities the original model, shrinking it to be a much smaller one during fine-tuning.

AK-LoRA

• Does not apply the same quantization over the whole model; instead, selectively applies different levels of quantization to different layers based on importance.

• Quantizes the Lora parameters.

• During inference, there is no need to unpack the model, and addresses the model as it is, both the large language models, and the Lora parameters are quantized.

Comparison Between LoRA, Q-LoRA, and AK-LoRA

• Lora: Add the A and B complement composition matrix to the pre-trained weights.

• Q-Lora: Original weights are quantized.

• AK-Lora: Adding more parameters to determine the dynamic quantization.