LLM Training and Fine-Tuning (contd.)

Topics Covered

• Review of fine-tuning logarithmic models.

• Reports on reinforcement learning from human feedback (RLHF).

• Gigantic learning to teach logarithmic models to use tools.

Recap of Last Time

• Training Alignment Model: Analogous to human education with four phases:

  • Pre-training: Broad world knowledge.

  • Fine-tuning: Domain-specific knowledge, professional skills.

  • Alignment: Adapting to real-world society based on feedback.

  • Tool Usage: Learning effective use of tools.

Pre-training

• Self-Supervised Learning:

  • Learning from the data itself (predicting part of the data).

  • Unsupervised learning (no explicit labels; part of the data serves as the label).

• Architectures:

  • Decoder Architecture: Predicts the next word (greedy approach).

  • Encoder Only: Predicts the masked token.

  • Transformer-based architecture is mainstream, especially for larger models.

Fine-tuning Strategies

• Different types:

  • Full fine-tuning.

  • Parameter-efficient fine-tuning (freezing weights).

  • Prompt tuning.

  • Prefix tuning.

• Supervision types:

  • Supervised fine-tuning.

  • Self-supervised fine-tuning.

  • Continue pre-training.

  • Reinforcement learning fine-tuning.

Parameter-Efficient Fine-Tuning

• Challenge:

  • Fine-tuning every weight in the model is memory and compute-intensive.

• Solution:

  • Freeze many weights and only fine-tune a portion of the model's weights.

• Bias Tuning: updating the biases, but not the weights.

• Low-Rank Adaptation (LoRA) and QLoRA:

  • Add decomposition matrices to important weights.

  • LoRA is based on precision transformers, while QLoRA is based on quantized transformers.

• Shade LoRA:

  • Dynamically decides quantization parameters for weights.

  • Less quantization for more important weights and more quantization for less important weights.

Prompt Tuning and Prefix Tuning

Prompt Tuning

• Lies between larger model fine-tuning and prompting.

• Tunes the prompts, not the model parameters (soft prompts).

• Soft Trainable Embedding Vectors:

  • Adds a "plug-in" to the prompts by setting up weights.

  • Values are continuous and don't correspond to concrete tokens.

  • Doesn't have a specific meaning for humans but is important for the model.

  • Prompt engineering is similar to hard prompt tuning.

• Process:

  • Add embedding vectors before the input/prompt.

  • These embeddings act as context to condition the prompts, making them better understood by the larger language model.

• Efficiency:

  • More efficient with longer models.

  • Reaches comparable performance to full fine-tuning and usually outperforms hard prompts with human engineering.

Prefix Tuning

• Values are added to vectors after multiplying the input with a shape matrix, rather than directly added to the prompts like in prompt tuning.

Supervised Fine-Tuning

• Training pre-trained language models on labeled data.

Instruction Tuning

• Training data:

  • Triplet [instruction, input, output]

Instructions

• After pre-training, models complete tasks given pretexts.

• Instructions tell the language model what to do.

• Example: To get a zip code, an instruction tells the model to come up with a zip code to go with a given address and city.

• Without instruction tuning, models continue writing from the prompt rather than solving a task.

Instruction Tuning Process

• Train/tune models on a triplet dataset (instruction, input, output).

• The model learns to predict the output given the instruction and input.

• Goal:

  • Enable trained models to follow instructions and not just completions.

• Example:

  • To translate, the instruction is "translate the sentence in French", input is the sentence, and output is the French translation.

Backend vs Frontend

• The instruction from the prompt is fulfilled in backend, but you only operate a front end.

Opinions on Instruction Tuning

• Superficial alignment hypothesis:

  • Instruction tuning only teaches formal interaction syntax.

• New knowledge hypothesis:

  • Instruction tuning brings new knowledge into the models; the more data, the more knowledge.

• FLAN hypothesis:

  • Instruction tuning enables models to generalize to new tasks.

  • Example: Training on 1000 tasks allows generalization to 500 new tasks.

Synthetic Data and Instruction Tuning

• Use synthetic data to fine-tune pre-trained language models for instruction tuning.

• Steps:

  • Seed a model with human-written instruction data.

  • Ask the model to generate new data via in-context learning.

  • Use generated data for instruction tuning.

• Example (Alpaca):

  • Feed LLaMA 7B with 175 seed inputs.

  • Ask it to generate more data in this style.

  • Supervised fine-tuning with this data results in Alpaca 7B.

Multi-Step Instruction Following

• Add multi-step instructions to guide what the model voted for and generate one modest step in instruction following.

Limitations of Instruction Tuning

• Requires supervised data, which can be costly to collect.

• Some questions have no single correct answer, introducing ambiguity.

• Models treat all tokens equally, regardless of bias.

Reinforcement Learning from Human Feedback (RLHF)

• Explicitly training the language model with human preferences.

Reinforcement Learning Basics

• Training an agent to take actions based on the environment, observe its state and the agent take an action based on its current strategy.

• Interaction with receives feedback and then will update their strategy after they receive feedback to maximize the rewards.

• Compare:

  • Supervised learning: Input data and output data to train a model to output.

  • Reinforcement Learning: Only get rewards not actually what output we want, but only get the feedback from the environment and rewards.

Supervised Learning vs. Reinforcement Learning

• Supervised Learning:

  • Has labeled inputs.

  • Trains the model to match the labels.

• Reinforcement Learning:

  • No labels; the model learns patterns from the data itself.

  • The model observes a state, takes actions, receives rewards, and repeats to maximize rewards.

Reinforcement Learning with Language Models

• Environment: task (e.g., solve the user's task).

• State: User prompt.

• Agent: The language Model itself.

• Policy: Its weights.

• Action: Generated text.

• Reward: Arbitrary feedback signal.

• The language model updates its weights for better generations.

Benefits of Reinforcement Learning

• Goes beyond token likelihood.

• Aligns with human properties, factual correctness, and effective tool usage.

• Can extend the language modes' goals as long as there is a reward for the actions.

• The state and rewards are often modeled by dedicated normal models.

Reinforcement Learning Process in Language Models

• Model (agent) interacts with environment (users).

• Users provide prompts, and the language model generates output.

• The output is evaluated, and a reward is given to stimulate better output.

• Magic Word Example: A language model generates a magic word which combines with the promise which is the state. It is passed to the target language models to evaluate the model to see how much performance boosted with magic word.

• The evaluation results provide rewards for better magic word hunting.

Reinforcement Learning with Human Feedback (RLHF)

• Example:

  • The user gives instructors and has paragraphs. The language models generate multiple outputs rated by the user liking this or not. A higher reward is given for outputs liked and low reward vice versa. The LLM will be trained to generate outputs we like.

• Principle:

  • Helpful - Good at problem-solving.

  • Honest - Admits what it does not know.

  • Harmless - Should not cause physical, psychological, or social harm to humans or the environment

RLHF Setup

• Reward model for Water Generation and S (e.g., human ratings or human-like reward model).

• Pre-trained mental models (possibly with instruction turning).

• Optimization algorithm (e.g., Proximal Policy Optimization - PPO).

Proximal Policy Optimization

• Fine-tunes larger models while keeping updates stable.

• Requires a critic model to predict expected future rewards.

Key Components

• Advantage: Reward minus expected reward with no update.

• Clipping: The process to prevent the updates deviating too far

Practical Ways to Collect Human Preference

• Gather questionnaires. 2 or 3 books more and more authors and then we give a rank better then the honest. Instead of give it a score based on rating.

• Training a substitute rewards model to replace human beings with a large enough data.

Direct Preference Optimization (DPO)

• Inspired by comparison-based learning.

• No reward model.

• Directly asks the logged in model to increase the likelihood.

Group Relative Policy Optimization

• Retains many designs of old enforcement call penalty and commitments but then you said that it does not use the critic the value model.

• Directly get the advantage by group statistics of model generators.

Reinforcement Learning from AI Preferences.

• Transitions from Am human intergroup.

• To do the we will again have all the instruction data and the multi generation for the same instructions.

• Will be rated uh will be compared to by for example 24.

• And again we will train a reward model to mimic 24 references.

Limitations of Reinforcement Learning

• Reward hierarchy and agent get me maximize by cheating by not true goal.

• If generate verbose and seemed authoritative will be love which be not what we want

InstructGPT Strategy

• The team will be instructed and compare based data.

• Trained and NLP rewards models

• And we will use PTO to optimism is the latan models based on the preference

Tragedy

• Huge decoder transformer model.

• Strong in context learning abilities.

• Add conversation data into the instruction truly data set.

Learning to Use Tools

• The language models do not execute the tour instead there are wrapping frameworks for a runtime

• The language models decide based on the user inputs that they need to do a web search and commander to ask the frameworks if do web searches and bring it back to the language models

React: Reasoning and Acting

• The models will reason first, and then decide want to use tour the action that execute observation after executing the tool the continue reasoning.

• We training the logic models to decide one which will by itself

Tool Learning

• We handcrafted some toy usage examples for the initial supervise fine-tuning Ask and data.

• To the for each of the input we would generate new modules candidates for each of the code

• The tail app we have to get the module port results and output we have to bought lead in 2023

Implementation

• You identify to many candidates.

• Each course to be evaluated and keep them in the data set

Gigantic to More Use Learning

• To teach latest models to all the mind autonomous and autonomous design aware how and which towards to use . They are not based on fixed prompts. Where to use that force.