NLP Pipeline

Working with Pre-trained Models

• In-context learning: Give few examples of the task that you want the model to solve.

• Crafting prompts that reformulate tasks to resemble those solved during pre-training.

• GPT-3 onwards, LLMs can perform tasks with and without limited examples, by predicting sequences or comparing probabilities.

• Unpredictable phenomena of emergent abilities of LLMs.

• Emergence is when quantitative changes in a system result in qualitative changes in behavior (Steinhardt 2022).

• Emergent abilities of LLMs: abilities that are not present in smaller-scale models but are present in large-scale models; thus, they cannot be predicted by simply extrapolating the performance improvements on smaller-scale models

• Examples of emergence in the few-shot prompting setting (L. Wei et al. Emergent Abilities of Large Language Models. Transactions on Machine Learning Research (TMLR), 2022).

New Benchmarks for Language Models: Big Bench

• BIG-Bench [Srivastava et al., 2022] with 200+ tasks.

• 80% of benchmark tasks are in JSON, which contains a list of examples made up of inputs and targets; 20% are programmatic.

New Benchmarks for Language Models: MMLU

• Massive Multitask Language Understanding (MMLU) [Hendrycks et al., 2021].

• New benchmarks for measuring LM performance on 57 diverse knowledge-intensive tasks.

• High-school questionnaire for different subjects.

Prompting

• Prompting is when a pre-trained LM is given a prompt (e.g., a natural language instruction) of a task and completes the response without any further training or gradient updates to its parameters.

• It is a method of providing an LLM with a specific input or cue to generate a desired output or perform a task.

• A well-crafted prompt can guide an LLM to generate more accurate, relevant, and contextually appropriate responses.

• The process can be iteratively refined.

• Prompt engineering: design effective prompts to make better use of LLMs and enhance their practical utility in real-world applications.

Prompt Engineering

• Prompt: input to the LLM: $x$

• Generation process: text $y$ by maximising $P(y|x)$

• Prompt acts as the condition on which predictions are made and can contain any information that helps describe and solve the task.

• Prompt template: a piece of text containing placeholders or variables, where each placeholder can be filled with specific information.

• Variable example: ${* premise *}$, where $premise = the weather is nice this weekend$

Prompt Engineering (2)

• Prompt with multiple variables: Example: compare the two sentences in terms of their semantic similarity.

• Attribute-based prompts: represent and store such data in key-value pairs; this format can be used to describe more complex tasks.

Prompt Engineering (3)

• Prompt with system information: assign a role to LLMs and provide sufficient context.

• System information helps the LLM understand the context or constraints of the task it is being asked to perform; example: LLM as an assistant to correct English sentences.

In-context Learning

• Learning during inference.

• Prompts involve demonstrations of problem-solving.

• LLMs can learn from these demonstrations how to solve new problems.

• No model parameters are updated in the process.

• It is a way to efficiently activate and reorganize the knowledge learned in pre-training without additional training or fine-tuning.

• Enables quick adaptation of LLMs to new problems.

• Pushing the boundaries of what pre-trained LLMs can achieve without task-specific adjustments.

In-context Learning: Zero/one/few-shot learning

• Zero-shot learning does not involve a traditional “learning” process; it directly applies LLMs to address new problems not observed during training.

• In practice, repetitively adjust prompts to guide the LLMs in generating better responses without demonstrating problem-solving steps or providing examples; example: LLM as an assistant to correct English sentences.

In-context Learning: Zero/one/few-shot learning

• One-shot learning extends the 0-shot prompt by adding a demonstration of how to correct sentences, allowing the LLM to learn from this newly-added experience; example: LLM as an assistant to correct English sentences.

In-context Learning: Zero/one/few-shot learning

• Few-shot prompting includes a few input-output examples in the model’s context (input) as a preamble before asking the model to perform the task for an unseen inference-time example (Brown et al. 2020).

• Essentially provides a pattern that maps some inputs to the corresponding outputs.

• LLM attempts to follow this pattern in making predictions, provided that the prompt includes a sufficient number of demonstrations.

In-context Learning: Zero/one/few-shot learning

• Effectiveness of in-context learning relies heavily on the quality of prompts and the fundamental abilities of pre-trained LLMs.

• Significant prompt engineering effort is required to develop appropriate prompts that help LLMs learn more effectively from demonstrations.

• Stronger LLMs can make better use of in-context learning for performing new tasks.

• Example: machine translation: weak understanding if not enough language data during pre-training leads to poor performance regardless of prompting strategy.

Prompt Engineering Strategies (1)

• Clear task description: a precise, specific, and clear description of the problem to instruct the LLM to perform as expected.

• Particularly important when we want the output of the LLM to meet certain expectations.

Prompt Engineering Strategies (2)

• Guiding LLMs to think: for problems that require significant reasoning efforts, add “Let’s think step by step” to the end of each prompt [Kojima et al., 2022].

• Instruct LLMs to generate steps for reasoning about the problem before reaching the final answer.

• Multiple rounds of interaction: evaluate the correctness of the answer and, if necessary, rework it to find a better solution.

Prompt Engineering Strategies (3)

• Providing reference information enables the model to produce outputs that are confined to the relevant text and avoid unconstrained predictions.

• RAG (Retrieval-Augmented Generation): relevant text for the user query is provided by calling an IR system, and LLMs are prompted to generate responses based on this provided relevant text.

Prompt Engineering Strategies (4)

• Prompt formats: define several fields for prompts and fill different information in each field.

• Use code-style prompts for LLMs, which can understand and generate both natural language and code.

• Use control characters, XML tags, and specific formatting to specify how the input and output should be formatted or structured.

Prompting for NLP Tasks: Text Classification

• Text Classification: assigning pre-defined labels to a given text.

• LLM gives the answer not in labels but in text describing the result, which requires an additional label mapping step to map the text output to the label space.

• Induce output labels from LLMs is to reframe the problem as a cloze task.

Prompting for NLP Tasks: Text Classification

• Text Classification: constrain the prediction to the set of label words and select the one with the highest probability, e.g., $Y: {positive, negative, neutral}$.

• Constrain the output with prompts.

• Issues with a large number of categories.

Prompting for NLP Tasks: Text Generation

• Text completion: continual writing based on the input text.

• Text transformation: transformation of the input text into another text.

Prompting for NLP Tasks: Text Completion

• Generating text based on user requirements where requirements can include style, tone, length, and any specific content that the text should contain.

• Examples: formal report, a creative story, or a piece of programming code.

Prompting for NLP Tasks: Text Completion

• Code generation example: Please write a Python function to calculate the average of a list of numbers.

Prompting for NLP Tasks: Text Transformation

• Typical tasks include machine translation, summarization, text style transfer.

• Other NLP tasks: question-answering, MMLU benchmark, GSM8K dataset, information extraction, named entity recognition, relation extraction.

Multi-step Reasoning

• Chain-of-thought (CoT) Prompting enables language models to solve reasoning tasks, especially those involving multiple steps, by guiding models to produce a sequence of intermediate steps before giving the final answer.

• Specialized prompting or finetuning methods can be emergent in that they do not have a positive effect until a certain model scale – experiments on the LaMDA model.

Chain-of-thought

• Chain-of-thought (CoT) Prompting avoids reaching a conclusion directly and encourages to generate reasoning steps or to learn from demonstrations of detailed reasoning processes provided in the prompts.

• Consider the problem of algebraic calculation: Calculate the average of the numbers 2, 4, and 6.

Chain-of-thought

• LLM can learn from the correspondence between questions and answers and gain more from detailed problem-solving steps that are used to derive the answers.

• Incorporate some reasoning steps into the prompt to obtain a CoT prompt.

• Both few- and zero-shot CoT methods are in play.

Chain-of-thought: benefits

• Allows LLMs to decompose complex problems into smaller, sequential reasoning steps.

• Somewhat mirrors human problem-solving behaviors, making it particularly effective for tasks requiring detailed, multi-step reasoning.

• Makes the reasoning process more transparent and interpretable.

• User Trust: if users can follow the logic behind the reasoning process, they will be more likely to trust the predictions of an LLM.

• In-context learning approach.

• Applicable to most well-trained, off-the-shelf LLMs.

• Provides efficient ways to adapt LLMs to different types of problems.

Working with LLMs

• Zero-shot / Few-shot Scenarios (learning and evaluation).

• No finetuning, prompt engineering/CoT can improve performance.

• Limited by context length (input token length).

• Complex tasks will probably need gradient steps.

• Instruction Fine-tuning.

• Reinforcement Learning with Human Feedback (RLHF).

Instruction Tuning

• Language models are not aligned with user intent [Ouyang et al., 2022]

• After fine-tuning on a task, which shows user intent, language models generate accordingly.

Instruction Tuning

• Fine-tune the pre-trained model to follow instructions.

• Enable LMs to perform new tasks simply by reading instructions describing the task (without few-shot exemplars).

• Fine-tuning on a mixture of tasks phrased as instructions – LMs respond to instructions describing an unseen task.

Working with LLMs: Instruction Tuning

• Instruction Fine-tuning: train models to follow natural language instructions.

• Data consists of several thousand (Task/Instruction, Output) examples.

• Specialized prompting or finetuning methods can be emergent in that they do not have a positive effect until a certain model scale.

Instruction Finetuning

• Collect examples of (instruction, output) pairs across many tasks and fine-tune an LM.

• Evaluate on Unseen Pairs for predictions [FLAN-T5; Chung et al., 2022].

Scaling Up Finetuning

• Step 1: Pretrain (on language modeling).

• Lots of text to learn general things.

• Step 2: Finetune (on many tasks).

• Not many labels to adapt to the tasks.

• Pretraining can improve NLP applications by serving as parameter initialization.

Self-Instruct

• Task pool: seed hand-crafted task pool consisting of .

• LLM-generated instructions and samples added to this pool as the algorithm proceeds.

• Sampling: randomly select a few human-written instructions and a few LLM-generated instructions to ensure diversity.

• Instruction generation: selected instructions are used as demonstration examples; LLM in-context learns from these examples to produce a new instruction.

• Sample generation: LLM prompted to complete the sample by filling in the remaining input fields and generating the corresponding output.

Self-Instruct (2)

• Filtering: newly-generated sample is examined by some heuristic rules.

• Filtering out samples or instructions that are similar to those already in the pool.

• If it passes, the sample and instruction are added to the pool.

• The generation process is repeated to obtain a sufficient number of fine-tuning samples.

Instruction Tuning: Self-Instruct

• Input inversion: LLM generates the input based on the specified output.

• Biased predictions for tasks such as classification.

• Most generated samples belong to a single class.

• Specify the output (i.e., the class) with some prior, and prompt the LLM to generate user input given both the instruction and the output.

Limitations of Instruction Finetuning

• One limitation is obvious: it’s expensive to collect ground-truth data for tasks.

• Problem 1: tasks like open-ended creative generation have no right answer.

• Problem 2: language modelling penalizes all token-level mistakes equally, but some errors are worse than others.

• Even with instruction finetuning, there is a mismatch between the LM objective and the objective of “satisfy human preferences”!

Working with LLMs

• Zero-shot / Few-shot Scenarios (learning and evaluation).

• No finetuning, prompt engineering/CoT can improve performance.

• Limited by context length (input token length).

• Complex tasks will probably need gradient steps.

• Instruction Fine-tuning.

• Simple and straightforward, generalize on unseen tasks.

• Collecting demonstrations for so many tasks is expensive.

• Mismatch between LM objective and human preferences.

• Reinforcement Learning with Human Feedback (RLHF).

• Reward model with human feedback.

• Use these to further improve the model.