Prompt Engineering - Comprehensive Study Notes

Introduction to Prompt Engineering

• Prompt: a set of instructions and context passed to a language model (LM) to achieve a task.
• Prompt engineering: the practice of developing and optimizing prompts to efficiently use language models for a variety of applications.
• Purpose: improve and efficiently use LMs; useful for AI engineers and researchers.
• Two perspectives:
  • General definition: prompt engineering is the process of designing prompts to guide user interaction toward a desired outcome.
  • Interactive design perspective: prompts in interactive systems help users interact with a product intuitively.

Why Prompt Engineering?

• Important for research, discoveries, and advancement in AI.
• Helps test and evaluate the limitations of LLMs.
• Enables innovative applications on top of LLMs.
• Source example: Anthropic job listing highlighting these motivations.

Key Concepts: In-context Learning

• Rise of in-context learning enables LMs to perform tasks with prompts without gradient updates.
• Zero-shot: perform task without examples in the prompt.
• One-shot / Few-shot: provide one or more exemplars in the prompt to guide the model.
• Observed trend (illustrative): accuracy improves with more in-context examples up to a point, with performance varying across model sizes.
• Typical model sizes discussed: 175B params, 13B params, 1.3B params.
• Foundational citation: Brown, Tom B. et al. Language Models are Few-Shot Learners, ArXiv:2005.14165 (2020).
• Conceptual takeaway: larger models tend to benefit more from few-shot prompts; adding context examples can significantly improve task performance, especially for generations that require generalization from examples.

Prompt Components and Structure

• A prompt comprises:
  • Instructions: what the model should do.
  • Context: background information or constraints.
  • Input data: the specific instance to process.
  • Output indicator text: cues that steer the model toward a neutral, negative, or positive classification.
• Example layout for sentiment:
  • Input: "The meal was decent."
  • Output indicator: "Sentiment: Neutral" (or Positive/Negative as needed).

Decoding/Generation Settings

• Decoding strategies:
  • Greedy/Beam search: often produce less surprising/boring responses; not ideal for open-ended tasks like dialogue or storytelling.
  • Sampling-based generation: encourages more diverse outputs.
• Temperature (T): controls sharpness of the next-token distribution.
  • Range: $0 \le T \le 1$
  • Lower temperature yields a sharper (more deterministic) distribution; higher temperature yields more diverse outputs.
• Top-p (nucleus sampling): controls the cumulative probability threshold for token selection.
  • Range: $0 \le p \le 1$
  • Select the smallest set of tokens whose total probability mass is at least p, then sample from that subset.
  • Lower p leads to more repetition; higher p yields more diverse outputs.
• Practical guidance: use lower temperature/top-p for exact answers; use higher values for creative or diverse responses.

First Basic Prompt Example

• Prompt example (model: text-davinci-003); parameters: temperature $= 0.7$, top-p $= 1$.
• Text: "The sky is blue. The sky is a beautiful blue color during the day. The blue of the sky is created by the Earth’s atmosphere scattering the sun’s light. The blue is a result of the air molecules in the atmosphere reflecting the shorter wavelength of blue light back to our eyes."
• Purpose: illustrate how prompts and parameter settings influence a model’s response.

Exemplars: Prompt Components in Practice

• Example shows how prompts can be structured to yield a final answer with clear output signals (e.g., Instruction + Context + Input + Output text).
• Use of neutral/positive/negative sentiment tags or classification labels to guide the model’s final formatting.

Tasks Covered by Prompt Design

• Text Summarization
• Question Answering
• Text Classification
• Role Playing
• Code Generation
• Reasoning

Text Summarization Example

• Original text (summary target): Antibiotics are a type of medication used to treat infections. They work by killing bacteria or stopping them from reproducing. They are usually taken orally but can be administered intravenously. Antibiotics are not effective against viruses and inappropriate use can lead to antibiotic resistance.
• Desired output: A concise one-sentence summary.
• Prompt structure demonstrated: Context + Instruction (summarize to one sentence).
• Concept: leverage a single-sentence summary to capture the essence while avoiding extraneous details.

Question Answering (QA) Example

• Context: Teplizumab traces its roots to a New Jersey drug company called Ortho Pharmaceutical. Scientists generated an early version dubbed OKT3. Originally sourced from mice, the molecule bound to the surface of T cells and limited their killing potential. In 1986, it was approved to prevent organ rejection after kidney transplants, making it the first therapeutic antibody allowed for human use.
• Question: What was OKT3 originally sourced from?
• Answer: Mice.
• Note: The slide demonstrates a concise QA with explicit answer extraction from provided context.

Text Classification Example

• Text: "I think the food was okay."
• Label: Neutral
• Demonstrates simple sentiment classification task.

Role Playing Prompting

• Setup: Define role (e.g., assistant with a technical, AI research, scientific tone).
• Example dialogue: Human asks about a topic; AI responds in a specialized tone.
• Purpose: control voice, style, and expertise level of the assistant.

Code Generation Prompt Example

• Task: Write a MySQL query given a schema.
• Example schema fragments:
  • Table departments, columns = [DepartmentId, DepartmentName]
  • Table students, columns = [StudentName, DepartmentId, StudentId]
• Target query: select all students in the Computer Science department.
• Sample solution:
  • SELECT StudentId, StudentName FROM students WHERE DepartmentId IN (SELECT DepartmentId FROM departments WHERE DepartmentName = 'Computer Science');
• Purpose: illustrate how to prompt for code generation with proper schema context.

Arithmetic Reasoning Prompts

• Example: Identify odd numbers in a set and sum them; determine if the sum is even or odd.
• Demonstrates stepwise reasoning and final numeric conclusion.
• Typical outputs show steps and final answer.

Prompt Engineering Techniques Overview

• Few-shot prompts: provide exemplars in prompts to guide model behavior.
• Chain-of-thought (CoT) prompting: instruct the model to reason step-by-step.
• Self-Consistency: sample multiple reasoning paths and select the most consistent final answer.
• Knowledge Generation Prompting: use knowledge generated within the context to improve reasoning on complex tasks; pick highest-confidence prediction.
• ReAct: combine reasoning traces with external actions to interact with tools or knowledge bases.

Few-shot Prompting Details

• Show multiple example pairs to guide the model on the pattern.
• Example snippets illustrate how responses align with demonstrated formats and reasoning patterns.
• Important: the quality and relevance of exemplars influence performance significantly.

Chain-of-Thought (CoT) Prompting

• Instructs model to reason about the task before answering.
• Especially useful for tasks requiring multi-step reasoning.
• Can be used with few-shot prompts or in zero-shot settings.
• Zero-shot CoT: add a prompt cue such as "Let's think step by step" without explicit exemplars.

Zero-shot CoT Example

• A multi-step reasoning prompt includes step-by-step reasoning to reach the final answer.
• Demonstrates how the model can produce structured reasoning without prior demonstrations.

Self-Consistency

• Idea: generate multiple reasoning paths via few-shot CoT and pick the most consistent answer.
• Rationale: reduces reliance on a single chain of thought by considering multiple plausible paths.
• Example dilemma: multiple CoT paths may yield different answers; self-consistency picks the one that recurs across samples.

Revisit: Self-Consistency Example (illustrative)

• Prompt: a math reasoning problem involving ages or sums.
• Observation: multiple generated reasoning paths may converge on a consistent final answer (e.g., age relationships, arithmetic sums).

Introduction to Advanced Techniques: Part 2 Demo

• Demonstrations of applying self-consistency and CoT in practice to improve accuracy on arithmetic and reasoning tasks.

Knowledge-augmented Prompting (Generate Knowledge Prompting)

• Concept: generate auxiliary knowledge samples within the prompt to support reasoning.
• Process:
  • Step 1: generate knowledge statements about the world relevant to the task.
  • Step 2: use those knowledge samples to reason and answer.
  • Step 3: select the highest-confidence final answer.
• Source: Generated Knowledge Prompting for Commonsense Reasoning.

Example: Generate Knowledge Prompting – Show How Knowledge is Generated

• Example pair: Greece is larger than Mexico.
  • Knowledge: Greece is approximately 131,957 sq km; Mexico is approximately 1,964,375 sq km; Mexico is ~1,389% larger than Greece.
• Example pair: A rock is the same size as a pebble; pebble size on Udden-Wentworth scale: 4 to 64 mm; pebble larger than granule (2-4 mm) and smaller than cobble (64-256 mm).
• Example pair: Part of golf is trying to get a higher point total than others; knowledge: golf course structure (18 holes, lowest strokes wins).
• Purpose: illustrate how domain-knowledge augmentations steer reasoned answers.

Knowledge Prompting: Example Use

• Question: Part of golf is trying to get a higher point total than others. Yes or No?
• Two possible outputs depending on knowledge: No (lowest-strokes objective) vs Yes (incorrect framing).
• Knowledge-based explanation helps disambiguate and produce a high-confidence answer.
• High-confidence vs low-confidence predictions illustrated.

Program-aided Language Models (PAL)

• Motivation: CoT is powerful but not always sufficient; PAL offloads intermediate reasoning to a runtime (e.g., Python) via generated programs.
• Workflow: LM reads the problem, generates a program that implements the solution, executes it in a runtime, returns the result.
• Comparison: CoT uses text-only reasoning; PAL uses executable steps to compute results, offering potential reliability gains for certain tasks.
• Example: two problems solved by CoT vs a programmatic approach; program outputs with explicit code and results.

ReAct: Reasoning and Acting in Language Models

• ReAct framework: LLMs generate reasoning traces (thoughts) and task-specific actions in an interleaved manner.
• Reasoning traces: help the model plan, track, and adjust actions; handle exceptions.
• Acting steps: interface with external sources (knowledge bases, environments) to retrieve information.
• Benefit: enables the model to interact with tools and data beyond its internal knowledge, improving reliability and factuality.

ReAct Example (Hotspot QA)

• Demonstrates a step-by-step reasoning path with actions: search the Apple Remote, fetch Front Row, evaluate results, and finish with final answer.
• Shows how the model can combine CoT style thinking with concrete actions and tool usage to reach a conclusion.

Directional Stimulus Prompting

• A prompting technique to guide the LM toward producing a desired summary or output.
• Involves training a tuneable policy LM to generate hints that steer a black-box frozen LM.
• Purpose: improve alignment of model outputs with task requirements through indirect guidance.

Directional Stimulus Prompting Example

• Article: The Price Is Right host change in 2007 (Bob Barker returning for 2-3 sentences).
• Hint-based prompt: includes a cue such as "Bob Barker: TV: April 1: The Price Is Right: 2007: 91." to guide the model toward a concise summary.
• Model output with hint vs standard prompting shows improved ROUGE-1 score (example metrics included: ROUGE-1 48.39 vs 34.48).
• takeaway: hints/tolicy cues can improve summarization quality.

Part 4: Risks in Prompting

• Major risk categories:
  • Prompt Injection: hijacking an LM’s output by injecting untrusted commands that override instructions.
  • Prompt Leaking: forcing the model to reveal its own prompt or hidden instructions.
  • Jailbreaking: bypassing safety and moderation features via crafted prompts.
• Why these occur: LMs can be vulnerable when prompts are concatenated or when security boundaries are not properly enforced in API deployments.

Prompt Injection

• Definition: injecting an untrusted command that overrides the original prompt's instructions.
• How it happens: simple concatenation of user input with the prompt can hijack outputs.

Prompt Leaking

• Definition: prompting the model to reveal its own prompt or internal prompts or confidential system messages.
• Risk: could leak sensitive or confidential information about model behavior or data.

Jailbreaking

• Definition: attempts to bypass safety and moderation features to produce disallowed content.
• Reality: even when models are served via APIs with safety features, vulnerabilities can exist due to training data, model architectures, or deployment choices.
• Example: a jailbreak prompt that tries to coax the model into producing content it should refuse.

Jailbreaking Examples (Illustrative)

• Example: A user asks for instructions to hotwire a car; initial model refusal is followed by a jailbreak attempt that rewrites content or produces a poem about hotwiring.
• Lesson: be aware of prompts that coax the model into unsafe content and implement robust safety layers.

Prompt Engineering Resources (Guide)

• Prompt Engineering Guide (DAIR-AI): a repo with guides, papers, tools, datasets.
• Announcements: full lecture, notebook, and exercises planned; community discussions on Discord.
• Table of Contents in the guide:
  • Guides: Prompt Engineering - Introduction, Basic Usage, Advanced Usage, Adversarial Prompts, Miscellaneous Topics.
  • Papers, Tools & Libraries, Datasets.
• Link: https://github.com/dair-ai/Prompt-Engineering-Guide

Conclusion & Future Directions (Synthesis)

• Prompt engineering remains a rapidly evolving field with ongoing research in:
  • More robust, verifiable reasoning in LLMs (CoT, PAL, ReAct variants).
  • Safer interaction patterns to mitigate prompt injection, leakage, and jailbreaking.
  • Methods to augment LLMs with external knowledge sources reliably (Knowledge Prompting, PAL, ReAct).
  • Efficient, scalable evaluation of prompts across tasks and domains.
  • Tools, libraries, and community resources to democratize access to advanced prompting techniques.

Quick References and Citations

• Brown, Tom B. et al., Language Models are Few-Shot Learners, ArXiv:2005.14165 (2020).
• HuggingFace blog on decoding strategies (temperature, top-p).
• Large-Language-Model prompting techniques literature: CoT, Self-Consistency, ReAct, PAL, Directional Stimulus Prompting, etc.
• Anthropic job posting highlighting prompt engineering as a key capability.