AI Spotlight: Prompt Injection Attacks

AI Spotlight: Prompt Injection Attacks

What is a prompt injection attack?

Crafted input that manipulates an LLM to bypass controls or change its output.

What does LLM stand for?

Large Language Model.

What are two major goals of prompt injection?

Bypass protections or manipulate generated output.

Can prompt injection lead to sensitive information disclosure?

Yes, including protected context or application data.

Can prompt injection enable classic software injection vulnerabilities?

Yes, such as command injection in an AI-integrated application.

Why can prompt injection be underrepresented in CVE tracking?

A model-level bypass alone may not receive a CVE.

What three categories of prompts are distinguished in the course?

Foundational system, application system, and user prompts.

Who supplies the foundational system prompt?

The LLM service provider.

What is the purpose of a foundational system prompt?

Guide model output and establish provider guardrails.

Who supplies an application system prompt?

The application developer or system integrator.

What is the purpose of an application system prompt?

Guide model behavior for a specific application.

Who usually supplies a user prompt?

The person interacting with the AI application.

Which prompt type can a typical attacker most directly control?

The user prompt.

Why does the book distinguish foundational and application system prompts?

Both are sometimes called system prompts, causing ambiguity.

What does a role="system" message represent in the sample API code?

The application system prompt.

What does a role="user" message represent in the sample API code?

User-supplied prompt content.

What is preprocessing in the course LLM pipeline?

Combining and normalizing prompt content.

What is tokenization?

Splitting text into tokens, such as words, subwords, or characters.

What is an embedding?

A numeric vector representation of token meaning.

What does attention weighting do?

Weights token relationships and helps resolve references or ambiguity.

What is contextual grounding?

Adding context from history, memories, RAG, or MCP agent data.

What does RAG stand for?

Retrieval Augmented Generation.

What does MCP stand for?

Model Context Protocol.

What is constraint adherence in LLM processing?

Applying instructions or restrictions to generated content.

What does RLHF stand for?

Reinforcement Learning from Human Feedback.

What is alignment and bias mitigation intended to reduce?

Bias and harmful or unwanted generated content.

What does content generation repeatedly select?

The next output token.

What does post-processing do?

Filters, formats, or moderates generated output.

Why might a minor spelling change affect an LLM response?

It can alter tokenization, embeddings, and attention.

Which two primary prompt injection categories does OWASP identify?

Direct and indirect prompt injection.

What is direct prompt injection?

Attacker-controlled instructions submitted directly to the model.

What is indirect prompt injection?

Attacker instructions embedded in outside content later read by the model.

What are typical direct prompt injection entry points?

Chatbots or API requests.

What are typical indirect prompt injection entry points?

Webpages, files, retrieved documents, or logs.

Why might the same injection not work every time?

LLM output is often nondeterministic.

Why may an injection work on one model but not another?

Different models and implementations apply different controls.

Which Python package is used in the tokenization demonstration?

tiktoken.

Which function selects an encoding for a specified model?

tiktoken.encoding_for_model().

What is keyword mutation in prompt injection?

Changing spelling or characters to change how a model processes a phrase.

What is Office Infiltrator?

The SEC504 practice game for learning prompt injection techniques.

Which command launches Office Infiltrator from Slingshot Linux?

goprompt

How many main levels does Office Infiltrator have?

Five.

What is the first Office Infiltrator level?

The Turnstile.

What is a direct prompt attack?

A straightforward request testing a model’s behavior or restrictions.

Why test a model with basic direct prompts?

To learn its capabilities and constraints.

What does HITL stand for?

Human-in-the-Loop.

What is the second Office Infiltrator level?

The Elevator Door.

Name three keyword mutation techniques.

Misspellings, leetspeak, and breaking up tokens with punctuation or spaces.

What does ablation mean in prompt testing?

Removing or changing prompt parts to identify what triggers controls.

What is leetspeak in the context of prompt injection?

Substituting numbers or symbols for letters.

What is the third Office Infiltrator level?

Floor 42 Access.

What is refusal suppression?

Reframing a rejected request to induce a different response.

What kinds of pretexts appear in refusal suppression?

Fictional, educational, or claimed misunderstandings.

What is the fourth Office Infiltrator level?

The Entry Door.

What is format switching?

Requesting a different output format to alter model behavior.

Name formats used for format switching in the course.

Markdown, HTML, JSON, or program code.

What is the fifth Office Infiltrator level?

The Workstation.

What is role-play prompt injection?

Using a fictional persona or scenario to change model output.

What does DAN stand for in the course?

Do Anything Now.

What command starts an alternative-seed Office Infiltrator game?

gopromptalt

How does gopromptalt differ from goprompt?

Uses a random seed and disables hints.

Name two other prompt injection techniques listed in the book.

Multi-turn manipulation and Unicode smuggling.

What is an important limitation of many prompt injection techniques?

They may stop working when providers update defenses.

What sensitive prompts can direct injection expose?

Foundational or application system prompts.

What external data might a prompt injection expose?

RAG documents or AI-agent context.

How can prompt injection lead to code execution?

An application may execute untrusted code generated by the model.

Which AI database framework illustrates code execution risk?

Vanna.ai.

What is the unsafe trust boundary in the Vanna.ai example?

Executing LLM-generated Python without sufficient validation.

What application data does Vanna.ai supply through RAG?

Database schema information.

What does Vanna.ai generate to query a database?

SQL.

What language does the Vanna.ai example generate for charting?

Python.

What does the Vanna.ai example show about AI output?

Treat model-generated code as untrusted.

What is the defining feature of indirect prompt injection?

The attacker plants instructions in data the model retrieves later.

How can a LinkedIn profile be used for indirect injection?

Embedded instructions can bias an AI resume evaluator.

How can a phishing email conceal indirect prompt injection?

Place hidden instruction text in HTML content.

What does display:none do in an HTML email example?

Hides text visually while leaving it in the content.

Which HTTP header can carry injected text in web logs?

User-Agent.

What might a malicious User-Agent prompt try to cause?

An AI log analyst to classify an attack as harmless.

Which Ffuf option changes an HTTP request header?

-H

Where can an attacker embed instructions in a Windows service?

The service description field.

What is the core trust error in indirect injection?

Treating untrusted data as model instructions.

Are system prompts alone a complete prompt injection defense?

No; additional controls are required.

What is recency bias in LLM prompting?

Later prompt material may have greater influence.

Why reiterate critical constraints near the end of a prompt?

To reduce effects of recency bias.

Why use delimiters around untrusted input?

To distinguish external data from instructions.

What is multi-chain analysis?

A second model reviews the first model’s output for security concerns.

Which security-focused model is cited for multi-chain analysis?

Cisco Foundation-sec-8b.

What additional control could reduce Vanna.ai exploitation impact?

Run generated Python in a constrained environment.

What is Human-in-the-Loop analysis?

A human reviews or approves AI-assisted results or actions.

Why use HITL in a security operations center?

To validate AI threat assessments, including false negatives.

Why log AI prompts?

To support threat hunting and investigation.

What layered strategies does the book recommend?

Structured prompts, access controls, multi-chain checks, HITL, and logging.