Exhaustive Notes on Context Engineering and Agentic LLM Systems

Speaker Introduction and Career Background

• Artemas's Professional Journey:     - Artemas has been in the data space for approximately $13 \text{ to } 14 \text{ years}$.     - He has occupied various roles, including Data Analyst, Data Scientist, Machine Learning (ML) Engineer, Data Engineer, and MLOps Engineer (focusing on infrastructure).     - He led teams across all these disciplines for several years.     - He served as the Chief Product Officer (CPO) at Neptune AI, which was recently acquired by OpenAI.     - Currently, he is the Founder and CEO of Squirtle AI.

• Squirtle AI Activities:     - Consulting and Implementation: Acts as forward-deployed AI engineers for organizations of various sizes to implement AI solutions.     - Educational Programs: Runs "Lightning Lessons" (such as this session) and intensive AI engineering boot camps.

Evolution of Large Language Model (LLM) Context Windows

• Model Context Capacities (as of late $2025$ / early $2026$):     - JEMA Free: An open-source model featuring a context window of $128,000 \text{ tokens}$.     - Claude 3.5 Sonnet: Features a context window of $200,000 \text{ tokens}$.     - 55 Mini: Features a context window of $400,000 \text{ tokens}$.     - Claude OPUS 4.6: Recently introduced a beta version of their $1,000,000 \text{ token}$ context window.

• The Context Problem Myth: While models are becoming larger with massive context windows, improvements in the model architecture alone do not solve the functional challenges of context management.

Components of the Context Window in Agentic Systems

• Instructional Elements:     - System prompts and core instructions.     - Structured output instructions (defining how a model should format response data to be used by downstream systems).

• Dynamic Context:     - Tool definitions: Lists of available tools for specific nodes in an agentic topology.     - User prompts: specific instructions provided by the user to initiate work.

• Memory and External Data:     - Retrieved Context (RAG): Data pulled from internal systems or databases using retrieval mechanisms.     - Long-term Memory: Historical actions taken by the system over time.     - Scratchpads / Conversational History: The record of action history and interaction turns which can quickly "bloat" the context.

Core Challenges in Context Management

• Accuracy and Reasoning Issues:     - Needle in the Haystack: As context grows, it becomes harder for the LLM to attend to a specific, small piece of information hidden within the mass of data.     - Lost in the Middle: Empirical evidence suggests that LLMs attend to information located in the middle of a long context window less efficiently than information at the beginning or end.

• Operational Constraints:     - Latency: Increasing the number of tokens pushed to the LLM increases the time required for a response.     - Cost: More tokens result in higher financial costs per API call, even with mechanisms like Key-Value (KV) caching for preceding tokens.

Definitions and Paradigms of Context Engineering

• Context Engineering: Defined as the practice of identifying and implementing the "minimal viable amount of context" required for an LLM call to generate a specific, high-quality output.

• Key Management Patterns:     - Tool Management: Managing the number, quality, and descriptions of tools to prevent agent confusion.     - Context Compression: Extracting only the necessary signals from text and discarding irrelevant noise.     - Retrieval Optimization: Ensuring only necessary data is retrieved and injected.     - Context Routing: Directing a user's intent to a specialized subsystem rather than processing it through a generic, bloated system.     - Progressive Disclosure: Loading capabilities and information to the agent only as they become necessary during execution.

Technical Deep Dive: Context Compression

• The React Agent Loop Example:     - A React (Reasoning and Action) agent analyzes a query, creates a plan, and if it cannot answer directly, it calls a tool.     - The result of the tool is added back to the context for the next turn.     - In loops repeating up to $n$ times, the "scratchpad" expands. If tools retrieve large database chunks, the prompt size can reach hundreds of thousands of tokens quickly, escalating costs significantly each iteration.

• Compression Implementation Strategies:     - Drop History: Keeping only the top $n$ interactions (low cost, but loses historical relevance).     - Summarization/Multi-tiered: Keeping the top $n$ turns as-is while compressing the older history or storing it in a long-term memory database.     - Hot Path vs. Cold Path:         - Hot Path: Compression occurs during the agent's turn, ensuring the freshest state but increasing latency.         - Cold Path: Compression runs as a background job, reducing latency but risking an inconsistent state.     - Rule-based Compression in RAG: If an LLM retrieves $20$ chunks but only references $3$ to answer a query, the system can automatically drop the remaining $17$ chunks from subsequent context calls.

Technical Deep Dive: Tool Management and MCP

• Native Tool Calling Principles:     - Function definitions are injected into the prompt.     - The LLM returns a suggestion for which tool to execute.     - The application (not the LLM) executes the tool and passes the output back to the LLM.     - Best Practice: Keep tools below $10$ to prevent confusion and overlapping capabilities.

• Model Context Protocol (MCP) Challenges:     - In MCP, a client connects to an MCP server which returns all tool definitions.     - Problem: Most frameworks (like LangChain or LangGraph) push every tool in the MCP server into the context.     - Dependency: Developers of the MCP server, rather than the AI engineer, effectively control the context engineering. If a server implements $90$ tools, all $90$ schemas are injected into the agent prompt, which can lead to "tool bloat" and rate-limiting issues.

Technical Deep Dive: Progressive Disclosure and Agent Skills

• Anthropic Agent Skills Protocol:     - This pattern loads capabilities only when needed rather than pre-baking all instructions and tools into the initial prompt.     - Initial Prompt: Contains only a system prompt, user query, and a list of available skill names/definitions (saving thousands of tokens).     - Triggering a Skill: When the agent identifies it needs a skill (e.g., a PDF handling skill), it uses a tool call to read the corresponding skill.md file.

• Structure of an Agent Skill (e.g., PDF Skill):     - The system looks for a skills folder containing subfolders for each skill.     - Each folder contains a skill.md file with a YAML front matter description used for the initial discovery.     - The .md file contains detailed text instructions and Python code snippets (tools).     - These files can contain pointers to further scripts or secondary files (e.g., license, reference, or form-handling scripts), which are loaded only if specified by the agent.

Synthesis: The Context Engineering Architecture Framework

• Layer 1: Initiation (Load-time):     - Agent Identity.     - High-level Skills discovery.     - Initial Tool Management settings.

• Layer 2: Run-time (Per Turn):     - Context Routing (intent selection).     - Context Compression (managing the history/scratchpad).     - Retrieval (RAG optimization).

• Layer 3: Execution (Dynamic):     - Progressive Disclosure (loading extra files/scripts during the task).

• Layer 4: Resilience:     - Error Tracing: Adding error messages from failed tool calls back into the context so the LLM can self-correct (e.g., the Instructor framework's retry logic for Pydantic schemas).

• Foundation: Evaluation:     - Requires golden datasets and production monitoring to validate if context engineering changes improve performance.

Questions and Discussion

• Weather and Logistics:     - Michael (Central Virginia): Reported sunny weather at $68^{\circ}F$.     - Artemas: Confirmed similar conditions at approximately $18\,^{\circ}C$.

• Technical Q&A:     - Will technology solve the context problem? Improvements may help "Needle in the Haystack" issues, but Transformer architectures have inherent physical limits (KV matrix expansion), and cost/latency will always persist as issues for large contexts.     - Is compression native to LLMs? While some labs might abstract this away, AI engineers must often implement it manually, especially when using open-source or custom-deployed models.     - Tool Bloat Case Study (Chris): Chris shared a scenario building a "personal assistant" agent with $40 \text{ to } 50$ tool definitions. He solved the rate-limiting and bloat issues by implementing "Tool Discovery" and "Tool Describe" as two meta-tools. The model performs a semantic search over embeddings of tool keywords, selects what it needs, and then retrieves the full JSON schema.     - UTCP CodeMode: A community project/shim mentioned as highly effective for MCP context management, reportedly offering up to $98\,\%$ token savings by presenting tool calls as typed functions rather than raw JSON schemas.     - Context Mode: A reverse-proxy tool for Claude that hooks into life-cycle hooks to aggressively reduce the number of return tokens from tool calls.     - Student Discount Query (Sam): Sam asked about discounts for the boot camp. Artemas clarified they offer a limited $15\,\%$ discount but do not have a dedicated student/PPP program yet; however, two-installment payment plans are available.