Large Language Model Operations (LLM Ops) and Agentic Systems Architecture

Professional Background and Industry Context

• Artemis has worked in the data domain for approximately $14$ years, occupying roles including Machine Learning Engineer, Data Engineer, Machine Learning Operations (MLOps) Engineer, and AI Engineer.

• Executive leadership experience includes serving as C-level manager at Neptune AI, directly collaborating with global AI research labs, and co-founding Swirl AI to consult enterprises on data/AI strategy, build hands-on solutions, and educate the broader developer community.

• The evolution from MLOps to AI Engineering represents a natural expansion of scaling practices, shifting focus from traditional model training and deployment pipelines toward managing application-layer orchestration, evaluation, and reliability of Large Language Models (LLMs).

Fundamentals of Large Language Model Operations (LLM Ops)

• LLM Ops is defined as the set of procedures, processes, and infrastructure systems that enable the reliable deployment and management of LLM-based applications in production at scale.

• Differences between MLOps and LLM Ops:

  • MLOps focuses on traditional machine learning lifecycle workflows, including training, versioning, feature stores, and model serving infrastructure.

  • LLM Ops governs application-layer processes built on top of foundation models, including dynamic orchestration, model routing, prompt observability, end-to-end evaluation, guardrails, and agentic memory management.

The Cowboy AI Agent's Hierarchy of Needs

• Architectural Layers of Agentic Infrastructure (from base to apex):

  • Bare Infrastructure Layer: Physical infrastructure hardware, including GPUs, CPUs, and low-level compute required to host foundation models.

  • Foundation Model Layer: Base foundational large language models accessed via local deployments or remote APIs.

  • Data Storage Layer: Databases handling dynamic retrieval and persistence, including Vector Databases, Graph Databases, SQL Databases, and key-value stores.

  • Orchestration & Routing Layer: Frameworks and middleware managing operational flow, dynamic routing, chain execution, and agentic tool usage.

  • Observability & Evaluation Layer: Systems tracking execution traces, individual spans, latency, token costs, and automated quantitative evaluation metrics.

  • Security & Guardrails Layer: Governance mechanisms protecting systems against adversarial prompt injections, data leakage, toxic outputs, and operational compliance risks.

  • Agentic Memory Layer: Automated dedicated short-term and long-term memory architectures built specifically for agentic execution loops.

  • Application Layer: User-facing frontend systems and enterprise interfaces.

• Product Rollout Stages Mapped to Infrastructure Needs:

  • Proof of Concept (POC): Requires only Foundation Model APIs without needing complex orchestration libraries or dedicated evaluation stacks.

  • Minimum Viable Product (MVP): Introduces Data Storage layers and basic Orchestration frameworks.

  • Beta Release: Integrates Observability and Evaluation tools to eliminate black-box execution behaviors and capture structured user feedback.

  • General Availability (GA): Demands full Security, Guardrails, and production-grade continuous Evaluation pipelines to protect customer trust and operational stability.

• The Paradigm Shift from "Cowboy" Development to Enterprise Rigor:

  • Historical approach (circa $1.5$ years prior): Rushed development focused exclusively on basic functionality via orchestration tools, skipping observability and evaluation layers.

  • Modern enterprise approach: Flips the hierarchy stack by positioning Observability and Evaluation directly above data storage, establishing them prior to introducing heavy Orchestration or Routing frameworks.

Evaluation-Driven Development (EDD) Framework

• EDD adapts classical software engineering Test-Driven Development (TDD) to non-deterministic LLM-based system development.

• Step-by-Step EDD Implementation Workflow:

  1. Define the Problem: Clearly articulate the business challenge. Derisk technical investment early by identifying if LLMs are strictly necessary or if non-LLM algorithmic approaches are superior.

  2. Build a Minimal Prototype: Create a simple, low-code artifact (e.g., Jupyter Notebook or Excel spreadsheet capturing basic inputs and outputs) to test basic technical feasibility before writing production code.

  3. Define Performance Metrics: Establish high-level product and business success metrics prior to user exposure (frequently owned or assisted by AI Product Managers).

  4. Define Evals: Map business performance metrics directly to quantitative evaluations across every operational node in the target agentic topology.

  5. Build a POC: Develop a lightweight functional interface (e.g., simple spreadsheet or minimal UI) designed solely to expose the solution to end users quickly.

  6. Expose to Target Users ASAP: Gather immediate real-world usage signals and direct feedback loops.

  7. Production Flywheel & Focus on Bad Data: Continuously ingest live traces, collect user feedback, aggregate metrics within an observability platform, isolate failing evals or negative user feedback ("bad data"), perform targeted prompt engineering/fine-tuning/topology adaptations, and redeploy.

Spectrum of Agentic Topologies

• Level 1: Simple LLM Application

  • Topology: Direct input prompt $\rightarrow$ LLM processing $\rightarrow$ Output answer.

  • Scope: Non-complex text transformations or basic generation tasks.

• Level 2: Retrieval-Augmented Generation (RAG)

  • Topology: User query $\rightarrow$ Embedding generation $\rightarrow$ Similarity search against private storage (Vector DB, Graph DB, SQL DB) $\rightarrow$ Context injection into prompt template $\rightarrow$ Grounded LLM output generation.

• Level 3: Agentic RAG

  • Topology: Introduces dynamic action loops into standard RAG pipelines.

  • Capabilities: Query rewriting, multi-query expansion, tool and source selection routing, post-retrieval re-ranking, and output analysis/reflection loops.

• Level 4: AI Agents

  • Components: Dynamic task planning strategies, explicit tool capabilities for external environment interaction, and structured short-term memory (in-context state) combined with long-term memory (external storage).

• Level 5: Semi Multi-Agent Systems

  • Architecture: Multiple specialized agents operating within the same process and runtime execution context. Agents share system state and memory, managed by a master/lead agent delegating execution steps.

• Level 6: Real Multi-Agent Systems ("Internet of Agents")

  • Architecture: Fully distributed network of heterogeneous autonomous agents operating across distinct runtimes, network boundaries, frameworks, and LLM backends communicating via standardized communication protocols.

Tracing, Instrumentation, and Observability

• Instrumentation: The process of adding hooks and code wrappers into application logic to capture runtime execution metadata.

• Core Tracing Definitions:

  • Span: The smallest atomic unit of work within an application step (e.g., generating an embedding vector, executing a database lookup, issuing a single model request).

  • Trace: An acyclic execution graph connecting all individual spans associated with a single end-to-end user request.

• Metadata Extraction Across a Trace (RAG Example):

  • Query Step: Ingests raw input text.

  • Embedding Step: Captures raw input text, output vector values, embedding model name/version, and computation latency.

  • Vector Database Search Step: Captures input search vector, parameter thresholds, pre-filtering rules, post-filtering rules, index hit count, retrieved text chunks, retrieved chunk vector values, and retrieval latency.

  • Prompt Construction Step: Captures exact prompt template string, system instructions, and merged dynamic context text.

  • Generation Step: Captures complete input prompt payload, generated output text, exact model identifier, overall execution latency, and advanced metrics including $\text{time to first token}$ and $\text{inter-token latency}$.

• Diagnostic Failure Modes Tracked via Tracing:

  • Database API unavailability or network timeouts.

  • Pre-filter over-constraining resulting in $0$ retrieved search hits.

  • Poor contextual relevance leading to downstream model hallucination.

• Human Feedback Integration:

  • Frontend user interactions (e.g., thumbs up/down, explicit text corrections) are collected and dynamically appended as context metadata directly onto the specific trace ID.

Evaluation Datasets and CI/CD Pipeline Practices

• Continuous Integration (CI) in LLM Ops requires managing three specific types of validation datasets:

  1. Development Dataset:

  • Purpose: Serves as the guiding target for active system engineering.

  • Composition: Contains hard edge cases and known failing inputs where current accuracy is intentionally low (e.g., $50\%$ to $70\%$ failure rate).

  • Objective: Drive system improvements until failure rates drop (e.g., down to $30\%$ or lower).

  1. Balanced Dataset:

  • Purpose: Serves as a standard validation benchmark reflecting real-world input distributions.

  • Target: Used to monitor overall health without expecting $100\%$ success rates. Reaching $100\%$ accuracy on a balanced dataset typically indicates evaluation flaws or systemic overfitting.

  1. Regression Dataset:

  • Purpose: Acts as a strict gatekeeper in CI pipelines to prevent functionality regressions.

  • Target: Must consistently achieve a $100\%$ pass rate or maintain strict evaluation threshold score baselines.

  • Maintenance Rule: Once a problem sample in the Development Dataset is successfully resolved, it is permanently transferred to the Regression Dataset to guarantee future updates never break previously established fixes.

Production Monitoring and Business Metric Hierarchy

• Automated Production Monitoring:

  • Derived automatically from code instrumentation, tracking aggregated latency, token usage, dollar costs, operational errors, and delayed ground-truth user evaluations.

• Prompt Monitoring:

  • Explicitly tracks prompt template versioning across individual nodes in production, monitoring metric oscillations tied to specific prompt changes.

• Hierarchical Alignment of Business and Technical Metrics:

  • Lagging / North Star Metric: High-level business impact objective (e.g., support twice the user volume with constant support headcount, or reduce support operational cost by $50\%$).

  • Leading Metrics: Measurable operational performance indicators that drive the North Star metric (e.g., average response time per query, average ticket closure duration).

  • Node-Level Technical Evals: Specific quantitative evaluation metrics ($0$ to $1$ score bounds or pass/fail rules) attached directly to inputs and outputs of individual sub-nodes within the agentic topology designed to optimize leading metrics.

Questions & Discussion

• Slide Availability & Recordings:

  • Presentation slides are not distributed prior to sessions. Session recordings are emailed to all registered participants within $48$ hours.

• Orchestration Tools & Strategy:

  • Current leading orchestrators: LangGraph and CrewAI.

  • Strategic guidance: Avoid heavy orchestrator frameworks in early stages until complex state management strictly requires them. Prefer lightweight libraries such as Instructor for enforcing structured outputs and retry logic.

  • Terminology Note: Historical references to "controllers" in early LLM diagrams are synonymous with modern "orchestrators".

• Defining Evals without Domain Expertise:

  • Automated evals should not be deployed without input from subject matter experts.

  • Teams should collaborate directly with domain experts or leverage historical enterprise data (e.g., $5$ to $10$ years of recorded human workflow logs) to construct valid ground-truth datasets.

• Impact of Foundation Model Upgrades:

  • Upgrading foundation model versions frequently degrades application performance due to prompt sensitivities. Strict regression datasets are essential prior to updating model versions.

• Observability Platform Selection:

  • Popular choices include LangSmith (established default choice), LangFuse (rapidly growing open-source adoption), and Opik.

• Role of AI Product Managers (AIPMs):

  • AIPMs drive high-level non-technical EDD phases: problem definition, initial spreadsheet prototyping, performance metric definition, eval dataset curation, prompt alignment, and domain expert coordination.

• Structure of LLM Ops Teams:

  • Dedicated standalone "LLM Ops Teams" should generally not exist. Unlike traditional MLOps, AI Engineers are expected to own the end-to-end stack, including infrastructure, instrumentation, tracing, and CI/CD evaluation pipelines.

Educational Initiatives and Resources

• Swirl AI End-to-End AI Engineering Bootcamp:

  • Teaches practical implementation of Evaluation-Driven Development, observability infrastructure, evaluation setups, model routing, and production guardrails.

• Exclusive Discount:

  • Code: LLMops 10 (applies a $10\%$ discount at checkout).