LangChain Advanced Retrievers, AI Agents, Tool Integration, and Memory Systems

Advanced Retrievers in LangChain

Multi-Query Retriever

• Definition: The Multi-Query Retriever is a retrieval strategy designed to solve the problem of phrasing mismatches between a user's query and the source documentation.
• Problem Statement: Users often use casual, vague, or short wording (e.g., "How many leaves do I get?") that does not match the formal or technical language of a document (e.g., "Annual entitlement of paid time off"). A standard similarity search on raw user queries might miss the relevant context.
• Mechanism:
  • It uses a Large Language Model (LLM) to rewrite the user's initial query into $3 \text{ to } 5$ different variations.
  • For each variation, it retrieves relevant document chunks from the vector store.
  • It then combines all retrieved chunks and deduplicates them to provide a comprehensive context for the final answer.
• Ideal Use Cases:
  • Casual or short user queries.
  • Documents using formal, technical, or legal language (e.g., HR policies).
• Trade-offs:
  • Performance: It is slower than a normal retriever because it requires multiple LLM calls for rephrasing.
  • Cost: It is more expensive due to the increased token usage from multiple LLM variations.
• Library Status: In LangChain $1.x$, this retriever is deprecated in the main library but remains accessible via the langchain-classic package. Users are encouraged to build similar patterns manually using LangChain Expression Language (LCEL) for better transparency and control.
• Conceptual Analogy: The speaker compares it to a "Tree of Thought" where one query branches into several, each generating its own context before being merged.

Contextual Compression Retriever

• Definition: A retriever that compresses retrieved chunks to extract only the sentences relevant to the query before feeding them to the LLM.
• Problem Statement: Standard similarity searches often retrieve the "right" chunk, but that chunk may contain a high volume of irrelevant "noise" or sentences mixed in with relevant ones.
• Consequences of "Bloated Context":
  • Waste of tokens.
  • Potential for LLM distraction or confusion caused by irrelevant details.
• Mechanism:
  1. It retrieves chunks normally using a "base retriever."
  2. It runs each chunk through a "compressor" (usually an LLM chain like LLMChainExtractor).
  3. The compressor filters out irrelevant information, providing a tighter, cleaner context.
• Use Cases:
  • Very large or noisy document chunks.
  • Local LLMs with small context windows.
  • Scenarios where the LLM is easily distracted by irrelevant data.
• Implementation Components:
  • Base Retriever: The standard vector store retriever.
  • Compressor: Specialized logic (e.g., LLMChainExtractor.from_llm(llm)) to filter text.

Optimization and Strategy Caveats

• Over-Engineering: The speaker warns against unnecessarily chaining multiple advanced retrievers (Multi-Query + Compression). For well-structured documents, a simple retriever tuned with the correct chunk size and $k$ (top-k) value is often sufficient.
• Key Parameters for Quality:
  • Chunk Size: Determines how much info is retrieved.
  • $k$ (Top-k): Determines how many documents are retrieved. High $k$ increases cost/context complexity; low $k$ risks losing context.
• Search Types: Vector stores typically offer two main search types:
  1. Similarity Search.
  2. MMR (Maximum Marginal Relevance): Focuses on diversity in results.

Understanding AI Agents

Core Definition and Components

• AI Agent: An application capable of reasoning, decision-making, and performing tasks through actions rather than just generating text.
• The Analogy of Components:
  1. LLM (The Brain): Handles the reasoning and decides which actions to take.
  2. Memory: Allows the agent to remember historical interactions.
  3. Tools (The Hands): Enables the agent to perform actions like API calls, sending emails, or searching the internet.

The Agent Execution Loop

1. Perception: The agent reads the user's request.
2. Reasoning: The LLM decides what should happen next based on the request.
3. Action: The agent uses a tool (e.g., hits a weather API).
4. Observation: The agent observes the result/data returned by the tool.
5. Result Generation: The LLM articulates the observation into a meaningful response for the user.

Agent Implementation Details

• create_agent Function: A pre-defined LangChain function used to initialize an agent.
• System Prompts: The "guardrails" or "grounding rules" of the agent. It defines:
  • Persona: Who the agent is (e.g., "Weather Assistant").
  • Constraints: What it should/should not do (e.g., "Only answer programming questions").
  • Tone/Style: Tone (humor, formal) and formatting (use emojis).
• Invoking the Agent: Messages are passed as a list of dictionaries containing a role (user/system) and content (the query).
• Extracting Output: To get the final text, developers usually grab the content of the last message in the response array: response["messages"][-1].content.

Memory in Agents

Short-Term vs. Long-Term Memory

• Short-Term Memory: Limited to a specific thread or conversation session. It tracks the immediate history of the current interaction.
• Long-Term Memory: Stores and recalls information (like user preferences or settings) across different conversations and sessions.

Checkpointers and Persistence

• In-Memory Saver: Uses InMemorySaver from langgraph.checkpoint.memory. This stores conversation history in the system RAM. It is lost when the application restarts.
• Database-Backed Persistence: Uses PostgresSaver (or SQLite equivalents) to store session history in a database.
  • Requires: langgraph-checkpoint-postgres and a driver like psycopg-binary.
  • Connection String Format: postgresql://[user]:[password]@[host]:[port]/[database_name].
  • Setup: The checkpointer.setup() function automatically creates the necessary tables (checkpoints, etc.) in the database if they do not exist.

Multi-User Management: The Thread ID

• thread_id: A unique identifier provided in the configurable dictionary when invoking an agent.
• Purpose: It allows the agent to distinguish between separate conversations. In a real-world multi-user app, developers must generate and track these IDs (often using UUIDs) to map conversations to specific users.

Tools for AI Agents

Tool Taxonomy

1. Built-in Tools: Plug-and-play tools available in LangChain (e.g., Wikipedia, DuckDuckGo, Tavily).
2. Custom Tools: User-defined Python functions decorated with @tool.

Creating Custom Tools

• Function Structure: A tool is a standard Python function.
• The Docstring: The most critical part of a custom tool. The LLM reads the docstring to understand what the tool does, what arguments it takes, and when to use it.
• Decorator: All custom functions must use the @tool decorator to be recognized by the agent.

Featured Tool: Tavily Search

• Purpose: A search engine optimized specifically for AI agents, providing real-time, factual results.
• API Configuration: Requires a TAVILY_API_KEY.
• Advantage: Provides current data (e.g., protests, news) that is beyond the LLM's training cutoff date.

Featured Tool: Open Weather Map API

• Scenario: A weather agent making live API calls.
• Logic:
  1. Hit api.openweathermap.org/data/2.5/weather.
  2. Pass a city as a query and an app_id (API Key).
  3. Set units (e.g., metric for degrees Celsius).
  4. LLM reasons that the user's travel plan requires a weather check and decides to invoke this function.

Questions & Discussion

• Q: Can we see the variations generated by Multi-Query?
  • A: Not directly in the console, but they are visible via observability tools like LangSmith through execution logs.
• Q: What is the difference between AI Agents and Agentic AI?
  • A: An AI Agent is a single application performing a task; Agentic AI refers to the broader complex workflow or ecosystem built to solve business problems.
• Q: Does LLM perform DB read/write operations directly?
  • A: No. LLMs only generate text. To interact with a database, you must provide the LLM with a Tool (a Python function) that contains the code to perform DB operations.
• Q: Can an agent call multiple tools?
  • A: Yes. The LLM can decide to use one or several tools (e.g., weather + search) to answer a single query.
• Q: What is the username/password for Postgres?
  • A: The default username is usually postgres. The password is set by the user during installation.
• Q: Is Tavily better than Google Search?
  • A: It is specifically optimized for agents and offers a free tier ($1,000$ credits/month), whereas Google Search is generally paid and structured differently.