Advanced Document Analysis and Project Management in AI Workflows

Introduction to Document-Heavy Workflows and AI Projects

  • The speaker discusses the shift towards "document heavy work" and how to manage large-scale data sets using segmented AI environments.

  • Concepts of Projects and Canvases:

    • Cloud Projects: A feature in Claude (implied by the interface description) that allows for segmented sets of files and a persistent system prompt.
    • ChatGPT Canvas: A similar feature in ChatGPT intended to help manage specific areas of work.
    • These product suites create a partitioned workspace that reads from a consistent set of files and system prompts on repeat for every chat. This eliminates the need to provide context or instructions manually in setiap new interaction.
    • These tools function effectively as a "walled garden" where the AI only cares about the specific data provided rather than the general knowledge of the "whole world."

  • Workflow Efficiency and Scale:

    • Users can upload a significant volume of data, ranging from a few files to 20, 40, or even 100 files.
    • Once a workspace is established with a persona, verification rules, and identified behavior, it creates a highly focused environment for solving specific business problems.
    • This approach transitions AI use from "prompting a random chat" to working within a highly specialized tool designed for day-to-day business operations.

Anatomy of System Prompts and Project Instructions

  • The "System Prompt" or "Project Instructions" acts as the architectural specification for how the AI should behave within that segmented space.

  • Key Components of a Robust Instruction Set:

    • Persona Assignment: Explicitly defining the role, such as "You are a commercial real estate research analyst."
    • Core Capabilities: Listing the specific functions the AI is expected to perform.
    • Output Standards: Listing specific expectations for the quality and style of the responses.
    • Verification Rules: Explicitly stating rules for how the AI must Verify data.

  • Injection and Templating:

    • These instructions are typically injected into the prompt using a templating system controlled by the application layer. This ensures that every query processed within the project includes the necessary guidelines.
    • The speaker notes that these instructions are often written in Markdown format to organize different sections of the prompt effectively.

  • Collaborative Benefits:

    • Projects can be shared with coworkers or project members. Once the system prompt and file sets are established, others can jump into the workspace and ask questions without needing to understand the underlying system-level prompting.

Advanced Extraction Challenges and Specialized Tools

  • Challenges with Edge Models:

    • General-purpose models (like standard Claude or ChatGPT interfaces) often struggle with highly nuanced data extraction.
    • Specific hurdles include:
      • Pie charts, bar charts, and graphs.
      • Tables or charts that are cut off and continue on the next page (e.g., in a 2,000-page zoning ordinance document).
      • Rotated tables or images.
    • Large models may "mangle" this data because they lack the specific logic to handle physical layout distortions.

  • Specialized SaaS Wrappers:

    • There are specialized products, such as Llama Cloud, that act as specialized extraction layers.
    • These tools are SaaS providers that wrap large models (LLMs) with proprietary logic designed for advanced data processing.
    • For instance, these systems can detect if a chart is rotated and self-correct it before passing the information to the final model for analysis.
    • Users with highly specialized workflows should consider these wrappers instead of generalized platforms for optimal data quality.

Discussion and Audience Q&A

  • Sean's Question regarding Prompt Interaction:

    • Question: How do a master project prompt and a secondary prompt (given in a specific chat) interact? Does one override the other, or is the behavior muddled?
    • Response: The speaker explains that while the exact application layer logic is proprietary, it is likely both are added. The system prompt is usually injected as a baseline, and the user's specific query/prompt is appended to it. This creates a "huge wall of text" for the model to process. The speaker warns that adding conflicting instructions can "muddle" the AI's behavior. This is related to "prompt injection," where users try to bypass safeguards or existing system instructions with new text.

  • Charlie's Question (University of Illinois, Finance and Data Science) regarding Instruction Creation:

    • Question: How exactly was the massive set of project instructions generated?
    • Response: The speaker manually wrote the instructions in advance of the course to serve as a template. He used a Markdown structure to categorize his needs (persona, capabilities, verification, formatting). The instructions are a "choose your own adventure" where the user identifies what they care about most (e.g., data referencing, specific formatting) and documents it as a string that is injected into every subsequent prompt.

  • Kush's Question (University of South Florida Graduate, Finance) regarding Excel Extraction Failures:

    • Question: Why does the AI (Claude or Copilot) sometimes say it is extracting info into an Excel template, but the final template is blank? Even when corrected, it acknowledges it is blank but won't fill it.
    • Response: This is a failure in the handoff between the LLM and the tool invocation layer. The model identifies that it should use a tool to write to Excel, but the deterministic code that executes the command fails. This happens because:
      1. The model might be "hallucinating" that it invoked the tool.
      2. There is a permissioning issue.
      3. The payload formatting is incorrect. Smaller or less powerful models often identify the correct tool but "butcher" the body of the data payload. If the format of the data does not perfectly match what the external software expects, the operation will error out, even if the AI thinks it completed the task.

Recap of Key Topics

  • Lease Abstraction: Using AI to pull specific details from legal and commercial documents.
  • Market Research: Using tools to identify trends (e.g., Biggest trends in the Austin industrial market).
  • Hallucination Detection: Using citation-based rules and verification guidelines to ensure accuracy.
  • Project Management: Leveraging segmented workspaces to handle long-lived, complex work streams.