Advanced Prompt Engineering and Technical LLM Architecture

Chain of Thought (CoT) Prompting and Reasoning

  • Conceptual Definition: Chain of Thought (CoT) refers to an internal mechanism added to Large Language Models (LLMs) that allows them to "log out" thoughts in the middle of a process. The system refers back to these intermediate thoughts to decide how it acts.
  • Mechanism of Reasoning: Contrary to the metaphor of "thinking," LLMs achieve this through training on vast amounts of data where they are taught to output tokens while answering inputs to facilitate self-reflection.
  • User Interface Indicators: Modern UIs often label this as "Extended Thinking."
  • Extended Thinking Metaphor: This is described as increasing the window or "size of the brain" by allocating more tokens to the reasoning steps. It essentially increases the model's processing window during a task.
  • Prompting Application: Using the phrase "Think step-by-step" forces the model to identify assumptions and trace out specific effects.
    • Example: Asking a model to trace the effect of a 150 basis point150 \, \text{basis point} rate increase on a real estate portfolio yields more granular and structurally different reasoning than a general prompt.

Persona Assignments and Behavioral Steering

  • Definition: Assigning a specific role to the LLM (e.g., "Act as a senior software engineer" or "Act as a senior asset manager") to influence the tone, scrutiny level, and output structure.
  • Functional Impact:
    • Senior Software Engineer: Causes the model to scrutinize code more critically based on its training on professional corpuses.
    • Senior Asset Manager: Shifting a general description of a complex (e.g., a garden-style apartment in Tampa, Florida) to an industry-specific analysis.
  • Output Comparisons:
    • General Prompt: Provides standard descriptions and correct but generic information.
    • Persona Prompt: Elicits specific metrics such as a $24,000,000\$24,000,000 sale price, NOIs (Net Operating Income), implied exit cap rates, and market softening details in basis points (bpsbps).

Technical Architecture: Context Windows and Retrieval

  • The Context Window: This is the limit on how much input can be provided in a single query.
  • The Attention Window: Even if a model like Gemini has a context window of 1×106 tokens1 \times 10^6 \, \text{tokens}, it does not pay equal attention to every token.
  • Retrieval-Augmented Generation (RAG): A method to manage context by selectively retrieving precise pieces of information (e.g., specific properties in a portfolio segmentation) to avoid context dilution and high costs.
  • Semantic Search: Using mathematical calculations like cos⁡\cos (cosine similarity) on vector numbers (lists of numbers) to find data relevance.
  • Memory Persistence: In agentic systems, "memory" is achieved by persisting data into a standard database between steps in a multi-node flow, decoupling long-running data from discrete calls.

Training Data and Fine-Tuning

  • Data Annotation: Companies like Scale AI (acquired by Meta) employ large groups of experts to tag and annotate data for training datasets.
  • Proprietary vs. Open Models: Large generalized models (OpenAI, Claude, Llama, Grok) use proprietary data.
  • Fine-Tuning: The process of taking an open base model and training it further on a custom, high-confidence dataset to improve reliability and specialized function.

Advanced Prompting Techniques and Pitfalls

  • Few-Shot Learning: Providing specific examples (shots) within a prompt to steer the output format and tone. This is distinct from fine-tuning because it happens within the context window.
    • Example: Inputting a previous email to ensure the AI-generated response matches a specific human tone.
  • Vague Instructions: LLMs may prompt users for more info as a UX feature, but specific structures yield better results initially.
  • Context Overload vs. Starvation:
    • Overload: Providing irrelevant noise (e.g., the seller's name in a financial analysis) dilutes the priority weights of the model.
    • Starvation: Providing too little information prevents the model from understanding how to solve the problem.
  • The Sycophant Problem: Models are often overly agreeable ("You are absolutely right"). Users should invite a "second opposing perspective" or ask the model to review itself as a critical expert (e.g., a senior engineer finding security issues in its own previously written code).

Questions & Discussion

  • Chris's Question: How do we manage context windows to avoid hallucinations in complex AI agents?
    • Response: Use memory flows and RAG. Fragment data to retrieve only what is necessary, saving money and improving accuracy. Use intermediate nodes to persist memory into databases.
  • Matthew's Question: How do we know the training data makes a persona (like an asset manager) reliable?
    • Response: For large models, you rely on the provider's annotation (Scale AI). For specific needs, use fine-tuning on your own datasets.
  • Andrew's Question: How to maximize productivity when switching between Claude and Gemini?
    • Response: Use an orchestration layer (e.g., Claude Cowork) rather than multiple browser windows. This allows for a "command center" approach to sharing context.
  • Charlie's Question: How to ensure data in generated market reports isn't made up?
    • Response: Validation is mandatory. AI can generate reports in seconds, but human verification prevents credibility risks, particularly in legal or high-stakes environments.
  • Ali's Question: Why does the model forget previous reports when I upload several for a new synthesis?
    • Response: This is a context window limitation. Large generalized tools like ChatGPT might not scrutinize every doc. Custom logic or specific GPTs that use deterministic code to retrieve relevant data are more effective.
  • Elijah's Question: How to prompt for mapping unstructured data (e.g., disbursements in PDFs) into specific values/hash maps?
    • Response: Use Vision LLMs and tools like Llama Index or Llama Extract. Define a schema (key-value pairs) for the model to target during extraction.
  • Sean's Question: Can you verify output with another AI? Is it better to cut context yourself or let the AI do it?
    • Response: Using a smaller, cheaper "judge model" to evaluate a larger model's sentiment or quality is standard practice. You should proactively ask the AI what information is relevant and then validate its selection before proceeding.
  • Andre's Question: Is it more accurate to check output across different platforms (ChatGPT vs. Claude)?
    • Response: Yes. Different base models excel at different things. Playing models against each other is a legitimate validation approach.

Prompt Management and Version Control

  • Categorization: Segment prompts by use case and function to prevent irrelevant context from leaking between projects.
  • Standardization: Create approved use cases, standard structures, and specific tones for team-wide use.
  • Version Control Best Practices:
    • Never overwrite the same file; save new versions (e.g., v1, v2) with dates.
    • Use Git for professional repository management if possible.
    • Centralize libraries in tools like LangChain to share high-value prompts with a broader team.