Summary of Advanced Rack Project Preparation Meeting

Introduction

  • Overview of project for advanced rack (RAC) research.
  • Focus for April: assist groups in starting their projects weekly.

Summary of Previous Topics

  • Introduction to RAC, agentic tooling, agentic reasoning, image RAC, agent evaluation.

Project Setup

  • Theodore will demonstrate how to set up the evaluation framework.
  • Project involves creating a notebook to guide research.
  • Participation encouraged for those interested in advanced rack.

Group Insights

  • Participants Check-In: Engagement with team members and assignment of project roles.
  • Exploration of research interest: Nikil interested in evolutionary games; Siddharth in cybersecurity.
  • Aim to have drafted outlines for projects by April 19.

Evaluation Framework Overview

  • Emphasis on the importance of evaluations in scientific research.
  • Queries can be classified as:
    • Single Query: Single lookup.
    • Multi-Hop Query: Multiple lookups required.

Query Classification Details

  • Single Hop Query:
    • Specific: Example: "What's the capital of France?"
    • Abstract: Example: "How do French people think about Americans?"
  • Multi-Hop Query:
    • Specific: Example: "What's the capital of France and its population?"
    • Abstract: Example: "How has the attitude of French people towards Americans evolved over the last 50 years?"

Concept of Evaluation Metrics

  • Precision: Proportion of retrieved chunks that answer the query.
  • Recall: Number of relevant chunks returned by the query.
  • Response Relevance: Relevance of the actual response to the query.
  • Faithfulness: Grounded nature of a response based on the source document.
  • Factual Correctness: Correspondence of the response to factual data.

Advanced Chunking Techniques (Presented by Theodore)

  1. Token Text Splitting:

    • Splitting text based on token counts (e.g., 512 tokens).
    • Limits adherence to language structure.

  2. Markdown Node Pass Parser:

    • Parses text based on markdown structure (headers and paragraphs).
    • Maintains document hierarchy.

  3. Markdown Element Node Parser:

    • Parses individual markdown elements (e.g., lists, headers).
    • Useful for structured content like tables.

  4. Semantic Splitter Node Parser:

    • Splits text based on meaning.
    • Preserves context and coherence.

  5. Sentence Splitter:

    • Divides text into sentences for clearer meaning and context.

Evaluation and Implementation Process

  • Skeleton setup leads to creating a vector store and query engine.
  • Implementation will focus on handling complex datasets and crafting queries for evaluation.

Future Considerations

  • Next week’s focus will be on Reasoning Techniques in conjunction with RAC.
  • Questions encouraged from team members to deepen learning on chunking techniques.

Conclusion

  • Invitation for group participants to collaborate on the advanced RAC.
  • Reminders for logistics and upcoming meetings to discuss reasoning techniques and project progress.