Client Meeting Notes 9/26

Introduction of Project Team and Scope

• Dave, the coordinator, introduces Dwight Williams as a dedicated technical resource for this project and the "text to 3D" project, as well as overseeing five GMU capstone projects.

• Dwight provides a brief introduction about himself:

  • Joined Leidos in $2019$. His background is in nuclear engineering, holding bachelor's, master's, and doctorate degrees, along with a PE license.

  • He has worked for the Department of Defense (DoD) and Department of Energy (DoE) in the DC area.

  • He previously taught at MIT, where he is still on staff.

  • He is familiar with AI, large language models (LLMs), and $3D$ tools.

  • He offered to answer questions about working as a federal employee or contractor.

• The project team comprises all members present in the meeting.

Project Deliverables and Initial Steps

• Request for Proposal (RFP) Review: The team had one week to review the RFP, deliverables, and next steps for the first deliverable, which is a Work Breakdown Structure (WBS).

• WBS and Trade Study Table Submission: The team submitted a draft WBS and a trade study table that researches LLMs for various input types.

Clarification on Project Scope: Data Formats and Standardization

• Initial Focus: The core project aims to convert data, specifically "JSON to the IFC GLB."

• Researching Other Input Types: The team is also researching raw data types, such as Building Information Modeling (BIM) data inputs.

• JSON as a Suggestion, Not a Requirement: JSON was initially suggested but is not a mandatory format. The overarching goal is to achieve "standardization" of file formats that are suitable for $3D$ modeling tools.

• Revit as an Example Tool:

  • Students are encouraged to use a free trial of Revit, which possesses IFC import/export capabilities.

  • Experimentation Task: The team should export a sample Revit design project as an IFC file to observe its textual format. Subsequently, they should re-import it to gain an understanding of the data flow.

  • Configuration Importance: It's crucial to note that different configurations for export exist, allowing for the inclusion or exclusion of specific data elements, data types, and assets. If certain data is not explicitly exported, it will not be imported back, leading to missing information.

  • This experimentation, to be conducted over the next $1-2$ weeks, will be instrumental in formulating the required format for successful import into a Revit $3D$ model.

• JSON Schema (or equivalent): The team is not mandated to use JSON but should design a schema or an efficient format to store data for subsequent import.

• LLM Response Adjustment: The response generated by the LLM will need to be adjusted into the desired format. For instance, if the AI (LLM) provides a textual response, an "adapter" (a software utility or routine) will be necessary to convert this output into an importable standard for applications like Revit.

• Revit as One of Many: Revit serves as a representative example. Many $3D$ applications allow for IFC file imports, but each may have unique nuances in metadata and tagging required for a successful import.

Raw Data Input Types

• Sources: Raw data can encompass text documents, standards documents, PDFs, drawings, and images.

• Data Package: An unclassified data package will be provided at a later stage.

• Focus on Structure: The initial focus for the project is on extracting structural information, including the number of floors, rooms, wall thickness, potential locations of doors and windows, and other specific structural details.

• Challenges with Images:

  • LLMs face difficulties in accurately interpreting sizing from images due to the absence of real-world frames of reference (e.g., distinguishing the height of a large wall or the distance of a person from a wall).

  • Spatial aspects are critically important for structural import (e.g., floors must be precisely aligned). Misalignments would indicate either AI errors or issues in the conversion process, necessitating troubleshooting.

• Validation: Students are required to develop routines and scripts to validate the imported data and identify any errors or points of failure.

Review of Work Breakdown Structure (WBS)

• The team shared a draft WBS, which was found in Section $8.2$ of a document.

• Feedback provided:

  • The current WBS depicts strictly linear predecessors. However, some tasks (e.g., preconditioning, LLM development and orchestration) could potentially be performed in parallel to facilitate a divide-and-conquer approach.

  • Documentation and reporting (task $1.5$) will likely be dependent on the completion of all preceding tasks ($1.1$, $1.2$, $1.3$, $1.4$, and $1.5$) rather than just a single predecessor.

  • The trade study (task $1.1$) is expected to significantly influence many subsequent decisions.

Review of Trade Study Table

• The team presented a completed trade study table detailing LLMs capable of processing text, image, PDF, and BIM files. The table included an overall score for each LLM.

• Proposed Approach outlined by the team:

  • Convert any text data into a standard format using specific LLMs.

  • Trace and extract information from images using other tools, followed by conversion to a standard format.

  • Trace scanned PDFs or convert them directly to a standard format.

  • Convert BIM files into a standard format.

  • Link all converted data together before proceeding with the processing for $3D$ generation.

• Language Translation Consideration:

  • Tools such as Tesseract and ABBYY (historically used for language translation) possess built-in translation engines.

  • It is probable that the majority of raw data will be in a foreign language. While LLMs are capable of ingesting foreign language data and providing responses in English (by internal interpretation), the team should account for potential explicit translation needs.

Deliverables: Reporting and Demonstrations

• Weekly/Bi-weekly Status: A high-level, one-page status update submitted before each call is encouraged. This allows for focused discussion and targeted questions, thereby improving meeting efficiency.

• Fall Semester Deliverables: Existing class-oriented deliverables (e.g., briefings, design documents that explain the problem and proposed solution) are considered sufficient. No additional work specifically for this project is required beyond the class assignments.

• Spring Semester Deliverables: Beginning in the spring semester, the focus will shift to bi-weekly demonstrations of the developed capabilities. These demonstrations can be straightforward, such as showcasing a design document or running a simple command-line routine to display input and output.

• Lessons Learned Document:

  • First Submission: Due at the end of the fall semester, this document should cover the project planning phase, including resolutions reached, processes learned, and insights derived from decisions made (both successful and unsuccessful paths taken).

  • Second Submission: Due at the end of the spring semester, this document should detail the development processes, pitfalls encountered, and how these challenges were overcome or led to adjustments in scope. The importance of timely communication of challenges to the team was emphasized.

• Preliminary Design Package: The final project submission for the fall semester's class assignment can serve as this deliverable.

• Final Design Package: The final project submission for the spring semester's class assignment can serve as this deliverable.

Next Steps for the Team

• Decision Making: Based on the insights from the trade study, the team needs to decide which LLMs or tools to commence work with (e.g., initiating with higher-scored options like GPT-$4.0$). They should also consider whether tasks can be split, for example, by data type (PDFs, images, text).

• Scheduling and Assignment: The team must develop a detailed schedule, define specific tasks, and provide time estimations for the planning portion of the current semester. They should also consider outlining parts of the schedule for next semester's development phase.

• Redefine Problem and Solution: The team is tasked with refining their understanding of the challenge and articulating the overall solution they aim to achieve, clearly stating its practical benefits.

Future Support and Meeting Logistics

• Continued Meetings: The team is scheduled to meet again next Friday to ensure continued progress.

• Additional Technical Resources:

  • A Cloud Environment Specialist will be provided to assist with setting up and utilizing the cloud environment for testing and development.

  • A Software Engineer will be available for support with specific coding challenges.

  • These additional resources are expected to be introduced on the next call.

• Dwight's Role: Dwight will provide ongoing oversight and direction throughout the project.

• Next Meeting: The upcoming meeting is scheduled for next Friday at $12:30$ PM and will last for $45$ minutes.