2.1 Architecture Design Patterns of LLM-Powered Application

Architecture Design Patterns for LLM Applications

Introduction

  • Focus: Architectural design patterns for applications built on Large Language Models (LLMs).
  • Goal: Understand different design patterns and their associated security controls to select the most suitable pattern for specific application needs.

Typical Design Patterns (Non-Exhaustive)

  • Four main patterns, with potential for:
    • Mix-and-match combinations for new designs.
    • Innovations from new LLM deployments, APIs, agents, and task management.

1. Prompt Engineering Only

  • Definition: Crafting prompts for LLMs (e.g., GPT) via APIs to generate desired responses.
  • Analogy: Natural language programming.
  • Implementation:
    • May involve direct use of OpenAI (GPT-4, GPT-3.5, or later versions like GPT-5).
    • Often incorporates a vector database for efficiency.
Vector Database
  • Functionality:
    • Embeds text (words to vectors).
    • Enables fast similarity searches (cosine similarity).
  • Uses:
    • Saving prompts for reuse.
    • Providing memory for LLMs, addressing context window limitations.
  • Context Window:
    • Limited input and output length.
    • Vector database and language chains help summarize and compress information to fit within the context window.
  • Benefits:
    • Saves data locally, avoiding reliance on OpenAI.
    • Improves memory usage and search speed.
Security Controls for Prompt Engineering
  • Importance: Often overlooked.
Access Control
  • Vector Database: Protect sensitive information (custom data, internal documentation).
    • Current Limitation: Lack of encryption in vector databases, making access control crucial.
  • OpenAI API: Secure API tokens under your organization.
    • Secret Management: Protect and rotate API keys.
      • Avoid hardcoding in code.
      • Use environment files (outside source control).
      • Employ secret management tools (HashiCorp Vault, AWS Secret Manager).
Prompt Validation
  • Purpose: Prevent prompt injection attacks (similar to SQL injection).
    • Input Validation: Check for malicious inputs.
    • Output Validation: Ensure desired and safe responses.
  • Defense Against Prompt Injection: Prevent jailbreaking and hallucinations.
  • Third-Party Tools:
    • Amazon Macie: Detect PII/PHI data.
    • Azure OpenAI suite: PII data detection.
  • Misuse Detection:
    • Before sending requests to LLMs, detect misuse, jailbreaking attempts, or offensive content.
  • Output Modulation:
    • Use OpenAI's Modulation API or other tools.
  • Logging and Monitoring:
    • Log user requests.
    • Actively monitor for flaws with monitoring tools (CSPM).
    • Leverage traditional cloud security measures.
Developing Applications with Prompt Engineering
  • Typical Tools: Langchain.
Key Considerations (8 points)
  1. Specify Role: Define the persona for the LLM (e.g., cybersecurity expert).
  2. System Message: High-level instructions for the LLM (using OpenAI playground or Langchain).
  3. Specific Instructions and Examples: Provide security scenario examples (e.g., defending against cyberattacks).
  4. Context: Information from the vector database.
  5. Query: User input (validate carefully).
  6. Output Formatting: Specify desired text-based output format (plain text, JSON, Markdown).
  7. Constraints: Set limits (token length), control the tone (positive, authoritative, polite).
  8. Temperature: Control creativity vs. precision (0 = precise, higher values = more creative/prone to hallucination).
  • Development Languages: Python (common), Java, C, C++.

2. Fine-Tuning

  • Definition: Adapting a pre-trained (foundation or base) model using additional training data.
Purpose of Fine-Tuning
  • Business-Specific Customization: Tailor models to specific domains (healthcare, airlines) with unique internal knowledge.
  • Improved Inference Time: Faster processing due to specialized training.
  • Computational Resource Savings: Reduces CPU/GPU usage.
  • Bias Reduction: Mitigate biases by providing more representative data (e.g., addressing bias in loan applications).
LORA (Low Rank Adapter)
  • Concept: Freeze most of the pre-trained model parameters and only retrain a small subset (adapters) to save computational resources (invented by Microsoft).
Quantization
  • Concept: Reduce model size by quantizing data into smaller chunks (e.g., 16-bit floating points to 4 bits).
Fine-Tuning Process
  1. Data Preparation: Clean and split data into training, validation, and testing sets.
  2. Model Training: Use training data for fine-tuning.
  3. Validation: Constantly validate the model and adjust to reduce bias.
  4. Testing: Final testing to ensure proper functioning.
  • Tools: Azure AI Studio.
Security Considerations for Fine-Tuning
  • Data Sanitization: Ensure data does not contain PII/PHI.
  • Model Validation: Prevent revealing sensitive information or generating harmful content.
  • Secure Deployment: Use tools like MLflow or Azure deployment tools.

3. Plug-in API and Function Calls

  • Plug-in API: Allows integration of real-time information into applications; overcomes the limitation of models like ChatGPT, which have data up to a certain point in time.
  • Function Calls: Allows specification of functions in prompts that the LLM model API can call.
  • Agent Creation: Utilizes function calls or plug-ins to create reasoning chains for complex tasks.
Security Considerations
  • API Security: Secure both application and LLM APIs.
    • Secret Key Management.
    • Industry Standards: OAuth (data access control), OIDC (authentication, JWT tokens).
  • API Gateway: Rate limiting, API formatting checks, cost optimization (use GPT-4 for high precision, GPT-3.5 for less critical tasks).
  • Third-Party Plug-in Validation: Validate third-party plug-ins, especially in regulated industries (HIPAA, HITRUST compliance).
  • Shift Left Security: Integrate security considerations early in the development process by talking with security architects, threat modeling.
Data Validation and Encryption
  • Validate data in transit and encrypt if necessary.
  • Validate the final response before presenting to the user.
Sample Validation Tools
  • OpenAI Modulation API.
  • Shilya's guardrail tool (GitHub).
  • NVIDIA's guardrail application.

4. Training From Scratch

  • Use Case: For very large enterprises dissatisfied with existing foundation models.
Key Steps
  • Data Security and Sanitization.
  • Large-Scale Model Validation.
  • LLM Operations (LLMOps): Integrate security into DevOps at each stage (DevSecOps).
DevSecOps Considerations
  • Shift Left: Consider security early in the design phase.
  • Secure Data Pipeline: Secure data from various sources through cleaning, splitting, and normalization.
  • Secure Coding: Use SAST tools (Checkmarx, Snyk, Fortify, Veracode) for code analysis.
  • DLP Detection: Detect sensitive data using DLP tools.
  • CICD Pipeline Integration: Integrate security tools into the CICD pipeline (e.g., Azure DevOps pipeline).
  • Monitoring and Model Evaluation: Continuously monitor and evaluate the model, with retraining and fine-tuning as needed.

Machine Unlearning

  • Concept: Remove harmful or biased information by adjusting weights in the neural network.
  • Mechanism: Deactivate harmful neurons by adjusting weights through unlearning.