2.2 Security in LLM Powered Application Architecture

LLM Powered Applications & Common Security Concerns

  • Focus: Universal security concerns for LLM (Large Language Model) applications, irrespective of the design pattern chosen.

Industry Leaders' Viewpoints on AI Security

  • Elon Musk:
    • In 2018, stated AI is far more dangerous than nukes.
    • Reasoning: AI's rapid learning capability, potentially surpassing human intelligence (super intelligence).
    • AI agents: may copy themselves and control industry control systems (ICS) via APIs, jeopardizing infrastructure (e.g., electrical grids, nuclear power plants).
    • Advocates for AI regulation.
    • March 2023: Open letter with Steve Wozniak and 1,000+ others to halt GPT training.
  • Sam Altman (OpenAI):
    • Confirmed pausing GPT-4 training.
    • Proposed an IAEA-like cross-country, UN-level control for AI implementation.
    • Acknowledges a non-zero probability of AI potentially destroying the human race.
    • Engaged in a world tour to discuss AI security, promoting global coordination.
  • Geoffrey Hinton (Godfather of AI):
    • Left Google to openly discuss the dangers of AI.
  • Yann LeCun:
    • Believes AI is not dangerous and it's okay. However, responsible AI development is crucial.

AI for Good Initiative

  • United Nations Level
  • AI for Good conference in Geneva
  • Sofia the robot: Granted citizenship in Saudi Arabia.
    • Affirmed alignment with human interests.

Importance of Caution & Security Measures

  • Critical for security professionals and AI developers to exercise caution.
  • Continuous testing of AI outputs.
  • Ensuring the security of AI connections.

Real-world Incidents

  • ChatGPT Data Leak (March 2023):
    • Prompt history leaked.
    • OpenAI uses Redis (in-memory database with embedding and similarity search capabilities).
    • User inputs cached in Redis for fraud/misuse detection.
    • Data usage (unless opted out) for search and other purposes.
    • Violation of GDPR (EU) and PCI DSS (Payment Card Industry Data Security Standard).
    • Led to JPMorgan and other major banks, as well as Samsung, banning ChatGPT use for professional reasons.

Cloud Security Alliance Research on ChatGPT Security Issues

  • Five potential security issues identified:
    1. Malicious Use: Social engineering, phishing emails, fake news.
    2. Prompt Injection: Direct attacks on LLMs using natural language (similar to SQL injection), aiming for harmful output, sensitive data exposure, or jailbreaking.
      • Defense: Checkpoints at the LLM and API level.
    3. Credential Stuffing & Data Leakage: Stealing credentials or leaking sensitive data (including credit card information).
    4. Misinformation: Spreading misinformation (e.g., during the 2024 US presidential election), similar to Cambridge Analytica.

User-Level Precautions for ChatGPT

  • Disable chat history and training in ChatGPT settings to prevent data usage for training purposes.
    • Tradeoff: Disabling training may remove prompt history access.
  • Plugin validation: Third-party plugins are not controlled by OpenAI, necessitating user validation.
    • Data sent to external apps requires caution concerning sensitive information.

Security Controls Applicable to All Applications

  1. Data Handling:
    • Avoid storing PHI (Protected Health Information) or sensitive data in prompts and training data.
    • Regular audits.
    • Security awareness training.
    • Promptly disclose and report any data leaks.
  2. Access Control:
    • Protect fine-tuned models and vector databases.
    • Implement and monitor access logs to track app access, timing, which API calls.
    • Review access control policies.
    • Ensure proper access control for business functions and fine-tuned models, considering intellectual property, trade secrets, and customer-sensitive data.
  3. API Security:
    • Refer to OWASP (Open Web Application Security Project) Top 10 API Security.
      1. Broken Object Level Authorization: Prevent unauthorized access to objects (e.g., accessing payment objects with credit card info when only address access is intended).
      2. Fine-grained access control: Implement this for sensitive business functions.
      3. In LLM sense: Object = weight of model. Safeguard with API access control to protect sensitive weights from model stealing.
      4. Authentication: Strong password policies and two-factor authentication for sensitive APIs.
      5. Broken Object Property Level Authorization: Return only necessary data instead of entire objects (mass assignment) such as Payment card information, even your address, even your name, birthday, and also the CVC code and credit card.
      6. OAuth and OIDC: Use these standards for API authentication and authorization.
      7. Rate Limiting: Prevent denial-of-service attacks and control costs associated with third-party API usage or GPU cycles.
  4. Logging:
    • Log all LLM application API calls, including access details.
    • Store logs securely.
    • Analyze and monitor data.
  5. Data Bias:
    • Clean data used for fine-tuning and training to reduce bias.
    • Regularly review potential biases.
    • Use diverse datasets (gender, age, race, ethnicity).
    • Maintain balance during data cleaning.
    • Tools: Wimpure, LoneGuardio.
      • LoneGuardio URL: (URL Provided)
  6. Guardrails:
    • Implement data guardrails to validate inputs and outputs.
    • Address bias, toxicity, and poisoned data from third-party applications.
    • Balance guardrail implementation with LLM performance.
    • Tools: GardaReal (GitHub), Nvidia, OpenAI Modulation API.
  7. Red Teaming:
    • Internal and external testing to identify vulnerabilities (penetration testing for models).
    • Perform red teaming tests after model updates and fine-tuning.
    • Example: OpenAI spent six months on red teaming for GPT-4.
  8. Prompt Injection:
    • Validate user input.
    • Escape malicious content using blacklists or whitelists.
  9. AutoGPT/Baby AGI Validation:
    • Validate outputs and incorporate human feedback or third-party API checks.
    • Address potential issues arising from unchecked self-prompting.

Summary

  • The video covered security concerns for LLM applications, industry leader perspectives, and practical guidance on implementing security controls, including API security.