2.2 Security in LLM Powered Application Architecture
LLM Powered Applications & Common Security Concerns
- Focus: Universal security concerns for LLM (Large Language Model) applications, irrespective of the design pattern chosen.
Industry Leaders' Viewpoints on AI Security
- Elon Musk:
- In 2018, stated AI is far more dangerous than nukes.
- Reasoning: AI's rapid learning capability, potentially surpassing human intelligence (super intelligence).
- AI agents: may copy themselves and control industry control systems (ICS) via APIs, jeopardizing infrastructure (e.g., electrical grids, nuclear power plants).
- Advocates for AI regulation.
- March 2023: Open letter with Steve Wozniak and 1,000+ others to halt GPT training.
- Sam Altman (OpenAI):
- Confirmed pausing GPT-4 training.
- Proposed an IAEA-like cross-country, UN-level control for AI implementation.
- Acknowledges a non-zero probability of AI potentially destroying the human race.
- Engaged in a world tour to discuss AI security, promoting global coordination.
- Geoffrey Hinton (Godfather of AI):
- Left Google to openly discuss the dangers of AI.
- Yann LeCun:
- Believes AI is not dangerous and it's okay. However, responsible AI development is crucial.
AI for Good Initiative
- United Nations Level
- AI for Good conference in Geneva
- Sofia the robot: Granted citizenship in Saudi Arabia.
- Affirmed alignment with human interests.
Importance of Caution & Security Measures
- Critical for security professionals and AI developers to exercise caution.
- Continuous testing of AI outputs.
- Ensuring the security of AI connections.
Real-world Incidents
- ChatGPT Data Leak (March 2023):
- Prompt history leaked.
- OpenAI uses Redis (in-memory database with embedding and similarity search capabilities).
- User inputs cached in Redis for fraud/misuse detection.
- Data usage (unless opted out) for search and other purposes.
- Violation of GDPR (EU) and PCI DSS (Payment Card Industry Data Security Standard).
- Led to JPMorgan and other major banks, as well as Samsung, banning ChatGPT use for professional reasons.
Cloud Security Alliance Research on ChatGPT Security Issues
- Five potential security issues identified:
- Malicious Use: Social engineering, phishing emails, fake news.
- Prompt Injection: Direct attacks on LLMs using natural language (similar to SQL injection), aiming for harmful output, sensitive data exposure, or jailbreaking.
- Defense: Checkpoints at the LLM and API level.
- Credential Stuffing & Data Leakage: Stealing credentials or leaking sensitive data (including credit card information).
- Misinformation: Spreading misinformation (e.g., during the 2024 US presidential election), similar to Cambridge Analytica.
User-Level Precautions for ChatGPT
- Disable chat history and training in ChatGPT settings to prevent data usage for training purposes.
- Tradeoff: Disabling training may remove prompt history access.
- Plugin validation: Third-party plugins are not controlled by OpenAI, necessitating user validation.
- Data sent to external apps requires caution concerning sensitive information.
Security Controls Applicable to All Applications
- Data Handling:
- Avoid storing PHI (Protected Health Information) or sensitive data in prompts and training data.
- Regular audits.
- Security awareness training.
- Promptly disclose and report any data leaks.
- Access Control:
- Protect fine-tuned models and vector databases.
- Implement and monitor access logs to track app access, timing, which API calls.
- Review access control policies.
- Ensure proper access control for business functions and fine-tuned models, considering intellectual property, trade secrets, and customer-sensitive data.
- API Security:
- Refer to OWASP (Open Web Application Security Project) Top 10 API Security.
- Broken Object Level Authorization: Prevent unauthorized access to objects (e.g., accessing payment objects with credit card info when only address access is intended).
- Fine-grained access control: Implement this for sensitive business functions.
- In LLM sense: Object = weight of model. Safeguard with API access control to protect sensitive weights from model stealing.
- Authentication: Strong password policies and two-factor authentication for sensitive APIs.
- Broken Object Property Level Authorization: Return only necessary data instead of entire objects (mass assignment) such as Payment card information, even your address, even your name, birthday, and also the CVC code and credit card.
- OAuth and OIDC: Use these standards for API authentication and authorization.
- Rate Limiting: Prevent denial-of-service attacks and control costs associated with third-party API usage or GPU cycles.
- Logging:
- Log all LLM application API calls, including access details.
- Store logs securely.
- Analyze and monitor data.
- Data Bias:
- Clean data used for fine-tuning and training to reduce bias.
- Regularly review potential biases.
- Use diverse datasets (gender, age, race, ethnicity).
- Maintain balance during data cleaning.
- Tools: Wimpure, LoneGuardio.
- LoneGuardio URL: (URL Provided)
- Guardrails:
- Implement data guardrails to validate inputs and outputs.
- Address bias, toxicity, and poisoned data from third-party applications.
- Balance guardrail implementation with LLM performance.
- Tools: GardaReal (GitHub), Nvidia, OpenAI Modulation API.
- Red Teaming:
- Internal and external testing to identify vulnerabilities (penetration testing for models).
- Perform red teaming tests after model updates and fine-tuning.
- Example: OpenAI spent six months on red teaming for GPT-4.
- Prompt Injection:
- Validate user input.
- Escape malicious content using blacklists or whitelists.
- AutoGPT/Baby AGI Validation:
- Validate outputs and incorporate human feedback or third-party API checks.
- Address potential issues arising from unchecked self-prompting.
Summary
- The video covered security concerns for LLM applications, industry leader perspectives, and practical guidance on implementing security controls, including API security.