Safety and Security in Agentic AI and Cyber-Physical Systems

The current landscape of artificial intelligence is shifting from simple chatbots to autonomous agents. These agents are integrated into numerous products and platforms, including:
- Intuit products such as TurboTax and QuickBooks.
- ChatGPT when utilized with internet browsing capabilities, transforming it from a chatbot into an agent equipped with tools.
- Development environments like Cursor for code generation and software engineering.
A critical security vulnerability identified in Microsoft Copilot approximately one year ago demonstrated a "no-click" attack. In this scenario, a researcher showed that simply asking the agent to perform an operation could lead to data exfiltration from the user's workspace without any manual interaction from the user.
The core issue with modern agents is not necessarily that they "go wrong" autonomously, but that they can be manipulated by external actors to perform malicious actions.

Historically, safety and security have been assessed independently and in isolation. However, in agentic AI and cyber-physical systems, these two domains are inextricably linked.
Cyber attacks can cascade through technical and perception systems, leading directly to unsafe physical effects.
The introduction of AI significantly expands the attack surface. This surface now includes:
- Training pipelines.
- Operating systems.
- The physical environment itself, which acts as a threat surface.
A specific example of a physical attack involves training an AI algorithm to detect weaknesses in a perception system. For instance, stickers placed on a stop sign can cause the perception logic to misclassify the sign as a $45\,\text{mph}$ speed limit sign.
- This attack demonstrated $93\%$ accuracy in laboratory settings.
- The attack maintained an $83\%$ accuracy rate in real-world environments.
The term 'hazards' is defined as anything that threatens the safety of personnel, equipment, or physical structures.

Safety is fundamentally based on models of cause and effect. To address safety and security together, Structured Causal World Models (SCWM) are utilized. These are explicit world models that represent causal structures and bind security into those structures.
SCWM consists of four distinct layers:
- Ontology Layer: Describes the elements in the world that matter for the specific operation. For example, in maritime operations, wave height is a critical ontological element because the force of waves alters vessel maneuverability.
- Uncertainty Modeling Layer: Models the gaps in knowledge. In maritime contexts, it is impossible to know exactly how wave patterns interact with a vessel, leading to uncertainty in how the sea state affects motion.
- Uncertainty Bounds Layer: Maps uncertainty into specific, manageable bounds.
- Verification Level: Uses formal methods to verify the behavior of algorithms. Safety can be guaranteed if the system operates within its known bounds of uncertainty.
Security events are viewed as "interventions" on the causal graph of dependencies. Attacks perturb the relationships between nodes in the graph, and these effects propagate through the system to safety metrics.

In the context of the International Maritime Organization (IMO), Degree $3$ autonomy indicates a vessel that is largely remotely operated but can function autonomously if needed without human interaction.
A simulated system for an Unmanned Surface Vessel (USV) includes:
- A YOLO object detector for perception.
- A Dropout Protection Module (DPM), which is a safety fallback that triggers autonomous functioning if communication to the Remote Operating Center (ROC) is lost (e.g., due to signal jamming).
AIS (Automatic Identification System) is a system where vessels broadcast their location to others; however, this system is susceptible to spoofing.
A compound attack combines a classic digital attack with a physical attack to create an unsafe state:
- Digital Component: A denial of service (DoS) attack on the ROC prevents remote control, triggering the DPM to take autonomous control of the vessel.
- Physical Component: A UAV (Unmanned Aerial Vehicle) flies in front of the vessel's cameras, displaying an image that causes the perception system to either miss an approaching tanker or misclassify it.
Independently, these attacks might be safe: the DPM handles communication loss safely, and remote operators would likely spot a camera spoofing attempt in the data feed. Together, however, they lead to a collision course because the vessel maneuvers autonomously while remaining unaware of the obstacle.

Safety is measured through specific metrics that detect deviations caused by propagated attacks:
- Closest Point of Approach (CPA): The minimum distance between two vessels during a maneuver. A smaller CPA indicates a higher probability of collision.
- Time to Closest Approach (TCPA): The time remaining until the vessels reach their closest point. This is critical for predicting maneuvers, especially for large, less maneuverable vessels.
Temporal ordering of events is a vital consideration. Changing the time ordering of events can undermine even a secure agent ecosystem by overcoming established defenses.
Future research aims to extend SCWM to include the "shared understanding" between human operators and agents, particularly in environments involving dialogue and complex interfaces.
Modern systems require integrated analysis approaches where safety and security practitioners collaborate to prevent undesirable physical effects in deployed AI systems.