Back to Feed
Agents / Safety & Alignment

Enforcing AI Safety Through Runtime Contracts

Original: Agent Safety Should Be a Runtime Contract

Listen to the summary

Uses a voice available on your device

Audio options
On this page 4 sections
Related concepts 4 concepts

Key Takeaways

  • Current AI safety depends too heavily on training-time methods like RLHF, which are insufficient for agents performing consequential actions in open-world environments.
  • A survey of 52 safety incidents revealed that 40 were fully preventable and 11 were partially mitigable using a functional runtime harness.
  • The proposed architecture utilizes a two-faced harness consisting of a preventive layer for blocking actions and an evidential layer for verifying task completion.
  • Only 2 of 12 analyzed public agent systems currently implement submission-like evidence gates, highlighting a significant gap in production-grade safety infrastructure.

Summary & Methodology Analysis

The researchers propose shifting safety enforcement from model weights to a system-level runtime contract. This architecture employs a two-faced harness designed to intercept and audit agent behavior. The preventive layer functions as a middleware component that utilizes sandboxes, permission gates, and input filters to halt dangerous actions before they reach execution. This prevents the agent from performing unauthorized or harmful operations in the host environment by strictly gating API interactions and system calls at runtime. The architecture avoids reliance on model self-reporting, which is often unreliable in autonomous workflows. To ensure accountability, the evidential layer introduces hash-chained trajectory logging and evidence-gated submissions. This layer generates a cryptographically verifiable record of the agent actions, ensuring that task completion is supported by concrete, tamper-evident data captured via a formal Agent Trajectory Schema. By composing these preventive monitors and evidential verifiers, the system creates an immutable contract for agent operation. The paper notes that current industry practices are lacking, as an audit of 12 public agent systems showed that only 2 implemented submission-like evidence gates. The methodology is computationally efficient only when the defined verifiers operate independently, as compositional verification can otherwise introduce overhead. A primary limitation is that the contract governs the external actions and submissions of the agent, rather than the internal objectives or goals the model is attempting to optimize.

Interactive System Flowchart

Click diagram to expand and zoom

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What is the core problem with current AI safety methods?

Current paradigms rely too heavily on training-time alignment, which is structurally insufficient for autonomous agents taking consequential actions in open environments.

Q2. How does the proposed solution differ from existing approaches?

It shifts the unit of safety from the model itself to a runtime contract enforced by the system infrastructure.

Q3. What components make up the proposed safety harness?

The harness includes a preventive layer for blocking actions and an evidential layer for verifying task completion using hard evidence.

Q4. What is the purpose of the Agent Trajectory Schema?

It captures every event with a hash to provide a tamper-evident log of agent behavior.

Q5. How effective is the proposed harness at stopping safety incidents?

A survey of 52 incidents found that 40 were fully preventable and 11 were partially mitigable using such a harness.

Q6. Does this approach account for the internal goals of the model?

No, the contract focuses on enforcing actions and submissions rather than governing the internal goals of the model.

Q7. What are the computational constraints of this architecture?

Compositional verification is only computationally efficient when the defined verifiers operate independently.

Q8. Are there existing agent systems that use this type of security?

An audit of 12 public systems showed that only 2 currently utilize submission-like evidence gates.

Q9. Does the paper suggest this replaces RLHF or DPO?

The paper does not explicitly state that this replaces RLHF or DPO, but argues that training-time alignment is insufficient for agents in open-world environments.