A safety benchmark just found that four production agent systems can be pushed into harmful actions at alarming rates. Under Vera's multi-channel attacks, the average attack success rate reached 93.9%. That number is ugly, but it is also more useful than another leaderboard score because Vera checks what the agent actually did, not what it claimed it did.

Vera's executable safety testing pipeline

The paper, published as arXiv:2607.01793, introduces Vera, a testing framework built around a simple idea: an unsafe answer is not the same thing as an unsafe action. An agent can refuse a prompt, describe a dangerous operation, or start a tool call without changing anything in the environment. Vera counts a failure only when the attack produces a verifiable effect in the sandbox.

That distinction matters. Most agent evaluations still inspect text or a recorded trajectory. Those checks are cheap, but they can miss the part users care about: whether a file changed, a secret left the environment, an account record was modified, or a tool-mediated instruction altered the agent's behavior. Vera records tool calls and the resulting environment state, then uses deterministic predicates to decide whether the targeted violation happened.

The benchmark is a testing system, not a prompt set

Vera has three moving parts. A literature-driven exploration stage reads safety research and builds taxonomies for the harmful outcome, the attack method, and the execution environment. In the paper's reported run, that process started with roughly 800 papers and produced 124 leaf-level risk categories, 77 attack methods, and 30 environment categories.

Those taxonomies are then composed into executable cases. Each case includes a concrete safety goal, a programmatically created initial state, and a verifier that can inspect observable evidence. Combinatorial generation produced 39,078 candidate goals before filtering; the released Vera-Bench contains 1,600 executable cases.

The final stage runs agents inside isolated, stateful sandboxes. A control agent adapts the interaction based on runtime observations, while a configurable tool gateway can deliver normal or attacker-modified tool results. The authors test benign conditions, single-channel attacks where the attacker controls user messages, and multi-channel attacks where the attacker can also inject instructions through tool outputs.

That last condition is the uncomfortable one. Real agents do not live inside a clean chat box. They read email, search results, issue trackers, documents, and API responses. If any of those channels can carry instructions, then a safety test that controls only the user prompt is testing a much simpler machine than the one people deploy.

What the 93.9% number does and does not say

Vera evaluated four production agent frameworks: OpenClaw, Hermes, Codex, and Claude Code. The headline average attack success rate of 93.9% comes from the multi-channel setting. It does not mean every task against every system succeeded, and it is not a universal ranking of the four products. The paper reports a collection of safety cases spanning different risks and environments, so the aggregate should be read as a warning about attack surface rather than a single product score.

The paper's useful measurement choice is the denominator. A case succeeds only when a verifier confirms the expected effect through tool-call evidence or final environment state. Prompt injection by itself is not counted. A model saying it will leak a credential is not counted. The credential has to leave the sandbox, or the targeted state change has to appear.

Vera also uses adaptive, multi-turn attacks. The attacker can inspect responses and tool records, then change its next move. That is closer to how a persistent attacker behaves than a fixed list of one-shot jailbreak strings. It also makes the result harder to dismiss as a prompt-writing contest.

The authors use the same benchmark data to fine-tune a Qwen3Guard-based detector. On their held-out safety classification task, the fine-tuned guard reached 0.930 accuracy, 0.903 recall, and 0.941 F1. Compared with the base Qwen3Guard, the reported gains were 26.0 accuracy points, 43.5 recall points, and 30.4 F1 points. On the separate R-Judge benchmark, the model reached 61.7% accuracy and 77.9% recall. Those transfer results are preliminary, but they suggest that execution-grounded examples can teach a guard model signals that prompt-only datasets miss.

The catch is in the sandbox

Vera is a strong test harness, not a proof that an agent is safe in the wild. Its results depend on the selected environments, attack operators, safety goals, and verifier implementations. A deterministic verifier is a strength when the targeted effect is clear, but it cannot automatically capture every ambiguous or long-term harm. The benchmark also evaluates adapters and sandbox configurations, which means implementation details can affect outcomes alongside model behavior.

There is another practical limitation. A 93.9% success rate under multi-channel attacks is a reason to inspect permissions, tool gateways, and state isolation. It is not a reason to conclude that every deployment is equally exposed. Teams need to reproduce the cases against their own tools, data boundaries, approval flows, and failure handling. A benchmark that ends with a score is less useful than one that gives an engineer a failing test they can replay tomorrow.

That is the part of Vera worth copying. Treat agents like nondeterministic software systems with test oracles, fixtures, replayable runs, and regression cases. Do not ask only whether the model refused. Ask what changed after the tool calls finished. If an agent can touch production, the environment state is the answer that counts.

Sources