The benchmark result is almost rude: the best agent in ORCA-bench reached only 10.0% RCA accuracy on the hard tasks. On the medium setting, which the authors call the realistic-input case, the top score was 25.3%. These are coding agents with terminal access, current frontier models, and a live-looking service stack. They still miss most of the incidents.
That is a much more useful result than another leaderboard where an agent edits a repository until the tests turn green. Oncall work begins with a person saying something vague like “the site is having issues,” often long after the problem started. The agent has to decide whether there is an incident, find the relevant time window, search telemetry, connect symptoms across services, and name every plausible cause. A passing test cannot tell you whether it did any of that.
The benchmark looks like a bad Friday
The paper, ORCA-bench, puts agents inside the OpenTelemetry Astronomy Shop, a distributed demo system with 19 microservices written in 13 languages. The environment uses Prometheus for metrics, OpenSearch for logs, Jaeger for traces, and Grafana as the interface. Agents get the source code too. That detail matters because the ablation removed source access and every measured metric got worse.
The testbed contains six days of telemetry totaling more than 50 GB. The authors create incidents by switching real feature flags in the service rather than dropping an isolated synthetic bug into a clean sandbox. The schedule includes 13 scenarios, including isolated faults, independent faults, cascading failures, conflicts, and sequential incidents. Day six is a deliberately ugly “FAFO Friday” with six flags active at once. That is closer to the kind of mess an SRE actually inherits than a single red alert with a neat root cause.
The task design changes two things that ordinary software benchmarks mostly ignore. First, the report can be precise, approximate, or uselessly broad. “The recommendation service is failing” is easier than “users are reporting site issues.” Second, the investigation can begin 15 minutes, one hour, eight hours, or 24 hours after the incident starts. A stale report and a delayed investigation create a search problem, not just a code problem.
The paper reports 1,079 RCA tasks in the abstract and 1,076 in the construction section. That small mismatch does not change the result, but it is worth flagging because benchmark papers should make the counting easy to audit. The public dataset link included in the paper also drew complaints in the Hacker News discussion after readers found it unavailable. A benchmark can be technically thoughtful and still need a boring release checklist.
Why the score collapses
ORCA-bench does not award full credit for vaguely pointing at the right service. Each task has a rubric describing the expected metrics, logs, traces, and mechanism. The agent gets partial credit for confirming symptoms or identifying some evidence, but full credit requires naming the root cause and explaining the supporting signals. RCA accuracy is stricter still: the agent must name every plausible root cause for the task.
That distinction exposes a failure pattern hidden by simpler evaluations. GPT-5.5 had the highest overall RCA depth at 48.8% across 884 incident tasks, which means it often made some progress. Claude Sonnet 4.6 led the stricter accuracy measure at 30.6% across the full incident set. On the realistic Medium and Hard settings shown in the headline results, however, the best accuracy fell to 25.3% and 10.0%. “Found a clue” and “diagnosed the incident” are not the same event.
The agents also get distracted. In one example, a persistent Playwright browser failure produced a huge volume of background noise. GLM-5 followed that loud signal and missed the active product catalog fault that was actually breaking checkout. Claude Opus 4.7 traced the request path through the product catalog service and connected the HTTP 500s to the enabled feature flag. The difference is not raw log-reading ability. It is deciding which evidence deserves attention when several things are wrong at once.
Hallucinations make the operational risk sharper. The weakest result in the paper shows an implausible root cause in 40% of incident reports. DeepSeek-V4-Pro was at 7% on that measure. A wrong diagnosis is often worse than an empty one because it can send a tired engineer toward the wrong deploy, the wrong service, or a rollback that hides the actual failure.
The researchers checked their GPT-5.4 judge against human scoring on a verified 40-task subset. Weighted Cohen’s kappa was 0.90 overall, with 0.96 on Medium tasks and 0.84 on Hard. That is a reasonable safeguard against a model grading another model by vibes. It does not make the benchmark perfect, but it makes the central comparison easier to trust.
The useful deployment pattern is less glamorous
The obvious lesson is that nobody should let a coding agent change production because it produced a confident incident report. The less obvious lesson is that an agent can still be useful before it reaches autonomous SRE status.
A sensible setup would let the agent collect evidence, build a timeline, list competing root causes, and show the exact telemetry behind each claim. A human would decide whether the evidence is enough to page someone, roll back a release, disable a flag, or leave the system alone. The agent becomes a tireless first pass over dashboards and code, not the person with the pager.
That distinction lines up with the mixed reaction on Hacker News. One commenter argued that agents could investigate first and give the human a head start. Others pushed back on any plan that lets an LLM make production changes without verification. Another commenter noticed the public dataset link had already failed. The skepticism is healthy. Oncall is full of incomplete context, old incidents, misleading alerts, and consequences that arrive after the original mistake has been forgotten.
There is also a trap in treating 10% as the final word. ORCA-bench is harder than a static repository task, but it is still a curated 50 GB environment with public code, known instrumentation, isolated investigations, and a finite set of fault mechanisms. Real systems have undocumented dependencies, changing traffic, partial deploys, permissions problems, and telemetry that quietly stopped working months ago. If agents score 10% here, their production score should not be assumed to be 10%. The benchmark authors call these results a lower bound on the engineering needed before agents can safely own reliability. That is the right framing.
The benchmark changes the question. The question is not whether an AI agent can read logs. It plainly can. The question is whether it can keep several plausible explanations alive, weigh noisy evidence, account for time, and refuse to invent a cause when the evidence is weak. ORCA-bench says that capability is still narrow. Until the accuracy curve moves much higher on concurrent, delayed, ambiguous incidents, the best place for an agent on the oncall rotation is beside the engineer, not above them.
The full paper includes the task construction details, scoring prompts, case studies, and supplementary results in the arXiv HTML version. The underlying service comes from the OpenTelemetry Astronomy Shop Demo, which is also a practical starting point for testing an internal incident agent without pretending a toy benchmark is production.