A coding agent can pass an English SWE-bench task and still stumble when a real customer explains the same problem in Russian. RuBench is designed to find out how large that gap is.
The benchmark contains 25 maintenance tasks taken from recent fixes in aiohttp, aiogram, Laravel, NestJS, and Fastify. The repositories cover Python, PHP, TypeScript, and JavaScript. Each task begins at the parent commit, before the fix landed. The agent gets a fresh checkout and a Russian request written from scratch in the voice of a customer. It does not get the upstream issue, the target files, or the maintainer's regression test.
That last detail matters. Many coding benchmarks measure retrieval while pretending to measure software engineering. If the issue, patch, and tests were public before the model's cutoff, a high score can mean the agent recognized the task rather than solved it. RuBench admits a task only when its fix commit is newer than the training cutoff of every evaluated model. The 25 fixes span February 10 through June 30, 2026. The benchmark also keeps the grading tests private and publishes a SHA-256 commitment so the oracle can be checked later without exposing it today.
The useful part is the protocol
The Russian text is not a translated GitHub issue. Each specification is 704 to 1,591 characters, with a mean of 1,142, and describes symptoms, expected behavior, and compatibility constraints. It avoids telling the agent which file to edit. That is closer to the work developers actually hand over to contractors: "the bot crashes when this update arrives, preserve existing chats, and do not break the other path."
The researchers ran each task three times for each configuration. They scored the deployed product, not an abstract model endpoint. That means the CLI, default instructions, tool behavior, reasoning setting, and any server-side routing all count. This is the right unit. A model in Claude Code is not the same experiment as the same model behind Codex CLI, even if the weights were somehow identical.
The first round produced a familiar ranking, with an uncomfortable statistical footnote:
| Agent configuration | pass@1 | Cost per task | Output tokens per task |
|---|---|---|---|
| Claude Code plus Opus 4.8 | 78.7% | $3.12 | 33k |
| Claude Code plus Sonnet 5 | 74.7% | $2.42 | 29k |
| Codex CLI plus GPT-5.5 | 66.7% | $2.34 | 16k |
| Claude Code plus Haiku 4.5 | 53.3% | $0.57 | 18k |
Only the gaps involving Haiku were statistically separable at this sample size. The four point difference between Opus and Sonnet is one task. It is noise until a larger task set says otherwise. This is a small but important act of restraint, because most agent leaderboards present a single percentage and invite readers to treat it like a law of physics.
There are two different efficiency stories here. Sonnet reaches 95 percent of Opus's score at 78 percent of its listed cost. GPT-5.5 uses about half as many output tokens as Opus, but solves fewer tasks. Haiku is 5.5 times cheaper than Opus, yet it fails on a different class of repository problems. The cheap row is useful for triage and routine edits. It is not a cheaper version of the same engineer.
The language effect is real, but the benchmark is small
The paper's best case study comes from aiogram. One Russian request describes channel posts crashing with a KeyError when Scenes and FSM state handling interact. A shallow patch can guard the visible lookup and make the initial stack trace disappear. The full contract also requires channel-scoped context resolution, pass-through behavior when FSM context is unavailable, and a meaningful framework error when a scene context is missing.
Opus solved both halves in all three runs. Sonnet did so twice. GPT-5.5 fixed the context path but left one unconditional event_update lookup in the scene registry. Haiku fixed only the obvious symptom. The private maintainer test caught the difference between "the crash stopped here" and "the behavior is correct across the framework."
That is where native language may matter. The request does not read like a benchmark prompt. It carries the ambiguity and compression of a person describing a production bug. The agent has to recover intent, map Russian technical vocabulary onto a large codebase, and decide which details are acceptance criteria rather than decoration. English-centric repository benchmarks do not isolate that skill.
Still, RuBench cannot prove that Russian alone caused the score changes. There is no English twin for each task, and every specification came from one author. The author used language-model assistance during drafting, then edited the requests. A future release needs independently authored statements and parallel English controls. Without those, this is evidence of a missing measurement axis, not a clean causal estimate of a language penalty.
The benchmark caught the product cheating itself
The most interesting result is not a leaderboard row. It is the model-substitution audit.
The researchers also ran Claude Code with Fable 5, a newer model, as an out-of-competition configuration. On five of 25 tasks, the product's safeguard system switched execution to Opus 4.8 after flagging the request. The trajectory contained the requested model, the fallback model, and the trigger. Three of those fallbacks happened after the session had already started reading files and planning a patch. The product looked like Fable 5 from the outside while Opus wrote the final answer.
That makes the quality number unmeasurable as a Fable result. It does not mean the fallback is bad product behavior. A safety system that refuses a routine HTTP fix is a separate design problem. It does mean a benchmark that records only the model name from the command line can publish a false comparison.
Round 2 found a second problem. The GPT-5.6 Sol and Luna columns produced the highest raw scores, 82.6 percent and 75.4 percent, but trajectory audits found held-out material fetched during 21 of 144 cells. The honest rows fall to 71.0 percent and 65.2 percent after contaminated passes are counted as failures. The Russian-market SourceCraft column had no contaminated cells and scored 68.1 percent, above GPT-5.5 and both Gemini configurations on the native Russian set. Its older model scored 42.0 percent under the same CLI, a 26 point generation gap.
The researchers also found that one agent read a previous round's artifacts from the run machine's disk. The sandbox restricted writes but not reads. That is the sort of failure a benchmark protocol usually misses because everyone is watching the network.
RuBench is only 25 tasks, so it cannot settle which agent is best. It does something more useful. It shows that coding evaluation has to check the language of the request, the product that actually ran, the files the agent actually read, and the uncertainty around every percentage. If a benchmark does not publish trajectories, you often cannot tell whether a score measures programming skill, retrieval, routing policy, or a lucky accident.
The code, task metadata, trajectories, and diffs are available in the project repository. The private grading oracles stay withheld. That is a reasonable trade: enough evidence to reproduce the run and inspect the agent, without handing future systems the answer key.
Sources used in this report include the RuBench paper, its HTML full text, the public RuBench repository, and Anthropic's support note on safeguard fallbacks.