The new coding agent harness benchmark makes a useful claim that most leaderboard charts hide: the same model can need a very different interface depending on how well it handles a shell. The paper tests 176 matched settings across four models, SWE-Bench Verified, and Terminal-Bench 2.1. The result is not that bash always wins. It is that a harness should be fitted to the model instead of treated as a permanent product identity.

Coding agent harness experiment overview

That distinction matters because most coding-agent comparisons change several things at once. A vendor may bring its own system prompt, tools, context policy, planning format, and stopping rules. If the agent wins, the score gets attributed to the model. If it loses, the model gets blamed. Fan and the other authors built a lighter harness that keeps the execution loop fixed while changing three pieces: planning, the action space, and context management. They tested Nemotron-3 at 30B, 120B, and 550B, plus Mistral-Medium-3.5-128B. The models were served locally in BF16 with temperature 0, and the study priced token usage using OpenRouter rates rather than pretending every successful task costs the same.

What the experiment actually measured

The test set is broad enough to expose different failure modes. SWE-Bench Verified contains 500 human-verified GitHub issues. Terminal-Bench 2.1 contains 89 end-to-end command-line tasks. The researchers report success rate and mean cost per task, then inspect the trajectories to see whether an agent localized the right file, edited code, verified it, or simply ran out of room.

The context experiment uses five strategies at four budgets: 32K, 64K, 96K, and 128K tokens. T0 does nothing. T1 elides stale tool output. T2 adds external storage and a recall tool. T3 summarizes with an LLM. T4 stages rule-based elision before summarization and keeps the other mechanisms available. This is a much better test than asking whether a generic long-context feature sounds useful. The relevant question is where the run dies.

At 32K, the answer is usually context overflow. The paper reports that managed tiers let trajectories reach roughly 50 to 180 turns depending on the model and strategy, while unmanaged runs often stop after 20 to 30 turns. At 128K, the differences shrink. For Nemotron-3 30B on SWE-Bench, median trajectory length stays between 39 and 42 turns across context tiers. For Nemotron-3 550B, it stays between 70 and 74. If your agent rarely reaches the context limit, adding a complicated memory layer is probably solving a problem you do not have.

T4 is the practical winner among the context policies. The staged design keeps the cheap, deterministic cleanup ahead of an LLM summarization call. Recall, the reversible storage mechanism, sounds safer, but the agents rarely use it and it does not improve accuracy over elision alone. That is a useful warning for harness builders: recoverability is not automatically useful just because it is technically available.

The planning result is more conditional. On the weakest model, Nemotron-3 30B, removing planning drops SWE-Bench success from 25.2% to 13.6% at the 128K T4 setting. The failure is visible in the trajectory. Runs without planning have a median length of 5 turns, 68.6% terminate without editing a file, and 58.4% never get past localization. Planning keeps the agent alive long enough to attempt the work, but it costs more because it adds turns and tool calls.

The stronger models tell a different story. For Nemotron-3 550B, planning changes SWE-Bench success from 65.8% to 69.0% when disabled, while mean cost changes only slightly in the paired setting. The trajectory analysis is clearer than the headline percentage: planning cuts redundant post-edit verification. For Mistral, disabling planning lowers cost by about 30% on SWE-Bench and 32% on Terminal-Bench, with only small success-rate changes. Planning is therefore an efficiency control for a capable model, not necessarily a reasoning upgrade.

The action-space ablation is where the paper becomes uncomfortable for tool-heavy agent products. Under T4 at 128K, the full interface gives Nemotron-3 550B a 65.8% SWE-Bench success rate at $3.25 per task. Bash-only reaches 68.6% at $3.14. On Terminal-Bench, the same model moves from 37.1% at $2.33 with the full tool set to 45.4% at $1.72 with bash-only. That is a useful result, but it is not a license to delete every file and search tool. The task distribution is command-line oriented, and the full interface includes validation and state tracking that bash-only removes.

Choosing a harness by model and budget

For a bash-weak model, keep predefined tools. The clearest warning is Mistral's bash-only SWE-Bench result: 32.8% of runs end without editing a file, versus 1.2% with the full tool set. The structured interface lowers the effort needed to read, search, and modify a workspace. It also reduces the chance that the model emits a shell command it cannot compose correctly.

For a bash-capable model working on command-line tasks, start with a small interface. Bash lets the model combine several low-level operations into one script. In the study, bash-only reduces repeated patches and makes file-writing actions coarser. For Nemotron-3 550B, the median Terminal-Bench trajectory is 31 actions under bash-only versus 47 with the full interface, while the share of write-code actions rises from 16% to 27%. Fewer actions are not automatically better, but they reduce the places where a tool schema, validation hook, or extra observation can derail the run.

For context budgets below 64K, add context management before adding more tools. Start with deterministic elision. Add staged summarization only when long tasks still hit the limit. Do not begin with external recall because the paper found little evidence that agents use it. For 128K contexts, measure overflow and trajectory termination first. If neither changes, keep the harness simple.

For planning, split the decision by model capability. A small or inconsistent model needs a persistent plan because it may never make the first edit. A stronger model may benefit from planning mainly because it stops checking the same patch repeatedly. In that case, judge planning by cost per successful task, not by whether the plan looks thoughtful in the transcript.

This also changes how benchmark scores should be read. A result named Model X on Benchmark Y is incomplete unless it identifies the harness, context budget, tool interface, planning mode, and cost assumptions. LangChain reported a fixed GPT-5.2-Codex system moving from 52.8% to 66.5% on Terminal-Bench 2.0 after changing only its harness. OpenAI describes a different production lesson: its Codex team built a million-line product with zero manually written code by investing in repository context, tests, CI, and feedback loops. Anthropic's long-running harness work uses task decomposition and separate generator and evaluator agents. Those are not contradictions. They are evidence that the harness is part of the system under test.

The paper has limits. It uses four models, two benchmarks, local serving, and a particular lightweight implementation. Its bash-only comparison removes more than a menu of named tools, including some tracking and validation behavior. The results should guide an experiment, not settle a universal ranking. The good news is that the experiment is easy to copy: hold the model and task set fixed, then test context policy, planning, and action space separately. Record success, cost, turns, tool calls, and where failed runs stop.

If you are already deciding whether a local coding agent should use Ollama or vLLM, the queue test for Ollama and vLLM covers the serving threshold. If the bigger question is which model should handle routine work, use routing routine jobs to cheaper models as the next layer. Harness choice comes after those constraints, not before them.

My default build from this evidence is boring: deterministic context elision, a plan for weaker models, bash plus a few safe inspection tools for strong models, and explicit limits on turns and spend. Add machinery only when a trace shows the missing component would have changed the outcome. Otherwise, you are benchmarking your framework's ambitions instead of your agent's ability.

Sources