Training an agent in a toy loop is easy. Training the thing people actually deploy is where the plumbing falls apart.

OpenForgeRL proxy and remote rollout architecture

Most serious agents are not just a model plus a prompt. Claude Code, Codex, OpenClaw, and their cousins keep state across turns, call tools through several processes, launch subagents, read files, drive browsers, and sometimes operate inside a remote desktop. The harness is doing a lot of the work. Yet most open reinforcement-learning stacks still expect a clean request, a clean response, and a reward that arrives at the end.

That mismatch is the problem OpenForgeRL attacks. The paper, released on arXiv on July 23, describes an open framework that trains agents inside the same kind of harness they will use after training. It does this without forcing researchers to rebuild every harness as a special-purpose RL environment.

The missing adapter between agents and training code

OpenForgeRL has two important pieces. A lightweight proxy stands between the harness and the model endpoint. When the harness asks the model for its next action, the proxy serves the request and records enough information to reconstruct the trajectory later. Those recorded prompt-response pairs can then move into a conventional RL stack such as veRL.

The second piece is a Kubernetes orchestrator. Each rollout gets its own remote container, with the CPU, memory, browser, tools, and network state needed for the task. The trainer does not need to host every environment beside the GPUs. That separation sounds mundane, but it is the difference between a demo and a system that can run thousands of messy episodes without turning the training node into a landfill of half-dead processes.

The design also preserves the deployment harness. Researchers can train with ZeroClaw, OpenClaw, Codex, or a ReAct loop instead of quietly swapping the real agent for a simplified imitation. That matters because the same base model can behave differently when the system prompt, context compaction, tool schema, retry behavior, or subagent policy changes. A benchmark score without the harness is often measuring the wrong object.

The proxy approach has a cost. It does not make credit assignment disappear. A browser task can run for dozens of turns before success or failure becomes clear. OpenForgeRL reconstructs the trajectory, but the trainer still has to decide which actions deserve credit. The paper is candid about this. Error recovery remains weak, even when other reliability behaviors improve.

What the numbers say about harness training

The authors test two versions of the system. OpenForge-Claw uses a 30B-A3B mixture-of-experts model and trains across three harnesses, ZeroClaw, OpenClaw, and Codex, plus a standard ReAct loop. On ClawEval it reports 31.7 pass^3 and 55.9 pass@3. It reaches 33.7 on QwenClawBench and 28.1 on MCPAtlas, which the paper treats as a held-out test of novel tool use.

OpenForge-GUI uses an 8B model for browser and computer interaction. It reaches 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager. The paper says these results beat open baselines of similar size on nearly all reported tasks. In the GUI setting, the 8B system matches or beats models several times larger.

Those scores need some restraint. They are not proof that an 8B model has suddenly replaced a frontier system. The experiments use hundreds to a few thousand tasks, and benchmark protocols differ. The more interesting claim is narrower: a smaller model can gain useful behavior when the training loop sees the real interface it must operate.

The behavioral analysis is even more revealing than the leaderboard. Reinforcement learning raises self-verification, tool coverage, format robustness, and completion of multi-step plans. The model is more likely to check whether a write action worked and more likely to touch every service required by a task. Error recovery is the stubborn exception. When a command fails, the agent still often collapses instead of diagnosing the failure and continuing.

That pattern feels familiar to anyone who has watched an agent work. Planning is easy to reward because the trace contains visible actions. Recovery is harder because the useful move depends on what went wrong, and the failure may not be obvious from the last tool response. OpenForgeRL makes that weakness measurable instead of hiding it behind a hand-written harness.

Why this matters for the next agent stack

The uncomfortable conclusion is that model training and harness engineering are no longer separable jobs. A model can have strong tool-use knowledge and still fail because the harness compacts context badly. Another model can look weaker in a bare chat loop and become much more reliable when the harness gives it a better retry policy and a read-back step. OpenForgeRL gives researchers a way to train against those real conditions.

It also gives us a better experimental question. Instead of asking only which model is strongest, we can ask which combination of model, harness, environment, and training method produces the fewest expensive failures. That is closer to how agents are bought and deployed.

There are limits. Kubernetes adds operational overhead. The framework depends on careful trajectory reconstruction. Sparse rewards remain difficult. The paper also shows that harnesses differ in how easy they are to learn, so results may not transfer cleanly between OpenClaw and Codex. A recipe that works for one interface can still fail on another.

Still, this is the kind of infrastructure the open agent community has been missing. The current agent race is often described as a contest between model checkpoints. OpenForgeRL suggests the more useful unit is the whole running system. Train the harness people use, in the environment it sees, and the benchmark starts to look less like a magic trick.

Sources