An agent can browse a page, call a tool, and produce a convincing answer. That is the easy part. Ask it to remember a constraint across several turns, move evidence between apps, and recover after a user changes the request, and the success rate drops below half.
That is the uncomfortable result in UniClawBench, a new benchmark from HKU-MMLab that evaluates proactive agents in live Docker environments instead of frozen web copies and tidy one-shot prompts. The paper describes 400 manually built tasks, split evenly between English and Chinese. Each of the five capability families gets 40 tasks in each language. The authors also published the code and task suite on GitHub.
The benchmark is aimed at a very specific problem: current agent scores often tell us that a system failed, but not why. An “office” task might fail because the model could not read a screenshot, lost a file path, forgot an earlier instruction, or handed a partial result to another agent. Those are different failures. Treating them as one scenario category makes the leaderboard look cleaner than the underlying systems really are.
The benchmark watches the whole attempt
UniClawBench divides tasks into Skill Usage, Exploration, Long Context, Multimodal Understanding, and Cross-Platform Coordination. The labels are less important than the design choice behind them. The task author tries to make one capability the main bottleneck while still allowing normal tool use around it. A long-context task can involve files and commands, but the hard part is keeping the relevant facts alive. A cross-platform task can use a browser and a local workspace, but passing information between those surfaces is the test.
The evaluation loop has three roles. The executor is the agent being tested. A hidden supervisor watches the trajectory and checks private completion criteria. A user simulator sees the visible trajectory and a coarse progress signal, then sends a follow-up instruction. The supervisor does not hand its rubric to the user simulator, and the executor never receives the hidden answer files. That separation matters because a benchmark that leaks its grading rules is just a prompt-following contest.
The live setup also avoids a problem that looks minor until you deploy an agent: real services change. A product price can move overnight. A page can reorder its content. A local file can be modified by an earlier step. Instead of comparing the final output with one permanent answer, UniClawBench uses fine-grained checkpoints. The system scores whether the required evidence and intermediate state appear at the right point in the run.
That does not make evaluation perfect. It makes the failure visible. The paper reports a reliability study over 50 trajectories, scored independently by three human experts. The automated supervisor reached Pearson correlation of 0.71 for pass or fail judgments and Spearman correlation of 0.68 for continuous completion scores. Those numbers are good enough to make the benchmark useful, while still leaving plenty of room for disagreement on messy tasks.
The ceiling is lower than the demos
The authors tested ten models under the same OpenClaw framework, then tested representative models across OpenClaw, EDICT, and Nanobot. The strongest reported overall pass rate stays below 50 percent. Claude Opus-4.8 reached 47.5 percent with OpenClaw, while GPT-5.4 reached 40.7 percent. The pass rate is the strict metric: the task either satisfies the required checkpoints or it does not. Average completion scores are higher because an agent can do most of the work and still miss one condition that makes the result unusable.
The pattern by capability is more revealing than the overall ranking. Claude Opus-4.8 with OpenClaw passed 55.0 percent of Skill Usage tasks and 82.5 percent of Exploration tasks. Its pass rate fell to 43.8 percent on Long Context, 17.5 percent on Multimodal tasks, and 38.8 percent on Cross-Platform tasks. GPT-5.4 showed the same shape: 51.2 percent on Skill Usage, 77.5 percent on Exploration, then 22.5 percent on Long Context, 17.5 percent on Multimodal, and 35.0 percent on Cross-Platform.
In plain English, agents can find things and operate familiar tools. They are much less reliable when the job requires them to preserve state, interpret visual evidence, or coordinate across application boundaries. That is exactly where product demos tend to become vague.
The framework comparison is the sharper finding. OpenClaw produced the highest pass rate for every tested model in the paper's cross-framework study. For GPT-5.4, OpenClaw reached 40.7 percent, compared with 33.8 percent for EDICT and 29.0 percent for Nanobot. For Claude Opus-4.8, OpenClaw reached 47.5 percent, while EDICT reached 41.5 percent and Nanobot 38.5 percent. The differences are not just about which model sits underneath. They come from how much of the task trajectory survives each handoff.
OpenClaw keeps the instruction, tool evidence, and user feedback in one main trajectory. EDICT uses more orchestration and handoffs, which can produce higher partial completion scores but lower strict pass rates. Nanobot uses fewer input and output tokens, yet its smaller context footprint comes with a cost on difficult tasks. This is a useful reminder for anyone building an agent stack: adding more agents does not automatically add more capability. Sometimes it just adds more places for state to disappear.
The benchmark's most practical lesson is that “model quality” and “agent quality” are now inseparable. A strong model can still be trapped by a weak runtime. A lightweight model can look better when the framework preserves context and gives it clean recovery paths. The table is not merely ranking models. It is measuring the plumbing around them.
What the paper leaves open
UniClawBench has 400 tasks, which is a serious manual effort but still a small sample for a broad claim about general assistants. Live environments introduce their own noise. Browser behavior, dependency versions, timing, and evaluator judgments can all affect a run. The benchmark also tests a particular family of proactive-agent assumptions, so it should not be treated as a universal measure of every coding or research system.
There is another limitation that matters more to builders than leaderboard position. The benchmark can tell you that multimodal or cross-platform work is weak, but improving those scores may require new tools, memory policies, or task decomposition rather than a better base model. That makes the next research step less glamorous and more useful: run the same model through several runtimes, inspect the lost state, and fix the handoff before buying a larger model.
UniClawBench is valuable because it makes agents fail in ways users recognize. The result is not that autonomous assistants are useless. It is that the polished demo is still measuring the least difficult part of the job.
References used in this analysis include the UniClawBench paper, the official project repository, and the Hugging Face paper page.