The worst way to improve an AI coding agent is to change five things, run one impressive task, and decide the new setup works. That is how teams end up defending a pile of hooks, memory files, and custom prompts that nobody can measure.

AWB, short for AI Workflow Benchmark, gives you a saner loop. Its useful idea is simple: hold the model and task prompt steady, then compare a bare tool invocation with the configured workflow around it. The new v1.7.0 release adds end-to-end Codex CLI support, but the bigger win is the small probe hiding inside the project. The fast-check runs eight representative tasks in roughly 12 minutes with parallelism and costs about $4 in metered spend. That is cheap enough to run before merging a configuration change.

AWB showing its coding-agent checkup workflow

If you want the background on why agent claims need stronger evidence, our earlier look at what agent evaluations need to prove is the useful companion. This article is narrower: how to test whether your own harness helps.

What the fast-check actually measures

AWB is not another model leaderboard. Its methodology gives every adapter the same task description, the same starting repository at a pinned commit, the same timeout, and the same verification commands. Each tool may use its normal hooks, rules, agents, and project instructions. The comparison is between the configured workflow and a vanilla baseline using the same underlying model.

That distinction matters. A model may be capable of fixing a bug, but your harness can still bury the relevant file under stale context, spend too many turns on a plan, or skip the final test. AWB records correctness, cost efficiency, speed, code quality, reliability, security, and efficiency. Correctness carries 55 percent of the default composite score, while cost efficiency carries 15 percent. The weights tell you what the result rewards. They also tell you what it does not settle. A higher total does not prove that the agent is pleasant to use, safe for production, or better on your private codebase.

The fast-check samples eight hand-picked tasks, one for each category in its probe list. It produces an estimated full-suite score with a 95 percent confidence margin, but the project explicitly treats that output as a quick signal rather than a publishable ranking. Eight tasks can catch a configuration that is obviously harmful. They cannot establish that a tiny two-point difference is real.

The repository's own example makes the point. A paired run reported Workflow Lift of +4.2 points with a p-value of 0.031, with the custom setup passing 68 percent of tasks versus 62 percent for vanilla. The same report says the custom workflow improved bug diagnosis by 12.3 points and multi-file reasoning by 8.1 points, but lost 4.2 points on cost discipline. That is a much better engineering conversation than “the agent feels smarter.” The configuration helped in specific places and charged rent for it elsewhere.

Run the eight-task probe

Install AWB in a fresh Python 3.11 or newer environment. The project pins its core dependencies, which is important because a benchmark that silently changes its own CLI or parser dependencies is not a stable benchmark. A minimal setup looks like this:

git clone https://github.com/xmpuspus/ai-workflow-benchmark
cd ai-workflow-benchmark
python3 -m venv .venv
. .venv/bin/activate
pip install -e .
awb quickstart

Run the warmup before judging the wall clock. AWB caches bare repository clones and prepared workspace templates under ~/.cache/awb/. The project says warm templates make repeated setup 10 to 30 times faster, and its published timing assumes the cache is warm and four tasks run in parallel. If you compare one cold run against one warm run, you are measuring disk and package installation behavior as much as agent behavior.

For the first pass, use the fast-check mode rather than the full suite:

awb warmup
awb run

Select the fast-check mode and four workers in the CLI options for this first pass.

The exact adapter and authentication flags depend on the tool installed on your machine. AWB currently lists Claude Code, Pi, Gemini CLI, and Codex CLI as full adapters. The v1.7.0 Codex path uses an ephemeral workspace-write execution mode with JSON event output, so token usage, command exits, file edits, model provenance, and trace events enter the same result format. That is useful, but it is also a reminder to pin the CLI and model before comparing runs.

To test a configuration change, keep the task set and model fixed, then run the vanilla and configured variants. Do not change the prompt, model reasoning level, timeout, hardware, and configuration in the same experiment. If you do, a positive Workflow Lift has no clear cause. Export the result and keep the run metadata, especially tool version, model identifier, repository commit SHAs, task-set hash, configuration hash, Python version, and hardware class.

The full suite is a different commitment. AWB describes 100 tasks run three times as roughly three hours and about $150 under its stated pricing assumptions. The fast-check is about 97 percent cheaper than that published full-suite estimate. Use the small run for configuration iteration. Promote only a stable change to the full evaluation.

A useful local gate is therefore: run fast-check before the change, make one harness edit, run it again, and keep the edit only if the task-level failures move in the direction you care about without an unacceptable cost increase. If bug diagnosis improves but token cost discipline collapses, the answer may be a narrower rule or a lazy-loaded skill, not a bigger prompt.

The main trap is treating the confidence margin as magic. The eight tasks are hand-picked and the result is still one run. Model output is stochastic. Network latency moves the speed score. Provider prices change the dollar estimate. A task can also resemble code seen during training, and AWB can flag contamination risks without proving novelty. For a serious claim, repeat the experiment at least three times, use the same hardware class, and inspect task-level results instead of staring only at the composite.

Zenodo archives v1.7.0 as a 1.7 MB software release with DOI 10.5281/zenodo.21753418. That archive does not make every score true, but it gives future readers a fixed version to retrieve. Reproducibility is a habit here: pin the code, pin the tool, pin the model, pin the tasks, and save the configuration hash.

The practical decision is not whether AWB proves your agent is good. It cannot. The decision is whether a proposed workflow change survives a cheap, paired test before it becomes permanent folklore. Eight tasks and $4 will not answer every question. They can answer the first one, which is whether the change deserves a longer experiment.

Sources