For four months, nothing beat Claude Opus in Ploy's production pipeline. Then GPT-5.6 Sol showed up and did the same work in less time, at lower cost, with half the tokens. The numbers sound clean. The migration was not.
Ploy builds and edits real marketing websites with an AI agent. The agent plans a page, reads the codebase, writes components, generates imagery, screenshots its own work, and decides when it is done. That job description sets a high bar. Every frontier model gets tested against it. For months, Opus 4.7 and then Opus 4.8 held the default slot. Nothing else came close. When GPT-5.6 launched in late June, Ploy ran it against their eval suite and found something unexpected: it was not just competitive, it was better. Then they tried to actually switch.
What the benchmarks miss
The head-to-head numbers are striking. On matched redesign builds (n=10-11 per model), GPT-5.6 Sol averaged 3 minutes 42 seconds per completed build versus Opus's 8 minutes. Cost per build dropped from $3.06 to $2.22. Output tokens fell from 33K to 17.1K. The visual quality score, measured by a judge running binary checks against reference designs, actually improved from 0.936 to 0.970.
On one matched pair, Opus produced a 17,957-character globals.css with 174 CSS variables, most unused. GPT-5.6 wrote 2,508 characters and 45 variables for a comparable rendered page. The model writes lean code. That efficiency compounds across an entire build.
On TerminalBench 2.1, GPT-5.6 Sol scored 88.8% versus Opus 4.8's 78.9%. But Ploy's numbers matter more than synthetic benchmarks because they measure production outcomes, not isolated tasks. The agent has to handle full-stack website builds, not just complete code snippets.
The migration nobody warns you about
Here is where the blog post gets interesting. Ploy uses Vercel's AI SDK, a universal LLM provider abstraction. You would think switching models would be a config change. It was not. The things developers think of as "the model" turned out to be provider-specific behaviors the entire stack had quietly specialized around.
The first problem was the eval harness itself. Ploy's tool-call budgets were sized for Opus's sequential calling style. GPT-5.6 fans out parallel calls and blew through those budgets on cases it was actually solving correctly. The eval executor did not support batched file reads, which Opus rarely used and GPT-5.6 uses constantly. Roughly a third of the raw failures in the first cross-model run were harness assumptions, not model behavior. They were not evenly distributed between models. If you evaluate a challenger model against an incumbent, you have to triage the traces before trusting the pass rate. Otherwise you are grading the new model on how well it imitates the old one.
One dataset omitted a minScore threshold and silently inherited a default of 1.0. GPT-5.6 "failed" a hero it scored 0.98 on. Opus "failed" a case while passing every individual check. Two defensible design choices, one invisible threshold. The kind of thing that hides in eval suites for months until a model switch forces it into the open.
The remaining surprises were in tool schemas, prompt caching mechanics, and reasoning replay between turns. Each one required investigation, a fix, and a re-run. None of them showed up in the benchmark numbers.
The real cost is not the API bill
The migration guide Ploy published reads like a field report from a model war, not a product announcement. Their conclusion is blunt: if you are evaluating a challenger model, triage every trace before you trust the pass rate. A dataset with hidden thresholds or harness assumptions baked into tool-call budgets will give you false negatives on the new model and false positives on the old one.
GPT-5.6 Sol is priced at $5 input and $30 output per million tokens. Terra sits at $2.50/$15, matching Claude Fable 5 on TerminalBench at 84.3%. Luna goes for $1/$6. The pricing structure is deliberately tiered to pull cost-sensitive workloads away from competitors, and the token efficiency gains make the price gap wider than it looks.
For teams running production agents, the Ploy migration is a useful case study because it surfaces the hidden dependencies. Switching the model is the easy part. Switching the eval harness, the tool schemas, the caching strategy, and the reasoning replay loop is the expensive part. And it only gets surfaced when someone actually tries the switch on real workloads, not on benchmark suites.
What this means for the model war
The leaderboard narrative is simple: GPT-5.6 Sol overtook Claude Opus on TerminalBench and in at least one production deployment. But the real story is about what "better" means in context. Opus held the top slot for four months not because it scored highest on benchmarks, but because it handled complex agentic workflows reliably. GPT-5.6 appears to do the same work faster and cheaper, which is a different kind of "better" than winning a leaderboard.
The model war is no longer about who scores highest on synthetic benchmarks. It is about which model you can actually deploy in production with your existing tooling, your existing eval suite, and your existing team's mental models. Ploy's experience suggests the gap between "wins the benchmark" and "works in production" is still wide enough to require a dedicated migration effort.