GPT-5.6 Sol leads the Artificial Analysis Coding Agent Index at 80 points

Ploy builds real marketing websites with AI agents. Full pages, components, generated imagery, screenshots, the whole loop. For four months, their eval suite could not find a model that beat Claude Opus on quality. Then GPT-5.6 Sol landed, and the numbers were immediate enough to justify ripping out months of Opus-tuned infrastructure.

The data they published is the most useful production migration benchmark I have seen this year.

What a real migration looks like

Ploy's agent runs against hundreds of fixture workspaces. Every build gets scored by a visual judge on ten binary checks (hero format, CTA styling, spacing, color usage) plus content assertions and tool-trajectory analysis. The runs below are from their redesign suite, where the agent rebuilds a brand's homepage against a reference design.

Mean per completed build Opus 4.8 (n=11) GPT-5.6 Sol (n=10)
Cost $3.06 $2.22
Wall time 8m 00s 3m 42s
Input tokens 2.60M 1.70M
Output tokens 33.0K 17.1K
Visual score 0.936 0.970

That is a 2.2x speedup and a 27% cost reduction, with better visual scores. The output token difference tells its own story. On one matched pair, Opus produced a 17,957-character CSS file with 174 mostly-unused CSS variables while GPT-5.6 wrote 2,508 characters and 45 variables for a page that rendered as good or better.

Speed comes partly from parallelism. Opus works sequentially; GPT-5.6 fans out parallel calls aggressively. But the speedup is not free. Ploy found their tool-call budgets and eval executor were tuned to Opus's style, and roughly a third of the initial cross-model failures traced back to harness assumptions, not model behavior.

The gotchas nobody warns you about

Three problems that cost Ploy real debugging time, and that you will hit too if you switch.

Tool call verbosity. Ploy's code tool has 25 optional parameters. Claude sends the two or three it is using. GPT-5.6 sends all 25, every time, with invented values like siteId: "00000000-0000-0000-0000-000000000000". Out of 6,635 production calls, every single one carried all 25 properties. This broke their file-read implementation. GPT-5.6 was reading blank files 52% to 64% of the time because offset: 0 looked real but was not. Prompting did not fix it. OpenAI's strict mode did not fix it. The only fix was redesigning the tool implementation to treat all params as real unless proven otherwise.

Prompt cache misconfiguration. OpenAI's cache-write pricing (first for GPT-5.6) means a cold cache costs real money. Ploy's first runs showed cost parity or worse with Opus, which was alarming until they realized their cache keys were not working across workspaces. They implemented a three-tier cache strategy: a static prefix that never expires, a workspace context that self-heals, and OpenAI's implicit chain cache within sessions. After the fix, first-call cache hits went from near zero to 83.7% and total uncached input tokens dropped 28%.

Reasoning replay state. GPT-5.6's Responses API replays prior-turn reasoning as server-side item references by default. Conversations started failing mid-stream with Item 'rs_...' not found. The fix was store: false, which makes the SDK request encrypted reasoning blobs instead of pointers. The lesson is that with server-side state in the loop, your effective prompt can change even when the bytes you send are append-only.

Where Sol fits among the current field

Independent benchmarks back up the production data. Artificial Analysis puts GPT-5.6 Sol at 59 on their Intelligence Index, one point below Claude Fable 5 at max reasoning but at roughly one third the cost per task. On the Coding Agent Index, Sol leads at 80 points across all three evaluations (DeepSWE, Terminal-Bench v2, SWE-Atlas-QnA), ahead of both Fable 5 in Claude Code and Opus 4.8.

The broader community take is more nuanced. The AI automation YouTuber Nate Herk summed it up as "Fable is the better manager; Sol is the really good worker that ships fast and costs a fraction." On Reddit, early adopters are split. Some call Sol "the real deal" that made improvements where 5.5 stalled, while others report more safety triggers on creative tasks.

Pricing tells its own story. GPT-5.6 Sol costs $5 per million input tokens and $30 per million output tokens, with 1.25x cache writes (also first for OpenAI) and a 90% discount on cache reads. Luna costs $1 and $6 and still pushes past GPT-5.5 on the Pareto frontier. Amazon's new Muse Spark 1.1 and xAI's Grok 4.5 both entered the chat this week too, but on coding agent benchmarks, Sol has the highest composite score in the field right now.

The practical takeaway is straightforward. If you are running production AI agents and have not rerun your eval suite on GPT-5.6 Sol, you are leaving money and time on the table. But swap in a Friday afternoon, not a Monday morning. The cache config and tool-call reshape will eat your weekend if you go in blind.

Sources