A cybersecurity firm ran a test that shouldn't have been close. They pointed two AI models at the same set of real-world applications and asked each one to find a specific class of security flaw: IDOR vulnerabilities, where an app exposes an internal identifier without checking whether the user is actually authorized to access it. It's one of the most common bugs in web applications, and finding it requires understanding not just code syntax but the business logic behind who should see what.

The results, published by Semgrep last week, landed like a grenade. GLM 5.2, an open-weight model from Chinese AI lab Zhipu AI, scored 39% F1 on the IDOR detection benchmark. Claude Code, running Opus 4.8, scored 28%. The previous best Claude variant, Opus 4.6, managed 37%. All three models ran in nearly identical conditions: a simple Pydantic AI harness with the same prompt, no fancy endpoint discovery, no guided navigation. Just "here's the code, find the bugs."

The cost per vulnerability found? About 17 cents on GLM 5.2. Claude Code runs roughly six times that.

Benchmark results showing GLM 5.2 outperforming Claude Code on IDOR detection

What the harness actually measures

Before anyone declares open-weight models superior to closed-source frontier models, the Semgrep team makes an important caveat that's easy to miss. Their own purpose-built multimodal pipeline, which combines endpoint discovery with guided navigation and runs on GPT 5.5, scored 61% F1. The same pipeline on Opus 4.8 hit 53%. Both blew away every "prompt-only" configuration, including GLM 5.2.

The takeaway isn't that GLM 5.2 is a better security model than Claude. It's that the gap between a raw model call and a well-orchestrated pipeline dwarfs the gap between any two models. "The largest performance gap in the table isn't between models," Semgrep wrote. "It's between configurations that get endpoint discovery and those that don't."

This matters for the economics of security tooling. If you're a security team deciding between paying for Claude Code's built-in agent or rolling your own pipeline with an open-weight model, the math now tilts toward the open option. You get comparable raw detection at a fraction of the cost, and you keep the option to swap models as better ones appear.

The verification problem nobody expected

Independent testing has mostly confirmed Semgrep's numbers. Design Arena's coding leaderboard now shows GLM 5.2 at number one, ten Elo points above Claude Fable 5. Fireworks reproduced a 91.4% score on GPQA-Diamond. The model runs on a single 256GB Mac Studio at 3-9 tokens per second, thanks to Unsloth's 2-bit dynamic quantization that compresses the 1.5TB weights down to about 239GB.

But there's a wrinkle. Graphistry's analysis, published on the same day as Semgrep's benchmark, raised a more uncomfortable question: where did GLM 5.2's intelligence come from?

Graphistry measured the correlation between model outputs using Cohen's Kappa, a statistical metric that captures agreement on both correct and incorrect answers. OpenAI's models and Anthropic's models show a Kappa of about 0.63 with each other, which is roughly what you'd expect from two independently trained systems. GLM 5.2 jumps to 0.80 with GPT 5.5 and 0.76 with Opus 4.8. That kind of correlation in both right and wrong answers is what you see when one model was trained on the outputs of another.

Graphistry's conclusion: GLM 5.2 "may be an illegal distillation of both GPT-5.5 and Opus 4.8." If true, it would mean the model's performance isn't a breakthrough in independent Chinese AI research but a very sophisticated copy operation. Zhipu AI hasn't responded to the allegation.

The silicon question

There's another dimension to the story that makes it more than a benchmark race. GLM 5.2 was trained on approximately 100,000 Huawei Ascend 910B chips using the MindSpore framework. No NVIDIA hardware involved. This is the first time a model competitive with frontier US labs has been built entirely outside the American-controlled chip supply chain.

The implications for export controls are significant. If Chinese labs can produce Opus-class models on domestic silicon, the strategic value of chip restrictions drops considerably. Zhipu's market cap reportedly reached HK$1 trillion after the release, with projected revenue growth of 534% for the year.

The Z.ai founder has publicly stated that open-weight Fable-class capabilities will arrive before Q1 2027. Given GLM 5.2's trajectory, that timeline doesn't feel ambitious.

What security teams should actually do

The practical implications are straightforward. Security teams running automated vulnerability scanning should test GLM 5.2 alongside their current models. The MIT license means you can deploy it locally, which matters enormously for teams handling sensitive codebases. The cost advantage is real: at 17 cents per vulnerability versus a dollar or more for frontier models, the math works even if you need to run 10x more scans.

But the harness point from Semgrep is the one that matters most. A raw model, no matter how capable, isn't a security tool. The endpoint discovery, the guided navigation, the business-logic context, that's where the real performance lives. GLM 5.2 makes the raw model part of that equation cheaper and more accessible. The orchestration layer is still where you should spend your engineering time.

The distillation question will probably never be fully answered. But the practical reality is that the weights are out there, they work, and they're free. As one researcher put it: "The weights are free and unbannable now. It's the silicon to run them that turned into the scarce part."

Sources