A frontier model can be perfectly open and still be impractical to run. The weights are only half the bill. The other half is the memory needed to keep conversations alive, serve several users at once, and stop one request from reading another request's cache.
Cloudflare's August 3 engineering writeup is useful because it treats this as an operations problem rather than another leaderboard contest. The company is serving Moonshot's Kimi K-series and Z.ai's GLM models through Workers AI, with SGLang handling the inference layer. Its approach has three parts: store the attention cache in a smaller format, compress the model weights, then add integrity checks so the shared cache does not become a data leak with a nice benchmark score.
The result is a lesson that many model launch posts skip: the cheapest token is often created by changing the serving system, not by choosing a smaller model.
The memory problem is two different problems
During inference, a long request has a prefill phase and a decode phase. Prefill processes the prompt. Decode generates the answer one token at a time. They stress the hardware differently, which is why one global precision setting is a bad compromise.
The KV cache belongs to the decode side. It stores the attention keys and values for tokens the model has already seen, so the model does not reread the entire conversation for every new token. On a long-context model, that cache can fill GPU memory before the weights do. The practical consequence is easy to miss: a model may have enough memory to answer one request, yet fail when a second wave of users arrives.
Cloudflare stores the cache in FP8 instead of BF16 for Kimi K2.6. That halves the cache footprint. The company says the same hardware can hold about 1.37 million tokens instead of roughly 686,000. FP8 does not make one request faster. In Cloudflare's direct kernel measurements, BF16 is slightly faster at the same concurrency. FP8 wins because it keeps more requests resident.
At 32 concurrent requests, BF16 reached 1,558 tokens per second. At 64, it ran out of memory. FP8 reached 2,192 tokens per second at 64 requests. That is about 41 percent above BF16's peak and, in Cloudflare's estimate, cuts cost per token by about 30 percent. The trade is capacity, not magic acceleration.
The accuracy check matters more than the headline. On the company's evaluation suite, the BF16 and FP8 cache results were nearly indistinguishable: MMLU was 89.11 versus 89.04, MMLU-Pro was 80.29 versus 79.29, and tool-call validity was 92.2 percent versus 92.6 percent. Those numbers are close enough that the cache format is not the part I would worry about first. I would worry about whether the benchmark reflects your prompts, your tool schemas, and your actual context lengths.
If the difference between a useful long-context system and an expensive demo is still fuzzy, our earlier piece on why long context became a memory problem gives the model-level background. Cloudflare's work starts where that article ends: at the point where several real users compete for the same GPU.
Four bits help, but only during the right phase
The second change targets the weights. Cloudflare compresses GLM 5.2 into INT4 weights. The checkpoint is 421 GB, compared with 705 GB in FP8, and an eight-way deployment needs roughly 52 GB per GPU instead of 88 GB. That recovered space can hold about 1.18 million tokens of KV cache on the same hardware.
Decode gets the obvious win. With one concurrent request, GLM reached 92 tokens per second in INT4, versus 60 in FP8, a 55 percent gain. At 32 requests, the gain was 27 percent, with throughput rising from 994 to 1,267 tokens per second. Smaller weights mean less data has to stream out of memory for each generated token.
Prefill is the awkward part. INT4 weights have to be expanded before the matrix operations, so the compressed version is slower there. Cloudflare measured about 10,160 prefill tokens per second in FP8 and 8,660 in INT4. A single precision choice would force the platform to lose somewhere. Separating prefill and decode lets the operator use FP8 where arithmetic dominates and INT4 where memory bandwidth dominates.
That is the genuinely useful idea here. Quantization is not a badge that says a model is efficient. It is a placement decision. Put the cheaper representation on the part of the pipeline that benefits from smaller weights. Keep the less compressed representation where decompression costs more than it saves.
The quality numbers also show why blanket claims about lossless quantization deserve suspicion. Cloudflare reports less than a point of movement across its evaluation set. GSM8K exact match changes from 94.39 percent to 93.56 percent, MMLU from 86.60 to 86.54, and MMLU-Pro from 80.80 to 80.47. That looks harmless for general chat. It may not be harmless for a narrow task that lives in the tail of the distribution. The right question is not whether INT4 preserves an average score. It is whether it preserves the behavior your application depends on.
More sharing creates a security problem
The first two optimizations pack more traffic onto shared hardware. That improves utilization and creates a new failure mode. Paged attention and continuous batching make many requests read and write physical cache pages. If the bookkeeping points at the wrong page, one request could receive data produced for another.
Cloudflare's fix is a generation tag for every cache page. The tag changes when the page is reallocated. Each request carries the page and tag it expects, and the server checks the mapping before supported decode operations read the cache. A mismatch aborts the request rather than returning data from the wrong page.
This is the part of the post that deserves more attention than the throughput chart. A cache isolation bug does not look like a model hallucination. It looks like an infrastructure bug, and the victim may have no idea that the returned text came from somebody else's context.
The check costs less than one percent in Cloudflare's measurements. At concurrency levels one, two, four, and eight, throughput changed by roughly 0.38 to 0.79 percent below baseline, while p95 latency rose by about 0.42 to 0.80 percent. The validation runs as a separate batch check instead of being fused into the attention kernel, avoiding a race between GPU thread groups. Deployments that do not enable the tracker use a no-op path.
That is a reasonable price for a shared cache. The uncomfortable detail is that the protection exists because the system is aggressive about sharing memory. Efficiency and isolation are coupled here. You do not get to optimize one and pretend the other is somebody else's concern.
The practical verdict
Cloudflare's numbers do not prove that every Kimi or GLM deployment should use FP8 KV caches and INT4 weights. They show how to reason about the tradeoffs.
Measure prefill and decode separately. Measure memory admission, not just tokens per second. Test at the concurrency where your service actually starts queueing. Run application-specific evaluations after quantization, especially if your model writes tool calls, handles long retrieved documents, or processes sensitive conversations. Add a cache integrity check before the incident report forces you to.
The model is only one component in the cost equation. A 705 GB checkpoint that serves one user slowly is not cheaper than a compressed deployment that serves many users safely. The interesting frontier is no longer just which model can solve the hardest benchmark. It is which serving stack can keep the model useful when the GPU is crowded.
Sources are linked in the article text: Cloudflare's production writeup provides the measurements, SGLang is the serving framework used in the experiments, and the Hacker News discussion records the first community reaction to the release.