Intelligence density comparison of Bonsai 27B vs other models

A 27-billion-parameter language model takes up about 54 gigabytes in standard 16-bit precision. Even with aggressive 4-bit compression you are still looking at roughly 18 GB. Your phone has 12 GB total, and the operating system lets apps use maybe 6 of those. So when PrismML announced on July 14 that their Bonsai 27B runs on an iPhone 17 Pro, the math was worth staring at.

The 1-bit variant weighs 3.9 GB. That leaves enough headroom for the KV cache and activations. It runs at 11 tokens per second. That's usable, not blazing. For a model class that was exclusively a cloud creature until yesterday, usable is the whole point.

PrismML is a Caltech spinout backed by a $16.25 million seed round from Khosla Ventures and Cerberus Ventures. They have been shipping compressed models since April: first the 8B, then a ternary image model, now this 27B flagship. Each release pushes the same idea further: what if you compress the entire network end to end, no high-precision escape hatches, and see what survives?

The architecture is built on Qwen3.6 27B, a 27.8-billion-parameter multimodal model. PrismML took the weights and constrained every single one (embeddings, attention projections, MLP blocks, and the language modeling head) to either binary values ({-1, +1}) for the 1-bit variant or ternary ({-1, 0, +1}) for the ternary variant. Group-wise scaling factors in FP16 preserve per-group magnitude, so the model is not literally 1-bit in the information-theoretic sense. The ternary variant clocks in at 1.71 effective bits per weight; the binary variant at 1.125. That puts the effective compression at 9.4x and 14x respectively.

The vision tower ships in compact 4-bit form so the model can accept screenshots, documents, and camera input. Context length is 262K tokens. Speculative decoding is supported for lossless acceleration.

The benchmark table from PrismML shows what gets lost and what does not. Ternary Bonsai 27B retains 95% of the full-precision Qwen3.6 baseline across 15 benchmarks in thinking mode. 1-bit retains 90%. The scores cluster by capability, and the pattern is telling. Math holds at 93.4 (ternary) and 91.7 (1-bit), within two points of the 95.3 baseline. Coding is at 86.0 and 81.9 against 88.7. A bigger drop, but still workable. Tool calling and agentic benchmarks take the hardest hit: 74.0 and 66.0 compared to 80.0. The model can still do agentic work, but it gets sloppier the more complex the chain of actions becomes.

A community benchmark on GitHub by a developer running these on a Jetson Orin found the same pattern. The model works. It fits. But it is not a free lunch. You are trading quality for footprint, and the tradeoff is measurable.

The speed numbers are encouraging. On an RTX 5090, 1-bit Bonsai hits 163 tokens per second in 1-bit and 134 in ternary. On an M5 Max laptop, 87 and 58 respectively. On the iPhone 17 Pro, 11 tokens per second. One user on Reddit reported closer to 30 on an M4 Pro and around 100 on an iPhone 17 Pro Max, suggesting the silicon matters a lot and Apple's Neural Engine might be doing some heavy lifting under the hood.

What makes this different from just downloading a heavily quantized GGUF is the end-to-end architecture. When you take a standard model and quantize it to Q2 or Q1, the quality falls off a cliff because the compression was an afterthought. PrismML trained with the constraint baked in, and the models were optimized specifically for this representation on Google v5 TPUs. The whitepaper claims that the most aggressive conventional low-bit build of Qwen3.6 27B scores significantly lower than 1-bit Bonsai while occupying 2.5x more memory. That is the comparison that matters.

Not everyone is sold. The r/LocalLLaMA subreddit has threads calling the earlier 8B Bonsai "pure hype" and "much dumber than Gemma 4 E2B." The 27B is getting a more measured reception . people acknowledge the compression achievement but question whether you would actually want to run this instead of a smaller model at higher precision. One commenter put it bluntly: 27B at Q1-ish quality behaves more like a very smart 8B than a real 27B.

That critique has teeth. The intelligence density number PrismML touts . 0.53 per GB for 1-bit Bonsai 27B, more than 10x the full-precision baseline . is a measure of efficiency, not absolute capability. If you have a laptop with 24 GB of RAM, you might get better results running a smaller model at Q4 than this one at 1-bit. The phone is the real target, and on a phone the comparison is not against full-precision 27B. It is against whatever tiny model you could run before, which was probably a 3B or smaller.

That is where the argument gets interesting. Bonsai 27B is not just a technical demo. It ships under Apache 2.0, with weights on HuggingFace, an API on Together AI (free tier), and an iOS app called Locally AI that wraps the model for on-device chat. PrismML is building an ecosystem around the idea that local agents should be the default, not the fallback. A model that fits on a phone and can reason across 262K tokens of private data without a network round trip changes what products you can build.

They also released a demo of Hermes Agent running on ternary Bonsai 27B on an RTX 5090, doing multi-step tool calling and MCP integration. The demo is polished, but demos always are. The real test will be whether agent frameworks start adopting these compressed models as first-class targets or whether developers stick with cloud APIs for anything that matters.

The timing is good. Apple has been signaling that on-device AI is a priority, with the iPhone 17 Pro's 12 GB of RAM explicitly pitched as headroom for local models. Google is pushing Gemini Nano for on-device Android. Microsoft is putting NPUs in every Copilot+ PC. The hardware is arriving before the models that can use it well. Bonsai 27B is an early answer to the question of what actually runs on that hardware, not as a toy but as something that can do real work.

The question I keep coming back to is not whether this specific model changes everything. It is whether the compression curve keeps bending. Ternary Bonsai 27B retains 95% of full precision at 9.4x compression. If the next generation retains 97% at 15x, and the one after that hits 98% at 20x, the cloud-to-device migration stops being a niche story and becomes the default architecture for anything privacy-sensitive or latency-bound. The 1-bit Bonsai 27B is a good model on its own terms. Its real significance might be as a proof point that the curve is bending at all.

Sources