Bonsai 2 27B: A 27B Model at 1.7 Bits, and the Fork It Forces

On 17 September 2026 PrismML published Bonsai 2 27B: a ternary-quantized build of Qwen3.8-27B under Apache 2.0. This is not a lab curiosity. It compresses the exact base model that has run our production inference lane as FP8 on 4× RTX 5090 since 18 August. I read the model card, the whitepaper and the repository contents, and pulled the repository numbers myself on 22 September. What follows is sorted into what I could verify and what the vendor asserts.

Who is selling this

PrismML grew out of the Caltech orbit (CEO Babak Hassibi, a compression researcher), raised a $22.25 million seed, and counts Ion Stoica among its advisers. Its earlier Bonsai models shipped from March onward; TechCrunch puts the family's cumulative downloads above 11 million and reports talks with Apple. The last part stays rumor, unconfirmed by the CEO's side.

What matters to us is the technique, not the company. Ternary weights were lab ware because early binary and ternary attempts collapsed precisely where reasoning lives. PrismML claims to have broken that wall with a rotated ternary representation. On a model we operate ourselves, down to the KV-cache arithmetic.

Numbers I can check myself

File sizes from the repository's LFS metadata on 22 September, measured, not from marketing:

On the community side: 2,227,879 downloads and 1,768 likes on the GGUF repo, five days after release. And an ecosystem that grew within 48 hours. CRACK, abliterated and uncensored variants already stack up in the hub. That is a mirror of the scene, not a recommendation. For a company under data-protection duties, a tampered variant of this file is not a model, it is a risk without a ticket number.

What the vendor claims

Everything below this line comes from the whitepaper, not from our bench:

• 84.78 average across 14 thinking-mode benchmarks against 86.32 for the FP16 reference. The widely quoted 98.2 %.
• Conventional 2-bit builds collapse selectively: IQ2_XXS holds MMLU-Redux (88.93) but drops to 57.5 on AIME26 and 56.4 on LiveCodeBench. Bonsai 2 is claimed to hold exactly those (95.83 and 90.07).
• Round trip on our card class: about 130 tok/s decode on a single RTX 5090 (PQ2_0, batch 1), 120.5 with PTQ1_0. Around 30 tok/s on a 72-watt L4.
• On laptops: 47 tok/s on an M5 Max, 27.5 W GPU draw on an M5 Pro.

What stands out in that table is not the peak value but the pattern: the conventional IQ2_XXS build does not lose evenly. It loses the thinking. MMLU-Redux, a knowledge store, stays intact; math and code break away. That is exactly why a number like 98.2 % is worthless without the breakdown. We saw the same effect with FP8 conversions on our own vLLM lane. Quantization is not uniform decay; it is a choice about which capabilities survive.

The independent test exists. For the wrong generation.

One independent Terminal-Bench run is already published, but against the first Bonsai generation (July, before the thinking focus): 7.9 % on Terminal-Bench 2.0 with an 8 GB laptop GPU. Behind Qwen3.5-9B (9.2 %), far behind Qwen3.6-35B-A3B (24.3 %). And the VRAM advantage shrank under real load: 7.29 GB resident for the 2-bit file at 16k context versus 5.68 GB for the 9B model at Q4. The r/LocalLLaMA thread documents the very question we ask ourselves: how does aggressive compression compare against a smaller model at ordinary quantization? For Bonsai 2, the answer under an agent harness is still out. PrismML itself named long-horizon coding as a weakness of version one, and for version two has so far measured only tool calling (BFCL v3: 74.92 against 76.74 FP16; again a vendor number).

What this means for our lane

Our 5090 lane streams FP8 weights at up to 1,790 GB/s across the bus during decode. Bandwidth is the limit, which we measured ourselves. Bonsai flips that arithmetic. With 5.9 GB of weight traffic per token, the vendor's table puts Blackwell cards out of bandwidth-bound territory and into instruction throughput and launch overhead. That is why PQ2_0 beats the denser PTQ1_0 there, and why the order reverses on Ada. The plausible consequences:

• Laptops and fringe hardware: a 27B agent with 262k context on an ordinary notebook. Offline, with no data leaving the building.
• Side services: a 72-watt L4 as an always-on endpoint for cron jobs and voice agents. The power bill knows the difference between 30 and 450 watts.
• Freed capacity: 21 GB of VRAM per card back for KV cache, batching, or a second model.

Three things stay open before this sees production. The fork is an operational risk: pinned binaries, our own mirror, one vendor's update tempo; our house pattern for exactly this is the SHA256 procurement in four files before installation. The throughput table is batch-1 measurements, our lane serves 64 concurrent sequences, and ternary behavior under batching is uncharted. And the vLLM path of our standard stack does not exist for these types. Bonsai means the llama.cpp fork, take it or leave it.

The one test that is missing

The plan sits on our list: PQ2_0 on a single 5090 in its own LXC. First stock llama.cpp, so we watch the new types get rejected with our own eyes. Then the fork binary under our pinning pattern, then the 34 work tasks from the self-hosting post in an A/B against the running FP8 lane. Measured: accuracy delta, tok/s at batch 1 and at 32, and the lane's total power draw. Only after that do we talk deployment. And if the result comes out against Bonsai, that result is worth a post too.

Further reading

• Model card prism-ml/Ternary-Bonsai-2-27B-gguf: sizes, bit counts, benchmarks, fork requirement
• Bonsai 2 27B whitepaper (PDF): methodology and the measurement table
• Bonsai-demo on GitHub: the model card's own source of truth, 2,978 stars
• PrismML-Eng/llama.cpp: the fork carrying the ternary kernels
• TechCrunch on PrismML: funding, background, family download counts
• Independent counter-test on r/LocalLLaMA: Terminal-Bench 2.0 on Bonsai 1, different context
• From this site: Qwen3.8-27B self-hosted, llama.cpp & GGUF