Qwen3.8-Flash-Next in production: how a community patch doubled the KV cache on 4× RTX 5090

A week ago the verdict in this spot was: Qwen3.8-Flash-Next runs stably, but not as a production lane, because llama.cpp pushes eight concurrent requests down to 50.7 tokens/s combined. That verdict was correct and it is out of date anyway. Since 3 September the model has served our lanes for text, reasoning, vision and the coding agent, and since 6 September it does so with an FP8 KV cache that has nearly doubled the number of concurrent full sessions. How we got there is the actual story, because it went differently than planned in three places.

First detour: the checkpoint that did not exist

vLLM has two usable editions of the model: the FP8 variant from Qwen with 180 billion parameters in total, and an NVFP4 quantisation by Inferact with 118 billion. Neither fits on its own onto four RTX 5090 with 128 GB of VRAM and 124 GB of main memory. The FP8 edition is too large; the NVFP4 edition carries the 51-billion n-gram table in a format the loader refuses on our hardware. A related loading problem with a different NVFP4 edition is open as Issue #54765.

So we mixed the two: the FP8 n-gram table from the Qwen checkpoint, the NVFP4 experts from the Inferact checkpoint, merged by script, plus a small patch in the embedding layer so that vLLM accepts the FP8 table alongside NVFP4 weights. Result: 82.8 GiB of weights across four cards, tensor parallelism 4, expert parallelism on.

Second detour: a measurement with empty weights

The first vLLM run on 1 September came up only with the slow Marlin backend. Every fast NVFP4 kernel aborted with a padding error, and we wrote down: no NVFP4 backend handles expert parallelism. That was wrong. The test had run with --load-format dummy, so without real weights, and that alone kept the detection of the FP8 table format from ever firing. With real weights, flashinfer_cutlass loaded cleanly with expert parallelism and delivered 205 tokens/s across 32 concurrent streams at 32,768 tokens of context. The dense 27B that had carried production until then saturates at around 172.

The lesson is unspectacular and was expensive regardless: a backend test without weights does not test the backend.

Third detour: the KV cache that only does BF16

The sparse attention in this model, which Qwen calls QSA, requires a BF16 KV cache as shipped. With --kv-cache-dtype fp8_e4m3, a matter of course on the 27B, all four workers die at startup. BF16 costs twice as much memory per token as FP8, and that gap grows linearly with context length. At the full 262,144 tokens, 2.12 concurrent full sessions were left. At 192,000 tokens and 94 percent memory utilisation it was 3.15, and that is the configuration the model went live with on 3 September.

Running it produced the next finding immediately: up to 87 requests in the queue at only 42 percent KV utilisation. The cause was the parameter --max-num-seqs 3, a hard cap on all concurrent requests, not only the long ones. On top of that, an internal service was sending 1,680 sentiment-scoring requests per weekday. That service has since been redeployed without an AI backend, and the cap now stands at 8.

The patch from the issue tracker

On 30 August a user opened an RFC in the vLLM repository: FP8 KV cache on the QSA path, working patch, measured on exactly one machine, a DGX Spark, with a request for confirmation on other hardware. The diff is small, two files, 17 hunks, and it also sidesteps the lock that FlashAttention otherwise puts on FP8 KV for our card generation. We laid it onto a copy of our image and started the container with the new data type. Nothing else changed.

Before the production lane was switched over, both BF16 and FP8 ran against the same prompts at temperature 0:

That gave the RFC a second piece of hardware, and one that does not appear in the official vLLM recipe at all. The recipe lists 4× GB300, 8× H200, 4× H100 and 4× MI355X, not a single consumer build. Two pull requests for the same feature are now open: #55557 for FP8 only, opened on 6 September, and #54846 for FP8 and NVFP4, which reports 1.9× and 3.1× KV pool on its own hardware. We deliberately passed on NVFP4: the perplexity penalty is real, and prefill doubles.

What the model does today, and what it does not

It serves 192,000 tokens per request, eight requests at a time, roughly 5.9 full sessions in the KV cache, or about 45 typical chats of 25,000 tokens. Vision works with real images, tool calling works. The 262,144 tokens from the spec sheet would be back within reach at roughly 4.2 full sessions with FP8 KV, on paper. We have not measured that.

Three things remain open. First, temperature 0 is not deterministic, because the reduction order varies across tensor and expert parallelism; the fix is filed as PR #55122. In practice that means the same request can answer once and burn its entire token budget on thinking the next time. Second, our 21-part correctness battery has never run on the hybrid checkpoint, neither with BF16 nor with FP8; the quality gate above is a smoke test. Third, community code is running on a development build. As soon as one of the two pull requests is merged, we move to an official image.

For comparison, what others report: on a single RTX 5090 with 128 GB of RAM the model reaches around 48 tokens/s under llama.cpp with a 2-bit quantisation at 64,000 tokens of context. The Intelligence Index from Artificial Analysis lists the model at 46 points, explicitly as an estimate. Both match what we saw a week ago: a 125B MoE with 6 billion active parameters plays in the league of a dense 27B. What changed is not the quality, it is throughput per card.

Where things stand

Flash-Next carries production. The way back has three stages and takes around six and a half minutes per stage: unpatched image with BF16 KV, then the 3 September configuration with three slots, then the dense 27B, which sits stopped on the same machine. If the queue reappears below 70 percent KV utilisation, the next step is not more memory, it is a higher slot cap.

Further reading

• Qwen3.8-Flash-Next on Hugging Face
• On the Design of Qwen3.8-Next Architecture, arXiv 2608.30320
• vLLM: Quantized KV Cache
• vLLM recipe for Qwen3.8-Flash-Next
• RFC #54426: FP8 KV on the QSA path
• llama.cpp PR #27742: Flash-Next support
• Our four-day test under llama.cpp
• The cloud test with vLLM and PLE offload