senn-techTwo vLLM builds for Qwen3.8-Flash-Next: faster decode, tighter memory
Since 16 September, vLLM can keep the Qwen3.8-Flash-Next KV cache in FP8 without an outside patch. Until then our production ran a build from the model's release day with the FP8 patch from an open GitHub issue added by hand. On the evening of 24 September we measured both builds on the same machine. The new one decodes much faster. It is not in production yet because it runs short of memory under long parallel requests.
What we compared
The machine has four RTX 5090 cards with 32 GB each, and the model runs with tensor and expert parallelism across all four. Both runs used the same settings: 220,000 tokens of context, eight concurrent requests, FP8 KV and a fixed KV budget of 7,440,208,896 bytes per card.
The old build is the vLLM version from the model's release day (0.1.dev20073) with four local changes: the FP8 patch from RFC 54426, a switch for the FP8 embedding table, a fix for released Mamba blocks and a fix in the tool-call parser. The new build is nightly 0.30.1rc1.dev48 from 24 September. It contains PR 55557 with FP8 KV on the QSA path and the FP8 index cache that shipped in 0.30.0. Of our changes it only still needs the parser fix, which went into the image as a small diff. The other three are upstream now or replaced by a config option.
Version 0.30.0 itself was not an option. Its code still rejects FP8 on the QSA path with "requires a BF16 main KV cache", which would have cut the KV pool in half.
How the switch ran
A script stopped the production container and renamed it without deleting it, then started the new build on the same port. Meanwhile the gateway sent requests to a single RTX 5090 in another server that is kept ready for this case. The script then checked, in order: whether the server reports healthy, how large the KV pool is, whether needles are found in texts of 35,000 to 180,000 tokens, whether streamed tool calls add up to valid JSON, how many of our 21 test questions come back correct, and whether eight parallel requests run without a memory warning. We had taken the old build's numbers an hour earlier with the same scripts under live load.
The numbers
Answer quality is the same: both builds get 18 of 21 test questions right and fail on the same three. Median time to first token dropped from 1.23 to 0.79 seconds. The new build decodes at 138.9 tokens per second against 94.3, about 47 percent faster.
At the same byte budget the KV pool grew from 1,041,901 to 1,064,800 tokens. That comes from the FP8 index cache, which stores the side structures of the sparse attention in less space. At 220,000 tokens of context this means 4.84 instead of 4.74 full sessions at once.
Reading long inputs is slower on the new build. A single 180,000-token request took 15.7 seconds against 15.1. Four concurrent requests of about 128,000 tokens each finished after 55 seconds against 39. Loading the weights also took longer, 151 seconds per card against 107.
Why production stays on the old build
In the parallel test with four long requests, the new build reported twelve times that a 205 MB allocation failed while about 200 MB were free per card. vLLM recovered, the server kept running and then passed the load test with 32 concurrent requests. The old build showed none of these warnings in the same test.
We take the warning seriously because it preceded a crash of the old build at 20:40 the same evening. At that time the KV budget was ten percent larger, eight requests were running, and a 188 MB allocation met about 100 MB of free memory. The server shut down and requests ran on the fallback card for a quarter of an hour. Afterwards we lowered the budget to today's value.
So outside the KV cache the nightly needs more working memory than the patched build. Prefill chunk size is not the cause, both use 2,048 tokens. We have not found the actual cause yet. To keep the same safety margin we would have to lower the KV budget by about one gigabyte per card, and the pool would drop to roughly 900,000 tokens.
Our load could probably live with that. On 24 September KV usage peaked at 70 percent during the day, the bottleneck was the eight slots for concurrent requests, and up to 41 requests were queued. Faster decode frees those slots sooner. Before we lower the budget and put the nightly into production, though, we want to know what it spends the extra memory on.
The script rolled back on its own after the seventh check. After seven minutes the patched build reported healthy again, and its restart policy was back at the original value.
How solid these numbers are
Both runs happened under live load from agents we do not control, and each measurement ran once. The differences in decode speed and memory warnings are large enough that we consider them real. The 0.6 second gap on the single 180,000-token request is within what one parallel agent call can cause.
What we do next
The image, the start script and the switch script are ready on the machine. When vLLM 0.31 ships with PR 55557, or a nightly lowers the memory demand, a new image name and another run with the same seven checks are all it takes. If you run Qwen3.8-Flash-Next on 32 GB cards and want to move to the nightly, send several long requests at once first and search the log for "memory allocation failed", because the health check stays green through it.
Further reading
Why does production still run the old vLLM build when the new one decodes faster?+
With four parallel requests of about 110,000 tokens each, the new build got down to roughly 200 MB of free memory per card and logged twelve allocation warnings. The old build logged none in the same test. The same warning preceded a crash of the old build earlier that evening. A smaller KV pool would make the new build safe, and we want to know where the extra memory goes before we make that trade.
Is FP8 KV for Qwen3.8-Flash-Next in a vLLM release now?+
It is on the main branch since PR 55557 was merged on 16 September 2026. Version 0.30.0 still rejects FP8 on the QSA path. Until the next release, running without a local patch means running a nightly build.
How did you protect production during the switch?+
A script parked the old container instead of deleting it, started the new build, ran seven checks and rolled back on its own when one failed. A second card in another server answered requests during the switch. The rollback took seven minutes.