Qwen3.8-Flash-Next, four days in: why production stays on the 27B

Four days ago Alibaba released the weights for Qwen3.8-Flash-Next, a MoE model with 125 billion main weights plus 51 billion n-gram embeddings, only 6 billion of which are active per token. We judged the model from its spec sheet in our first post about it. Since then we've run it for four days on our 4×RTX 5090 host kipc5090, and the verdict has shifted, though not in the direction one might expect.

A custom build was unavoidable

On release day, no serving framework officially supported the new architecture. vLLM, llama.cpp, and SGLang all only had open pull requests. We built llama.cpp from the PR branch inside a CUDA devel container, since the host itself has no nvcc, and pulled the matching GGUF checkpoint (UD-Q4_K_XL, 104 GiB) from Unsloth as soon as the upload finished.

The first request worked, tool calling succeeded, and the quality battery passed 8 of 9 tests. Then came two crashes: a segfault with speculative decoding enabled, and an assertion failure where the sparse indexer's internal state desynced across parallel slots. The second one only went away with --parallel 1 — one request at a time instead of real concurrency. The very next day, the fix landed in the PR, and slot state has stayed clean since in tests with four and eight concurrent requests.

The RAM upgrade did more than expected

On day four, two extra RAM sticks brought the host from 62 to 94 GiB. The expectation was that only prefill would benefit, since the n-gram table moves from SSD into page cache on cold start. Decode speed rose too, by 18 percent, from about 77 to 96.8 tokens/s. The reason: the table is host-memory-resident by design and gets read on every single token, not just during prefill. A larger page cache pays off during ongoing generation as well.

The real bottleneck is no longer a stability question

With four and eight concurrent requests, the model has run without a single CUDA error. The older upstream report the --parallel 1 rule was originally based on came from entirely different hardware with CPU offload — it didn't reproduce on ours at all.

What remained was a throughput question. Eight concurrent requests totalled 50.7 tokens/s, a single one alone reached 96.8. --parallel also divides the usable context by the number of slots — at eight slots, 131,072 tokens shrank to 16,384 per request. For a lane meant to serve several agents at once, that's a queue, not concurrency. The production 27B model on vLLM saturates at 32 to 64 concurrent requests, delivering around 172 tokens/s.

Why the dense 27B ties the much larger MoE

Flash-Next's GGUF metadata shows the active-expert count: 512 experts total, 10 active per token, which works out to roughly 6 billion active parameters. The dense 27B model activates all 27 billion instead, 4.5 times the compute per token. A rough rule of thumb for a MoE model's "dense equivalent" is the geometric mean of total and active parameters: √(125 × 6) ≈ 27 — exactly the dense model Flash-Next tied with in the quality battery.

We'd seen the same pattern before, in the previous switch from a 122-billion MoE with 10 billion active parameters to the current 27B: another tie in A/B testing, at a similarly predicted dense equivalent. Total parameters seem to store breadth of knowledge; active parameters seem to set how much computation goes into a given answer.

One test neither model passes

Given a heavily repetitive prompt asking for a count of a specific word repeated 440 times, both models produced wrong counts, and with reasoning enabled, both burned their entire token budget on internal reasoning without ever emitting an answer. Not a difference between the models, but an operational trait: a long, repetitive prompt can exhaust the reasoning budget entirely before any content gets generated.

Where things stand

Flash-Next runs stably for us now, but it isn't a production lane. What's still open is a clean comparison against the day-0 vLLM image now available for the same model — only that would separate whether the batching weakness sits in llama.cpp or in the model architecture itself. Until then, our production system keeps serving text, reasoning, and vision through the dense 27B model.