senn-techQwen3.8-Flash-Next on vLLM 0.32: MTP doubles single-user speed, the cache gets smaller
vLLM 0.32.0 came out on 18 October. It supports Qwen3.8-Flash-Next without the fork we have run since September, and it ships multi-token prediction (MTP) for the model. We tested it the same day on the machine with four RTX 5090 that hosts the Flash-Next lane, then carried our NVFP4 cache patch over to the new version.
Series: Qwen3.8-Flash-Next in our own operation, six posts:
- Qwen3.8-Flash-Next: a 125B MoE for the local stack? (31 August), the announcement.
- Qwen3.8-Flash-Next: a four-day test (3 September), first tests with llama.cpp.
- A rented GPU instead of our own production (4 September), offloading the embedding table in vLLM.
- Qwen3.8-Flash-Next in production (9 September), the FP8 KV cache.
- One word in the vLLM fork (14 September), a bug in block accounting.
- This post (18 October), vLLM 0.32, MTP and NVFP4.
The measurements
Every run used the same machine and the same model (a hybrid checkpoint with FP8 embeddings and NVFP4 experts), 220,000 tokens of context, eight slots and four cards with tensor and expert parallelism. The load came from the scripts of our earlier measurements and a replay of 156 real requests from our agent lane, taken from the logs. The reference is the previous production setup, measured warm.
The new engine alone adds 25 percent. MTP predicts three tokens per step and verifies them in one pass.
The mean accepted length was 2.1 to 2.6 tokens per step. With eight concurrent streams the total rose from 605 to between 1,148 and 1,184 tokens per second. Prefill of 35,000 tokens with FP8 stayed at 11,400 to 11,800 tokens per second, against 9,650 on the old production setup.
In the replay of the 156 requests, response times dropped:
| Configuration | Median | 95th percentile | Garbled output |
|---|---|---|---|
| previous production | 7.8 s | 41.4 s | 0 |
| vLLM 0.32, FP8 KV | 5.2 s | 30.5 s | 0 |
| vLLM 0.32, FP8 KV, MTP3 | 3.4 s | 12.7 s | 0 |
| vLLM 0.32, NVFP4 KV, MTP3 | 4.3 s | 22.5 s | 0 |
What the cache costs
The speed is paid for with cache. In total, fewer sessions with a full context fit into memory at the same time.
MTP costs the pool 17.5 percent. With the same 7.6 GB reserve it holds 1,638,571 tokens without MTP and 1,351,242 with it, because the draft layer keeps its own KV cache in the same pool and its weights take 0.46 GiB per card. We did not pin down the rest of the gap to the old production setup (8.23 sessions). Block size is the same at 1,600 tokens. What we know is that the port needs more activation memory during prefill, so we can give the cache less reserve.
The NVFP4 cache had to be ported
vLLM 0.32 aborts startup with NotImplementedError: Qwen4Exp QSA requires a BF16 or FP8-e4m3 main KV cache as soon as you set --kv-cache-dtype nvfp4. The pull request that adds support, vLLM #54846, is still open; its FP8 half landed in #55557. We carried the NVFP4 parts over to 0.32 with a three-way merge: kernel branch, cache geometry, write path, warmup. Kernel tests against a float64 reference passed for every tile profile, and we measured the scale layout against vLLM's own writer (K linear, V interleaved).
The second failure was the dangerous one. For this hybrid model vLLM picks a memory layout in which the K and V heads of each token are interleaved. The NVFP4 scales then overlap the data. The server started, answered, and after a few words produced letter noise without reporting any error. With VLLM_KV_CACHE_LAYOUT=BLHNC the layout is head by head and the answers are correct. After that, a 35,000-token prefill ran out of memory at 8.2 GB of reserve, but not at 7.6 GB, and not at 150,000 tokens either (25 seconds). Anyone using the port needs both settings.
Prefill is still open: with NVFP4 it drops to 6,170 tokens per second, because the port lacks the optimisations of our earlier patch. To see whether vLLM issue #58422 (silent freezing with MTP, NVFP4 and GDN on SM120) affects us, we ran a burst test: 120 requests in twelve rounds of ten at once, zero errors, no stalled token counter.
What we did not measure
Answer quality against the BF16 model has not been re-measured. Our replay catches garbled output and aborts, not small losses. NVIDIA's ready-made FP4 model (Qwen3.8-Flash-Next-FP4-Local) runs only in a community container according to its model card, and that container's repository has no licence file, so we did not test it.
The NVFP4 port with MTP3 is what runs on the machine right now. A week of real load is still to come. If you want to avoid any extra patch, take vLLM 0.32 with FP8 KV and MTP3: fastest prefill, best 95th percentile, but 4.3 instead of 6.1 sessions at full context.
Further reading
- Flash-Next in production: the FP8 KV patch of 6 September
- One word in the vLLM fork: the KV block leak
- vLLM v0.32.0 release notes
- vLLM PR #54846: fp8 and nvfp4 KV cache on the QSA path
- vLLM PR #59533: merged QSA projections
- vLLM issue #58422: silent freeze with MTP, NVFP4 and GDN
- Qwen3.8-Flash-Next on Hugging Face
What does vLLM 0.32 change for Qwen3.8-Flash-Next?+
The model runs without a fork. The release notes list a merged projection in the sparse attention (1.45 to 1.68 times faster according to vLLM) and an NVFP4 KV cache on SM12x. On our machine single-stream speed rose from 104 to 130 tokens per second, and to 209 with multi-token prediction. Boot takes 280 instead of 394 seconds.
Why is the cache smaller with MTP?+
The draft layer used for speculation keeps its own KV cache in the same pool. With an identical memory reserve the pool holds 17.5 percent fewer tokens (1.64 million versus 1.35 million), and the draft weights take 0.46 GiB per card. Speculating fewer tokens does not change that, because the layer is cached per token.
Is the NVFP4 cache in vLLM 0.32 ready for this model?+
No. On Qwen's attention layer vLLM 0.32 aborts at startup with NVFP4, because PR 54846 is still open. We ported it to 0.32. It runs stably in our tests, but it is a patch with one author and not part of the release.