One word in the vLLM fork: why our Flash-Next lane paid for every token three times over

On Saturday evening, 13 September, the lane was gone. The vLLM container on our machine with four RTX 5090, in production with an FP8 KV cache since 6 September, restarted every second and aborted every second. No human had shut it down, our operations agent had, and it had been cleared to do so. It was supposed to find out how many more concurrent sessions the card can carry, and it had come in with a measurement that turned out to be correct in hindsight. Its explanation was wrong. Finding the right one took a day, and it ends at a single word.

How the lane went down

The agent had written a start script for a debug container and passed serve in it as the first argument. The container image already carries vllm serve as its entrypoint. The result was vllm serve serve …, a command the parser answers with an error, and a container with restart automation that repeated exactly that every second. The agent's own session died shortly afterwards on a gateway error, because its own model lane has no fallback path. Nobody rolled anything back.

Recovery was routine: container gone, production image up with the known start script. Two things about it were not. The start script still carried the values from 7 September, 192,000 tokens of context and eight slots, but production had been running since the 12th with 220,000 tokens and four slots. The correct state came from the agent itself of all places, which had saved the container with docker inspect before killing it. And the lanes for chat and reasoning stayed reachable during the outages, because the gateway diverts them to the standby box with a single RTX 5090. We checked that with real requests before the second restart: three seconds of response time over the detour.

Each restart costs about 15 minutes, a good twelve of them for loading the weights through the n-gram table offload into main memory.

What the agent had measured correctly

The lane's KV pool has 1,356 blocks of 800 tokens and 5.53 MB each per card. From that, the vLLM start log computes 1,050,422 tokens and 4.77 concurrent full sessions at 220,000 tokens. The agent had used the kv_cache_usage_perc metric to count how many blocks a request actually holds: not one per 800 tokens, but one per 308. At 16,000 tokens that was right, above it the agent had not measured. We repeated it with a request of 84,495 tokens: 275 blocks, where the geometry calls for 111. So the number in the start log was too optimistic by a factor of 2.5, and a real full session of 220,000 tokens would have fit into the pool only a bit more than twice.

As the cause the agent named the rounding of the Mamba state blocks up to the page size of the attention blocks. The start log itself contradicts that: Padding mamba page size by 0.25%. A quarter of a percent does not explain a factor of 2.6, and the padded blocks are constant per request, they do not grow with context length.

Three more measurements on the running lane, without a restart, closed the circle. The same request a second time, with 84,000 tokens out of the prefix cache: 237 blocks, so almost as many, although there was hardly anything left to compute. A short request with 4,000 generated tokens: one block per 800 tokens, exactly attention and nothing else. And a cold request, sampled twenty times a second: every 190 milliseconds, with every prefill step of 2,048 tokens, 6.5 blocks were added. Attention needs 2.56 of them. The other four are one state block each for the model's four Mamba groups, three for the Gated DeltaNet layers and one for the PLE convolution, and none of them was freed before the end of the request.

The bug fits on one line

vLLM stores the states of the linear layers in “align” mode in the same block table as attention, indexed by position. Each prefill step appends a new state block, and the positions in between are filled with null blocks. The intent is that only the current and the previous state stay held; the capacity arithmetic in the start log assumes exactly that.

This is what the start of one Mamba group's table looks like after three prefill steps, when the fourth step triggers a release up to position 5:

The state from step 2 sits at position 5, outside the range, and survives in both cases. The one from step 1 is only reached with continue.

Freeing goes through a helper function that walks the range of the table to be released backwards. It is written for sliding-window attention, where null blocks sit only at the start of the table, and so it breaks at the first null block:

for i in range(last_block - 1, first_block - 1, -1):
    if blocks[i] == self._null_block:
        break          # senn 2026-09-14: continue
    freed.append(blocks[i])
    blocks[i] = self._null_block

With a Mamba table shot through with null blocks, the backward walk hits a null block almost immediately during prefill and frees nothing. In decode the previous state block sits directly below the release boundary, gets freed, and only after that comes the break. That is exactly the pattern from the measurements. The fix consists of skipping null blocks instead of breaking. Nothing changes for sliding-window attention, since that range contains only blocks to be freed anyway.

While checking the sources for this post it came out that we were not the first. In the vLLM main branch the same diagnosis has been fixed since 11 September, pull request #55450, “Retire Mamba states across null gaps”. The version there skips the gaps as well, and additionally remembers how far the table has already been cleared, so it does not read from the start every time. Our build is from 26 August and does not know the patch; to this day it is not in any vLLM release either. Three days of lead would have saved us a day of searching, and conversely, without the search nobody would have known that this particular patch is worth a factor of 2.3 to us.

One detour beforehand is worth the mention. The fork knows an environment variable VLLM_PREFIX_CACHE_RETENTION_INTERVAL that sets how densely Mamba states are kept for later prefix hits. We suspected it first and set it to 0 with a restart: 275 blocks, unchanged. It only controls which states get a hash, not which blocks a running request holds on to.

The fix and the proof

An image with the changed file, a restart of 15 minutes, the same measurements:

The check series from 6 September then ran through unchanged, at temperature 0 and without reasoning:

On top of that a needle at 175,383 tokens that held 233 blocks, and zero preemptions in the scheduler over the whole series.

What this means for operations

A request now costs one block per roughly 710 tokens. At contexts around 100,000 tokens, the everyday case for our coding agent, about nine sessions fit into the pool at once, four before. At 220,000 tokens it is 4.7, and that is the number the start log claimed from the beginning. The arithmetic there was never wrong, it only assumed a behaviour the code did not have. The slot cap has been back at eight since.

What has not changed is what the model itself stays bound to: 262,144 tokens per request, the attention layers cost per token what they cost, and the weights are the same. What the agent had originally been looking for, a trick along the lines of DeepSeek-V4.1-Flash with its 890 bytes per token, is a property of training and cannot be bolted on afterwards. The factor of two in capacity sat instead in a release function written for a different attention pattern.

Where things stand

The lane has been running since 14 September, 11:05, with the corrected image, 220,000 tokens of context and eight slots. The way back is the same start command with the image from 6 September, 15 minutes, and then the leak is back and nothing else is different. Two small things remain open in the configuration, a start script with stale default values and a context limit in the chat frontend that sits above the one in the backend. As soon as a vLLM release brings the patch from #55450 and the FP8 KV path from #54426 together, we move to an official image and throw both home builds away.

Further reading

• Flash-Next in production: the FP8 KV patch of 6 September
• Prefix caching in vLLM, explained from our own operations
• vLLM PR #55450: Retire Mamba states across null gaps
• vLLM Engine Arguments: --mamba-cache-mode, --mamba-block-size
• Marconi: Prefix Caching for the Era of Hybrid LLMs, arXiv 2411.19379
• Qwen3.8-Flash-Next on Hugging Face
• RFC #54426: FP8 KV on the QSA path