KV cache in 4 bits: INT4 failed in production, NVFP4 holds 8.2 full sessions
Addendum, 3 October: INT4 is out again, NVFP4 is running
On the morning of 3 October several agent runs returned garbage: hex strings, a leaked control token of the model and Chinese fragments, up to the limit of 16,384 tokens. At 08:20 we switched back to FP8. The checks described below had passed, and INT4 still did not hold up in production.
The session logs show 3 failed model steps out of 248 under INT4. Under FP8 it was 0 out of 24,565 since 20 September. The failure starts at the first generated token and hits different requests on every repeat. Our first guess was that the quantization error grows over long outputs. This pattern fits that guess poorly. The cause in the INT4 read path is open.
Our checks from 2 October had little chance of finding it. They ran at temperature zero, without reasoning, with at most 700 output tokens and about 20 requests. At a failure rate of a little over one percent, 20 requests look clean in about four out of five cases. The new check replays 169 real agent requests from the morning of 3 October with their original overlap, 156 of them with real output. INT4 failed it with 3 clear and 2 suspicious responses. FP8 with the same block size of 1600 stayed clean, so the block size is not the cause.
Since 13:25 the lane runs NVFP4, a different 4-bit format with block scales. The read path for it comes from the open vLLM pull request #54846. The pool holds 1,810,884 tokens, which is 8.2 full sessions of 220,000 tokens. FP8 reached 1,146,478 tokens and 5.2 sessions on the same day. Three passes of the new check, 468 responses in total, came back without garbage.
NVFP4 costs time. In the replay the median response time was 12 to 14 seconds against 8.9 with FP8, and production load was running during the NVFP4 passes as well. The author of the patch also measures perplexity 0.4 to 0.6 percent worse, which our check does not capture. Going back to FP8 remains one command. By the evening of 3 October NVFP4 had run for seven hours without a restart and answered 8,661 requests without a server error.
The post from 2 October
From here on the text stands as it was published on 2 October. Only one chart label and the last section have changed.
Since the evening of 2 October our production lane for Qwen3.8-Flash-Next runs with a 4-bit KV cache. The pool grows from 1,041,901 to 1,944,081 tokens. At 220,000 tokens of context per request that means 8.8 instead of 4.7 full sessions fit into GPU memory at the same time. Answer quality stayed unchanged in every check, single-stream speed dropped by eight percent. Same model, same four RTX 5090s, same memory budget per card.
What we changed
The KV cache is the part of GPU memory where the model keeps the history of a request. Until now we stored it in FP8, one byte per value. The int4_per_token_head switch stores every value in 4 bits and keeps the scaling factors per token and attention head. That works out to 264 bytes per head and token at a head dimension of 256, roughly half of the previous footprint.
Two details had to be solved for this. Our vLLM build already knows the write path for INT4, but the model's sparse-attention layers kept reading the cache as FP8. We brought the read side to the same level and verified it against the reference implementation. On top of that, the server refused to start with the usual block size of 800 tokens. The Mamba state takes 408,576 bytes per session and shares pages with the KV cache. An 800-token page holds only 211,200 bytes at INT4. With a block size of 1600 a page fits 422,400 bytes, and the server starts.
The same Mamba state is also why the pool grows by a factor of 1.7 to 1.87 instead of exactly two. Its footprint stays constant while the KV part shrinks.
The measured values
Decoding is slower with INT4 in a single stream: 90.7 against 98.6 tokens per second, measured on the same prompts at temperature zero. Under parallel load this hardly matters, because requests wait less often for free cache space. The first start with the new image took almost ten minutes, because new kernel paths are compiled. Later starts take about five.
The number of concurrent requests stays at eight, that cap is unchanged. What changes: all eight slots may now hold the full context at the same time. Before, eight slots had to share 4.7 full sessions, and very long documents had to wait or be shortened.
How we checked quality
Four checks, all run on 2 October. A kernel harness compares the INT4 output directly against the torch reference: minimum cosine similarity 0.99982. Then needle tests, where a hidden sentence has to be recovered from long texts. The needle was found at 180,000 tokens in the middle and at 215,000 tokens at depths 0.3 and 0.9. Deeper is not possible on this configuration, because the engine caps at 220,000 tokens.
Four coding tasks at temperature zero returned the same algorithms and structures on both cache formats, with small differences in variable names and comments. We extracted and ran two of the solutions, an LRU table and a Dijkstra, both functionally correct. The last test was a mixed run with four parallel requests of about 128,000 tokens each, which completed cleanly.
The kernel incident on the same day
The day had a second construction site. A routine run of apt full-upgrade installed kernel 7.0.0-38. In the process DKMS rebuilt the NVIDIA module, from the unmodified source tree. The patch that enables direct memory transfer between our four cards was missing afterwards. Without it, traffic between the cards goes through host memory, and that direct path is exactly what makes tensor parallelism fast.
We built the patched modules from our clone of open-gpu-kernel-modules against the new kernel headers, backed up the original modules first, rebooted and checked with nvidia-smi topo -p2p r: all card pairs OK again. After that we pinned the kernel and NVIDIA packages with apt-mark hold. A kernel change now happens only deliberately, with the rebuild in the same maintenance window.
What this means for operations
The bottleneck of the last weeks was memory for long parallel requests, not compute. Since 3 October NVFP4 holds a little over eight full sessions in the cache, so all eight slots may hold the full context at the same time. The INT4 image stays on the machine and is not used while the cause of the failed responses is open. FP8 remains the way back. Before any further change to the KV cache we replay the real requests first.
Further reading
- vLLM pull request #54846: fp8_e4m3 and nvfp4 KV cache on the QSA path
- vLLM on GitHub
- aikitoria/open-gpu-kernel-modules: the P2P patch for GeForce cards
- AI with Eric: Qwen3.8-Flash-Next sparse attention under test (video, 27 Aug 2026)
- Cloud Codes: Qwen3.8-Flash-Next and the VRAM limit (video, 28 Aug 2026)
- The build comparison with FP8 KV from 24 September
- How FP8 KV reached our production
Why does the pool not exactly double?+
The linear part of the model, the Mamba state, takes 408,576 bytes per session and shares memory pages with the KV cache. Its footprint stays the same while the KV part is halved. Measured across boots we land between 1.7 and 1.87 times the old pool.
Does answer quality suffer with a 4-bit KV cache?+
With INT4, yes. On 3 October, 3 of 248 model steps in production were unusable, although the kernel comparison, needle tests and coding tasks had passed. With NVFP4, 468 replayed real responses came back without a failure. The author of the patch measures perplexity 0.4 to 0.6 percent worse.
What happens if a problem shows up in production anyway?+
That happened on 3 October. Going back to FP8 was one command and took six and a half minutes. The FP8 image stays on the machine, and the same way back now applies to NVFP4.
senn-tech