Skip to content
senn-techsenn-tech
Back to blog
AI & Development
AI in production2026-11-17· By Franz Senn

vLLM 0.34.0 measured: the KV cache default changes, our lane stays as it is

vLLM 0.34.0 was released on November 16, 2026 at 23:22 UTC. The change that matters most for running your own inference lane sits in a single table row: the KV cache is now sized from the memory that is actually free after warmup. Our lane does not use this automatic sizing, because of its own start parameters. We checked that on November 17, 2026 in four places.

The extensible KV cache is a memory allocation mode in vLLM: the engine reserves the address range of the KV cache up front, commits physical memory only after warmup and CUDA graph capture, and sizes the cache from the memory that is then really free instead of from an estimate. Since 0.34.0 it is the default on CUDA and ROCm, and gpu_memory_utilization counts as 1.0 when unset (PR 56492).

In short

  • 0.34.0 brings ten new models, the extensible KV cache as default, Model Runner V1 removed, block size 64 as default, and wheels and images for CUDA 13.0.
  • Measured on our production lane: the automatic sizing does not apply to us, because we set the KV budget by hand. The fixed size also hides the percentage setting.
  • The ReDoS fix in the sanitizer for validation errors only arrives in 0.34.0. Measured on the tag sources, it is missing from 0.32.0 and from 0.33.0.
  • Our base already meets the new dependency floors: huggingface_hub 2.2.0 and tokenizers 0.23.3, measured in the running container.
  • The headline figures in the notes belong to GB200, H200, MI355X and B200. For our four RTX 5090 they show a direction, nothing more.

What 0.34.0 changes

The release notes list eight breaking changes; the eighth removes the models Keye-VL and Keye-VL-1.5. The table shows the seven that touch configurations. The right column holds our own measurement of November 17, 2026, taken on the container and on the source of the tag.

ChangePRDoes it affect our lane?
Extensible KV cache as default, gpu_memory_utilization then 1.0 when unset#56492No, because --kv-cache-memory-bytes 7600000000 is set
Model Runner V1 removed, VLLM_USE_V2_MODEL_RUNNER=0 fails at startup#63373No, the variable is not set in our configuration
Default block size 64, was 16#63231Yes, we do not pass --block-size at all
huggingface_hub >= 2.0.0, tokenizers >= 0.23.3, timm >= 1.0.30#64779Met, measured in the image: 2.2.0 and 0.23.3
Wheels and images for ROCm 7.2 dropped#61939No, our lane runs on CUDA
QuTLASS and fp_quant removed#65230No, no fp_quant checkpoint in use
Marlin W4A8 removed, new route via --quantization humming#65542No, we quantise weights and the KV path differently

The install defaults change as well: PyPI ships CUDA 13.0, and the default image is vllm/vllm-openai:v0.34.0. Checked through the Docker API: the tag exists, 11.3 GB, two architectures, last updated November 16, 2026 at 22:23 UTC.

Extensible KV cache: size from measured free memory

According to PR 56492, the extensible KV cache brings 4 to 10 percent more cache tokens on GB200; on our lane it does not apply. In detail: vLLM reserves the address range of the cache through virtual device memory addresses and creates the physical pages only after warmup. The cache size therefore follows from the memory that is free after CUDA graph capture, where before it came from an estimate made in a profiler pass. On paper that promises more cache and more throughput for most models.

How 0.34.0 determines the cache sizeReserve address rangevirtual, no memory behind itWarmup, graphsonly touched blockscommittedMeasure free memoryafter capture, with a marginMap in pagessame addresses, graphs stayvalid
Simplified from the description of PR 56492. The estimate before capture is dropped when the sizing applies. (Quelle: vLLM PR 56492)

The PR description gives the gain as 4 to 10 percent more KV cache tokens on GB200 (4.9 to 11.3 GiB per card). With an explicitly set 0.92 the cache there comes out 1 to 4 percent smaller than before. The release notes give MiniMax-M3 on H200 as an example and warn about the price: time per output token can rise by 6 to 18 percent on some H200 models at 128 concurrent requests. --no-enable-extensible-kv-cache switches it off.

KV cache of MiniMax-M3 on H200, thousand tokensbefore 0.34.0433 · 433,000 tokens0.34.0 default672 · 672,000 tokens0700
Vendor figure on H200, not re-measured by us. (source: vLLM v0.34.0 release notes)

In the configuration module of v0.34.0 the mechanism appears in two places, and both concern our lane. First, kv_cache_memory_bytes makes the engine ignore gpu_memory_utilization, according to the same docstring, as soon as the size is set. Second, the extensible KV cache is enabled unless stated otherwise, except when kv_cache_memory_bytes is set. We give 7,600,000,000 bytes per card, that is 7.6 GB, or 28.3 GiB over four cards. So on our lane the old estimate stays active and the new sizing stays off.

A warning merged only on November 12, 2026 fits in here (PR 66704): anyone who sets a percentage above 0.7 and below 1.0 and thereby holds back a GiB or more gets it in the log. The release notes and the PR description give 0.8 as the threshold; the source of v0.34.0 and the docs say 0.7. We go with the code. Our 0.92 would hold back 2,555 MiB per card on 32,607 MiB, which is 2.5 GiB per card and 10.2 GiB over four cards (derived from the measured card sizes). Because on our lane the fixed byte value hides the percentage, this calculation leads nowhere. It hits a lane that took its 0.92 from an older README and sets no fixed size.

Anyone who wants the automatic sizing has to drop the manual budget. The v0.34.0 docs name --no-enable-extensible-kv-cache, --kv-cache-memory-bytes and --num-gpu-blocks-override among the cases that fall back to the estimate. A byte value taken from the startup log is only valid for the same GPU with the same free memory at the start; if the start fails after a hardware change, the docs say to remove the flag and let vLLM profile again. For us more hangs on dropping it: our own build adds extra device memory through VLLM_PLE_CPU_OFFLOAD=1 and VLLM_PLE_FORCE_FP8=1, and the docs of 0.34.0 say explicitly to keep the percentage low when something needs memory that vLLM does not measure. That is exactly our case.

Prompt tokens by origin, lane .180.3, as of November 17, 2026From prefix cache (76.4M)76.4Computed anew (13.0M)13084
Own measurement from the metrics of the running engine, counter since the start on November 16 at 11:40 UTC. Share calculated from 89,416,791 prefix cache queries counted in tokens: 85.4 percent cache hits. (source: Own measurement, Senn IT, November 17, 2026)

The two numbers also say how this lane works. In a good 18 hours, 89.4 million prompt tokens went through, just under 5 million an hour, over 3,490 completed requests. That is 25,621 prompt tokens per request (derived), so long contexts. A cache hit rate of 85.4 percent is why this lane can answer interactively at all. That same cache depends on the block size, which jumps from 16 to 64 in 0.34.0. Whether 64 hits better or worse on our mix of long system parts and changing shares cannot be read from the notes. That is the measurement that runs on the standby card before any upgrade.

Measured on the source of the tags

For three questions, reading the notes is not enough. So we compared the same files at three tags, each through the raw delivery from GitHub, on November 17, 2026.

Checkv0.32.0 (our base)v0.33.0v0.34.0
Atomic group (?> in the sanitizer for validation errorsmissingmissingpresent
extensible_kv_cache in the argument module0 hits0 hits (rechecked November 19)3 hits
nvfp4 as KV dtype in the configuration module4 hits4 hits (rechecked November 19)4 hits

Positive control for the method: the same count in the same file of 0.34.0 finds VLLM_KV_CACHE_LAYOUT three times and VLLM_ENABLE_CUDA_COMPATIBILITY four times. A zero for a term that does occur would therefore be a counting error, not a finding. Two things follow. Our KV dtype nvfp4 is also in the configuration module of 0.34.0, so that path stays open. And VLLM_USE_V2_MODEL_RUNNER no longer appears in envs.py of 0.34.0 at all, which matches the row on Model Runner V1.

Our two PLE variables do not appear in envs.py of 0.34.0 either. They have been there since the custom build we described in vLLM 0.32 with MTP and nvfp4. An upgrade for us therefore means: carry the patch to the new version, then measure. The base of our lane is an official nightly image pinned to build commit 5df63a7, with our changes on top. That is why we do not jump from one version to the next with a custom patch, without lead time.

A security fix that only arrives with 0.34.0

Under security, the notes list a ReDoS fix in the sanitizer for validation errors (#52120, merged October 28, 2026). According to the PR description, a 100 KB input with slash segments kept an API worker busy for more than 3 seconds per rejected request; claim by the PR authors, unchecked. The merge date is before the tag of 0.33.0, whose commit dates from October 31 (PyPI on November 2, 2026), so one would expect the fix in 0.33.0. Measured on the source, it is not there, and the GitHub compare API explains why: the merge commit of #52120 and the one of the extensible KV cache (#56492) are not in the history of v0.33.0. The tag is a branch that lies 159 and 471 commits behind the main branch (rechecked November 19, 2026). The atomic group is missing from v0.32.0 and v0.33.0 and present in v0.34.0. The fix shows that the merge date and the first version that ships the change can lie apart.

We looked at our lane directly. In the running container the function sits in vllm/entrypoints/serve/exception_handling/utils.py, and its body contains the four substitutions with no pre-check for a dot and no atomic group, so the unfixed version. The advisory number named in the PR, GHSA-f2g9-pmwr-xwc7, is not in the public advisory list of the repo; there we found 100 entries from November 20, 2025 to November 9, 2026, and no CVE entry can be read from it. The neighbouring advisory GHSA-m9fg-w9qh-67pc on the same sanitizer affects versions from 0.14.0rc1 up to below 0.31.0; our 0.32.0 lies outside that range.

To judge how urgent this is, we measured the endpoint: it binds to 0.0.0.0:8000, the health path answers locally with 200, and from network 181 no connection comes through (measured value 000). For this host there is no public name and no edge route. The path runs for requests that fail validation before they are queued into the engine. The engine does not count rejected requests in its metrics; the counters for completed requests stand at 1,576 with stop and 1,914 with length cut-off, with zero errors and zero preemptions. A company network without an open inference endpoint is therefore in no hurry, but a reasoned update is still worthwhile.

The vendor figures and what they are worth to us

All figures in this section are vendor claims from the release notes of November 16, 2026, on hardware of other operators. The notes report up to 1.95 times decode gain (step 5 on GB200), 1.6 times throughput (GLM-5.3-Flash) and 1.24 to 1.40 times faster first response on long prompts (GLM-5.3). Add the RoPE cache shared across layers, with 11.5 GiB saved per card and 14.5 percent more KV cache on eight B200. All of it is vendor data on hardware we do not own.

The comparison axis is our own baseline: speed and cache size of the current version are recorded in the series on vLLM 0.32 with MTP and an NVFP4 KV cache on four RTX 5090. A jump in numbers on foreign hardware is therefore no argument for an upgrade; it is an argument to do the measurement yourself. The test that counts: standby card, frozen arguments, real traffic from the gateway, the same context mix, then hit rate, time to first response and tokens per second against the current version.

Official vLLM office hours from August 2025 (channel Red Hat) on memory management in the KV cache. It predates 0.34.0 and covers the hybrid allocator, not the extensible cache.

What the versioning shows

Besides the technology, the way this release was shipped stands out. On PyPI, 0.32.0 is dated October 18, 2026, 0.33.0 November 2, 2026 and 0.34.0 November 16, 2026, with no 0.33.1 in between; the newest release is 0.34.0. The Docker tag v0.33.0 exists, measured through the Docker API with answer 200. The GitHub release page for v0.33.0 does not exist, the interface answers 404 there, while for v0.34.0 it returns 47,575 bytes of notes.

Anyone who uses release notes as the trigger for an update has no text at all for 0.33.0, although the image and the wheels were available. For an operation with several lanes it follows: query version and build commit from the runtime, not from what a news site currently takes to be current.

How many prompt tokens a prefix cache saves in self-hosting we measured in vLLM prefix caching in self-hosting, and the hub AI in production holds the further measurements on our own hardware. Anyone who wants to build an inference lane on their own cards will find our offer at Corporate LLM: running your own models in-house.

Further reading

Questions?
Do I have to move to vLLM 0.34.0 because of the ReDoS fix?+

The path sits in the sanitizer for validation errors and, on our side, is reachable only from the network of the AI hosts; nothing public is attached to it. For a company network without an open inference endpoint that is no reason for an emergency change, but it is a reason to justify the next planned update with it. Measured: the atomic group of the fix is missing from the source of v0.32.0 and v0.33.0 and present in v0.34.0.

What happens to gpu_memory_utilization when I set a fixed KV cache size?+

The fixed size wins and hides the percentage. The configuration module of v0.34.0 says that kv_cache_memory_bytes makes the engine ignore gpu_memory_utilization, and that the extensible KV cache is not enabled when that size is set. Anyone who sets the KV budget by hand keeps the old behaviour and does not get the new sizing.

How do I switch off the extensible KV cache in vLLM 0.34.0?+

With `--no-enable-extensible-kv-cache` and an explicit `--gpu-memory-utilization`, the old estimate returns. According to the v0.34.0 docs, vLLM also falls back to the estimate with `--kv-cache-memory-bytes` and with `--num-gpu-blocks-override`. Then `gpu_memory_utilization` is the share of device memory, and under the estimate its default is 0.92.

Is the jump from 0.32 to 0.34 worth it for a 5090 lane?+

The figures in the release notes come from GB200, H200, MI355X and B200, which is different hardware from a lane with four RTX 5090. Without your own measurement on your own hardware they are no basis for a decision. We would measure on the standby card with frozen arguments and real traffic before the production lane gets a new image.