vLLM Prefix Caching in Self-Hosting: Measurably Fewer Tokens

Prefix caching is one of those optimizations that sound impressive in benchmarks but only pay off in production if you measure them. We enabled --enable-prefix-caching and --enable-prompt-tokens-details on our local Qwen3.8-27B on 19 August 2026 — and we now see concrete numbers.

How prefix caching works in vLLM

vLLM organizes the KV cache in PagedAttention blocks. Each block represents a slice of already-computed attention. When a new request arrives, vLLM compares its prefix with existing blocks. If they match, the blocks are referenced instead of recomputed.

This is especially valuable for workloads with long, recurring context: system instructions, RAG documents, tool schemas, or repeated structural prompts. The server computes the shared part once and reuses it for all subsequent requests.

Measurability was the missing piece

Without this flag, the API only returns:

"usage": {
  "prompt_tokens": 12543,
  "completion_tokens": 892,
  "total_tokens": 13435
}

With --enable-prompt-tokens-details:

"usage": {
  "prompt_tokens": 12543,
  "prompt_tokens_details": {
    "cached_tokens": 11204,
    "new_tokens": 1339
  },
  "completion_tokens": 892,
  "total_tokens": 13435
}
}

The cached-to-new ratio immediately tells you whether a prompt template fits the cache. A value like 11,204 out of 12,543 cached tokens means only about 11% of the prompt had to be recomputed.

Local implementation

Activation is just two extra flags in the vLLM command:

- "--enable-prefix-caching"
- "--enable-prompt-tokens-details"

With Qwen3.8-27B, this runs stably. The --reasoning-parser qwen3 and --tool-call-parser qwen3_xml do not affect the cache mechanism; what matters is that the token representation of the prefix stays identical between requests.

What we observe

After four days of continuous operation on the 4×-RTX-5090 host, the container's Prometheus endpoint shows:

vllm:prompt_tokens_total: roughly 78 million prefill tokens
vllm:generation_tokens_total: roughly 5.9 million generated tokens
Prefix cache hit rate: about 43–45 %

The API details add the per-request cache share. Where long documents or tool schemas are reused, the cached share typically falls between 70 and 90 percent. For short ad-hoc requests, it is near zero — exactly as expected.

The effect is even more dramatic on the separate hermes host (192.168.180.202), where a single RTX 5090 agent runs with prefix caching enabled. A 150,000-token replay prompt — with identical system instructions and tool schema — dropped from 53.2 s to 2.3 s, with a measured cache hit rate of 93.5 %.

Limitations

Prefix caching is not a universal fix. It does not help when:

every request has a completely different context,
the cache is constantly invalidated by many parallel, distinct prompts,
or prompt ordering varies (nested prefixes are not always recognized).

Memory usage also increases slightly because vLLM retains cache blocks longer. On our host with 128 GB VRAM, this is not an issue; on smaller cards it may marginally reduce the maximum available KV cache size.

Our take

Prefix caching is a free improvement for workloads with recurring context. The key insight: the cache share must be measurable, otherwise you optimize blind. --enable-prompt-tokens-details is the switch that makes the effect visible. If you self-host vLLM and reuse long prompts, enable both flags.