senn-techHigh-Throughput LLM Inference with vLLM on Custom Hardware
Local LLMs are tools in daily operation now, no longer an experiment. But raw inference with Transformers is slow. vLLM addresses exactly that.
PagedAttention: The Key
Classic inference stores the key-value cache of each token linearly, leading to memory fragmentation and inefficiency during batching. vLLM uses PagedAttention: the KV cache is managed in blocks (pages), like virtual memory. This enables sharing, reuse, and most importantly continuous batching.
Concretely this means: new requests do not have to wait for the current batch to close. They slot into the next free block as soon as space is available. Throughput follows the GPU, not the rigidity of the batch grid.
Result: 10 concurrent requests don't run 10× slower, but perhaps only 2×, because memory and compute time are shared dynamically.
Running on an RTX 5090
The 5090 with 32 GB VRAM is our standard inference card. With vLLM, it effortlessly runs a 70B model with Q4 quantization or a 13B model with full weights. The OpenAI-compatible API works with any tool that expects openai as a backend, no adjustment needed.
python -m vllm.entrypoints.openai.api_server \
--model mistralai/Mixtral-8x22B-Instruct-v0.1 \
--tensor-parallel-size 1 --max-model-len 8192
Then every tool that is "OpenAI-compatible" simply points to http://<server>:8000/v1.
What Really Matters in Operation
For highly available deployment three things are decisive: set the VRAM budget carefully (--max-model-len and --gpu-memory-utilization tuned to the concrete host, or you get OOM under load), monitor token rates (vLLM's Prometheus endpoint delivers latency and throughput per model), and choose quantization deliberately. For text, Q4 is often enough; for structuring and digit extraction we prefer FP8, because it raises the hit rate of results.
When vLLM, When llama.cpp?
vLLM excels at high throughput, many parallel requests, and server operation. For single requests, CPU offload, or exotic quantizations, llama.cpp is better. Both have their place, see the next article.
Why Self-Host Instead of a Cloud API?
For the SME, self-operation pays off quickly. A single RTX 5090 in your own rack covers the load of dozens of users, with no per-token billing and without a single document leaving the house. The cloud API pays off at small volume or for models that will not run locally; as soon as data sovereignty, consistent latency, and predictable costs count, on-prem is the clearer solution.
Then there is predictability: a local card scales linearly up to its limit, while cloud prices fluctuate depending on tariff, region, and model. Those running an extraction pipeline around the clock know at the end of the month exactly what the card cost, power included.
Conclusion
If you want to serve LLMs for more than a single user, vLLM is the first choice. The architecture with PagedAttention saves VRAM and measurably increases throughput. For the SME this means: a single in-house card replaces several cloud API subscriptions, and the data stays in-house.
Further Reading
Why run our own card instead of a cloud API?+
Self-hosting is a utilization question, not a matter of principle: the card only beats the API once enough load lands on it. It buys no headroom beyond that card, and models too large for 32 GB of VRAM remain with the API.
When is llama.cpp the better choice than vLLM?+
The split runs through operations, not answer quality. vLLM earns its place once several users share one OpenAI-compatible endpoint; llama.cpp stays the tool for single runs and for quantizations the server stack does not carry.
What really matters in around-the-clock operation?+
Three things: set the VRAM budget carefully (tune --max-model-len and --gpu-memory-utilization to the host, or you get OOM under load), monitor token rates via vLLM's Prometheus endpoint, and choose quantization deliberately. For text, Q4 is often enough; for structuring and digit extraction we prefer FP8, because it noticeably raises the hit rate.