High-Throughput LLM Inference with vLLM on Custom Hardware

Local LLMs are no longer hype — they are tools. But raw inference with Transformers is slow. vLLM addresses exactly that.

PagedAttention: The Key

Classic inference stores the key-value cache of each token linearly — leading to memory fragmentation and inefficiency during batching. vLLM uses PagedAttention: the KV cache is managed in blocks (pages), like virtual memory. This enables sharing, reuse, and most importantly continuous batching.

Concretely this means: new requests do not have to wait for the current batch to close. They slot into the next free block as soon as space is available. Throughput follows the GPU, not the rigidity of the batch grid.

Result: 10 concurrent requests don't run 10× slower, but perhaps only 2× — because memory and compute time are shared dynamically.

Running on an RTX 5090

The 5090 with 32 GB VRAM is our standard inference card. With vLLM, it effortlessly runs a 70B model with Q4 quantization or a 13B model with full weights. The OpenAI-compatible API integrates seamlessly into any tool that expects openai as a backend.

python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mixtral-8x22B-Instruct-v0.1 \
  --tensor-parallel-size 1 --max-model-len 8192

Then every tool that is "OpenAI-compatible" simply points to http://<server>:8000/v1.

What Really Matters in Operation

For highly available deployment three things are decisive: set the VRAM budget carefully (--max-model-len and --gpu-memory-utilization tuned to the concrete host, or you get OOM under load), monitor token rates (vLLM's Prometheus endpoint delivers latency and throughput per model), and choose quantization deliberately. For text, Q4 is often enough; for structuring and digit extraction we prefer FP8, because it raises the hit rate of results.

When vLLM, When llama.cpp?

vLLM excels at high throughput, many parallel requests, and server operation. For single requests, CPU offload, or exotic quantizations, llama.cpp is better. Both have their place — see the next article.

Why Self-Host Instead of a Cloud API?

For the SME, self-operation pays off quickly. A single RTX 5090 in your own rack covers the load of dozens of users — with no per-token billing and without a single document leaving the house. The cloud API pays off at small volume or for models that will not run locally; as soon as data sovereignty, consistent latency, and predictable costs count, on-prem is the clearer solution.

Then there is predictability: a local card scales linearly up to its limit, while cloud prices fluctuate depending on tariff, region, and model. Those running an extraction pipeline around the clock know at the end of the month exactly what the card cost — power included.

Conclusion

If you want to serve LLMs for more than a single user, vLLM is the first choice. The architecture with PagedAttention saves VRAM and measurably increases throughput. For the SME this means: a single in-house card replaces several cloud API subscriptions, and the data stays in-house.