senn-techOne RTX 5090 Instead of Four Old GPUs: Consolidating Vision Inference
We run our own AI inference — and run into the same questions as anyone who does. For the vision stage of our extraction pipeline, the question was: more cards, or the right card? The answer was instructive.
The Starting Point
The vision part of our pipeline processes scanned PDFs and images: receipts, delivery notes, hand drawings. Four older GPUs carried this load together — and still delivered only meager token rates, which formed the bottleneck of the entire extraction.
The first reflex was: more hardware. The wrong reflex, as it turned out. Before we ordered a fifth card, we measured. Repeatedly, over several days.
Measure First, Buy Second
The diagnosis took less than two hours, but it could have cost us thousands of euros in a bad investment. We looked at what the cards were actually doing: VRAM utilization, streaming-multiprocessor utilization, token rate per card, PCIe traffic between the devices, and the effective execution time per request. The picture was unambiguous and quite unexpected.
The hardware was not saturated. The cards ran idle most of the time. At 16 tokens/s neither the memory was full nor the compute at its limit. The problem sat one layer deeper, in the software and the driver.
The Real Root Cause
- Not raw power was the problem, but the software path: the driver stack forced an inefficient execution mode (enforce-eager) that throttled the cards to around 16 tokens/s. CUDA graphs — the usual acceleration — were disabled.
- On the RTX 5090, CUDA graphs kick in again because its driver path is current. This multiplies throughput without needing more cards.
- A single card also avoids the tensor-parallel and PCIe overhead that four cards create among themselves. Every cross-GPU hop costs latency that a single card simply does not have.
- FP8 instead of aggressive 4-bit quantization keeps the digits stable — decisive when numbers must be extracted correctly from documents. A wrongly recognized amount is worse than a slower throughput.
What This Means for the SME
Running four cards means four power connectors, four cooling loops, four sources of failure, four drivers that need maintaining. In on-prem operation, each card costs not only its purchase price but continuous effort: rack space, the power bill, heat, spare-part holding.
A single, current card consolidates that down to a fraction of the complexity. The RTX 5090 sits in the same inference host as the rest of our pipeline — one device, one driver stack, one upgrade path. When it is time to move to a newer model later, you swap one card instead of four.
Our Take
The lesson is old but keeps repeating: measure the cause before buying hardware. A current card beats four older ones here — and along the way, complexity, power draw, and maintenance all drop. In self-operation, consolidating is often better scaling than adding. Those who scale on-prem get further with the right card than with four old ones. The two hours of diagnosis paid for themselves overnight.
Further Reading
Is a new expensive card really necessary, or can the old GPUs be sped up?+
Measure first, buy second. For us, raw power wasn't the problem — the driver stack forced an inefficient enforce-eager mode that throttled the cards to around 16 tokens/s, with CUDA graphs disabled. Diagnosis took under two hours. On an RTX 5090, CUDA graphs work again because its driver path is current. Often the right software layer solves the problem instead of new hardware.
Why one card instead of several older ones?+
A single card avoids the tensor-parallel and PCIe overhead that multiple cards create among themselves — every cross-GPU hop costs latency. Then there's operations: four cards mean four power connectors, cooling loops, drivers, and failure sources. FP8 instead of aggressive 4-bit quantization keeps the extracted numbers stable, decisive for receipts. In on-prem operation, consolidating is often better scaling than adding.
What does self-operated inference mean for data protection?+
We run our AI inference ourselves, in our own inference host. Images, PDFs, and receipts never leave our network to call cloud APIs — the entire vision stage of the extraction pipeline runs on the RTX 5090 on-site. For anyone processing sensitive receipts or drawings, this avoids handing data to external providers and keeps processing in a GDPR-compliant self-operated setup.