senn-techOpen models running on proprietary hardware versus closed cloud APIs
The question comes up in almost every initial meeting: in-house hardware or API? And almost always, it’s framed as a matter of faith, even though it’s a computational problem.
We use both—in-house GPUs for everything that needs to stay internal, and vendor APIs for tasks where top-tier quality is essential. What you’ll find here is the decision-making framework we use ourselves.
The Quality Gap, Honestly Quantified
Two things can be gleaned from this, and both are important:
The gap to the top has narrowed. The best open-source models are just a few index points behind what the major providers deliver. That wasn’t the case two years ago.
What fits on a single GPU is still far behind. The powerful open-source models are large. A model with 27 to 30 billion parameters runs in quantized form on a 32 GB card—the 200-billion-parameter class requires multiple cards and a lot of main memory. This is where most calculations fall apart: People compare the open-source top performers with the proprietary top performers and then buy hardware on which the open-source top performers won’t even run.
Where the gap doesn’t matter
When querying your own documents. When a model is supposed to provide answers based on a quote, a contract, or a maintenance log, the quality of the retrieval is what matters, not the model size. A medium-sized open model with a well-designed vector search significantly outperforms a top-tier model without access to your data.
That’s the real reason why on-premises inference is worthwhile for businesses: The typical use case for small and medium-sized businesses isn’t “write me a novel,” but rather “what is the payment deadline in this contract?”
The Cost Threshold
An RTX 5090 with 32 GB costs around 2,500 euros. Add to that a case, power, and operating costs. A 30-billion-parameter model runs on this card at a usable speed once quantized.
Now for the math: What does the same volume cost via an API? With budget models costing cents per million tokens, it takes a long time to recoup the cost. With top-of-the-line models costing double-digit dollar amounts per million output tokens, the card pays for itself in a matter of months.
The decisive factor is utilization, not the purchase price. A GPU running at five percent capacity is more expensive than any API. A GPU running continuously is unbeatably cost-effective.
Hence the practical advice: measure first, then buy. Run operations via an API for two to three months, log the actual token volume, and base your decision on that.
Arguments in favor of your own hardware, beyond the cost
The data stays on your network. No data processing agreement, no transfers to third countries, no discussions with the works council about employee data in a third-party cloud.
No dependence on model changes. Providers phase out models, change behavior with an update, or withdraw a model on short notice. A model stored on your hard drive doesn’t do that. Anyone who has built an application around a specific model’s behavior knows how valuable that is.
Predictable costs. A card costs what it costs. An API budget can skyrocket overnight due to a faulty agent run.
The Case for the API
No capital tied up. For many businesses, this is the most important point—and the reason why “on-premise first” is the wrong blanket recommendation.
Top-tier quality. For difficult reasoning tasks, long autonomous agent runs, and image understanding on complex diagrams, the major providers are still ahead.
No operational overhead. Driver selection, quantization, context lengths, and model switching during runtime—that’s work. Those who don’t want to or can’t do it are right to pay someone else to handle it.
The right mix is usually the answer
Here’s how we do it: Anything involving customer data, contracts, or HR data runs in-house. Anything requiring general knowledge and maximum quality goes through an API—deliberately and without sensitive content.
A gateway in front of this setup makes this separation manageable: The application communicates with a single address, and the switch determines which model responds based on the task. If a model changes, a single configuration line is updated—not the application code.
If your own hardware doesn’t fit the budget, there’s a third option that’s often overlooked: rented GPU capacity in the EU. The same open model stack, the same control over the software, a standard data processing agreement within the EU—just without the upfront purchase.
Further Reading
How significant is the difference in quality between open and closed models?+
In the Intelligence Index, the top open-ended models in 2026 lag about ten to fifteen points behind the top closed-ended models. For queries regarding one’s own documents, this gap is practically insignificant, because the quality of retrieval is the deciding factor there. For open-ended creative work and difficult reasoning tasks, however, the gap is noticeable.
At what point does having your own GPU pay off?+
An RTX 5090 with 32 GB costs around 2,500 euros and can handle a model with 30 billion parameters in a quantized format. When factoring in API costs, it often pays for itself in less than a year with continuous use—but never with sporadic use. The key factor is not the price, but the utilization rate.
Is on-premises inference automatically GDPR-compliant?+
It eliminates two problems: no processing by a third party and no transfers to third countries. It does not eliminate the obligations you already have—legal basis, purpose limitation, data erasure policy, and transparency toward data subjects. Using your own hardware makes things easier, but it doesn’t automatically take care of everything.
Can I start small?+
Yes, and that's the sensible approach. Start with an API and measure your actual volume over a few weeks. That number will determine your hardware choice—or the decision to stick with the API. Otherwise, you'll end up buying expensive equipment that isn't being fully utilized.