Company AI portal on its own GPUs

Starting point

ChatGPT has become part of everyday work — but uploading company documents to someone else's cloud? Not an option for a company responsible for customer and employee data. At the same time the pressure grows: whoever doesn't use AI works slower than the competition.

What was needed was the convenience of modern AI assistants — without a single document leaving the premises.

The hard part

Data protection is the easy justification for running your own. The hard requirement is usefulness: an in-house model that answers noticeably slower or weaker than what people know from private use will not be used but bypassed — and then the documents sit in someone else's cloud after all, only this time nobody knows about it.

No single model is good at everything. A large reasoning model is strong at analysis and inappropriately slow for a translation; a vision model reads PDFs and makes a poor conversational partner. What is needed, then, is not one model but a mediating layer that picks the right one per task and keeps the applications from knowing which one is behind it.

And it needs an answer to who may see what. An assistant that searches company documents is only as good as the permissions it respects while doing so.

Solution

An in-house AI portal was built on the company's own hardware: six GPUs running five self-hosted models — from a large reasoning model through a vision model for images and PDFs to embedding and reranking — behind a gateway with 15 model routes.

The portal does more than chat: knowledge search across indexed company documents with source citations, image and PDF understanding, and 32 connected tools — from ERP lookups to web search — which the models use autonomously.

Depending on the task, the gateway picks the right model. Chats and documents are processed exclusively on the company's own models — only the optional web search naturally queries external search engines.

How it is built

The models run on our own GPU servers, each behind an inference service. Above them sits a gateway offering named routes instead of model names: “reasoning”, “vision”, “fast”. Applications only ever know the route, and that is why a model underneath can be swapped without touching a single application. It has happened several times since the build.

The portal itself is the interface: chat, knowledge search across indexed company documents with source citations, image and PDF understanding. 32 tools are connected which the models call on their own, from ERP lookups to web search.

Sign-in runs through the central company login rather than separate portal accounts. Someone leaving the company loses access in one place. Not in five.

Every request is logged and metered at the gateway. That is not employee surveillance but the precondition for judging cost and utilisation at all, and for noticing when an application is producing requests in idle for which nobody ever reads an answer.

Day-to-day operation

The GPUs are a limited resource and are scheduled accordingly. One model is swapped on a timetable depending on what is needed during the day and what at night; for the applications the route name stays the same.

What is watched is not only availability and response times but behaviour. If a route suddenly answers noticeably faster, that is rarely progress. Usually something other than the intended model is answering.

Updates run through a maintenance script that restarts the whole stack in order and afterwards checks that every route is reachable again — because a partial start that leaves half the routes silently down is the most unpleasant state of all: it looks like operation.

Outcome

Around 90 employees have AI access at their own workplace today — GDPR-clean and without a per-seat cloud subscription.

Cost control is built in: every request is routed and measured internally instead of billed externally.

What we learned

The gateway answered a completely invented model name with a valid response instead of an error. A typo in a configuration therefore ran unnoticed on a different route than intended, and nobody could see it, because answers kept coming. Since the finding, a test checks that an unknown name genuinely fails. The absence of an error is no proof.

Two nightly analyses stopped delivering for no visible reason. The cause was not in the code. It was in the calendar: the model behind the route is swapped at fixed times, during that window it is unreachable for a few minutes, and both jobs started exactly on the changeover minute. The schedules now know about the swap windows.

Both cases share a shape typical of AI infrastructure: it rarely fails at the model. It fails where something answers quietly instead of failing loudly — and that is considerably harder to find than a crash.