Enterprise AI stack, built in-house

Starting point

Running your own AI models on your own hardware means making every decision a rented API would otherwise make for you: which model, how much context, which card, which fallback path. In return, the data stays in-house, and nobody else decides when a model gets retired or swapped.

This reference comes from our own operations: the fleet that carries LibreChat, our own agents, and internal document processing every day — when a card here hits its limit, we find out first, not a customer left waiting.

The hard part

It started with a single large model that a cron job swapped morning and evening. That held up right until the schedule outlived the model: the switcher kept trying to resurrect a model that had long since been replaced, weeks after the fact, because nobody had told it otherwise.

A second card read documents for invoice processing, deliberately on its own independent model architecture, so two readers wouldn't sit on the same foundation and be wrong together. That card was specialised hardware incompatible with the rest of the fleet, and it had to make way for another project.

A third service, a Telegram bot for the daily briefing, moved across three different cards within three weeks, always to wherever capacity happened to be free. On the last one it ran blind for five days, because a single file — the vision module — was simply forgotten during the move. No error, no warning, just an answer that never matched the picture.

Solution

The fleet today runs a single model family across every chat lane — reasoning on, reasoning off, agent traffic, document vision — from one host with four graphics cards, instead of seven different model names for seven different purposes.

If the main host fails or takes too long, a second card with the same model family steps in automatically — smaller context, different quantisation, but nothing a user would notice.

The small, constantly-used services — embedding for search, re-ranking results, text recognition — now sit together on one small card instead of scattered across the fleet.

How it is built

A gateway is the only thing any caller talks to — LibreChat, the agents, internal automation, even the CNC line on the shop floor. Which card actually answers stays hidden behind it, and that's exactly what makes the fallback possible without any client ever needing to change.

The road to today's 262,144-token context wasn't a straight one. A new model first needed a lower, more honest ceiling, because the old maximum no longer applied to the new architecture — a mistake that would otherwise have surfaced only at the first conversation that got cut off. A patch to the cards' cache format then nearly doubled concurrent capacity, from 3.2 to 5.9 full-length conversations, before context was raised back to its real maximum.

A different piece of serving software for the same new model looked excellent running alone — and collapsed to less than half its solo throughput under eight concurrent requests. What runs today instead saturates past 30 concurrent conversations rather than collapsing at eight. That comparison alone decided which serving software the fleet runs.

In document processing, four independent readings vote on every field — two language models, two deterministic text-recognition passes. Since the specialised card was retired, two of those four votes now share the same server. That's documented openly, not glossed over: it's three independent opinions today, not four.

Day-to-day operation

A watchdog script automatically restarts stopped services — useful, until it misidentified a model and resurrected one that had long been replaced. The rule since is as simple as it is strict: a service only counts as 'should be running' if its own restart policy says so too.

The automatic fallback isn't just configured, it's regularly tested for real: the main host gets deliberately knocked out, and the only thing that counts is whether the second one actually takes over — not whether the configuration looks plausible.

Outcome

Every chat lane now runs on one model family with automatic failover, at a native context ceiling of 262,144 tokens — measured, not claimed.

The fleet's former head node no longer carries a single graphics card; its entire specialised driver stack was removed, freeing 22 GB of disk.

What we learned

The missing image file from the bot's move is the most expensive lesson: a service that answers a missing module with a plausible-sounding but wrong response looks like a working service for five days straight. Since then: send a known test image after every move and actually check the expected answer, not just the health check.

Two of the four verification passes in document processing sharing one server wasn't a planned trade-off — it was the fallout of a hardware removal somewhere else entirely. It only surfaced when we re-measured, for this very reference, which model actually runs behind which name, rather than trusting what the configuration claimed.

The watchdog that resurrects services once brought up the wrong, much larger model in the middle of maintenance and filled the little spare graphics memory that was left. Since then, the same first step precedes every change to this fleet: stop the watchdog before touching anything else.