EUR 261 instead of EUR 7,640 in API fees — a month of agent traffic, measured

People writing about agentic coding models like superlatives. Someone recently posted "900 tokens per second" with a few sentences about sovereignty on top. The number isn't wrong, it's just a different sum than the one we care about: it counts the aggregate across three parallel agents, on a model with far fewer active parameters per token.

The question behind it is what interests us: what actually runs across such a pipeline, in what kind of tokens, and what does the month cost? Rather than invent an arithmetic example, we recorded one day of production. Every figure below comes from our gateway's spend logs, the inference engine's own counters, or the harness itself — measurement window stated each time.

The pipeline: agents on top, model in-house

Three kinds of consumer hang off the same pipeline:

• Windows servers with VS Code. On the ERP box, DeepSeek Harness runs as a VS Code extension, and it runs on the Windows Server itself: the extension declares extensionKind: ["workspace"], so it loads in the remote extension host and needs the target machine — a win32-x64 VSIX on the server, not on the laptop. The mechanism is what matters: the extension has no model access of its own, it points at our internal gateway with provider slug litellm (dsh and vision, capacity set by hand to 245,760 / 16,384 because the extension's context table doesn't know our model). No model, no cloud key, no receipts leaving the building — unless you miss the provider switch: if the panel header stays on deepseek-official, the request really does go to api.deepseek.com, key and prompt included. It shows up as "Authentication Fails", which is precisely the kind of error message we want to be the one we get.
• The browser. The same harness exists as a web UI — terminal input, file editing, kanban, token usage, all in the browser, behind a Keycloak login. From there the agent walks the company over SSH: 59 hosts in the SSH config, Proxmox, Unifi, backups, mail, databases, the GPU boxes. That is the working mode nobody would have trusted a local model with two years ago.
• Scheduled runs. Nightly job, weekly job, audit runs, receipt extraction, a sentiment scorer. All through the same gateway, so that later you can see who consumed what.

In front of all of them sits a LiteLLM gateway. Not out of sentiment, but for two hard reasons: vLLM cannot attribute anything per consumer (engine-wide counters only), and without a gateway there is no reason to hold a separate key per tool. The router deliberately listens on loopback only; the public side is a small auto-router in front of it. Unknown model names have failed loudly for half a year now — the wildcard lane, which used to answer any invented name with HTTP 200, is gone. For weeks afterwards the logs contained requests such as gpt-4o and lane-that-does-not-exist from tools that had made up a name.

# LiteLLM, abridged — one lane plus the caps the harness respects
- model_name: dsh
  litellm_params:
    model: openai/flashnext-hybrid
    api_base: http://<gpu-host>:8000/v1
    api_key: <gateway-key>
    drop_params: true
  model_info:
    max_input_tokens: 245760
    max_output_tokens: 16384

The model and what the four cards actually do

In production we run Qwen3.8-Flash-Next as a hybrid checkpoint: a Mixture-of-Experts with 125 billion parameters and roughly 6 active per token, plus a 51-billion n-gram embedding table (PLE) that lives in CPU RAM, because 47 GiB won't fit into four consumer cards. The experts are quantised to NVFP4 — sm_120 does that natively, and our image forces the native FP4 MoE kernel via --moe-backend flashinfer_cutlass. Don't "tidy up" that flag: vLLM's automatic backend selection doesn't know SM120 for NVFP4 and will silently fall back to Marlin.

# Startup on the GPU box, abridged — this is how the lane has run since early September
docker run -d --gpus all --shm-size 16g -p 8000:8000 \
  -e VLLM_PLE_CPU_OFFLOAD=1 \
  vllm-flashnext-hybrid:kvq-fp8 \
  /models/flashnext-hybrid/merged \
  --tensor-parallel-size 4 --enable-expert-parallel \
  --moe-backend flashinfer_cutlass \
  --max-model-len 262144 --max-num-seqs 8 \
  --gpu-memory-utilization 0.92 --kv-cache-dtype fp8_e4m3 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_xml \
  --enable-auto-tool-choice --enable-prompt-tokens-details

The KV pool is the number that makes the difference. Agent sessions are long: around 38,000 prompt tokens per request is our normal, and whoever can't hold a 262k context in KV evicts and recomputes from scratch. FP8 KV nearly doubled the pool for us — at identical consumption, because after startup the KV cache takes virtually all the VRAM that's left anyway. The price is concurrency: our predecessor, a dense 27B in FP8, ran roughly double the simultaneous requests on the same hardware at the same context length. We traded concurrency for context length and tool reliability — and we say so plainly, because in live operation the slot ceiling was the bottleneck, never the model. We ran the quality gate for that change (needle-in-a-haystack up to 180k, four parallel 128k sessions, tool calls, reasoning) against BF16 before switching: identical.

The token bill: 111:1 and a cache that hits almost everything

One day of production at the gateway, all lanes, 15,914 requests:

Split by lane (same window):

Two observations matter more to us than any benchmark number:

Those 10,469 embedding requests are 66% of all requests but only 3% of the tokens. Whoever prices their cost per request misunderstands this pipeline.

Agent load is cache-shaped. vLLM's prefix cache hits because the agent carries almost its entire context forward from step to step: tool definitions, AGENTS.md, the same files. On the engine side that means 94.17% of 103.4 million tokens queried since the last restart — that counter is token-based, not per-request; on the gateway side 90.6% of prompt tokens across a whole day. The gap between the two is the share of small, non-repeating requests — embeddings, test runs, cold starts.

Over thirty days that adds up to 2.70 billion tokens across 209,622 requests on ten model lanes, on 30 out of 30 days.

Speed: measured, not projected

Decoding is the value every display shows and the one that says least about working speed. Our measurements:

Plus the time shares from a 21-hour log of 1,901 requests: 0.6 seconds to the first token on average, 0.44 seconds of precompute on a cache hit, 7.6 seconds of decoding per request. For a model moving 6 billion active parameters per token, that is workable for an interactive agent. Where the time actually goes is visible in the session that wrote this article: 30 minutes of model time against just over 5 minutes of tool time, 143 steps, 19.5 million input tokens against 116 thousand output, 97% cache hit. The bottleneck is the model itself — not SSH, not the tool.

We also corrected ourselves while measuring this. Our own benchmark log from August has a column "measured total" with values around 100 tokens/s that circulated as a reference for weeks. It included prefill — with a production prompt of 62,000 tokens against 64 response tokens in that test, it measures prefill, not decode. We don't quote it any more. Second lesson: 78 engine steps per second at exactly 1 token per step is the proof that our predictive fast path (MTP) is not active, even though the build registers it. It stays off because there is an open upstream bug on our card generation (#55357: occasionally 0% draft acceptance, plus thinking blocks that repeat themselves) and because NVIDIA's own measurement on four H100s came out negative (≈36% acceptance, 8–36% lower throughput, up to 173% worse per-token latency). Not a win button, more an open experimental door.

What the same month would have cost through an API

Here is the calculation everyone wants — with its assumptions directly underneath, so it can be picked apart.

Three things about this calculation deserve an honest word. First, the API variants are not our numbers, they are list price (claude.com/pricing, retrieved 10 September 2026) × our token mix. We did not run a head-to-head against Claude Sonnet 5 — we don't hold an API key for it, and without the same prompt on both ends every "quality advantage" is an opinion. Second: the most cache-friendly variant assumes the provider recognises those repetitions as cleanly as our automatic prefix cache does — prompt caching there needs maintained breakpoints, and cache lifetime is short. At 37,900 prompt tokens per request, that behaviour is the entire stake. Third: our €261 is only the GPU box. Gateway host, embedding card, storage, backups and the operating hours aren't in it — and operating time is the reason everyone doesn't run this, not the electricity.

The sentence we take from it is small and pragmatic: as long as our traffic has this shape — long, repeating context, little output — the marginal cost per task is close to zero for us. That is exactly why we can run tasks that would never amortise per task: the weekly run that walks 40 machines every night, the scorer that checks receipts, the audit run that reports itself when it finds something.

Does this replace Claude Sonnet 5? The honest answer

For the work we actually do: largely yes. And "largely" isn't a marketing word here, it's the place where we would still pay if we had to.

What argues for it, from operation rather than benchmarks:

• Volume and type of work. 464 million prompt tokens in a single day on the agent lane alone, 93.5% of them cache hits — that's long multi-step jobs across SSH, files, configuration and logs, not line completion.
• Tool reliability. tool_calls come back cleanly with a correct finish_reason; forced tool calls work; the chat template doesn't open a thinking block unless you ask for one. Two years ago that was the most common reason local models failed in agentic operation.
• Test batteries on real tasks. On 24 real ERP receipts with an unchanged production prompt, an MoE with 3 billion active parameters matched a dense 27-billion model on quality — same line items, same arithmetic check, and on our most critical field (address-rule violation) it was even better: 5.0% versus 17.4%. On another occasion: 21 test tasks (arithmetic, follow-the-instruction, tool calls, needle across 32k), 19 of 21 for our model at the time, 18 for the candidate.

What argues against it, and will continue to:

• Planning over very long horizons. Frontier models decompose a twenty-step task into correct intermediate steps more reliably. Our countermeasure isn't a model, it's discipline: project rules, bans on reusing instructions already spent, verification steps, hard exit conditions. It measurably helps — it doesn't replace the quality of the first draft.
• Non-determinism at temperature 0. Our build can return a thinking block and leave the answer text empty at temp = 0. For extraction that writes into an ERP, that is a production risk, and it still isn't fixed upstream.
• Hard ceilings. Eight concurrent requests, no automatic failover to the second box; the fallback lane has to be chosen by the client. An outage of the GPU box is an outage of the AI, not a switchover.
• Specialised lanes stay specialised. Reading receipts, OCR, re-parsing documents run on their own models and services, not through the chat lane.

So our answer isn't "replace". It is: we moved off the high-volume, repetitive, privacy-sensitive tasks, and that is the part that cost the most through an API. What remains are the rare moments when a model doesn't notice its own reasoning error. We keep buying those with controls and verification steps — not with token prices.

What we'd do differently next time

• Actually test MTP instead of leaving it alone. The build can do it; nobody ever tried. Even if the outcome is a loss, that's a result — the upstream thread: vllm#55357.
• Report prefill and decode separately in benchmarks. One of our own numbers misled us for three weeks.
• Raise slots before memory gets tight. The bottleneck today is the slot counter, not the KV pool. While utilisation stays under 30%, 16 slots is the cheapest win of the quarter.
• Deliberately not NVFP4 KV. The jump to a triple-sized pool was tempting; the measured perplexity regression and double prefill weren't worth it to us.

Bottom line

Between "we need a cloud key" and "we run an AI factory" sits a gateway, a container and a very unspectacular truth: most modern agent tokens are repetition. Whoever keeps that repetition in-house gets speed classes they don't have to care about, data sovereignty as a side effect, and a bill that consists mainly of depreciating one graphics card.

If you're planning a comparable pipeline, don't start at the model — start at the gateway: measure which lane uses what, and why. Only then do you know whether the model is the problem at all. In our case it hadn't been for months; the problems were called KV pool, backend flag and slot ceiling.

Related posts: vLLM on own hardware, vLLM or llama.cpp, comparing coding agents on Windows servers, running MCP servers yourself, the EU AI Act and local AI.