DeepInfer audited: two places in the code decide whether it could run here

At 04:09:42 UTC on 20 October 2026, DeepSeek pushed the contents of a new repository, DeepInfer, as a single commit (commit history on GitHub). Apache-2.0, three commits by the evening of 20 October. This is the inference engine behind DeepSeek V4.1 Flash, the layer we fill with vLLM and llama.cpp (how we speed up inference). We read the clone and installed or built none of it.

DeepInfer is the inference engine DeepSeek says it uses to serve its V4.1 Flash model in production. The open release under Apache-2.0 contains the offline core: an entrypoint class without a server, distributed execution, speculative decoding and structured output. The KV store and the load balancer are missing, and the model registry knows a single architecture, DeepSeek V4.1 Flash.

Status 5 November 2026: our statements about our own stack hold for 20 October. Since 4 November we have also been testing DeepSeek V4.1 Flash on our production host, with the engine Strata and not with DeepInfer. DeepSeek weights therefore sit there now, and the gateway carries a name with deepseek in it. That does not change the finding about the code: DeepInfer still loads nothing here, because the hardware check rejects our cards. The repository has moved: on 5 November 25,434 stars and 3,108 forks (GitHub API), six commits, the latest of 3 November with the message "update V4.1 Flash Q&A to v0.2". The quotes below come from that current version.

On the evening of 20 October the repository stood at about 11,450 stars and 1,016 forks, with 95 pull requests from 40 authors (our reading, 20 October at 20:06 and 20:26 UTC) and not a single one merged. The issues tab is switched off. Merging has started since, for example pull request #157 on 21 October. On the coverage: the primary sources are the README and the deployment Q&A in the repository, and the trade press followed, for example TechNode on 21 October. We found no Bloomberg report on DeepInfer and therefore do not cite one.

What the release actually contains

What ships is the offline core: an entrypoint class for inference without a server, distributed execution, speculative decoding, structured output. There is no HTTP server in the package, no KV store and no load balancer. The deployment Q&A says so itself: the components for KV cache management and load balancing are tightly coupled to internal infrastructure and therefore not yet fully open.

The repository does contain tests. Under tests/ sit 19 Python files, among them a KL test that compares step-by-step decode logprobs against a full prefill, a batch-invariance test, a regression file offline_decode_ppl.json with pinned perplexity values and a test for forbidden CUDA synchronisation points. What is missing is a pipeline to run them: there is no .github/ directory in the first commit, and the test markers in pyproject.toml (ci, ci_npu, benchmark) point at pipelines that were not shipped.

Anyone tempted to build this into an existing image should read the build script first. build.sh ends by force-reinstalling nvidia-nccl-cu13==2.32.3, which on a running inference image means swapping the communication library under the engine.

Item	Value	Unit	Source and measuring host
Model architectures in the registry	1	entries	Our own source reading, first commit d585e65, 5 November 2026, local clone
Python files with a "Derived from vLLM" marker	23 of 270	files	Our own count, same state
Word-for-word identical lines against vLLM v0.32.0	748 of 6,817 (11 %)	lines	Our own comparison of 20 October 2026, not re-checked
Tokens per day, one cluster	155 trillion	tokens/day	DeepSeek's figure, not re-measured by us
Revenue per GPU-hour	38	US dollars, off-peak price	DeepSeek's figure, not re-measured by us
FlashMLA decode kernel, open against internal	5 to 10 % slower	percent	DeepSeek's figure, not re-measured by us

Two places in the code decide

The first is a hardware check. The function that returns the compute ceiling lives in deep_cluster/hardware/device.py and knows two cases (shortened here):

def get_bf16_tflops_limit() -> int:
    if accelerator_kind() == "ascend":
        return 432
    if is_sm_major_10():          # capability[0] == 10
        return 2500 if arm else 2250
    raise RuntimeError("unknown GPU type")

It is called at line 304 of deepinfer/engine/llm_engine.py while the statistics logger is built inside the engine constructor. The README lists "NVIDIA SM100 GPUs" and Huawei Ascend 950 as supported platforms. Our four cards report compute capability 12.0, consumer Blackwell on PCIe buses, 32,607 MiB each (queried on the production host, 20 October 2026). On that hardware the engine would, going by the code, raise an exception before a first token exists. We did not run it; this is a reading of the source.

The second place is the model registry, and for us it weighs more. The _MODELS table in deepinfer/model_executor/models/__init__.py holds exactly one entry, DeepseekV41ForCausalLM, and the README names only DeepSeek-V4.1-Flash as a supported model. Then our measurement of 20 October on the production host: the gateway configuration on our LiteLLM host listed 16 model names, none of them contained deepseek, and all of them pointed at our own cards. The one inference process running there started its model with --reasoning-parser qwen3 --tool-call-parser qwen3_xml, served eight sequences at a time and speculated three tokens per step. No DeepSeek weights sat on that machine, and the open file of this model class would be 510.3 GB. DeepInfer was therefore not an offer we would run too slowly, it was one that would never load our model (our vLLM lane in production).

The optimisations landed in our pin weeks ago

DeepSeek links pull requests from the vLLM project inside its documentation. On 20 October we ran a git ancestor check against the state we were running, v0.32.0 (pin since 18 October). Five of the six named techniques are inside. We re-read the merge dates against the GitHub API on 5 November: the DSpark fix of 17 September (#57432), encoder replay of 18 September (#56227), top-k selection of 13 September (#56464), mega-attention with an NVFP4 compressed KV cache of 16 September (#56935) and asynchronous prefetching of the engram table of 13 September (#56512). The decoder branch (#56752) was closed without a merge, and the tracking issue #57448 ticks it anyway. That checkbox habit is the most useful finding here, because it applies to every checklist we would call an adoption decision. A newer state exists by now: vLLM published v0.33.0rc3 on 2 November (no final v0.33.0 release as of 5 November), and our statement refers to v0.32.0.

The second calculation concerns origin. The README itself says DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. The derived-from marker sits on 23 of 270 Python files. Against v0.32.0, 748 of 6,817 lines of those were still word-for-word identical on 20 October, eleven percent. That comparison is ours, and we have not published the script. DeepInfer is its own engine built on an older frame, and an operator carries the security and bugfix maintenance from day one.

The published numbers, recomputed

The deployment Q&A names 155 trillion tokens per day for one cluster and US$38 in revenue per GPU-hour at off-peak pricing, both without a measurement log. Our back-of-envelope arithmetic: at the most productive point of the published decode curve, around 26,300 tokens per second per GPU (read off the chart), 155 trillion tokens a day works out to roughly 68,200 cards. Taking the 13,730 from the measurement table at 60 sequences per rank instead, it is roughly 130,700. The document does not say whether the daily figure counts cached input tokens. The result is therefore an order of magnitude, an installation in the six- to seven-figure card range, and not a finding.

A range circulating on the topic does not appear in the documentation as such. The advantage of wide expert parallelism (EP64) over eight ranks (EP8) gets quoted as 25 to 35 percent, and the text of the documentation only refers to a chart. We read the points off the chart in the documentation, accurate to about one percentage point: roughly 6 percent at 256 requests per rank, 19 at 128, 24 at 64, 35 at 32, 37 at 16 and 39 at 8. The 25 to 35 percent band therefore sits in the middle of the curve, and at the operating point DeepSeek itself names, around 60 sequences per rank, the advantage is about 24 percent. The reading is ours, not DeepSeek's, and the curve applies to a context of 131,072 with a fixed draft of three tokens.

Well documented is the speculative verification, every value verbatim from the measurement table in the documentation:

Percentage gain in estimated throughput per rank (tokens/s/rank), from the measurement table DeepSeek published, retrieved 5 November 2026, Open-PerfectBlend dataset. The same table lists about 30 percent fewer verified tokens per step and a mean accepted length of 3.29 against 3.20. (Quelle: DeepInfer, doc/inference_deployment_qa.md)

Also documented, and often shortened in coverage: bounded replay (SWA bounded replay) recomputes only the last 128 tokens of the prompt in the decoder layers instead of running exactly the last 2,560 tokens through 20 layers, which DeepSeek says nearly halves the prefill computation. Smaller than usually reported is the distance between the open kernels and the internal ones: the public version of FlashMLA is 5 to 10 percent slower on the decode kernel because one hand-tuned SASS optimisation is missing, and the end-to-end difference is at most 5 percent according to the documentation.

What we do with it

Nothing changes at our gateway because of DeepInfer, and that is the result. There is nothing to buy here, we are not planning a build in that size class, and on 20 October our production lane ran a model this engine never loads. One free observation stays open: tracking issue #57448 in vLLM, still open on 5 November. We check there whether further techniques will reach our next pin.

What transfers is the craft. A comparison of decode logprobs against a full prefill, a regression file of pinned perplexity values and a test that reports forbidden synchronisation points are patterns for any lane, independent of the model. Our own quality checks on the lane include none of these three so far (our 27B lane in self-hosting). DeepSeek describes its own parallelism: decode runs at EP64 to 128 and DP64 to 128, and DeepSeek does not use tensor parallelism as long as the model fits in memory and the latency target is met. That width needs decode groups of 64 to 128 GPUs and traffic that fills them. Four cards on PCIe do not reproduce that (two DGX Spark in our own test).

A different look at the same model family is DeepSeek 1017: it covers the swapped weights and their cost per task according to Artificial Analysis, while this post covers the published engine code and whether it runs here. All our posts on running models in-house are collected under the topic AI in production. If you face the question of which engine holds up on your hardware and your model, that is the core of our work on company-owned language models.