Skip to content
senn-techsenn-tech
Back to blog
AI Hardware
Local AI and GPU hardware2026-11-15· By Franz Senn

Inkling 2 Small: 366 GiB of weights against our 127 GiB of VRAM

Thinking Machines announced Inkling 2 Small on 5 November 2026 and released the weights on Hugging Face under Apache 2.0 on 12 November: 594 billion parameters, 17 billion of them active, up to a million tokens of context. For us the question is whether it fits on our cards.

A fit test here means: weight bytes from the Hugging Face API, quantization from config.json, KV cache from layer and head counts, plus about 15 percent headroom for activations and CUDA graphs (our assumption, not a vendor figure). The sum is held against real VRAM, not against the parameter count on the model card.

The numbers

QuantityValueSource
Weight files, total366.5 GiB (393,510,049,944 bytes, 42 files)measured, HF API, 15 Nov 2026
Parameters total / active594B / 17Bvendor, total confirmed by the API
Experts384 per layer, 8 per tokenconfig.json
Layers with full attention8 of 48config.json (40 local layers counted)
KV cache, BF1632 KiB per token, 32 GiB at 1,048,576 tokensderived
KV cache at 220,000 tokens6.7 GiBderived
VRAM, production host with 4x RTX 5090127.4 GiB (4 x 32,607 MiB)measured, nvidia-smi, 15 Nov 2026
Vendor requirement, NVFP4at least 500 GB aggregatevendor

The KV figure follows from eight full-attention layers with eight KV heads of 128 dimensions each, K and V per token in BF16: 8 x 2 x 8 x 128 x 2 bytes = 32 KiB. The window of the 40 local layers adds about 0.16 GiB per session. The 17 billion active parameters match the configuration: 8 experts of 31.9 million parameters each across 48 layers make 12.2 billion, and attention plus embedding add roughly four to five more (our own arithmetic, leaving out gates and convolutions).

Inkling 2 Small needs 458 to 496 GiB, our three GPU hosts offer about 175Host with 4x RTX 5090127.4 · 127.4 GiB VRAM, measuredAll three GPU hosts175 · about 175 GiB, derivedInkling 2 Small weights366.5 · 366.5 GiB, measuredNeed with KV and reserve458 · 458 to 496 GiB, derived0520
The need is our own sum: weights plus KV cache (32 to 64.5 GiB) plus roughly 15 percent reserve, which is an assumption. (Quelle: Hugging Face API and our own measurement, 15 Nov 2026)

What follows

The KV cache is the good news. Since 40 of 48 layers hold only a 1,024-token window, the full million tokens cost 32 GiB on paper, half that with an FP8 cache. For the context of our production lane (max_model_len 220000, measured on the running process on 15 November 2026) it would be 6.7 GiB. The vLLM recipe, however, reports 64.47 GiB for one full-length request and recommends two nodes with TP8, so our 32 GiB is a lower bound.

The weights are the problem. At 366.5 GiB the model is 2.9 times our host with four 5090 cards. With KV cache and headroom we land between 458 GiB (32 GiB KV) and 496 GiB (64.5 GiB KV), roughly 490 to 530 GB. That agrees with the vendor's 500 GB. On 5090 cards of 31.8 GiB each that is 15 to 16 cards; the recipe works in groups of four and eight, so in practice it comes to 16 (derived). Eight cards give only about 255 GiB.

Host RAM does not rescue it either: the production host has 91 GiB (free -g, 15 November 2026), so VRAM plus RAM comes to about 218 GiB. All GPUs in our three GPU hosts add up to roughly 175 GiB: 127.4 GiB in the production host, 31.8 GiB in the standby machine with one 5090 and 16 GiB in the KVM machine with one RTX 4060 Ti (source: our host inventory as of 10 November 2026, taken from the file, not recounted; the sum is derived). Running the model would take new hardware.

The runtime path is still open. Until a release carries support, the vLLM recipe asks for the image vllm/vllm-openai:inkling-2-small and names 8x B200 as its default; it lists GB200 as verified. Whether the inkling2 model type runs on sm_120 we have not tested. For vLLM our rule is to use pinned vX.Y.Z tags only; a special image like this would be an exception.

Recommendation

Anyone who wants 17 billion active parameters and long contexts locally should check the 366.5 GiB and the runtime path before ordering anything. Without new hardware, Inkling 2 Small does not run here. The post on the DGX Spark cluster with DeepSeek V4.1 Flash deals with the same question for another large model and a memory cluster. More posts on the hardware side collect in the local AI hardware topic hub, and if you want to know what your model needs on your cards, the corporate LLM consulting page is where to start.

Open points: We did not test a run on sm_120, nor the actual KV cache volume in vLLM (32 or 64.5 GiB per million tokens). The 15 percent headroom is an assumption. The 175 GiB total rests on our internal inventory of 10 November 2026 and was re-measured only for the production host.

Read next: DSpine as a draft model for Qwen3.8-27B · Huawei MACaQ for GGUF, re-measured · NVIDIA snapshot: restore versus cold start · H100 rental prices by provider

Further Reading

How we measured our own values: SSH to the production host on 15 November 2026 around 16:48 CET, nvidia-smi --query-gpu=name,memory.total, free -g, max-model-len from the process list; weight size from huggingface.co/api/models/thinkingmachines/Inkling-2-Small?blobs=true, sum of the 42 file sizes.

Questions?
Does Inkling 2 Small fit on four RTX 5090 cards?+

No. The weight files add up to 366.5 GiB (Hugging Face API, 15 November 2026), while four RTX 5090 cards offer 127.4 GiB. Even all the GPUs in our three GPU hosts together, about 175 GiB, fall short, and they do not form one pool.

Why is the KV cache no problem at a million tokens?+

Only 8 of 48 layers use full attention; the other 40 use a window of 1,024 tokens. Derived from config.json, that is 32 KiB per token, or 32 GiB at 1,048,576 tokens in BF16. The vLLM recipe's own error message quotes 64.47 GiB for a million tokens, twice that figure, so a safe planning value lies between the two.

What hardware does the vendor ask for?+

At least 500 GB of aggregate VRAM for the NVFP4 checkpoint; the model card names 2x B300 or 4x H200 as examples. The vLLM recipe lists 8x B200 as its default configuration and points to a dedicated Docker image until a release carries support.