Inkling 2 Small: 366 GiB of weights against our 127 GiB of VRAM
Thinking Machines announced Inkling 2 Small on 5 November 2026 and released the weights on Hugging Face under Apache 2.0 on 12 November: 594 billion parameters, 17 billion of them active, up to a million tokens of context. For us the question is whether it fits on our cards.
A fit test here means: weight bytes from the Hugging Face API, quantization from config.json, KV cache from layer and head counts, plus about 15 percent headroom for activations and CUDA graphs (our assumption, not a vendor figure). The sum is held against real VRAM, not against the parameter count on the model card.
The numbers
| Quantity | Value | Source |
|---|---|---|
| Weight files, total | 366.5 GiB (393,510,049,944 bytes, 42 files) | measured, HF API, 15 Nov 2026 |
| Parameters total / active | 594B / 17B | vendor, total confirmed by the API |
| Experts | 384 per layer, 8 per token | config.json |
| Layers with full attention | 8 of 48 | config.json (40 local layers counted) |
| KV cache, BF16 | 32 KiB per token, 32 GiB at 1,048,576 tokens | derived |
| KV cache at 220,000 tokens | 6.7 GiB | derived |
| VRAM, production host with 4x RTX 5090 | 127.4 GiB (4 x 32,607 MiB) | measured, nvidia-smi, 15 Nov 2026 |
| Vendor requirement, NVFP4 | at least 500 GB aggregate | vendor |
The KV figure follows from eight full-attention layers with eight KV heads of 128 dimensions each, K and V per token in BF16: 8 x 2 x 8 x 128 x 2 bytes = 32 KiB. The window of the 40 local layers adds about 0.16 GiB per session. The 17 billion active parameters match the configuration: 8 experts of 31.9 million parameters each across 48 layers make 12.2 billion, and attention plus embedding add roughly four to five more (our own arithmetic, leaving out gates and convolutions).
What follows
The KV cache is the good news. Since 40 of 48 layers hold only a 1,024-token window, the full million tokens cost 32 GiB on paper, half that with an FP8 cache. For the context of our production lane (max_model_len 220000, measured on the running process on 15 November 2026) it would be 6.7 GiB. The vLLM recipe, however, reports 64.47 GiB for one full-length request and recommends two nodes with TP8, so our 32 GiB is a lower bound.
The weights are the problem. At 366.5 GiB the model is 2.9 times our host with four 5090 cards. With KV cache and headroom we land between 458 GiB (32 GiB KV) and 496 GiB (64.5 GiB KV), roughly 490 to 530 GB. That agrees with the vendor's 500 GB. On 5090 cards of 31.8 GiB each that is 15 to 16 cards; the recipe works in groups of four and eight, so in practice it comes to 16 (derived). Eight cards give only about 255 GiB.
Host RAM does not rescue it either: the production host has 91 GiB (free -g, 15 November 2026), so VRAM plus RAM comes to about 218 GiB. All GPUs in our three GPU hosts add up to roughly 175 GiB: 127.4 GiB in the production host, 31.8 GiB in the standby machine with one 5090 and 16 GiB in the KVM machine with one RTX 4060 Ti (source: our host inventory as of 10 November 2026, taken from the file, not recounted; the sum is derived). Running the model would take new hardware.
The runtime path is still open. Until a release carries support, the vLLM recipe asks for the image vllm/vllm-openai:inkling-2-small and names 8x B200 as its default; it lists GB200 as verified. Whether the inkling2 model type runs on sm_120 we have not tested. For vLLM our rule is to use pinned vX.Y.Z tags only; a special image like this would be an exception.
Recommendation
Anyone who wants 17 billion active parameters and long contexts locally should check the 366.5 GiB and the runtime path before ordering anything. Without new hardware, Inkling 2 Small does not run here. The post on the DGX Spark cluster with DeepSeek V4.1 Flash deals with the same question for another large model and a memory cluster. More posts on the hardware side collect in the local AI hardware topic hub, and if you want to know what your model needs on your cards, the corporate LLM consulting page is where to start.
Open points: We did not test a run on sm_120, nor the actual KV cache volume in vLLM (32 or 64.5 GiB per million tokens). The 15 percent headroom is an assumption. The 175 GiB total rests on our internal inventory of 10 November 2026 and was re-measured only for the production host.
Read next: DSpine as a draft model for Qwen3.8-27B · Huawei MACaQ for GGUF, re-measured · NVIDIA snapshot: restore versus cold start · H100 rental prices by provider
Further Reading
How we measured our own values: SSH to the production host on 15 November 2026 around 16:48 CET, nvidia-smi --query-gpu=name,memory.total, free -g, max-model-len from the process list; weight size from huggingface.co/api/models/thinkingmachines/Inkling-2-Small?blobs=true, sum of the 42 file sizes.
- Inkling 2 Small on Hugging Face, model card with hardware requirement and parameter counts, primary source, verified on 15 November 2026
- config.json of Inkling 2 Small, layers, KV heads, experts, primary source, verified
- Thinking Machines announcement, 5 November 2026 with an update of 12 November, vendor statement, reported (bot-walled for curl, content read through a browser fetch)
- vLLM recipe, default 8x B200, special image, 64.47 GiB message, vendor-claimed
- vLLM blog on Inkling 2 Small, 12 November 2026, vendor-claimed architecture and performance
- SGLang recipe, alternative runtime per the model card, vendor-claimed
- Apache License 2.0, license text, primary source
Does Inkling 2 Small fit on four RTX 5090 cards?+
No. The weight files add up to 366.5 GiB (Hugging Face API, 15 November 2026), while four RTX 5090 cards offer 127.4 GiB. Even all the GPUs in our three GPU hosts together, about 175 GiB, fall short, and they do not form one pool.
Why is the KV cache no problem at a million tokens?+
Only 8 of 48 layers use full attention; the other 40 use a window of 1,024 tokens. Derived from config.json, that is 32 KiB per token, or 32 GiB at 1,048,576 tokens in BF16. The vLLM recipe's own error message quotes 64.47 GiB for a million tokens, twice that figure, so a safe planning value lies between the two.
What hardware does the vendor ask for?+
At least 500 GB of aggregate VRAM for the NVFP4 checkpoint; the model card names 2x B300 or 4x H200 as examples. The vLLM recipe lists 8x B200 as its default configuration and points to a dedicated Docker image until a release carries support.
senn-tech