Skip to content
senn-techsenn-tech
Back to blog
AI Hardware
Local AI and GPU hardware2026-11-22· By Franz Senn

Inkling 2 Mini on Two Radeon Cards: Speed, Context and a PDF Extraction

Thinking Machines released Inkling 2 Mini with open weights on 20 November 2026. We measured it on 21 and 22 November on a Linux host with two AMD Radeon AI PRO R9700s: It passed 18 out of 21 tests, achieving 34.4 to 39.6 tokens per second on a single card, and 512,000 tokens of context only when using both cards. In PDF extraction, the model's reasoning, which cannot be switched off, got in the way.

Speculative decoding with DSpark means that a small draft model (here, 0.4 GB as a Q8 file) predicts several tokens in advance, and the large model evaluates them in a single step and adopts the correct ones. This speeds up output without changing the content, as long as the large model confirms each suggestion. The parameter n determines how many tokens the draft predicts per step.

Thinking Machines Lab logo
Thinking Machines Lab built Inkling 2 Mini and released it on 20 November 2026. (Quelle: Wikimedia Commons, public domain (artist field: Thinking Machines Lab))
AMD logo
AMD makes the Radeon AI PRO R9700. All measurements in this post are our own. (Quelle: Simple Icons, file amd.svg (CC0), mark: AMD)

What the vendor says

The model card (retrieved 22 November 2026) lists 26.4 billion parameters, 3.4 billion of which are active; with three passes through the middle twelve layers, there are 5.4 billion effective active parameters per token, a context size of up to 1 million tokens, Apache 2.0 licensing, and a checkpoint of approximately 20 GB in NVFP4 and MXFP8. For local runs, it specifies 32 GB or more of VRAM and lists llama.cpp with GGUF files as supported. Two of its listed limitations mattered to us: the lowest reasoning-effort value does not fully disable reasoning, and speed in local runtimes varies with runtime and hardware. The architecture and checkpoint memory sizes are covered in the post on the Inkling 2 Small Fit Test for the larger sibling; this post covers only what we measured ourselves.

How we measured

Device class: a Linux host with two AMD Radeon AI PRO R9700s (each with about 32 GB of VRAM). Software: the Unsloth fork of llama.cpp, release b12588, ROCm build for gfx120X, weights as Unsloth-GGUF in Q4 (UD-Q4_K_XL, 17.6 GB), plus the vision projector and the DSpark draft. The Radiance engine, which we normally use with these cards, cannot load the model because the checkpoint uses NVFP4 and MXFP8. The previous Radiance measurement pertains to a different model and a different engine; the numbers are not comparable.

Test dates: 21 November 2026 (speed, context, 21-task battery) and 22 November 2026 (PDF runs). One run per setup, temperature 0. Speed: three prompts with 600 output tokens each, median. Quality: our own battery of 21 tasks (math, code, German technical texts, tool calls, needle-in-a-haystack searches in long contexts).

Speed and context

SetupContextDraftTokens/s (decode)Measured by
1 card, HIP65knone34.4own measurement, 21 Nov 2026
1 card, HIP65kDSpark n=238.4own measurement
1 card, HIP65kDSpark n=339.6own measurement
1 card, HIP65kDSpark n=732.9own measurement
1 card, HIP131kDSpark n=3, KV q8_038.2own measurement
2 cards, layer split512kDSpark n=332.7own measurement
2 cards, layer split1Mnone23.5own measurement
1 card, Vulkan (RADV)65knone5.6own measurement
Decode speed per setup, Inkling 2 Mini on R97001 card, DSpark n=339.6 · 65k context1 card, DSpark n=3, 131k38.2 · KV q8_01 card, DSpark n=238.41 card, no draft34.41 card, DSpark n=732.92 cards, 512k, DSpark32.72 cards, 1M, no draft23.51 card, Vulkan5.6045
One run with three prompts each. The setups differ in context and card count and are not like-for-like comparisons. (Quelle: Own measurement, 21 Nov 2026, two AMD Radeon AI PRO R9700, llama.cpp b12588, one run each)

Three takeaways from this. First, with n=3, DSpark outperforms the run without a draft by 15 percent (39.6 vs. 34.4); with n=7, however, it performs worse than no draft at all (32.9). Anyone who knows the vLLM recipe with seven draft tokens should not carry that value over to llama.cpp, a different engine. Second, more context costs speed, but on two cards 512,000 tokens with a draft (32.7) leave more than one million without one (23.5). We did not compare the two at the same context. Third, with 65k context, Vulkan is about six times slower at 5.6 compared to HIP’s 34.4,a dead end for these cards.

Limits of a single card: without a draft, a context of 262,000 tokens ran; with a draft, only about 131,000; at 196,000, there wasn’t enough memory. The memory goes to a compute buffer that scales with context: At 512,000 tokens, it occupies 15 GB, even with a smaller batch and a quantized KV cache. One million tokens ran only with layer split across both cards, and DSpark running in addition to that failed due to memory constraints. The -sm tensor option (true tensor parallelism) still reports “not implemented” for this architecture. The two cards also report different amounts of VRAM (29.9 and 31.9 GiB); the cause is unclear.

Quality on the 21-task battery

One card with 262k context and two cards with 1 million tokens each passed 18 out of 21 tasks; the same three failed: a discount calculation and two counting tasks in long context. Qwen3.8-27B also achieved 18 out of 21 in our previous evaluation. This does not mean that the models are equally good: 21 tasks, each run only once, can only distinguish between two models in a rough way. This battery gives no evidence that either model is better.

PDF extraction without a thinking brake

As a practical test, we had the model convert order documents into structured JSON: Text from 17 PDF documents, approximately 20 fields (parties, locations, dates, goods, references), temperature 0, maximum of 2,500 output tokens, reasoning effort set to “none.” We do not say what the documents look like or where they come from; only aggregates count here.

Runvalid JSONTotal timeOutput tokensThroughputMeasured by
1 request at a time14 of 171,008 s (59.3 s per document)33,55533.3 tok/sown measurement, 22 Nov 2026
4 requests at a time15 of 17619 s31,18550.4 tok/sown measurement
Page images instead of text, 4 at a time, limit 4,50015 of 17761 s43,08256.6 tok/sown measurement

On average, the model required 1,974 output tokens per document (33,555 divided by 17). Qwen3.8-27B managed with 520 for the same texts and the same task (8,838 tokens, 3-bit variant without a thinking phase, measured on the same day on a card with a different engine; only the token count is comparable). The difference is the thinking time that Inkling expends despite the “none” setting, which the model card warns about. The three terminations at 2,500 tokens are therefore a token-budget effect and say nothing about extraction accuracy. Working in parallel helps, but only to a limited extent: Running four queries simultaneously reduced the total time by a factor of 1.6 (1,008 vs. 619 s); each individual query then took an average of 142 s per document.

Regarding quality: Of the 278 values provided in the 14 evaluable runs, 275 appeared verbatim in the PDF text (98.9 percent). This is an origin check. It shows the model invented nothing, and it leaves open whether every value sits in the right field. The three non-evaluable documents are excluded from the denominator; if they are included, the figure is 14 out of 17 usable responses. When reading the page images, 274 out of 293 values (93.5 percent) were present in the text; the errors were reading errors in city and street names and numbers. First attempt with a limit of only 1,500 tokens: 4 out of 17 responses were valid; the limit was the cause.

What we do not know

One run per setup, no variation. The 17 documents are a small sample from one area. We have not tested how the model performs under sustained load. The raw data for the speed runs is not available as a file, only the log. And we haven’t compared it to the faster engines for NVIDIA cards.

Recommendation

On two R9700s with this build: a single card with DSpark at n=3 and a context size of up to about 131,000 tokens yields the fastest results (38 to 40 tokens per second); both cards with layer split starting at 200,000 tokens of context. For extraction tasks, set a token limit of at least 4,500 and count responses that end at the limit as errors. If you’re looking for more options for local cards, check out the Local AI Hardware Topic Hub, and the Corporate LLM Consultation explains what such a model requires on your cards.

Further reading

Questions?
Does Inkling 2 Mini run on AMD Radeon AI PRO R9700?+

Yes, with a llama.cpp build for ROCm (gfx120X). On one card we measured 34.4 tokens per second, 39.6 with the DSpark draft. The Radiance engine cannot load the NVFP4/MXFP8 checkpoint, so the model ran as a Q4 GGUF. The Vulkan build managed 5.6 tokens per second, about six times slower. Measured on 21 November 2026, one run each.

How much context does Inkling 2 Mini handle on two R9700?+

With layer split, 1 million tokens ran without a draft model (23.5 tokens per second) and 512,000 tokens with DSpark (32.7). On one card with the draft model we reached only about 131,000 tokens; at 196,000 memory ran out. The cause is a compute buffer that grows with context. Tensor parallelism is not implemented for this model in the build.

Is Inkling 2 Mini good for extracting structured data from PDFs?+

Only with a large token limit. On 17 test documents, 14 returned valid JSON and 3 stopped at 2,500 tokens because the model kept reasoning although thinking was switched off. Of 278 returned values, 275 (98.9 percent) appeared verbatim in the source text. That checks origin only, not whether a value sits in the right field. A sample of 17 documents is a hint, not proof.