Skip to content
senn-techsenn-tech
Back to blog
AI Hardware
Local AI and GPU hardware2026-11-17· By Franz Senn

Two Radeon AI PRO R9700 with Radiance: what Qwen3.8-27B really does on AMD

Our host .202 now holds two Radeon AI PRO R9700 instead of one RTX 5090. On 16 and 17 November 2026 we measured Qwen3.8-27B on three inference engines there. Radiance reaches 59 to 61 tokens per second on one card, vLLM on ROCm 19 to 28. We also have a failure of the second card that we cannot explain yet, and six abliterated variants of the model, four of which passed the full test.

Radiance is an inference engine for AMD's RDNA4 cards. It loads models in its own format (.rad), uses its own compute kernels for these cards and offers an OpenAI-compatible API. From what we read in the issue tracker, the project has one main author. That belongs with the numbers below.

AMD logo
AMD makes the Radeon AI PRO R9700. The measurements in this post are ours, not AMD’s. (Quelle: Simple Icons, file amd.svg (CC0), trademark: AMD)

The setup

The host has an EPYC 7502 with 184 GB of DDR4 RAM, Ubuntu 26.04.1 with kernel 7.0.0-43, and the two cards (gfx1201, 32 GB each) in PCIe 5 x16 slots with ReBAR. ROCm is not installed on the host itself; the engines run in containers. All numbers come from this host. We measured with Qwen3.8-27B, 256 output tokens and short prompts, and the throughput figures are the sum of all concurrent streams.

Three engines compared

Engine (as of 16 Nov 2026)Setup1 stream, tok/s8 parallel, tok/s16 parallel, tok/s
vLLM v0.33.0, ROCm 7.21 card, AWQ-INT4, 4-bit KV, no drafter19 to 2869not measured
llama.cpp with MTP2 cards, tensor split, MTP speculation38109not measured
Radiance 2.1.01 card, 4.9 bit, KV i8, with drafter59 to 61266not measured
Radiance 2.1.02 replicas, one card each, with drafter57 (at 2 streams)371546
Radiance 2.1.02 cards, tensor parallel, with drafter89 to 104405363

Own measurement, 16 Nov 2026, 2× Radeon AI PRO R9700, Qwen3.8-27B. For a single stream, memory bandwidth sets the speed. Why Radiance leads there we can only guess. Its own RDNA4 kernels are one obvious reason. Radiance also ran with a drafter (speculative decoding) and a different quantisation than vLLM, and both skew the comparison. We did not separate the causes.

Two more things came with the vLLM numbers. The Composable Kernel variant of FlashAttention segfaulted during graph capture; with FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE it ran. Also, rocm-smi inside the container could not read the power cap, and hwmon answered "Device or resource busy" after boot. We could therefore not set a power limit.

Throughput at 8 parallel streams: Radiance TP2 leadsvLLM ROCm, 1 card69 · AWQ-INT4llama.cpp + MTP, 2 cards109Radiance, 1 card266Radiance, 2 replicas371Radiance, TP24050450
Sum of all streams in tokens per second, Qwen3.8-27B, 256 output tokens, 2× Radeon AI PRO R9700, 16 Nov 2026. (source: senn-tech, own measurement on .202)

Level1Techs also covers the R9700’s speed in the video “The CUDA hour is Late. Also, Who Made the R9700 this fast?”. We did not take numbers from the video; every figure in this post is our own.

Two cards, one model

Two replicas with one card each reach 546 tokens per second at 16 parallel streams, while one tensor-parallel instance across both cards reaches 363. At 8 streams the tensor-parallel setup is ahead, 405 against 371. A single stream gets faster with TP2, 89 to 104 against 59 to 61. That matters for an agent waiting alone on a long answer. For many short requests at once, two replicas do better.

Memory speaks for TP2. The 4.9-bit weights spread over both cards, and 19.46 GiB per card remain for the KV cache. The pool holds 1,103,104 tokens, which is 4.2 sessions at 262,144 tokens of context. Qwen3.8-27B keeps token KV in only 16 of its 64 layers; at 8 bit that is 32,768 bytes per token (a calculation from the model configuration).

Radiance supports only i8 and fp8 as KV formats, no 4-bit KV. Eight sessions at 262k tokens each need a pool of about 2.1 million tokens. With halved KV that works out to about 8 sessions for the 4.9-bit model and about 6 for the Int8 variants. These figures are calculated and were never measured. With realistic slots of 100,000 to 130,000 tokens, the 1.1 million tokens cover six to eight slots.

KV pool in sessions of 262,144 tokens of context, TP2Int8, Radiance 2.1.03 · 3.0 sessions4.9 bit, Radiance 2.1.04.2 · 4.2 sessionsMXFP4, Radiance 1.17.04.4 · 4.4 sessions05
KV pool as measured at startup, KV format i8 (Radiance 2.1.0) or fp8 (1.17.0), 17 Nov 2026. (source: senn-tech, own measurement on .202)

Abliterated variants

Abliterated models have had their refusal behaviour weakened by editing the weights. We wanted one as a reserve and as its own lane for dsh, our agent harness. For each variant we checked throughput, five tool calls with required arguments and a needle (ROT-4711-LUCHS) hidden in 135,000 tokens of context, all with TP2.

Variant (as of 17 Nov 2026)Format, engine1 / 8 / 16 streams, tok/sKV sessions at 262kTool callsNeedle at 135k
slybase Swift-1.5 hereticMXFP4, Radiance 1.17.0105 / 493 / 6514.45 of 5found, 41 s
mlasli Heretic-UncensoredInt8, Radiance 2.1.076 / 387 / 5113.05 of 5found, 47 s
Jiunsong SuperQwen3.8-27bInt8, Radiance 2.1.069 / 370 / 5283.05 of 5found, 47 s
akumaburn Swift-Qwen3.8-27bInt8, Radiance 2.1.069 / 391 / 5643.05 of 5found, 47 s
hotdogs Qwen3.8-27B-abliteratedInt8, Radiance 2.1.069 / 378 / 549not measured5 of 5card 2 dropped out
asfgsdfg Qwen3.8-27B-HereticInt8, Radiance 2.1.076 to 83 / 376 to 388 / 543 to 5483.0not measurednot measured

Own measurement, 16 and 17 Nov 2026. Single-stream speed follows the quantisation: Int8 takes about 29 GiB, the 4-bit version clearly less. The finished .rad from slybase runs only on Radiance 1.17.0, because the 2.x line rejects MXFP4 and FP8 containers. We converted the five Int8 variants ourselves from the BF16 sources, which takes about 8 minutes per model with the Int8 recipe. The 4-bit conversion with Hessians is CPU-bound: after 27 minutes on 32 cores not a single tensor was finished, and we gave up on it.

We did not measure refusal behaviour or quality (KL divergence, benchmarks). For Swift-1.5 the model card names the licence swift-open-license-1.0 with the note "research-only", and the model is text only, without vision. Two TP2 models do not fit on the cards at the same time, since each takes about 29 GiB per card. It is either one model across both cards or one model per card.

Throughput at 16 parallel streams, abliterated variants (TP2)slybase (MXFP4, 1.17.0)651 · tok/sakumaburn (Int8)564hotdogs (Int8)549asfgsdfg (Int8)545 · 543 to 548SuperQwen (Int8)528mlasli (Int8)5110700
Sum of all streams in tokens per second, 16 and 17 Nov 2026. For asfgsdfg the bar shows the midpoint of the range 543 to 548. (source: senn-tech, own measurement on .202)

The card failure

On 16 Nov at 11:33 the host reset itself while vLLM was loading a model, without a kernel log. We have not reproduced it. After that the same card, 0000:c3:00.0, dropped off the bus three times under load: amdgpu: device lost from bus!, with no AER entry. The first time was in the 135k prefill of the hotdogs model, the second and third about 11 and 2 minutes after boot during the test of the asfgsdfg model. The host sometimes restarted on its own; once systemctl reboot hung on processes in D state. A cold start through the BMC (ipmitool chassis power cycle) helps until the next run.

The power supply is rated at 1600 W on its datasheet, we checked the cables once and found nothing, and a 40-second full load on both cards ran stable. Against a pure software fault speaks that it was always the same card, across different models and quantisations. That is not proof. The next step is to swap the cards between the slots and reseat the 12V-2x6 connectors. If the same card fails again, that points to the card or its cable; if the failure follows the slot, the slot is at fault.

How we use .202 now

Since 17 Nov 2026 the container rad-prod runs on .202 with Radiance 2.1.0, TP2, 262,144 tokens of context and qwen3.8-27b-iq4r-l-v4.rad. In LiteLLM it is the second tier of the dsh lane, which serves the subagents of dsh, our agent harness. The first tier remains our host with four RTX 5090; .202 steps in only when the first tier is full at 8 concurrent requests or returns an error. In a test with 12 parallel requests, 6 went to the first tier, 4 to .202 and 2 ended in an error because both tiers were full.

How a dsh request ends up on .202dsh requestLiteLLM on .180.2Tier 1: .180.3Flash-Next, 4x RTX 5090, upto 8 requestsTier 2: .202Qwen3.8-27B, 2x R9700, up to4 requestsErrorwhen both tiers are full
Simplified view of the LiteLLM configuration of 17 Nov 2026. (source: senn-tech, own operation)

The first tier runs Qwen3.8-Flash-Next, the second Qwen3.8-27B. One session can be served by both models depending on load. The watchdog probes .202 every five minutes with a real request and sends an alert through SentinelLog on failure. It does not check the kernel log for device lost. That is on our list.

We describe Qwen3.8-27B on four RTX 5090 in Qwen3.8-27B in self-hosted operation. More measurements on local accelerators are in our posts on local AI hardware. What else we offer around local AI operation is under Services.

What we check next

Still open are the 135k needle for hotdogs, tool calls and the needle for asfgsdfg, a twelve-hour endurance run and the quality of the abliterated variants. Until then .202 is an overflow with a failure risk and not a reliable main path. As soon as Qwen 4.0 is out, we want to find out first whether Radiance loads it.

Anyone planning their own R9700 should run each card alone under sustained load and check the power connectors before comparing models.

Further Reading

Questions?
Is the Radeon AI PRO R9700 usable for local language models?+

With the right engine, yes. On our host Qwen3.8-27B reached 59 to 61 tokens per second on one card with Radiance and 89 to 104 on two cards in tensor-parallel mode. The second card dropped off the PCIe bus three times under load. We do not know whether the card, the slot or the power cable is at fault.

Why not vLLM on AMD?+

vLLM v0.33.0 with ROCm 7.2 only ran after a workaround: the Composable Kernel variant of FlashAttention crashed during graph capture, and only the Triton variant worked. After that it reached 19 to 28 tokens per second on one card, Radiance 59 to 61. This holds for our model, our cards and short prompts.

Can abliterated models be used in production?+

We have not checked that. We measured throughput, tool calls and a 135,000-token needle test. Refusal behaviour and quality are untested. For Swift-1.5 the model card names a research-only licence; we did not check the licences of the others. None of them runs in production for us today.