senn-techA rented GPU instead of our own production: testing vLLM's offload path for Qwen3.8-Flash-Next
After Qwen3.8-Flash-Next was released, one question couldn't be answered from the spec sheet: can vLLM actually offload the model's huge n-gram embedding table to host RAM, as its VLLM_PLE_CPU_OFFLOAD config flag promises? The relevant pull requests were still open, and nobody had publicly confirmed the combination actually loads and runs.
Running that test directly on our own production box would have meant taking the live 27B lane offline for the duration, without knowing in advance whether anything workable would come out of it. Instead, we rented a machine on Vast.ai with the same GPU type and the same PCIe generation as our own.
The rental caused more friction than the actual test
The first attempt failed on a misunderstanding in the Vast.ai interface: the Docker image and tag are separate fields, and picking a tile template doesn't automatically set the right image. The second attempt had the right image but too small an overlay filesystem for a 170 GB checkpoint, because the disk-size field and the volume-attachment mechanism are independent, and a saved template doesn't necessarily carry the volume through when re-picking an offer. Only the third attempt — a machine in Sweden with an EPYC 9755 CPU, 773 GB RAM, and PCIe 5.0 x16 — matched the saved template exactly.
The first crash sat deeper than expected
The NVFP4 checkpoint loaded cleanly: GPU weights in 101 seconds, the embedding table moved into host RAM as designed, host RAM climbing from 42 to 141 GB. The actual offload question was already answered by that point — it works. The crash came one step later, during inference kernel setup:
NotImplementedError: ('Intermediate size padding for w1 and w3, for %s NvFp4 backend,
but this is not currently supported', 'FLASHINFER_CUTLASS')
vLLM's automatic backend selection for NVFP4 MoE layers tries several kernel implementations in order, lands on FLASHINFER_CUTLASS for this card generation, and that one fails on this exact tensor shape instead of falling through to the next option in the chain.
A manual workaround, then a new error
A command-line flag lets you skip the automatic selector entirely: --moe-backend marlin. With it, all four GPU workers loaded cleanly and offload registered on every rank — and a new error appeared, a genuine out-of-memory during CUDA graph capture. The RTX 5090 isn't recognized by marlin's native FP4 support check, likely because that check doesn't yet cover consumer Blackwell cards. As a result, marlin falls back to pure software emulation of 4-bit weights, which uses noticeably more memory per card than a real 4-bit format — enough to exhaust the remaining VRAM needed for graph capture.
Only with --enforce-eager added, skipping CUDA graph capture altogether, did the server come up cleanly: a 200 health check, and a real chat completion with visible reasoning steps.
Working, but slow
Compared head-to-head with our own production lane on the same task, the rented machine reached about 14 tokens/s, the production 27B lane 85.7. That's expected: --enforce-eager gives up the CUDA graph speedup entirely, and marlin's software emulation adds further cost. A second rental with the identical configuration but capped to PCIe 4.0 instead of 5.0 delivered roughly a third less throughput again — a sign that the link itself matters for this model, not just raw compute.
Our existing llama.cpp setup reached about 46 tokens/s over the same period. The comparison isn't fully controlled, since different quantization and different hardware were involved, but the gap is large enough to count as a preliminary result.
What the test settled
The actual question, whether vLLM and PLE offload even load and run together, was answered after the first of three failed attempts, and the answer was yes. Getting it production-ready still needs a working native FP4 kernel for this card generation, without the detour through marlin and without giving up CUDA graphs. Until then, our own production stays untouched on its proven configuration, and the next test can again run on a rented machine instead of our own.
Further reading
What is PLE CPU offload and why does it matter for this model?+
Alongside its MoE weights, Qwen3.8-Flash-Next carries a 51-billion-parameter n-gram embedding table that would normally need to sit in GPU memory for fast access. PLE CPU offload moves that table into host RAM and keeps only the actual compute weights on the card — without it, the model doesn't fit in VRAM on any hardware we have access to.
Why rent a cloud GPU instead of testing locally?+
The test would have taken the production 27B lane offline for its duration, and at the time it wasn't publicly confirmed that PLE offload even worked at all. A rented machine with the same GPU type and the same PCIe link let the test run independently of our own production.
Is vLLM faster than llama.cpp for this model yet?+
Not yet. In our test, the vLLM configuration reached about 14 tokens/s, our existing llama.cpp setup about 46. The comparison isn't fully controlled — different quantization, different hardware — but the gap is large enough to count as a preliminary result.