senn-techQwen3.8-27B Self-Hosted: Running Locally on 4× RTX 5090
Alibaba released the weights of Qwen3.8-27B on 13–14 August 2026 under the Apache 2.0 license: a dense 28-billion-parameter model with a native 262k context window, multimodal, and — crucially for us — self-hostable. We performed the cutover on 18 August.
What ran before
Our inference host kipc5090 (192.168.180.3) previously ran a Qwen3.5-122B-A10B-NVFP4 as a Mixture-of-Experts model. That sounded like more capacity, but in practice it was neither faster nor more effective: in a direct A/B test on real work tasks, both models tied at 33/34 and 19/22 correct solutions, respectively. Meanwhile, the 122B model produced significantly more reasoning tokens and only activated around 10 billion parameters per token, compared to the new dense model's 27 billion.
The new configuration
The model runs as ki_training-qwen38-vllm with vLLM 0.27.1 and tensor-parallel size 4 across all four RTX 5090s:
command:
- "--model"
- "/models/Qwen3.8-27B-FP8"
- "--served-model-name"
- "thinking"
- "text"
- "nothink"
- "vision"
- "gemma"
- "gemma-4"
- "qwen3.8-27b"
- "--tensor-parallel-size"
- "4"
- "--kv-cache-dtype"
- "fp8_e4m3"
- "--max-model-len"
- "262144"
- "--max-num-seqs"
- "64"
- "--gpu-memory-utilization"
- "0.92"
- "--enable-prefix-caching"
- "--enable-prompt-tokens-details"
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "qwen3_xml"
- "--reasoning-parser"
- "qwen3"
The main differences from the previous lane: --reasoning-parser qwen3 cleanly extracts thinking tokens; --tool-call-parser qwen3_xml matches the Qwen 3 XML format (not the 122B's coder parser); and --enable-prompt-tokens-details combined with --enable-prefix-caching splits prompt usage into cached and new tokens.
Memory and throughput
Each card shows around 30.6 GB of occupied VRAM under load. That fits: the FP8 weights of the 27B model occupy about 31 GB, with the rest going to KV cache and scheduling reserve. After four days of continuous operation, the server had processed roughly 78 million prefill tokens and around 5.9 million generated tokens — so the model is actively used, not idling.
Measured average generation throughput is about 96 tokens/s, with prompt-throughput peaks around 16,000 tokens/s. The prefix cache hits in roughly 43–45 % of cases.
Gateway and lanes
Via the LiteLLM gateway at 192.168.180.2:4000, two chat lanes are publicly exposed:
- thinking: reasoning enabled, default route for complex tasks.
- nothink: same endpoint, but with
enable_thinking: false— for tasks where unnecessary reasoning just burns tokens.
A third internal lane, thinking-cnc, is a nothink clone for a specific production agent so its session is not blocked by overthinking. Vision aliases are now served from the same host; the separate vision card at 192.168.180.202 was retired on 19 August.
Our take
Qwen3.8-27B is the model we expected for the local stack: dense, hostable, Apache-licensed, and good enough to replace the previous 122B MoE. The gain is not a higher task-success rate but lower complexity: one lane instead of several, one parser instead of two, a clear upgrade path. If you self-host, the switch is worth evaluating — provided the hardware fits.
Further reading
Why replace a 122B MoE with a 27B dense model?+
In a real-world A/B test on actual work tasks, the old Qwen3.5-122B-A10B-NVFP4 tied with Qwen3.8-27B-FP8 — roughly equal task success rates. The difference: the 27B model emits 3-11x fewer reasoning tokens, activates 27 billion parameters per token instead of only 10 billion, and simplifies infrastructure by running on a single host lane.
Does a 27B model really fit on four 32 GB RTX 5090s?+
Yes. The FP8 weights take around 31 GB, with the rest going to KV cache and headroom. With --gpu-memory-utilization 0.92 and fp8_e4m3 KV cache, there is enough room for 64 concurrent sequences and contexts up to 262,144 tokens. BF16 would not fit; FP8 is the right precision here.
Which endpoints does the gateway expose?+
vLLM advertises thinking, text, nothink, vision, gemma, gemma-4, and qwen3.8-27b. The LiteLLM gateway at 192.168.180.2:4000 only exposes thinking and nothink publicly; nothink explicitly disables reasoning via chat_template_kwargs. The vision family was consolidated from the separate vision box onto the same 4×-5090 lane on 19 August 2026.
What does the switch mean for data sovereignty?+
Nothing changes in principle: text, images, and documents never leave the local network. Consolidating to one host actually simplifies logging and monitoring. Prompt-token details (--enable-prompt-tokens-details) even show transparently which parts of the prompt were reused from the KV cache.