senn-tech4× RTX 5090: When DRAM Bandwidth Is the Real Bottleneck
Our inference host kipc5090 (192.168.180.3) carries four RTX 5090 GPUs, each with 32 GB VRAM — 128 GB of graphics memory in total. On paper, enough to run almost any open model locally. In practice, the numbers point to an unexpected limit.
The setup
- Motherboard: ASRockRack SIENAD8-2L2T
- CPU: AMD EPYC 8024P (8 cores)
- GPUs: 4× NVIDIA GeForce RTX 5090, 32 GB GDDR7 each
- Driver: NVIDIA 580.173.02, CUDA 13.0
- Memory: 2× 16 GB DDR5-4800 ECC UDIMM
- Operation: vLLM with tensor-parallel size 4 across all four cards
The current workload is Qwen3.8-27B-FP8, running as a consolidated lane for text, reasoning, and vision.
No P2P
The RTX 5090 is a consumer card. It has no NVLink, no NVSwitch, no peer-to-peer data transfer. nvidia-smi topo shows NS or CNS — the cards cannot talk directly to each other.
What does that mean for tensor-parallel operation? Every all-reduce, every gradient synchronization, every inter-GPU activation exchange must travel through host memory:
GPU 0 → PCIe → Host memory → PCIe → GPU 1
A reduce step therefore costs at least two bus traversals. That is slower than NVLink by large factors, but still acceptable with fast host memory.
The host memory is half empty
The real problem is not the GPUs, but the host memory population. The board offers six DDR5 channels, but only two 16 GB modules are installed. That means only two of six channels are active.
We measure about 110 GB/s instead of the roughly 230 GB/s possible with all six channels populated. The value is reproducible — and it is why tensor-parallel across four cards does not reach the throughput that raw GPU power would suggest.
Why MoE models hurt especially
A dense model like Qwen3.8-27B keeps its weights in GPU VRAM and communicates mainly activations between cards. A MoE model with large expert routers and additional embedding tables moves far more data. Qwen3.8-Flash-Next with its 51 billion n-gram embeddings would be a typical candidate that stresses the host memory bus more heavily.
Prefix caching also benefits from fast host memory: the managed KV cache must be allocated and freed quickly. A slower memory bus shows up as higher latency when starting new requests.
What we learned
- Measure first, buy later: before considering a fifth GPU, we measured. The result: more GPUs will not help if the bus is the bottleneck.
- Memory channels are not optional: in multi-GPU inference with tensor parallelism, the host memory bus is part of the critical path. Every unused channel is lost bandwidth.
- Consolidate before expanding: a single well-connected GPU is often more efficient than four cards forced to communicate over a halved bus. For our current vision and reasoning lane, the existing configuration is sufficient; for larger MoE models, a memory upgrade is the next logical step.
Our take
The RTX 5090 is an impressive consumer card, but it was not designed as a multi-GPU inference building block. Anyone using it that way has to look at the entire data path — and that path leads through the host memory bus. In our case, halved DRAM bandwidth costs more throughput than missing GPU cores. The next sensible upgrade is therefore not a chip, but memory.
Further reading
Why does DRAM bandwidth matter with 4× RTX 5090s?+
Tensor-parallel operation across four GPUs requires constant data exchange. Because RTX 5090s do not support peer-to-peer transfer, every all-reduce goes through host memory. When the host memory bus only uses two of six DDR5 channels, available bandwidth drops to roughly half — and that becomes the bottleneck, not the CUDA cores.
What does 'no P2P' mean for tensor parallelism?+
Professional cards like H100 or A100 exchange data directly via NVLink. The RTX 5090 lacks NVLink; inter-GPU communication goes over PCIe into host memory and back out. Every all-reduce step therefore costs two bus traversals — one write, one read.
How much bandwidth do we lose with only two populated channels?+
We measure about 110 GB/s instead of the roughly 230 GB/s possible with all six DDR5-4800 channels populated. That is not a theoretical value; it is a real throttle that shows up in latency and throughput — especially with MoE models that move a lot of data between host and device.
Is upgrading to six memory channels worth it?+
For our current Qwen3.8-27B-FP8, the bottleneck is noticeable but not fatal. If Qwen3.8-Flash-Next or other MoE models with high memory traffic join the stack, populating the remaining memory channels will be the cheapest upgrade — far cheaper than adding more GPUs.