Storage Journey, Part 3: Arrived at DRBD & LINSTOR

After Ceph (Part 1) and Vitastor (Part 2) we have arrived at DRBD and LINSTOR: and stayed.

Why We Stayed With DRBD & LINSTOR

DRBD/LINSTOR gives us the best of what we tested: low latency on our own hardware, true high availability, clean Proxmox integration, and a commercial support path via LINBIT. Exactly this mix is what we were looking for after the journey through Ceph and Vitastor, and a direct comparison with real numbers confirmed the decision.

In our own load test with 100,000 mixed SQL write and read operations, DRBD and Vitastor came out practically even: both around 200 seconds. Broken down, that's roughly 2 ms per operation, of which only about 0.15 ms is the synchronous storage replication itself, DRBD mirrors every write in real time to the second site; made possible by a dedicated 2×100 Gbit link between our two sites. The rest is database overhead. Projected onto Ceph, with typical all-NVMe RBD latency (Ceph's block storage layer) of 1–2 ms, the same workload would have taken roughly 40–90% longer.

We didn't drop Vitastor for speed, but for the maturity of its write path: in our own crash-consistency testing we found that the version under test had only fixed fsync bugs (bugs in reliably writing data to disk) in its default store about five weeks earlier, and that development is concentrated almost entirely in one person, over 2,900 commits against fewer than ten for the second-most-active contributor. For production customer data, that was too early for us. A plain ZFS-over-iSCSI setup was ruled out too: single-headed, no built-in replication, no clean way to make it highly available across sites, and running iSCSI over the link between our two sites would have wiped out exactly the low latency DRBD gives us today.

What DRBD is

DRBD is a shared-nothing, replicated storage solution: It mirrors the content of block devices in real time between hosts. Technically, DRBD sits as a kernel module at the very bottom of the I/O stack; directly above the backing device, transparent to the layers above.

Real-time & synchronous: Write operations are considered complete only when they reside on all connected nodes (synchronous mode), or locally, with asynchronous replication over distance.
Shared-nothing: no central SAN, no single point of failure.
Quorum & Split-Brain Handling: DRBD comes with its own quorum (typically three replicas or a diskless tiebreaker), which precisely mitigates the classic HA pitfalls.

DRBD: Block-Replikation zwischen Knoten, shared-nothing, kein zentrales SAN

LINSTOR as Orchestration

Manually managing individual DRBD resources doesn't scale. LINSTOR manages the volumes cluster-wide, and the LINSTOR-Proxmox-Plugin integrates the whole directly into Proxmox:

drbd: drbdstorage
    resourcegroup defaultpool
    content images,rootdir
    controller 10.11.12.13

This allows active VMs to be live-migrated in seconds, without downtime and without central SAN. Via resource groups, the replica count is set centrally. Combined with a Proxmox-HA-Cluster this yields a full-fledged, SAN-free HA platform.

Operational Tip

When LVM, DRBD and LINSTOR work together, the global_filter belongs in lvm.conf:

global_filter = [ "r|^/dev/drbd|", "r|^/dev/mapper/[lL]instor|" ]

Otherwise high CPU load or hanging LVM operations loom, because LVM scans the DRBD devices as well.