senn-techProxmox HA Cluster: Quorum, Fencing and the End of Single-Host Woes
A single Proxmox host is practical—until it fails. Then everything comes to a standstill. An HA cluster with three nodes is the answer, but it requires understanding a few key concepts. Here’s what matters.
The Minimum Anatomy
Three nodes. With two nodes there is no quorum, and split-brain is preordained. Three nodes can withstand a failure and still decide: the remaining two have a majority, and orphaned VMs are restarted.
pvecm create <clustername>
pvecm add <ip-des-zweiten-knotens>
pvecm add <ip-des-dritten-knotens>
The cluster is ready. But HA needs more than Corosync.
Only Two Nodes? The QDevice
Not everyone has the budget for three full-featured servers. For this case there is the QDevice (corosync-qnetd): an external, lightweight third instance—a small VM or even a Raspberry Pi outside the cluster—that acts as an arbiter, providing an additional quorum vote. This makes a two-node cluster quorate and able to survive the failure of one node. The QDevice stores no VMs; it only casts a vote.
Corosync: The Quorum Wants Redundancy and Low Latency
Corosync is latency-sensitive. It belongs on a dedicated, fast network and ideally on two separate rings (link0/link1 on different NICs), so a single network failure doesn’t immediately topple the quorum.
Fencing: The Underestimated Lifesaver
Fencing means: a failed node is forcibly shut down by the rest of the cluster—power off, reboot. Without fencing, the cluster cannot tell whether a node is truly dead or just experiencing a network hang. A VM that still appears to run on the supposedly dead node, while the cluster starts it on another, causes data corruption.
We use IPMI fencing: the server’s iDRAC/iLO is instructed by the cluster to power the host off and back on. This sounds brutal, but it is the only clean solution.
Shared Storage: Without It, It Doesn’t Work
HA without shared storage is a joke. The VM disk must reside on storage reachable by all nodes. We use Ceph for this—in the storage VLAN with jumbo frames, as described in the network article. Alternatively, ZFS replication works, but it is not synchronous shared storage; it is asynchronous copy.
HA Groups: Placing VMs Purposefully
Via HA groups with priorities, you define on which node a VM preferably runs and where it should migrate in case of failure—for example, to prevent all heavy VMs from landing on the same backup node.
What We Did Wrong
- Too Little Corosync Redundancy: We ran Corosync on the same physical interface as storage traffic. When jumbo-frame issues arose, not only storage but also quorum hung. Now Corosync runs on a second, separate NIC.
- HA Without Testing: After setup we didn’t pull the plug. You should do this exactly once. During the first real incident we discovered fencing didn’t trigger because iDRAC credentials were incorrectly configured.
What HA Cannot Do
It doesn’t catch application errors. If the ERP crashes due to a faulty transaction, no cluster helps. HA handles hardware and host failures—the rest is application knowledge.
Conclusion
A Proxmox HA cluster is the endpoint of the virtualization journey. Three nodes (or two plus QDevice), shared storage, fencing—then VMs keep running even at 3 a.m. when no one is watching. Construction takes one day. Testing takes the next.
Do we really need an HA cluster, or is a single host enough?+
A single Proxmox host is practical — until it fails, then everything stops. An HA cluster is worth it when downtime at night or on weekends isn't acceptable and VMs should restart automatically. However, it only covers hardware and host failures, not application errors: if the ERP crashes on a faulty transaction, no cluster helps. For pure test or off-hours workloads, a single host is often sufficient.
We only have two servers — is HA still possible?+
With two nodes alone there is no quorum, and split-brain is preordained. The solution is the QDevice, an external, lightweight third instance like a small VM or a Raspberry Pi outside the cluster. It stores no VMs but casts an additional quorum vote as an arbiter. This makes the two-node cluster quorate and able to survive the failure of one node.
Why is fencing so important, and what happens without it?+
Without fencing, the cluster can't tell whether a node is truly dead or just experiencing a network hang. A VM that still runs on the supposedly dead node while the cluster starts it on another causes data corruption. We use IPMI fencing: the server's iDRAC or iLO is instructed by the cluster to power the host off and back on. It sounds brutal, but it's the only clean solution.