senn-techStorage Journey, Part 1: What Ceph Taught Us
When switching to Proxmox, the storage question arose immediately: no more central SAN, but software-defined storage on the nodes themselves. The obvious starting point was Ceph — Proxmox integrates it directly.
What Ceph Does Right
Ceph is a distributed object, block, and file system with self-healing and arbitrary scalability. Three properties convinced us:
- No Single Points of Failure — data is spread across many OSDs via CRUSH.
- Scale-out — add nodes, capacity grows with them.
- Block Storage (RBD) integrates cleanly with Proxmox.
The Hardware Truth
Ceph rewards good hardware and punishes stinginess mercilessly. Three things are non-negotiable:
- Network: Ceph only really gets going with 10 GbE. A dedicated storage network (separate public and cluster networks) and end-to-end jumbo frames (MTU 9000) are mandatory — otherwise, even a single rebalancing operation can saw through VM performance.
- Storage Media: Enterprise SSDs/NVMe with Power-Loss Protection. Consumer NVMe relies on SLC caching and collapses once the cache is full — and Ceph generates exactly the small, random writes and metadata operations that fill this cache.
- RAM and CPU: Each OSD daemon wants several gigabytes of RAM. When a node fails, CPU loads spike because data must be redistributed (backfill/recovery).
Replication and Failure Domains
The default is size=3 / min_size=2: three copies, writes allowed with at least two copies available. The failure domain should be at the host level so each replica lands on a different server. Three nodes are the absolute minimum — Ceph only feels comfortable with significantly more.
Where It Hit a Snag for Us
Ceph is built for large clusters. In smaller setups, the cost shows:
- Resource Hunger: RAM and CPU per OSD add up; the recommendation of many nodes is real, not academic.
- Latency: For very write-intensive, latency-sensitive VM workloads, write latency was higher than desired — without a dedicated 10 GbE storage network, the plateau was reached quickly.
- Operational Overhead: Recovery and rebalancing operations need to be understood and monitored —
ceph -sand Prometheus/Grafana are mandatory, not nice-to-have.
Conclusion
Ceph was no mistake — it taught us to think in distributed storage. But for our node count and latency profile, we sought something leaner and faster. That led us to Vitastor — to Part 2.
What hardware does Ceph really demand?+
Ceph punishes stinginess mercilessly. Non-negotiable are at least 10 GbE with a dedicated storage network and end-to-end jumbo frames (MTU 9000), enterprise SSDs or NVMe with power-loss protection — consumer NVMe collapses once the SLC cache is full — plus several gigabytes of RAM per OSD daemon. When a node fails, CPU load also spikes from backfill.
Are three nodes enough for a Ceph cluster?+
Three nodes are the absolute minimum, with size=3 and min_size=2 and a failure domain at host level so each replica lands on a different server. But Ceph is built for large clusters and only feels comfortable with significantly more nodes. In small setups you pay the price in resource hunger.
Why did you move away from Ceph?+
Ceph was no mistake; it taught us distributed storage. But for our node count and latency profile, write latency for write-intensive, latency-sensitive VMs was higher than desired. Add to that resource hunger per OSD and the operational overhead of recovery and rebalancing. We sought something leaner and faster — which led us to Vitastor.