Ceph Hardware Requirements: CPU, RAM, Disks, Network

Most Ceph hardware questions are really one question asked in four different ways: how much of this machine does the storage software take before anything useful happens? Ceph is designed for commodity hardware, but “commodity” is not the same as “whatever is in the spare rack”. The daemons have specific, published appetites, and a cluster built below them does not fail loudly on day one. It fails months later, during a recovery, when the margin that was never budgeted turns out to be the thing that keeps the cluster serving reads.

This is a walk through what the project documentation actually specifies, what those figures mean in practice, and which of them are worth exceeding.

Start with node count, not part numbers

Before any component decision, decide how many failure domains the cluster will have. That number constrains everything else, and no amount of good hardware compensates for getting it wrong.

Three hosts is the practical floor for a cluster that survives losing a machine, because the default replicated pool keeps three copies and a host-level failure domain requires three distinct hosts to place them. The documentation’s simplest erasure-coded pool likewise requires at least three hosts. Monitors want an odd count so a majority can always be formed; the documentation is explicit that the cluster needs more than half of the provisioned monitors available, so three monitors tolerate one loss and five tolerate two.

Four hosts is materially better than three, because at three the cluster has nowhere to re-replicate to when a host dies. It stays degraded until you fix the hardware. That distinction, not raw capacity, is usually what separates a homelab cluster from something that can be left alone over a weekend. The way copies are distributed across those hosts is decided by the CRUSH rule, which is covered in how Ceph places data across CRUSH, pools, and placement groups.

CPU: the metric changed

Older Ceph guidance was expressed as cores per OSD. The current hardware page says plainly that this metric is no longer as useful as cycles per I/O and IOPS per OSD, and advises selecting for IOPS per core. The reason is that a spinning disk saturates long before a modern core does, while an NVMe device can absorb far more work than one core can generate. The documentation notes that on real clusters Ceph can easily use five or six cores for a single NVMe OSD, and up to roughly fourteen cores for a single OSD in isolation.

The published minimums, stated as threads rather than physical cores because hyperthreading is treated as beneficial for Ceph, are:

Daemon	Processor guidance
ceph-osd (HDD)	1 thread minimum, 3 recommended per OSD
ceph-osd (NVMe SSD)	4 threads minimum, 6 recommended per OSD
ceph-mon	2 cores minimum
ceph-mds	2 cores minimum, higher clock preferred over more cores

Two qualifications matter. First, those OSD figures are stated as being before replication, and erasure coding and compression push them up, because parity computation is real CPU work on every write and on every degraded read. If the plan is an erasure-coded pool, budget above the recommended column rather than at it; the tradeoff is laid out in erasure coding versus replication. Second, the metadata server is single-threaded and benefits from clock rate, so a high-core low-clock part is the wrong shape of CPU for CephFS metadata.

RAM: the number that people underestimate

BlueStore does not lean on the operating system page cache. It manages its own memory, governed by osd_memory_target, which defaults to 4 GiB per OSD daemon. The documentation is direct about the shape of that curve:

• Below 2 GB is not recommended, and extremely slow performance is described as likely.
• Between 2 GB and 4 GB typically works but may degrade performance, because metadata has to be read from disk during I/O unless the working set is small.
• 4 GB is the default, chosen to balance RAM cost against OSD performance.
• Above 4 GB helps when there are many small objects or large per-OSD data sets, especially on fast NVMe.
• An effective target of at least 6 GiB is described as helping to mitigate slow requests on HDD OSDs.

The sizing rule for a whole server is the part worth writing down: total server RAM greater than the number of OSDs multiplied by osd_memory_target multiplied by two. The doubling covers the operating system, other Ceph daemons, and the fact that memory use rises during recovery and rebalancing rather than during the calm period you sized against. The documentation’s own worked example is a 1U server with eight to ten OSDs being well provisioned with 128 GB.

On top of that, roughly 20% extra system memory is suggested as headroom, because OSD memory management is explicitly best-effort: the daemon may unmap memory without the kernel reclaiming it promptly. Swap is discouraged on modern systems, on the reasoning that a crashed daemon is easier to live with than one that slows to a crawl.

Monitor and manager memory scales with cluster size: 32 GB suffices for very small clusters, 64 GB for clusters up to roughly 300 OSDs, and 128 GB beyond that. Metadata servers want at least 8 GiB per daemon, with cache behaviour controlled by mds_cache_memory_limit. Placement group count feeds into this too, since OSD memory consumption is related to the number of placement groups a daemon serves, which is one reason an unmanaged PG count is a memory problem as well as a balance problem.

Drives: one OSD per device, and a floor on size

The default assumption is one storage drive per OSD. The documentation recommends a minimum OSD size of 1 TiB, on the grounds that much smaller devices spend a significant fraction of their capacity on metadata. At the other end, PCIe Gen 4 and newer SSDs larger than 30 TB may benefit from being split into two or more OSDs, because a single OSD daemon becomes the bottleneck before the device does.

For hybrid nodes, the DB/WAL offload ratios are specific: one SSD partition per HDD OSD, four to five HDD OSDs per SATA SSD carrying DB/WAL, and no more than fifteen HDD OSDs per NVMe SSD. Exceeding those ratios turns the offload device into a shared bottleneck and a shared failure domain, since losing it takes down every OSD that depends on it.

Monitors want about 100 GB per daemon, and the documentation strongly urges SSD for that, because monitor stores are latency-sensitive and a slow monitor store affects the whole cluster. Where a node has a single drive, the guidance is to keep the OS on a separate partition at minimum and separate drives by preference. Enterprise-grade media is called out specifically for production workloads, which is a polite way of saying consumer SSDs without power-loss protection are a poor fit for a system that fsyncs constantly.

Network: 10 Gb/s is the starting line

The recommendation is at least 10 Gb/s between Ceph hosts and between clients and the cluster, 25 Gb/s for substantial workloads, and 100 Gb/s links for dense nodes. Active/active bonding across separate switches is strongly recommended, both for throughput and to survive a switch failure or a maintenance window, with attention to whether the bonding hash policy actually spreads traffic.

The documentation frames the cost of getting this wrong as recovery time rather than throughput, which is the more useful framing. Replicating 1 TiB across a 1 Gb/s network takes about three hours; 10 TiB takes about thirty. On a 10 Gb/s network the same transfers take about twenty minutes and about three hours. Recovery speed is a durability property, not a convenience one: the faster a placement group returns to active and clean, the smaller the window in which a second overlapping failure can cost data.

One subtlety worth planning for: a 40 Gb/s link is effectively four 10 Gb/s channels and a 100 Gb/s link is effectively four 25 Gb/s channels, so a single packet on 25 Gb/s can have marginally lower latency than on 40 Gb/s. Also budget a cheap separate 1 Gb/s path for out-of-band management, since baseboard management controllers rarely offer faster than 1 Gb/s and there is no reason to spend expensive switch ports on them.

Turning specifications into a bill of materials

Two numbers decide the shape of a first cluster: how much usable capacity is needed, and which durability scheme delivers it. Raw capacity is not usable capacity, and the gap is large. A three-replica pool has a space amplification factor of 3.0, so a third of raw capacity is usable. A 4+2 erasure-coded pool has a factor of 1.5, so two thirds is usable, at a real cost in write latency and CPU.

The Ceph storage and erasure coding sizer on this site takes an OSD count, a drive size, and a redundancy profile and returns raw capacity, usable capacity, and the OSD memory the daemons will want, which is the quickest way to check whether a proposed parts list actually meets a capacity target before anything is ordered.

Then sanity-check the result against the software prerequisites, which are modest but non-negotiable: cephadm needs Python 3, systemd, Podman or Docker, LVM2, and working time synchronisation on every host. Clock discipline is not optional in a quorum-based system, and skewed monitor clocks are one of the recurring health warnings covered in diagnosing and clearing Ceph HEALTH_WARN.

The mistakes that cost the most

Building on exactly three hosts and treating that as fault tolerant, when it only tolerates a failure rather than recovering from one. Sizing RAM against the 4 GiB default without the doubling rule, then discovering the shortfall during a rebalance when every OSD wants more than its steady-state footprint. Running a cluster network at 1 Gb/s because the throughput seemed adequate, and finding out during a disk failure that recovery now takes a day. Loading fifteen or more HDD OSDs behind a single DB/WAL device. Choosing consumer SSDs on price and inheriting their fsync behaviour. And sizing capacity from raw terabytes rather than from the durability scheme that will actually be configured.