How Ceph Places Data: CRUSH, Pools, and Placement Groups

Ceph is a distributed storage system that presents object, block, and file interfaces over one cluster. What makes it behave differently from a traditional array is that no central component holds a table of where each piece of data lives. Clients compute the location. Understanding that computation explains most of Ceph’s operational behaviour.

The pieces

An OSD, or object storage daemon, manages one storage device and is the unit of failure and capacity. Monitors maintain the authoritative cluster map and the consensus that keeps it consistent. Managers handle metrics and orchestration. A metadata server is added only when the file interface is used.

Data is written into pools. A pool is a logical namespace with its own durability scheme, its own placement rules, and its own placement group count. Block volumes, object buckets, and file data all end up as objects inside pools.

Placement groups sit between objects and devices

Ceph does not map each object directly to devices. Objects hash into placement groups, and placement groups map onto sets of OSDs. That indirection is what keeps the mapping cheap to compute and recovery manageable, since the cluster tracks and repairs a bounded number of placement groups rather than an unbounded number of objects.

Placement group count is a real tuning decision. Too few and data distributes unevenly, leaving some OSDs much fuller than others while capacity sits idle elsewhere. Too many and monitors and OSDs carry more peering state and memory overhead than they need to. Ceph includes an autoscaler that adjusts counts based on how much data a pool actually holds relative to the cluster, and using it is usually better than a hand picked number that stops being right as the cluster grows. Both extremes surface as named health checks, POOL_TOO_FEW_PGS and TOO_MANY_PGS, and what to do about each is set out in diagnosing and clearing Ceph HEALTH_WARN.

CRUSH turns a name into a location

CRUSH is the algorithm that maps a placement group onto specific OSDs. It takes the cluster topology, expressed as a hierarchy of devices inside hosts inside racks inside rooms, plus a set of rules, and deterministically computes which OSDs should hold a given placement group. Every client runs the same algorithm against the same map and gets the same answer, so there is no lookup service in the data path and no metadata bottleneck.

Because CRUSH is computed rather than looked up, the shape of the physical cluster is an input to correctness and not merely to performance. How many hosts, how they are racked, and what each one is built from is the first decision rather than the last; the published figures for OSD processors, memory, drives and networking are collected in Ceph hardware requirements.

The most consequential thing in a CRUSH rule is the failure domain. It states the level of the hierarchy across which copies must be separated. With host as the failure domain, no two copies of the same data land on the same server, so losing a server never loses all copies. With rack as the failure domain, copies are spread across racks and a rack level power or switch failure is survivable. The failure domain must be matched by real capacity: a rule demanding three copies in distinct racks cannot be satisfied by two racks, and the affected placement groups will simply stay undersized.

Device classes let one cluster hold mixed media and route pools to the right tier, with rules selecting only flash or only spinning devices.

Replication or erasure coding

Replicated pools store whole copies. Writes go to a primary OSD which forwards to the others, and reads are served locally. This costs the most raw capacity per usable byte and recovers quickly, because recovery is a copy.

Erasure coded pools split an object into data chunks plus computed parity chunks, spread across more OSDs. Usable capacity per raw byte is much better, but every write and every degraded read involves computation and touches more devices, so latency and CPU cost rise and recovery is heavier. Erasure coding suits large sequential workloads such as object storage and backups. Small random writes are the worst case for it.

The choice is worth making with numbers rather than instinct, because the profile cannot be altered after a pool is created. Erasure coding versus replication sets out the space amplification of each common profile, how many failure domains each one demands, and where the write penalty actually lands. To see what a specific parts list yields, the Ceph storage and erasure coding sizer converts an OSD count and drive size into raw capacity, usable capacity, and the memory the OSD daemons will want.

Common mistakes

Setting a failure domain the physical layout cannot satisfy and leaving placement groups permanently degraded. Treating near full OSDs as a warning to be dismissed, when they block writes and stall recovery. Building a cluster with too few failure domains to tolerate the loss of one. Putting latency sensitive block workloads on erasure coded pools. Ignoring uneven OSD utilisation instead of investigating placement group counts and device weights.