Ceph Erasure Coding vs Replication: How to Choose

Every Ceph pool is created with a durability scheme, and that choice is the single largest lever on how much of the hardware turns into usable capacity. Replication is the default and stores whole copies. Erasure coding splits each object into data chunks plus computed parity chunks and spreads them wider. The capacity difference is dramatic, the performance difference is real, and one of the two decisions cannot be undone after the pool exists.

The mechanics, briefly

A replicated pool with size=3 writes an object to a primary OSD, which forwards it to two others. Every copy is a complete, directly readable object. Recovery after a failure is a copy operation from a surviving replica.

An erasure-coded pool is defined by a profile with two key numbers: k, the number of data chunks an object is split into, and m, the number of coding chunks computed from them. The object is reconstructible from any k of the k+m chunks, so m is exactly the number of simultaneous chunk losses the pool tolerates. The default erasure-code profile ships as k=2 m=2 with crush-failure-domain=host, which the documentation describes as equivalent in durability to a replicated pool of size three while needing 2 TB rather than 3 TB to store 1 TB.

Two lines of the documentation are worth reading together before copying either. The sample command ceph osd pool create ecpool erasure is introduced as the simplest erasure-coded pool, “similar to RAID5”, requiring at least three hosts. Naming no profile, it gets the default k=2 m=2, which by the k+m failure-domain rule below actually wants four. Name the profile explicitly and count hosts against k+m rather than against the three-host figure attached to the sample.

Space amplification, side by side

The overhead factor of an erasure-coded pool is (k+m)/k, meaning the raw storage consumed per unit of user data. Replication with size=N has an overhead factor of exactly N. Putting the common choices in one table makes the tradeoff concrete:

Scheme	Overhead factor	Usable share of raw	Simultaneous losses tolerated	Minimum failure domains
Replicated, size=2	2.00	50%	1	2
Replicated, size=3	3.00	33%	2	3
EC k=2 m=2 (default)	2.00	50%	2	4
EC k=4 m=2	1.50	67%	2	6
EC k=6 m=3	1.50	67%	3	9
EC k=8 m=3	1.38	73%	3	11
EC k=8 m=4	1.50	67%	4	12

Read the “losses tolerated” column as data survival, not as uninterrupted service. Whether I/O keeps flowing is governed separately by min_size, the number of active replicas or shards a placement group needs before it will serve requests. For erasure-coded pools the documentation recommends min_size be K+1 or greater, precisely so that a pool never keeps accepting writes with zero redundancy left, which means a k=4 m=2 pool stops serving I/O one loss before it would actually lose data. For replicated pools the same logic is why min_size 1 in production is called out as an emergency-only setting.

The last column is the one that quietly disqualifies most small clusters. The documentation states that most erasure-coded deployments require at least k+m CRUSH failure domains, which in practice means hosts or racks, and that there are operational advantages to having at least k+m+1 so the cluster has somewhere to recover to rather than merely somewhere to sit degraded. A 4+2 pool with a host failure domain therefore wants six hosts as a floor and seven to be comfortable. That is a very different bill of materials from a three-node replicated cluster, and it belongs in the hardware planning conversation rather than being discovered afterwards.

Two rows in that table are traps rather than options. Replicated size=2 tolerates one loss with no margin during recovery. And m=1 profiles are strongly discouraged for production data: the documentation warns to expect data unavailability during maintenance and data loss when component failures overlap, because taking one host down for patching consumes the entire tolerance budget.

Where erasure coding costs you

Space efficiency is not free, and the documentation says so in as many words.

Write path. A replicated write touches size OSDs and copies bytes. An erasure-coded write computes parity and touches k+m OSDs. Without the newer optimisations, higher values of k and m both reduce performance, and the guidance for block and file workloads is to keep m no larger than 3.

Partial writes. By default erasure-coded pools only support full object writes, which suits the RADOS Gateway. RBD, CephFS and librados need allow_ec_overwrites set to true on the pool, and that flag is only safe on BlueStore OSDs, because BlueStore checksums are what detect corruption during deep scrubs. The documentation states there is no downside to enabling EC overwrites and that doing so routinely is best practice, on BlueStore.

Layout constraints. Even with overwrites enabled, RBD and CephFS are not fully supported on erasure-coded pools directly. RBD images are created against a replicated pool with the EC pool passed as --data-pool; CephFS keeps metadata on a replicated pool and directs file data to the EC pool through the default data pool or file layouts.

Small objects. Erasure coding amplifies space for very small objects, because chunking and padding overhead is proportionally larger. CephFS and RGW deployments with many tiny files are called out specifically. Since the Tentacle release a per-pool allow_ec_optimizations flag reduces padding and improves small I/O; the default stripe unit is 4K, and with optimisations enabled the documentation recommends raising it to at least 16K for general-purpose I/O, with larger values up to 256KB favouring reads at the expense of small sequential writes. The flag requires all monitors and OSDs on Tentacle or later, cannot be disabled once set, and the stripe unit cannot be changed after pool creation.

Recovery. Rebuilding a lost replica is a copy. Rebuilding a lost erasure-coded chunk requires reading k surviving chunks from k different OSDs and recomputing. Backfill and recovery are heavier, especially on HDDs, which is the phase where a cluster is least able to absorb extra load.

The decision that cannot be reversed

The erasure-code profile of a pool cannot be modified after the pool is created. Changing k or m means creating a new pool with a new profile and moving every object into it. The failure domain is the one part that can be revised in place, by swapping the pool’s CRUSH rule for an otherwise identical rule with a different failure domain, though ceph osd pool ls detail will keep displaying the original profile because crush-failure-domain in the profile is only consulted at creation time.

Practically, this means the profile decision deserves more scrutiny than almost anything else in pool configuration, and it deserves it before the cluster has data. Getting it wrong is not a config change, it is a migration.

A rule for picking one

The documentation’s own fallback is useful: if erasure coding is wanted but the profile is unclear, choose 4+2 or 6+3, which roughly double usable space versus size=3 with a tolerable write and recovery penalty. It also advises against k > 4 or m > 2 unless the ramifications are fully understood, and discourages k > 8 in most cases.

Mapped onto workloads:

• RBD volumes for virtual machines and databases. Replicated, size=3. Small random writes are the worst case for erasure coding, and latency here is user-visible.
• CephFS home directories and general file shares. Replicated metadata pool always. Data pool replicated if the file population skews small, erasure coded if it skews large.
• RGW object storage, backups, media, archives. Erasure coded, 4+2 or 6+3. Large sequential objects are the best case, and the capacity saving is the entire point.
• Mixed clusters. Both, in separate pools on the same cluster, with device classes routing latency-sensitive pools to flash and bulk pools to spinning media.

Before committing, run the numbers. The Ceph storage and erasure coding sizer converts an OSD count and drive size into raw capacity, usable capacity under 3x replication or a 4+2 or 8+4 profile, and the OSD memory the daemons will want, which makes the capacity gap between schemes obvious at the specific scale being planned.

What to check after the pool exists

A new erasure-coded pool that cannot place all its chunks will sit with undersized or degraded placement groups indefinitely rather than failing loudly, because CRUSH simply cannot satisfy a rule that asks for more distinct failure domains than the topology provides. That symptom, and how to distinguish it from an OSD that is merely down, is covered in diagnosing and clearing Ceph HEALTH_WARN. The underlying reason a rule can be unsatisfiable at all is worth understanding first, in how Ceph places data across CRUSH, pools, and placement groups.

Erasure coding is the right default for bulk capacity and the wrong default for latency. The documentation’s own caution is the sentence to keep: do not mistake erasure coding for a free lunch. The capacity is real, and so is the bill.