Ceph HEALTH_WARN: Diagnose and Clear Warnings

HEALTH_WARN is not one condition. It is a bag of named health checks, some of which mean data is at risk right now and some of which mean a scrub is running late. Treating them as equally urgent wastes time; treating them as equally ignorable eventually costs data. The useful skill is knowing which bucket a given check falls into before touching anything.

Every diagnosis starts in the same place:

ceph status
ceph health detail

ceph health detail names the check codes and lists the affected objects, and the code is the thing to look up. Everything below is organised by the order in which those codes deserve attention.

Triage order

1. Availability. Data cannot be read or written. Fix immediately.
2. Durability. Data is readable but has less redundancy than configured. Fix today.
3. Capacity. The cluster is approaching or past a fullness threshold that will block writes or recovery. Fix this week, before it becomes availability.
4. Hygiene. Scrubs, autoscaler recommendations, version skew. Fix deliberately.

The reason capacity outranks hygiene is that a full OSD does not degrade gracefully. Once the full ratio is crossed, writes stop.

Availability: PG_AVAILABILITY

PG_AVAILABILITY means one or more placement groups are in a state that cannot service reads or writes. The documentation lists the problematic states as peering, stale, incomplete, and any absence of active where those states do not clear quickly. Short bursts of peering after an OSD restart are normal; persistent ones are not.

In most cases the root cause is that OSDs are down, which will show as OSD_DOWN alongside it. That check points at three ordinary explanations: a stopped or crashed daemon, a down host, or a network failure that prevents peer OSDs from reaching each other over the public or cluster network. The daemon log under /var/log/ceph/ceph-osd.* is the next stop. To inspect one specific placement group:

ceph tell <pgid> query

OSD_HOST_DOWN and the related OSD_<crush type>_DOWN variants mean an entire CRUSH subtree went away at once, which almost always points at the host or its networking rather than at storage.

Durability: PG_DEGRADED and undersized placement groups

PG_DEGRADED means redundancy is reduced: some placement groups carry fewer replicas or fewer erasure-coded shards than the pool asks for, or have not reached clean for a long time. Again, the usual cause is an OSD being down, and the cluster will normally repair itself once the OSD returns or is marked out and the data is re-replicated elsewhere.

The case that does not self-heal is the one worth recognising on sight: placement groups stuck undersized when nothing is down. That means CRUSH cannot satisfy the pool’s rule against the current topology. A rule asking for three copies in three distinct racks cannot be placed in two racks, and an erasure-coded pool needs at least k+m distinct failure domains to place every shard. The cluster does not fail the pool creation; it simply leaves the affected placement groups permanently short. The fix is topology or rule, not restarting daemons. Both the mechanism and the failure-domain arithmetic are covered in how Ceph places data across CRUSH, pools, and placement groups and, for erasure-coded profiles specifically, in erasure coding versus replication.

Capacity: the three fullness thresholds

Three defaults govern what happens as OSDs fill, and knowing all three explains behaviour that otherwise looks arbitrary:

Threshold	Default	Effect when crossed
mon_osd_nearfull_ratio	0.85	Raises OSD_NEARFULL
mon_osd_backfillfull_ratio	0.90	Backfills will not start
mon_osd_full_ratio	0.95	Clients are prevented from writing

The middle one is the trap. At 90% the cluster stops being able to rebalance or complete recovery onto that OSD, so a capacity problem quietly becomes a durability problem: PG_BACKFILL_FULL and PG_RECOVERY_FULL are exactly this, placement groups that cannot migrate or recover because an OSD is over threshold.

Start with per-OSD utilisation rather than cluster totals:

ceph osd df
ceph df

The documentation is specific that the most full OSD is what matters, not the percentage of raw space used overall, because when ceph df reports space available to a pool it considers the ratios relative to the fullest OSD in that pool. A single outlier can therefore stall writes for an entire pool while the cluster looks half empty.

Flattening the distribution has two documented approaches: reweight-by-utilization, which progressively moves data off excessively full OSDs, and the ceph-mgr balancer module, which does the same job automatically and is the better default on any modern release.

Raising a threshold is a stopgap, not a fix:

ceph osd set-nearfull-ratio <ratio>
ceph osd set-backfillfull-ratio <ratio>
ceph osd set-full-ratio <ratio>

The documented use of set-full-ratio is a short-term restoration of write availability by a small amount, bought so that capacity can be added or data deleted. One easily missed source of reclaimable space: objects left behind by rados bench runs, which can be found with rados ls per pool by looking for names beginning with bench. Delete those very carefully, and only after confirming what they are.

The durable answer is more OSDs in appropriate failure domains, which is a hardware conversation rather than a configuration one; the sizing side of it is in Ceph hardware requirements, and the Ceph storage and erasure coding sizer will show how much usable capacity a given expansion actually adds under the pool’s redundancy scheme.

Hygiene: scrubs that fell behind

PG_NOT_DEEP_SCRUBBED and its shallow counterpart mean placement groups have not been scrubbed within a percentage of osd_deep_scrub_interval, with the percentage set by mon_warn_pg_not_deep_scrubbed_ratio. On large clusters with big placement groups, deep scrubs simply take longer than the default interval assumes, and the documented remedy is to change osd_deep_scrub_interval globally rather than to chase individual placement groups.

One detail explains most confusion here: only placement groups flagged clean are eligible for deep scrubbing. So if a cluster is simultaneously reporting degraded or misplaced placement groups and missed deep scrubs, the scrub warning is a downstream symptom. Clear the degradation and the scrub backlog starts draining on its own. A single placement group can be scrubbed on demand:

ceph pg deep-scrub <pgid>

Deep scrubbing is what detects bitrot, so suppressing the warning without addressing the interval trades a nuisance for a silent risk.

Hygiene: placement group count alerts

POOL_TOO_FEW_PGS fires when a pool holds enough data that its placement group count is producing uneven distribution, and it is only raised when pg_autoscale_mode for that pool is set to warn. The three modes are off, on, and warn:

ceph osd pool set <pool-name> pg_autoscale_mode on
ceph osd pool autoscale-status

autoscale-status is the command worth learning, because it shows each pool’s stored size, its space amplification rate (3.0 for a three-replica pool, 1.5 for a k=4 m=2 erasure-coded pool), the raw capacity available to it, and any recommended new pg_num. That single table answers both “is the placement group count wrong” and “how much raw capacity is this pool really consuming”.

The opposite check, TOO_MANY_PGS, means the cluster is above mon_max_pg_per_osd. Beyond that threshold Ceph refuses to create new pools, increase pg_num, or increase replication, because all three would add more placement groups. High placement group counts also raise OSD memory use and slow peering after any cluster change, which is why placement group count belongs in memory planning and not only in balance planning.

Hygiene: monitors and flags

MON_CLOCK_SKEW means monitor clocks differ by more than mon_clock_drift_allowed. Fix it with chrony or ntpd rather than by raising the threshold, and note the documented constraint: the allowed drift must stay well below mon_lease for the monitor cluster to work at all. Syncing monitors against each other as well as against upstream sources is called out as beneficial, because monitors agreeing with each other matters more than any of them being right in absolute terms.

MON_DOWN means the cluster has lost one or more monitors and is running on a thinner majority. It requires more than half the provisioned monitors to be available, so restoring a downed monitor promptly is what keeps the next failure from being an outage.

OSDMAP_FLAGS is the check that most often reflects an operator rather than a fault. Flags such as noout, nobackfill, norecover, norebalance, noscrub and nodeep_scrub are routinely set during maintenance and then forgotten. A cluster with noout still set will never mark a dead OSD out, so it will never re-replicate its data. All of these except full clear the same way:

ceph osd unset <flag>

Before every maintenance window, write down which flags were set. Before declaring the window closed, unset them.

Muting, done responsibly

Some warnings are expected and temporary, such as version skew mid-upgrade. Ceph supports muting a specific check:

ceph health mute <CHECK_CODE> [duration]
ceph health unmute <CHECK_CODE>

The documentation’s own guidance around DAEMON_OLD_VERSION is the right pattern generally: mute with intent, and unmute as soon as the underlying condition is resolved, so that a future genuine instance is not masked. A permanently muted check is an alert that has been deleted rather than handled.

The shortest version

Read ceph health detail and classify the code before acting. Availability and durability checks usually resolve to an OSD or a host being down; the exception is undersized placement groups with nothing down, which is a CRUSH rule the topology cannot satisfy. Capacity checks are governed by three ratios, and the fullest single OSD, not the cluster average, decides when writes stop. Scrub and autoscaler checks are real signals about long-term health, but they are frequently downstream of a durability problem that should be fixed first.