Read Ceph HEALTH_WARN causes with ceph health detail

2026-10-19·Ceph docs (Tentacle) · 2026-10·7 steps

Read HEALTH_WARN codes with ceph health detail, learn six common codes and the first command for each, and mute a known warning for a set time.

Ceph reports problems as upper-case codes such as OSD_DOWN. Read the code first and you know where to look. Every command here is read-only; run them on a node that has an admin keyring.

[1] Check the overall cluster status. If the health: line is not HEALTH_OK, continue.

console

# ceph -s

Same as ceph status. Under services: check the mon and osd counts, and on the pgs: line of data: look for PGs that are not active+clean.

[2] List the warning codes and what they refer to. Each code gets one entry, followed by the OSDs, PGs or daemons involved.

console

# ceph health detail

The documentation shows the shape of an entry. A muted OSD_DOWN entry lists osd.1 is down under it.

output

HEALTH_OK (muted: OSD_DOWN)(MUTED) OSD_DOWN 1 osds down    osd.1 is down

[3] OSD_DOWN and PG_DEGRADED usually appear together. OSD_DOWN means one or more OSDs are marked down: the ceph-osd daemon or its host stopped, or peers cannot reach it over the network. Find the OSD, then check the host, the daemon and the network in that order.

console

# ceph osd tree down

PG_DEGRADED means some data has fewer replicas (or erasure-code fragments) than desired, most often because of a down OSD. Inspect one PG with:

console

# ceph tell <pgid> query

If the OSD daemon crashed, read /var/log/ceph/ceph-osd.* on its host. Once the OSD is back, PG_DEGRADED normally clears as recovery finishes.

[4] OSD_NEARFULL is an early warning: one or more OSDs passed the nearfull threshold. Check usage before the cluster fills up.

console

# ceph df

POOL_NEARFULL means a pool is approaching a configured limit; pool quotas, mon_osd_nearfull_ratio and mon_osd_full_ratio all play a part. View and change a pool quota as below. A value of 0 disables the quota.

console

# ceph osd pool get-quota <pool># ceph osd pool set-quota <pool> max_bytes <bytes>

Check the pool's current usage with ceph df before changing a quota.

[5] MON_CLOCK_SKEW means the clocks on hosts running Monitors differ by more than mon_clock_drift_allowed. The documented fix is to synchronize the clocks with ntpd or chrony. Check the sync state on each monitor host.

console

# chronyc tracking

Raising mon_clock_drift_allowed would hide the warning, but the documentation says there are very few occasions when that is appropriate. Fix the clocks first.

[6] RECENT_CRASH means a daemon crashed recently and the crash has not been acknowledged (archived). List new crashes and read one in full.

console

# ceph crash ls-new# ceph crash info <crash-id>

Once you know the cause, archive the report to clear the warning, one at a time or all at once.

console

# ceph crash archive <crash-id># ceph crash archive-all

Archived crashes still show in ceph crash ls but not in ceph crash ls-new. "Recent" is set by mgr/crash/warn_recent_interval, which defaults to two weeks.

[7] Mute a warning you understand and can leave for now. With a duration, the mute expires on its own.

console

# ceph health mute OSD_DOWN 4h# ceph health mute MON_DOWN 15m

If nothing else is wrong, the status becomes HEALTH_OK, and ceph health lists the mute.

output

HEALTH_OK (muted: OSD_DOWN)

Remove a mute with:

console

# ceph health unmute OSD_DOWN

When the muted problem is resolved, the mute goes away; if the problem returns it is reported as usual. Add --sticky to keep the mute after the check clears (for example ceph health mute OSD_DOWN 1h --sticky). Per the documentation, mutes on checks with a threshold are dropped when things get worse, such as more OSDs going down.

Scope: No Ceph cluster was available, so no command was run. Commands and example output come from the documentation above; ceph osd tree down, ceph osd pool get-quota and chronyc tracking are read-only commands for the same purpose, not quotes from those pages.

#ceph#health