How to Add an OSD to Ceph Safely with cephadm

If you are searching for how to add an osd to ceph, the short answer is: verify the disk twice, let the cluster’s existing orchestrator create the OSD, and monitor the rebalance. The command is easy. Avoiding the wrong disk and a badly timed recovery is the actual job.

This guide uses cephadm, the normal choice for a current standalone Ceph cluster. The target disk will be erased.

Who this is for / who should skip

Use this procedure if the cluster is already managed by cephadm, is healthy enough to rebalance, and has a new local block device on an enrolled host. Run the commands inside cephadm shell or from a host with the admin configuration and keyring.

Skip it if Rook or Proxmox manages Ceph. Those systems have their own reconciliation loops. Rook watches the CephCluster resource, while Proxmox creates OSDs through its UI or pveceph; using cephadm beside either is configuration drift with extra steps. Ceph’s manual OSD documentation likewise warns that manual procedures conflict with cephadm.

Also skip Ceph for one N100 mini-PC plus one NAS unless the point is learning Ceph. One host is not a useful host failure domain. ZFS or the NAS’s native storage stack is less glamorous and usually the saner weekend.

Hardware that works

Ceph expects a locally attached block device. Its hardware guidance recommends separate OS and OSD storage and, in most cases, one storage device per OSD.

A GMKtec NucBox G3 is reasonable lab hardware because the vendor lists separate M.2 NVMe and M.2 SATA slots plus 2.5GbE. Once one device holds the OS, however, it is still basically a one-OSD node. A Beelink S12 Pro is also a lab box, but its listed 1GbE interface makes recovery traffic the obvious constraint.

A Synology DS923+ is better used as the backup target. Synology documents four SATA bays, dual 1GbE, and an optional 10GbE upgrade in its DS923+ data sheet. Do not turn an NFS or SMB share from it into an OSD. That stacks a distributed storage system on a network filesystem and makes failures impressively confusing.

Check the cluster and identify the disk

Start with read-only checks:

ceph -s
ceph df
ceph osd tree
ceph orch host ls
ceph orch device ls --hostname ceph-03 --wide --refresh

Replace ceph-03 with the real host. On that host, compare the model, size, serial, filesystem, and mount state:

lsblk -o NAME,SIZE,MODEL,SERIAL,FSTYPE,MOUNTPOINTS

Stop if the cluster has unexplained degraded or inactive PGs, lacks free space for recovery, or the serial does not match the drive in your hand. According to the cephadm device rules, an available disk has no partitions, filesystem, mount, LVM state, or existing BlueStore OSD. --wide shows why Ceph rejected a device. Do not “fix” that with a blind zap.

The stack: a narrow OSD service spec

There is no docker-compose.yml here. Cephadm already owns the containers, and Watchtower should not get a vote. Create osd-ceph03.yaml, changing both the host and device to values you verified above:

service_type: osd
service_id: ceph03_sdb
placement:
  hosts:
    - ceph-03
spec:
  data_devices:
    paths:
      - /dev/sdb

Preview the declarative spec, read the proposed host and data device, then apply the same file:

ceph orch apply -i osd-ceph03.yaml --dry-run
ceph orch apply -i osd-ceph03.yaml

For one deliberately selected disk, this is safer than ceph orch apply osd --all-available-devices. Every applied spec is persistent: this one remains limited to the named host and path, while the broader command can consume any future available drive. The official OSD service guide documents both the dry run and cephadm’s ongoing reconciliation behavior.

Verify the OSD and rebalance

Watch Ceph rather than immediately starting another maintenance job:

ceph -w
ceph osd tree
ceph osd df tree
ceph orch ps --daemon_type osd --hostname ceph-03 --refresh

The new OSD should become up and in, appear beneath the correct host in the CRUSH tree, and have the expected hdd, ssd, or nvme class. Temporary remapped, backfill, or recovery states are normal after adding storage. The job is finished when the OSD and PG checks settle and the cluster returns to its pre-change health baseline, normally with PGs active+clean.

Storage and backups

Adding an OSD increases capacity; it does not create a backup. Before the change, make sure critical RBD, CephFS, or RGW data has an off-cluster copy. For the small homelab described here, a practical target is a versioned backup repository on the Synology NAS, with a second copy off-site if the data matters.

The restore drill is simple and non-negotiable: restore one directory, object set, or VM image to a scratch destination, open it, and verify its checksum or application boot. A successful backup log is not a restore test. Do this before disk work, not during the exciting part when half the PGs say backfilling.

Networking and operations

An OSD is not an HTTP service. Reverse proxies, ACME, wildcard certificates, and DNS-01 challenges do not belong in its data path. Keep Ceph’s public and cluster traffic on trusted LANs or VLANs as described in the network reference. A separate cluster network can help only when the links and switching are genuinely separate; another VLAN on the same small NIC does not manufacture bandwidth.

For remote administration, Tailscale SSH is a sensible fit. Cloudflare Tunnel is reasonable for an authenticated web dashboard, but not for OSD replication. Raw port-forwarding Ceph services is all downside. Keep up with upstream Ceph notices and broader infrastructure security reporting such as Tech Sentinel, but patch deliberately.

Use cephadm’s coordinated upgrade workflow, not Watchtower: the upgrade documentation explains how it orders daemon restarts and pauses when a host is unavailable. If the new daemon fails, run cephadm logs --name osd.ID on its host; the cephadm troubleshooting guide explains the journald-backed logs. For alerts, the Prometheus module exports Ceph health checks, including OSD and PG warnings. Manual updates plus a maintenance note beat waking up to an enthusiastic container updater and three unavailable OSDs.

If Cephadm is not your manager

• Rook: add the device to the CephCluster storage selection, or increase the device-set count for PVC-backed OSDs. The Rook OSD guide says the operator provisions devices that match the resource.
• Proxmox VE: use Node > Ceph > OSD > Create or pveceph osd create /dev/sdX, following the Proxmox administration guide.
• Package-managed Ceph: use ceph-volume lvm create --data /dev/sdX only if that is already how the cluster is managed. Do not bolt this path onto cephadm because an old blog post looked reassuring.

Related across the network

• Proxmox Ceph vs ZFS Replication: When Ceph Is Worth It — proxmoxguide.com
• Storage Options for a Mini PC Homelab — minilabhq.com
• Proxmox High Availability: Fencing, HA Rules, and Limits — proxmoxguide.com
• Proxmox Snapshots: How They Work and When to Use — proxmoxguide.com
• Proxmox Troubleshooting: Fix Quorum, Storage & VM Errors — proxmoxguide.com