# Runbook: Disk Failure

Applies once real hardware/ZFS exists — see `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4 for
the storage decision framework this assumes (mirrored vdevs, never single-disk for Postgres).

```bash
# 1. Confirm the failure
zpool status -v
smartctl -a /dev/sdX   # the suspect drive

# 2. If it's a mirrored vdev (the required layout for anything holding Postgres data — see §4),
#    the pool stays online in DEGRADED state. No immediate data-loss risk, but no redundancy
#    either until replaced — treat DEGRADED as urgent, not routine.
zpool status   # confirm DEGRADED, not FAULTED/UNAVAIL (those mean act immediately, possibly
                # restore from backup if the redundant mirror side is also compromised)

# 3. Replace the physical drive, then:
zpool replace <pool> <old-device> <new-device>
zpool status   # watch the resilver progress

# 4. If a whole pool was lost (both mirror sides failed simultaneously — the scenario mirroring
#    exists to make vanishingly unlikely, but plan for it anyway):
#    - rn-lab-host-01's disposable data: nothing to recover, just rebuild the VM
#      (docs/RUNBOOK_VM_REBUILD.md)
#    - rn-platform-01's Postgres data: restore from the latest off-host backup
#      (docs/RUNBOOK_POSTGRES_RESTORE.md) — this is exactly the scenario off-host backup
#      copies exist for; a local-only backup on the SAME failed pool would be useless here
```

**Pre-deployment disk inspection checklist** (run once when the real drives are known, before
they go into service — `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4):

```bash
for d in /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1; do   # adjust to real device names
  echo "== $d =="
  smartctl -i "$d" | grep -E 'Model|Serial|Capacity|Firmware'
  smartctl -A "$d" | grep -iE 'power_on_hours|reallocated|pending|uncorrectable|wear|percentage.used|temperature'
  smartctl -H "$d" | grep -i overall
done
```
Record model, size, power-on hours, wear/endurance percentage, reallocated/pending sector
counts, temperature, and firmware version for every drive before committing to a ZFS layout —
a drive already showing wear or reallocated sectors shouldn't be trusted with Postgres data
even in a mirror.