ced4568
bf1e1b5c76
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs (PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR, OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST, and 12 runbooks (first deployment, normal deploy, rollback, Postgres restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure). PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine architectural gap this phase surfaced: the Gateway's existing env.internal_ip mechanism for reaching a Lab container assumes Gateway and Lab share a Docker daemon, which breaks once Labs run on a genuinely separate rn-lab-host-01 VM. The network-policy half is built (a firewall allowance for Gateway->Lab-Host on a published port range); the application half (Provisioner/Gateway address handling) is an explicit, reviewed, NOT-YET-BUILT follow-up flagged there rather than silently decided. Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and platform/README.md with Phase 4 pointers, consistent with how every prior phase updated these. Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
45 lines
2.2 KiB
Markdown
45 lines
2.2 KiB
Markdown
# Runbook: Disk Failure
|
|
|
|
Applies once real hardware/ZFS exists — see `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4 for
|
|
the storage decision framework this assumes (mirrored vdevs, never single-disk for Postgres).
|
|
|
|
```bash
|
|
# 1. Confirm the failure
|
|
zpool status -v
|
|
smartctl -a /dev/sdX # the suspect drive
|
|
|
|
# 2. If it's a mirrored vdev (the required layout for anything holding Postgres data — see §4),
|
|
# the pool stays online in DEGRADED state. No immediate data-loss risk, but no redundancy
|
|
# either until replaced — treat DEGRADED as urgent, not routine.
|
|
zpool status # confirm DEGRADED, not FAULTED/UNAVAIL (those mean act immediately, possibly
|
|
# restore from backup if the redundant mirror side is also compromised)
|
|
|
|
# 3. Replace the physical drive, then:
|
|
zpool replace <pool> <old-device> <new-device>
|
|
zpool status # watch the resilver progress
|
|
|
|
# 4. If a whole pool was lost (both mirror sides failed simultaneously — the scenario mirroring
|
|
# exists to make vanishingly unlikely, but plan for it anyway):
|
|
# - rn-lab-host-01's disposable data: nothing to recover, just rebuild the VM
|
|
# (docs/RUNBOOK_VM_REBUILD.md)
|
|
# - rn-platform-01's Postgres data: restore from the latest off-host backup
|
|
# (docs/RUNBOOK_POSTGRES_RESTORE.md) — this is exactly the scenario off-host backup
|
|
# copies exist for; a local-only backup on the SAME failed pool would be useless here
|
|
```
|
|
|
|
**Pre-deployment disk inspection checklist** (run once when the real drives are known, before
|
|
they go into service — `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4):
|
|
|
|
```bash
|
|
for d in /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1; do # adjust to real device names
|
|
echo "== $d =="
|
|
smartctl -i "$d" | grep -E 'Model|Serial|Capacity|Firmware'
|
|
smartctl -A "$d" | grep -iE 'power_on_hours|reallocated|pending|uncorrectable|wear|percentage.used|temperature'
|
|
smartctl -H "$d" | grep -i overall
|
|
done
|
|
```
|
|
Record model, size, power-on hours, wear/endurance percentage, reallocated/pending sector
|
|
counts, temperature, and firmware version for every drive before committing to a ZFS layout —
|
|
a drive already showing wear or reallocated sectors shouldn't be trusted with Postgres data
|
|
even in a mirror.
|