ced4568
bf1e1b5c76
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs (PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR, OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST, and 12 runbooks (first deployment, normal deploy, rollback, Postgres restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure). PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine architectural gap this phase surfaced: the Gateway's existing env.internal_ip mechanism for reaching a Lab container assumes Gateway and Lab share a Docker daemon, which breaks once Labs run on a genuinely separate rn-lab-host-01 VM. The network-policy half is built (a firewall allowance for Gateway->Lab-Host on a published port range); the application half (Provisioner/Gateway address handling) is an explicit, reviewed, NOT-YET-BUILT follow-up flagged there rather than silently decided. Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and platform/README.md with Phase 4 pointers, consistent with how every prior phase updated these. Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
32 lines
1.7 KiB
Markdown
32 lines
1.7 KiB
Markdown
# Runbook: Backup Failure
|
|
|
|
Triggered by the `BackupFailed`/`BackupAging` alerts (`infra/monitoring/prometheus/rules/rootnode.yml`).
|
|
|
|
```bash
|
|
# 1. See what actually happened — the cron wrapper logs to syslog
|
|
journalctl -u rootnode-backup --since "24 hours ago" -o cat
|
|
# or, if the cron job never even ran:
|
|
grep rootnode-postgres-backup /var/log/syslog
|
|
|
|
# 2. Run it manually to see the failure live
|
|
source /etc/rootnode/backup.env
|
|
/opt/rootnode/backup/backup-postgres.sh
|
|
```
|
|
|
|
**Common causes:**
|
|
- Disk full on `{{ rootnode_data_root }}/backups` — check `df -h`, prune more aggressively
|
|
(`BACKUP_RETENTION_DAYS`) or the ZFS dataset needs more space (see `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4).
|
|
- Off-host target unreachable (`BACKUP_OFFSITE_TARGET`) — check the SSH key/credentials
|
|
haven't expired, the target host/bucket is reachable: `ssh <offsite-target-host> echo ok` or
|
|
the equivalent for whatever adapter is configured (`docs/BACKUP_DR.md`).
|
|
- `BACKUP_ENCRYPTION_PASSPHRASE` missing/wrong in `/etc/rootnode/backup.env` — re-check against
|
|
the value in Ansible Vault (`vault_backup_encryption_passphrase`); a mismatch here means even
|
|
a "successful" old backup becomes unrestorable, so treat this specific failure mode as urgent.
|
|
- The dump itself is suspiciously small (script's own sanity check tripped) — Postgres may be
|
|
unhealthy; check `docker exec rootnode-platform-postgres-1 pg_isready` first.
|
|
|
|
**After fixing:** run the backup manually once to confirm success (metric updates immediately,
|
|
alert clears within one Prometheus scrape interval), then confirm the cron job is still
|
|
correctly scheduled (`crontab -l` as root, or re-run the `backup` Ansible role if the cron entry
|
|
itself is suspect).
|