# Runbook: Backup Failure

Triggered by the `BackupFailed`/`BackupAging` alerts (`infra/monitoring/prometheus/rules/rootnode.yml`).

```bash
# 1. See what actually happened — the cron wrapper logs to syslog
journalctl -u rootnode-backup --since "24 hours ago" -o cat
# or, if the cron job never even ran:
grep rootnode-postgres-backup /var/log/syslog

# 2. Run it manually to see the failure live
source /etc/rootnode/backup.env
/opt/rootnode/backup/backup-postgres.sh
```

**Common causes:**
- Disk full on `{{ rootnode_data_root }}/backups` — check `df -h`, prune more aggressively
  (`BACKUP_RETENTION_DAYS`) or the ZFS dataset needs more space (see `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4).
- Off-host target unreachable (`BACKUP_OFFSITE_TARGET`) — check the SSH key/credentials
  haven't expired, the target host/bucket is reachable: `ssh <offsite-target-host> echo ok` or
  the equivalent for whatever adapter is configured (`docs/BACKUP_DR.md`).
- `BACKUP_ENCRYPTION_PASSPHRASE` missing/wrong in `/etc/rootnode/backup.env` — re-check against
  the value in Ansible Vault (`vault_backup_encryption_passphrase`); a mismatch here means even
  a "successful" old backup becomes unrestorable, so treat this specific failure mode as urgent.
- The dump itself is suspiciously small (script's own sanity check tripped) — Postgres may be
  unhealthy; check `docker exec rootnode-platform-postgres-1 pg_isready` first.

**After fixing:** run the backup manually once to confirm success (metric updates immediately,
alert clears within one Prometheus scrape interval), then confirm the cron job is still
correctly scheduled (`crontab -l` as root, or re-run the `backup` Ansible role if the cron entry
itself is suspect).