# Backup & Disaster Recovery

## What's backed up (Tier 1) vs. not

| | Backed up | Why |
|---|---|---|
| Postgres data | Yes, daily | The only irreplaceable data — student records, cohort/session state, scenario attempts, assessments, achievements, audit events |
| `{{ rootnode_data_root }}` config/secrets (Vault files, env templates) | Yes, via the infra repo + Ansible Vault itself | Small, static, versioned |
| Infra code (`infra/`, Ansible, Terraform state) | Yes — it's in git; Terraform state should be backed up alongside Postgres if not using a remote backend | Needed to rebuild either VM from scratch |
| Disposable student Labs | **No, intentionally** | Ephemeral by design — a Lab is meant to be destroyed and recreated; backing it up would contradict the whole isolation/disposability model |
| Prometheus/Loki metrics & log history | No | Operational telemetry, not source-of-truth data; losing 30 days of dashboards on a rebuild is an inconvenience, not data loss |

## Backup mechanism

`infra/backup/backup-postgres.sh` (deployed to `/opt/rootnode/backup/` by the `backup` Ansible
role, run daily at 03:15 via cron):

1. `pg_dump --format=plain --no-owner --no-privileges | gzip -9`
2. Sanity check: reject if the compressed dump is under 1KB (almost certainly means Postgres
   was unreachable or empty, not a real backup).
3. `gpg --symmetric --cipher-algo AES256` — encrypt, then delete the plaintext dump immediately.
4. Optional off-host copy via `rsync -az` over SSH — works for Hetzner Storage Box or any SSH
   target; that's the extension point for S3-compatible alternatives (swap the rsync line for
   `aws s3 cp`/`rclone` if a different target is chosen later).
5. Retention pruning (`find ... -mtime +{{ backup_retention_days }} -delete`, default 30 days).
6. Writes Prometheus textfile-collector metrics
   (`rootnode_backup_last_run_success`, `..._last_success_timestamp_seconds`, `..._size_bytes`)
   atomically (tmp file + rename) so a half-written metric file is never scraped.

## Restore

`infra/backup/restore-postgres.sh` — requires an explicit target DB URL (never infers or
defaults to production, to prevent an accidental overwrite), shows the target's current tables,
requires typing `RESTORE` to confirm, then decrypts and pipes into `psql`.
`infra/backup/verify-restore.sh` — confirms all 8 expected tables exist plus the
`schema_migrations` row count, so a "successful" restore that silently missed a table is caught.

**Genuinely tested**, not just written: `infra/backup/test-backup-restore.sh` runs the full
cycle against two disposable local Postgres containers, with the actual production migrations
applied (3 Provisioner + Platform API, in the correct cross-service order), a
sample row inserted, backed up, restored into the second container, and verified byte-for-byte.
Confirmed passing, exit 0.

**Phase 4.1's destructive drill goes one step further**
(`infra/backup/test-destructive-restore-drill.sh`): realistic data (owner, admin instructor,
cohort, 3 students, progress, attempts, an assessment definition, achievements) seeded through
the REAL Platform API, backed up, the source Postgres **container genuinely destroyed**
(`docker rm`, not just left running alongside a second one), a fresh Postgres created from
nothing, restored, and the real Platform API booted again against the restored database and
used to log in as the restored admin instructor over HTTP — proving the whole stack recovers,
not just the SQL. Never touches the developer's real persistent database. Confirmed passing.

## Disaster recovery scenarios

| Scenario | RPO | RTO (estimate) | What happens |
|---|---|---|---|
| Platform VM lost, disks intact | 0 | ~30 min | `docs/RUNBOOK_VM_REBUILD.md` — Ansible re-run against the same disks |
| Platform VM + its disks lost | Up to 24h (since last backup) | ~1-2h | Rebuild VM (Terraform), re-run Ansible, restore Postgres from the latest off-host backup (`docs/RUNBOOK_POSTGRES_RESTORE.md`) |
| Lab Host VM lost, any reason | 0 (intentional) | ~30 min | Rebuild VM, re-run Ansible — no data to restore, all Labs were disposable and get recreated on next student login |
| Single disk failure in a mirror | 0 | 0 (no outage) | `docs/RUNBOOK_DISK_FAILURE.md` — pool stays online DEGRADED, replace and resilver |
| Whole storage pool lost (both mirror sides) | Up to 24h | ~1-2h | Same as "Platform VM + disks lost" above — this is precisely the scenario the off-host copy exists for; a local-only backup on the same failed pool is useless here |
| Encryption passphrase lost | N/A | Unrecoverable | The single most important secret to protect and back up separately (e.g., in the same secrets manager as other Vault secrets, not only in `production.vault.yml`) — losing it makes every existing backup file permanently unreadable |

RPO is bounded by backup cadence (daily) — a mid-day incident can lose up to a day of student
activity. If that's ever unacceptable, the fix is a shorter cron interval or WAL archiving for
point-in-time recovery, not decided or built here since daily has not yet been proven
insufficient against real usage.

## What backup does NOT cover

Application code and container images — those live in Gitea and the image registry, both
already durable and versioned independently of this backup mechanism. TLS certificates — those
are regenerable (`docs/RUNBOOK_DOCKER_TLS_ROTATION.md`, `docs/RUNBOOK_TLS_RENEWAL.md`), not
backed up.