rootnode-academy/docs/BACKUP_DR.md

Backup & Disaster Recovery

What's backed up (Tier 1) vs. not

	Backed up	Why
Postgres data	Yes, daily	The only irreplaceable data — student records, cohort/session state, scenario attempts, assessments, achievements, audit events
{{ rootnode_data_root }} config/secrets (Vault files, env templates)	Yes, via the infra repo + Ansible Vault itself	Small, static, versioned
Infra code (infra/, Ansible, Terraform state)	Yes — it's in git; Terraform state should be backed up alongside Postgres if not using a remote backend	Needed to rebuild either VM from scratch
Disposable student Labs	No, intentionally	Ephemeral by design — a Lab is meant to be destroyed and recreated; backing it up would contradict the whole isolation/disposability model
Prometheus/Loki metrics & log history	No	Operational telemetry, not source-of-truth data; losing 30 days of dashboards on a rebuild is an inconvenience, not data loss

Backup mechanism

infra/backup/backup-postgres.sh (deployed to /opt/rootnode/backup/ by the backup Ansible role, run daily at 03:15 via cron):

1. pg_dump --format=plain --no-owner --no-privileges | gzip -9
2. Sanity check: reject if the compressed dump is under 1KB (almost certainly means Postgres was unreachable or empty, not a real backup).
3. gpg --symmetric --cipher-algo AES256 — encrypt, then delete the plaintext dump immediately.
4. Optional off-host copy via rsync -az over SSH — works for Hetzner Storage Box or any SSH target; that's the extension point for S3-compatible alternatives (swap the rsync line for aws s3 cp/rclone if a different target is chosen later).
5. Retention pruning (find ... -mtime +{{ backup_retention_days }} -delete, default 30 days).
6. Writes Prometheus textfile-collector metrics (rootnode_backup_last_run_success, ..._last_success_timestamp_seconds, ..._size_bytes) atomically (tmp file + rename) so a half-written metric file is never scraped.

Restore

infra/backup/restore-postgres.sh — requires an explicit target DB URL (never infers or defaults to production, to prevent an accidental overwrite), shows the target's current tables, requires typing RESTORE to confirm, then decrypts and pipes into psql. infra/backup/verify-restore.sh — confirms all 8 expected tables exist plus the schema_migrations row count, so a "successful" restore that silently missed a table is caught.

Genuinely tested, not just written: infra/backup/test-backup-restore.sh runs the full cycle against two disposable local Postgres containers, with the actual production migrations applied (3 Provisioner + Platform API, in the correct cross-service order), a sample row inserted, backed up, restored into the second container, and verified byte-for-byte. Confirmed passing, exit 0.

Phase 4.1's destructive drill goes one step further (infra/backup/test-destructive-restore-drill.sh): realistic data (owner, admin instructor, cohort, 3 students, progress, attempts, an assessment definition, achievements) seeded through the REAL Platform API, backed up, the source Postgres container genuinely destroyed (docker rm, not just left running alongside a second one), a fresh Postgres created from nothing, restored, and the real Platform API booted again against the restored database and used to log in as the restored admin instructor over HTTP — proving the whole stack recovers, not just the SQL. Never touches the developer's real persistent database. Confirmed passing.

Disaster recovery scenarios

Scenario	RPO	RTO (estimate)	What happens
Platform VM lost, disks intact	0	~30 min	docs/RUNBOOK_VM_REBUILD.md — Ansible re-run against the same disks
Platform VM + its disks lost	Up to 24h (since last backup)	~1-2h	Rebuild VM (Terraform), re-run Ansible, restore Postgres from the latest off-host backup (docs/RUNBOOK_POSTGRES_RESTORE.md)
Lab Host VM lost, any reason	0 (intentional)	~30 min	Rebuild VM, re-run Ansible — no data to restore, all Labs were disposable and get recreated on next student login
Single disk failure in a mirror	0	0 (no outage)	docs/RUNBOOK_DISK_FAILURE.md — pool stays online DEGRADED, replace and resilver
Whole storage pool lost (both mirror sides)	Up to 24h	~1-2h	Same as "Platform VM + disks lost" above — this is precisely the scenario the off-host copy exists for; a local-only backup on the same failed pool is useless here
Encryption passphrase lost	N/A	Unrecoverable	The single most important secret to protect and back up separately (e.g., in the same secrets manager as other Vault secrets, not only in production.vault.yml) — losing it makes every existing backup file permanently unreadable

RPO is bounded by backup cadence (daily) — a mid-day incident can lose up to a day of student activity. If that's ever unacceptable, the fix is a shorter cron interval or WAL archiving for point-in-time recovery, not decided or built here since daily has not yet been proven insufficient against real usage.

What backup does NOT cover

Application code and container images — those live in Gitea and the image registry, both already durable and versioned independently of this backup mechanism. TLS certificates — those are regenerable (docs/RUNBOOK_DOCKER_TLS_ROTATION.md, docs/RUNBOOK_TLS_RENEWAL.md), not backed up.