ced4568
5bc0743b08
Master overview (PHASE4_1_PREPRODUCTION_HARDENING.md) covering all 27 substantive items, plus REMOTE_AND_SELF_PACED_ARCHITECTURE.md (reviews what already works for remote/self-paced delivery - student-driven provisioning, cohort-less students, idle-reaper-destroy-then-continue - and the one real gap: student authentication, explicitly Phase 5 scope, not built here). Updated PRODUCTION_NETWORK_SECURITY.md (the Gateway-to-Lab gap is now RESOLVED, with the full design writeup and a new control-plane-vs-data-plane section), PRODUCTION_DEPLOYMENT.md (four-cert TLS bundle), PRODUCTION_VALIDATION_CHECKLIST.md (Router mTLS check, admission control checks), PHASE4_PRODUCTION_INFRASTRUCTURE.md and ENGINEERING_SOP.md and LAB_ARCHITECTURE.md (stale 'known gap' pointers updated to point at the resolution), BACKUP_DR.md (destructive drill), README.md and platform/README.md (Phase 4.1 pointers, lab-router component, updated known-limitations). Also fixes a real pre-existing bug found while auditing what gets committed: infra/.gitignore's patterns were written repo-root-relative (e.g. "infra/terraform/...") but the file lives in infra/, so git resolved them relative to THAT directory and none of them ever matched anything - a real .terraform/ provider cache showed up as untracked when it should have been ignored. Fixed by dropping the redundant leading "infra/" from every pattern. Status: PHASE 4.1 COMPLETE / PRE-PRODUCTION HARDENING COMPLETE / READY FOR HARDWARE DEPLOYMENT.
75 lines
5.3 KiB
Markdown
75 lines
5.3 KiB
Markdown
# Backup & Disaster Recovery
|
|
|
|
## What's backed up (Tier 1) vs. not
|
|
|
|
| | Backed up | Why |
|
|
|---|---|---|
|
|
| Postgres data | Yes, daily | The only irreplaceable data — student records, cohort/session state, scenario attempts, assessments, achievements, audit events |
|
|
| `{{ rootnode_data_root }}` config/secrets (Vault files, env templates) | Yes, via the infra repo + Ansible Vault itself | Small, static, versioned |
|
|
| Infra code (`infra/`, Ansible, Terraform state) | Yes — it's in git; Terraform state should be backed up alongside Postgres if not using a remote backend | Needed to rebuild either VM from scratch |
|
|
| Disposable student Labs | **No, intentionally** | Ephemeral by design — a Lab is meant to be destroyed and recreated; backing it up would contradict the whole isolation/disposability model |
|
|
| Prometheus/Loki metrics & log history | No | Operational telemetry, not source-of-truth data; losing 30 days of dashboards on a rebuild is an inconvenience, not data loss |
|
|
|
|
## Backup mechanism
|
|
|
|
`infra/backup/backup-postgres.sh` (deployed to `/opt/rootnode/backup/` by the `backup` Ansible
|
|
role, run daily at 03:15 via cron):
|
|
|
|
1. `pg_dump --format=plain --no-owner --no-privileges | gzip -9`
|
|
2. Sanity check: reject if the compressed dump is under 1KB (almost certainly means Postgres
|
|
was unreachable or empty, not a real backup).
|
|
3. `gpg --symmetric --cipher-algo AES256` — encrypt, then delete the plaintext dump immediately.
|
|
4. Optional off-host copy via `rsync -az` over SSH — works for Hetzner Storage Box or any SSH
|
|
target; that's the extension point for S3-compatible alternatives (swap the rsync line for
|
|
`aws s3 cp`/`rclone` if a different target is chosen later).
|
|
5. Retention pruning (`find ... -mtime +{{ backup_retention_days }} -delete`, default 30 days).
|
|
6. Writes Prometheus textfile-collector metrics
|
|
(`rootnode_backup_last_run_success`, `..._last_success_timestamp_seconds`, `..._size_bytes`)
|
|
atomically (tmp file + rename) so a half-written metric file is never scraped.
|
|
|
|
## Restore
|
|
|
|
`infra/backup/restore-postgres.sh` — requires an explicit target DB URL (never infers or
|
|
defaults to production, to prevent an accidental overwrite), shows the target's current tables,
|
|
requires typing `RESTORE` to confirm, then decrypts and pipes into `psql`.
|
|
`infra/backup/verify-restore.sh` — confirms all 8 expected tables exist plus the
|
|
`schema_migrations` row count, so a "successful" restore that silently missed a table is caught.
|
|
|
|
**Genuinely tested**, not just written: `infra/backup/test-backup-restore.sh` runs the full
|
|
cycle against two disposable local Postgres containers, with the actual production migrations
|
|
applied (3 Provisioner + Platform API, in the correct cross-service order), a
|
|
sample row inserted, backed up, restored into the second container, and verified byte-for-byte.
|
|
Confirmed passing, exit 0.
|
|
|
|
**Phase 4.1's destructive drill goes one step further**
|
|
(`infra/backup/test-destructive-restore-drill.sh`): realistic data (owner, admin instructor,
|
|
cohort, 3 students, progress, attempts, an assessment definition, achievements) seeded through
|
|
the REAL Platform API, backed up, the source Postgres **container genuinely destroyed**
|
|
(`docker rm`, not just left running alongside a second one), a fresh Postgres created from
|
|
nothing, restored, and the real Platform API booted again against the restored database and
|
|
used to log in as the restored admin instructor over HTTP — proving the whole stack recovers,
|
|
not just the SQL. Never touches the developer's real persistent database. Confirmed passing.
|
|
|
|
## Disaster recovery scenarios
|
|
|
|
| Scenario | RPO | RTO (estimate) | What happens |
|
|
|---|---|---|---|
|
|
| Platform VM lost, disks intact | 0 | ~30 min | `docs/RUNBOOK_VM_REBUILD.md` — Ansible re-run against the same disks |
|
|
| Platform VM + its disks lost | Up to 24h (since last backup) | ~1-2h | Rebuild VM (Terraform), re-run Ansible, restore Postgres from the latest off-host backup (`docs/RUNBOOK_POSTGRES_RESTORE.md`) |
|
|
| Lab Host VM lost, any reason | 0 (intentional) | ~30 min | Rebuild VM, re-run Ansible — no data to restore, all Labs were disposable and get recreated on next student login |
|
|
| Single disk failure in a mirror | 0 | 0 (no outage) | `docs/RUNBOOK_DISK_FAILURE.md` — pool stays online DEGRADED, replace and resilver |
|
|
| Whole storage pool lost (both mirror sides) | Up to 24h | ~1-2h | Same as "Platform VM + disks lost" above — this is precisely the scenario the off-host copy exists for; a local-only backup on the same failed pool is useless here |
|
|
| Encryption passphrase lost | N/A | Unrecoverable | The single most important secret to protect and back up separately (e.g., in the same secrets manager as other Vault secrets, not only in `production.vault.yml`) — losing it makes every existing backup file permanently unreadable |
|
|
|
|
RPO is bounded by backup cadence (daily) — a mid-day incident can lose up to a day of student
|
|
activity. If that's ever unacceptable, the fix is a shorter cron interval or WAL archiving for
|
|
point-in-time recovery, not decided or built here since daily has not yet been proven
|
|
insufficient against real usage.
|
|
|
|
## What backup does NOT cover
|
|
|
|
Application code and container images — those live in Gitea and the image registry, both
|
|
already durable and versioned independently of this backup mechanism. TLS certificates — those
|
|
are regenerable (`docs/RUNBOOK_DOCKER_TLS_ROTATION.md`, `docs/RUNBOOK_TLS_RENEWAL.md`), not
|
|
backed up.
|