ced4568
bf1e1b5c76
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs (PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR, OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST, and 12 runbooks (first deployment, normal deploy, rollback, Postgres restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure). PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine architectural gap this phase surfaced: the Gateway's existing env.internal_ip mechanism for reaching a Lab container assumes Gateway and Lab share a Docker daemon, which breaks once Labs run on a genuinely separate rn-lab-host-01 VM. The network-policy half is built (a firewall allowance for Gateway->Lab-Host on a published port range); the application half (Provisioner/Gateway address handling) is an explicit, reviewed, NOT-YET-BUILT follow-up flagged there rather than silently decided. Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and platform/README.md with Phase 4 pointers, consistent with how every prior phase updated these. Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
66 lines
4.7 KiB
Markdown
66 lines
4.7 KiB
Markdown
# Backup & Disaster Recovery
|
|
|
|
## What's backed up (Tier 1) vs. not
|
|
|
|
| | Backed up | Why |
|
|
|---|---|---|
|
|
| Postgres data | Yes, daily | The only irreplaceable data — student records, cohort/session state, scenario attempts, assessments, achievements, audit events |
|
|
| `{{ rootnode_data_root }}` config/secrets (Vault files, env templates) | Yes, via the infra repo + Ansible Vault itself | Small, static, versioned |
|
|
| Infra code (`infra/`, Ansible, Terraform state) | Yes — it's in git; Terraform state should be backed up alongside Postgres if not using a remote backend | Needed to rebuild either VM from scratch |
|
|
| Disposable student Labs | **No, intentionally** | Ephemeral by design — a Lab is meant to be destroyed and recreated; backing it up would contradict the whole isolation/disposability model |
|
|
| Prometheus/Loki metrics & log history | No | Operational telemetry, not source-of-truth data; losing 30 days of dashboards on a rebuild is an inconvenience, not data loss |
|
|
|
|
## Backup mechanism
|
|
|
|
`infra/backup/backup-postgres.sh` (deployed to `/opt/rootnode/backup/` by the `backup` Ansible
|
|
role, run daily at 03:15 via cron):
|
|
|
|
1. `pg_dump --format=plain --no-owner --no-privileges | gzip -9`
|
|
2. Sanity check: reject if the compressed dump is under 1KB (almost certainly means Postgres
|
|
was unreachable or empty, not a real backup).
|
|
3. `gpg --symmetric --cipher-algo AES256` — encrypt, then delete the plaintext dump immediately.
|
|
4. Optional off-host copy via `rsync -az` over SSH — works for Hetzner Storage Box or any SSH
|
|
target; that's the extension point for S3-compatible alternatives (swap the rsync line for
|
|
`aws s3 cp`/`rclone` if a different target is chosen later).
|
|
5. Retention pruning (`find ... -mtime +{{ backup_retention_days }} -delete`, default 30 days).
|
|
6. Writes Prometheus textfile-collector metrics
|
|
(`rootnode_backup_last_run_success`, `..._last_success_timestamp_seconds`, `..._size_bytes`)
|
|
atomically (tmp file + rename) so a half-written metric file is never scraped.
|
|
|
|
## Restore
|
|
|
|
`infra/backup/restore-postgres.sh` — requires an explicit target DB URL (never infers or
|
|
defaults to production, to prevent an accidental overwrite), shows the target's current tables,
|
|
requires typing `RESTORE` to confirm, then decrypts and pipes into `psql`.
|
|
`infra/backup/verify-restore.sh` — confirms all 8 expected tables exist plus the
|
|
`schema_migrations` row count, so a "successful" restore that silently missed a table is caught.
|
|
|
|
**Genuinely tested**, not just written: `infra/backup/test-backup-restore.sh` runs the full
|
|
cycle against two disposable local Postgres containers, with the actual production migrations
|
|
applied (all 17 — 3 Provisioner + 14 Platform API, in the correct cross-service order), a
|
|
sample row inserted, backed up, restored into the second container, and verified byte-for-byte.
|
|
Confirmed passing, exit 0.
|
|
|
|
## Disaster recovery scenarios
|
|
|
|
| Scenario | RPO | RTO (estimate) | What happens |
|
|
|---|---|---|---|
|
|
| Platform VM lost, disks intact | 0 | ~30 min | `docs/RUNBOOK_VM_REBUILD.md` — Ansible re-run against the same disks |
|
|
| Platform VM + its disks lost | Up to 24h (since last backup) | ~1-2h | Rebuild VM (Terraform), re-run Ansible, restore Postgres from the latest off-host backup (`docs/RUNBOOK_POSTGRES_RESTORE.md`) |
|
|
| Lab Host VM lost, any reason | 0 (intentional) | ~30 min | Rebuild VM, re-run Ansible — no data to restore, all Labs were disposable and get recreated on next student login |
|
|
| Single disk failure in a mirror | 0 | 0 (no outage) | `docs/RUNBOOK_DISK_FAILURE.md` — pool stays online DEGRADED, replace and resilver |
|
|
| Whole storage pool lost (both mirror sides) | Up to 24h | ~1-2h | Same as "Platform VM + disks lost" above — this is precisely the scenario the off-host copy exists for; a local-only backup on the same failed pool is useless here |
|
|
| Encryption passphrase lost | N/A | Unrecoverable | The single most important secret to protect and back up separately (e.g., in the same secrets manager as other Vault secrets, not only in `production.vault.yml`) — losing it makes every existing backup file permanently unreadable |
|
|
|
|
RPO is bounded by backup cadence (daily) — a mid-day incident can lose up to a day of student
|
|
activity. If that's ever unacceptable, the fix is a shorter cron interval or WAL archiving for
|
|
point-in-time recovery, not decided or built here since daily has not yet been proven
|
|
insufficient against real usage.
|
|
|
|
## What backup does NOT cover
|
|
|
|
Application code and container images — those live in Gitea and the image registry, both
|
|
already durable and versioned independently of this backup mechanism. TLS certificates — those
|
|
are regenerable (`docs/RUNBOOK_DOCKER_TLS_ROTATION.md`, `docs/RUNBOOK_TLS_RENEWAL.md`), not
|
|
backed up.
|