Files

T

ced4568 bf1e1b5c76 docs(phase4): production infrastructure documentation + runbooks

Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs
(PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR,
OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST,
and 12 runbooks (first deployment, normal deploy, rollback, Postgres
restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner
recovery, stuck Lab cleanup, failed migration, backup failure, disk
failure, Lab Host resource pressure).

PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine
architectural gap this phase surfaced: the Gateway's existing
env.internal_ip mechanism for reaching a Lab container assumes Gateway and
Lab share a Docker daemon, which breaks once Labs run on a genuinely
separate rn-lab-host-01 VM. The network-policy half is built (a firewall
allowance for Gateway->Lab-Host on a published port range); the
application half (Provisioner/Gateway address handling) is an explicit,
reviewed, NOT-YET-BUILT follow-up flagged there rather than silently
decided.

Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and
platform/README.md with Phase 4 pointers, consistent with how every prior
phase updated these.

Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.

2026-08-18 17:13:51 -05:00

4.7 KiB

Raw Blame History

Backup & Disaster Recovery

What's backed up (Tier 1) vs. not

	Backed up	Why
Postgres data	Yes, daily	The only irreplaceable data — student records, cohort/session state, scenario attempts, assessments, achievements, audit events
`{{ rootnode_data_root }}` config/secrets (Vault files, env templates)	Yes, via the infra repo + Ansible Vault itself	Small, static, versioned
Infra code (`infra/`, Ansible, Terraform state)	Yes — it's in git; Terraform state should be backed up alongside Postgres if not using a remote backend	Needed to rebuild either VM from scratch
Disposable student Labs	No, intentionally	Ephemeral by design — a Lab is meant to be destroyed and recreated; backing it up would contradict the whole isolation/disposability model
Prometheus/Loki metrics & log history	No	Operational telemetry, not source-of-truth data; losing 30 days of dashboards on a rebuild is an inconvenience, not data loss

Backup mechanism

infra/backup/backup-postgres.sh (deployed to /opt/rootnode/backup/ by the backup Ansible role, run daily at 03:15 via cron):

pg_dump --format=plain --no-owner --no-privileges | gzip -9
Sanity check: reject if the compressed dump is under 1KB (almost certainly means Postgres was unreachable or empty, not a real backup).
gpg --symmetric --cipher-algo AES256 — encrypt, then delete the plaintext dump immediately.
Optional off-host copy via rsync -az over SSH — works for Hetzner Storage Box or any SSH target; that's the extension point for S3-compatible alternatives (swap the rsync line for aws s3 cp/rclone if a different target is chosen later).
Retention pruning (find ... -mtime +{{ backup_retention_days }} -delete, default 30 days).
Writes Prometheus textfile-collector metrics (rootnode_backup_last_run_success, ..._last_success_timestamp_seconds, ..._size_bytes) atomically (tmp file + rename) so a half-written metric file is never scraped.

Restore

infra/backup/restore-postgres.sh — requires an explicit target DB URL (never infers or defaults to production, to prevent an accidental overwrite), shows the target's current tables, requires typing RESTORE to confirm, then decrypts and pipes into psql. infra/backup/verify-restore.sh — confirms all 8 expected tables exist plus the schema_migrations row count, so a "successful" restore that silently missed a table is caught.

Genuinely tested, not just written: infra/backup/test-backup-restore.sh runs the full cycle against two disposable local Postgres containers, with the actual production migrations applied (all 17 — 3 Provisioner + 14 Platform API, in the correct cross-service order), a sample row inserted, backed up, restored into the second container, and verified byte-for-byte. Confirmed passing, exit 0.

Disaster recovery scenarios

Scenario	RPO	RTO (estimate)	What happens
Platform VM lost, disks intact	0	~30 min	`docs/RUNBOOK_VM_REBUILD.md` — Ansible re-run against the same disks
Platform VM + its disks lost	Up to 24h (since last backup)	~1-2h	Rebuild VM (Terraform), re-run Ansible, restore Postgres from the latest off-host backup (`docs/RUNBOOK_POSTGRES_RESTORE.md`)
Lab Host VM lost, any reason	0 (intentional)	~30 min	Rebuild VM, re-run Ansible — no data to restore, all Labs were disposable and get recreated on next student login
Single disk failure in a mirror	0	0 (no outage)	`docs/RUNBOOK_DISK_FAILURE.md` — pool stays online DEGRADED, replace and resilver
Whole storage pool lost (both mirror sides)	Up to 24h	~1-2h	Same as "Platform VM + disks lost" above — this is precisely the scenario the off-host copy exists for; a local-only backup on the same failed pool is useless here
Encryption passphrase lost	N/A	Unrecoverable	The single most important secret to protect and back up separately (e.g., in the same secrets manager as other Vault secrets, not only in `production.vault.yml`) — losing it makes every existing backup file permanently unreadable

RPO is bounded by backup cadence (daily) — a mid-day incident can lose up to a day of student activity. If that's ever unacceptable, the fix is a shorter cron interval or WAL archiving for point-in-time recovery, not decided or built here since daily has not yet been proven insufficient against real usage.

What backup does NOT cover

Application code and container images — those live in Gitea and the image registry, both already durable and versioned independently of this backup mechanism. TLS certificates — those are regenerable (docs/RUNBOOK_DOCKER_TLS_ROTATION.md, docs/RUNBOOK_TLS_RENEWAL.md), not backed up.

4.7 KiB Raw Blame History