docs(phase4): production infrastructure documentation + runbooks
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs (PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR, OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST, and 12 runbooks (first deployment, normal deploy, rollback, Postgres restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure). PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine architectural gap this phase surfaced: the Gateway's existing env.internal_ip mechanism for reaching a Lab container assumes Gateway and Lab share a Docker daemon, which breaks once Labs run on a genuinely separate rn-lab-host-01 VM. The network-policy half is built (a firewall allowance for Gateway->Lab-Host on a published port range); the application half (Provisioner/Gateway address handling) is an explicit, reviewed, NOT-YET-BUILT follow-up flagged there rather than silently decided. Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and platform/README.md with Phase 4 pointers, consistent with how every prior phase updated these. Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
This commit is contained in:
@@ -0,0 +1,31 @@
|
||||
# Runbook: Platform VM / Lab Host VM Rebuild
|
||||
|
||||
Both VMs are designed to be disposable and rebuildable from code — nothing irreplaceable
|
||||
should ever live only on either VM's disk outside of what's backed up (`docs/BACKUP_DR.md`).
|
||||
|
||||
## rn-platform-01 lost
|
||||
|
||||
Academic data survives (it's in Postgres backups, off-host). Instructor/student accounts and
|
||||
progress are recoverable; anything since the last backup is not (`docs/BACKUP_DR.md`'s RPO).
|
||||
|
||||
```bash
|
||||
cd infra/terraform && terraform apply # recreates the VM from the template
|
||||
cd ../ansible && ansible-playbook site.yml -i inventory/production.ini --limit platform --ask-vault-pass
|
||||
# Then restore Postgres — docs/RUNBOOK_POSTGRES_RESTORE.md, step 5b variant (fresh DB this time,
|
||||
# no need to rename since it's a fresh instance)
|
||||
./infra/scripts/production-smoke-test.sh
|
||||
```
|
||||
|
||||
## rn-lab-host-01 lost
|
||||
|
||||
**No academic data is lost** — this host only ever holds disposable Lab containers by design
|
||||
(`docs/BACKUP_DR.md`'s Tier distinction). Every enrolled student's Lab needs re-provisioning
|
||||
after this, which is a normal "Prepare Labs" click, not a recovery operation.
|
||||
|
||||
```bash
|
||||
cd infra/terraform && terraform apply
|
||||
cd ../ansible && ansible-playbook site.yml -i inventory/production.ini --limit lab_host --ask-vault-pass
|
||||
# Re-generate Docker TLS material for the new host if its identity (IP/DNS) changed —
|
||||
# docs/RUNBOOK_DOCKER_TLS_ROTATION.md
|
||||
# From /admin: Prepare Labs again for any cohort that needs it.
|
||||
```
|
||||
Reference in New Issue
Block a user