docs(phase4): production infrastructure documentation + runbooks

Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs
(PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR,
OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST,
and 12 runbooks (first deployment, normal deploy, rollback, Postgres
restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner
recovery, stuck Lab cleanup, failed migration, backup failure, disk
failure, Lab Host resource pressure).

PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine
architectural gap this phase surfaced: the Gateway's existing
env.internal_ip mechanism for reaching a Lab container assumes Gateway and
Lab share a Docker daemon, which breaks once Labs run on a genuinely
separate rn-lab-host-01 VM. The network-policy half is built (a firewall
allowance for Gateway->Lab-Host on a published port range); the
application half (Provisioner/Gateway address handling) is an explicit,
reviewed, NOT-YET-BUILT follow-up flagged there rather than silently
decided.

Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and
platform/README.md with Phase 4 pointers, consistent with how every prior
phase updated these.

Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
This commit is contained in:
2026-08-18 17:13:51 -05:00
parent 1d3afc736b
commit bf1e1b5c76
24 changed files with 1049 additions and 0 deletions
+10
View File
@@ -486,3 +486,13 @@ All four open questions this doc originally ended with are resolved — see the
at the top and §2–§5 above. Next: `docs/PHASE1_PLATFORM_FOUNDATION.md` — the concrete build
plan for the platform foundation (Postgres environment registry, provisioner skeleton,
networking, egress, resource policy, reaper) against these decisions.
## Phase 4: this boundary in production
Phase 4 (`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`) codifies the Platform/Lab-Host split this
document designed as two real, separately-firewalled Proxmox VMs (`rn-platform-01`,
`rn-lab-host-01`) connected only by mutual-TLS Docker and a tightly scoped firewall allowance —
see `docs/PRODUCTION_NETWORK_SECURITY.md`. That document also records a gap Phase 4 surfaced but
did not resolve: the Gateway's `env.internal_ip` mechanism for reaching a Lab container assumes
Gateway and Lab share a Docker daemon, which breaks once they're genuinely on separate VMs. Read
that document's "Known gap" section before treating two-VM production as fully wired.