ced4568/rootnode-academy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(phase4): production infrastructure documentation + runbooks

Browse Source 
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs
(PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR,
OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST,
and 12 runbooks (first deployment, normal deploy, rollback, Postgres
restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner
recovery, stuck Lab cleanup, failed migration, backup failure, disk
failure, Lab Host resource pressure).

PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine
architectural gap this phase surfaced: the Gateway's existing
env.internal_ip mechanism for reaching a Lab container assumes Gateway and
Lab share a Docker daemon, which breaks once Labs run on a genuinely
separate rn-lab-host-01 VM. The network-policy half is built (a firewall
allowance for Gateway->Lab-Host on a published port range); the
application half (Provisioner/Gateway address handling) is an explicit,
reviewed, NOT-YET-BUILT follow-up flagged there rather than silently
decided.

Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and
platform/README.md with Phase 4 pointers, consistent with how every prior
phase updated these.

Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
This commit is contained in:
ced4568
2026-08-18 17:13:51 -05:00
parent 1d3afc736b
commit bf1e1b5c76
24 changed files with 1049 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ENGINEERING_SOP.md
+25
View File
@@ -280,3 +280,28 @@ Full design: `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`.
- **Known limitations:** see the Phase 3B doc §20 — no automatic abandoned-attempt detection, no
per-cohort needs-attention threshold configuration, raw-JSON pool_spec authoring in the admin
UI (correct, not yet a guided form).
## 13. Production Infrastructure (Phase 4)
Full design: `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`, plus detail docs
`PRODUCTION_DEPLOYMENT.md`, `PRODUCTION_NETWORK_SECURITY.md`, `BACKUP_DR.md`,
`OBSERVABILITY.md`, `CAPACITY_PLANNING.md`, `PRODUCTION_VALIDATION_CHECKLIST.md`, and 12
`RUNBOOK_*.md` files, all under `docs/`. Everything lives in `infra/` (Ansible, Terraform,
Compose, firewall, monitoring, backup, CI, load-test tooling).
- **What it is:** the infrastructure-as-code needed to run rn-platform-01/rn-lab-host-01 on a
real Hetzner/Proxmox host — buildable and locally testable now, even though the dedicated
server hasn't been purchased. Status is explicitly **PHASE 4 BUILD COMPLETE / PRODUCTION
DEPLOYMENT PENDING HARDWARE**, not "deployed."
- **Where the boundary held:** the Platform VM / Lab Host VM security split from
`docs/LAB_ARCHITECTURE.md` is codified in firewall rulesets and mutual-TLS Docker config, not
weakened anywhere.
- **Known gap surfaced, not silently fixed:** the Gateway's Lab-reachability mechanism doesn't
survive the move to two real separate VMs — see `docs/PRODUCTION_NETWORK_SECURITY.md`'s
"Known gap" section for what's built (firewall policy) vs. what's an explicit, reviewed,
not-yet-built application follow-up (Provisioner/Gateway address handling).
- **Validated without the hardware:** real `tofu validate`, `ansible-playbook --syntax-check`,
`docker compose config`, `caddy validate`, `promtool`/`amtool`, and a genuine end-to-end
backup→encrypt→restore→verify cycle against disposable local Postgres with the real
production migrations applied. None of this proves the real Hetzner network boundary — that
needs `docs/PRODUCTION_VALIDATION_CHECKLIST.md` walked against the real server.