docs(phase4): production infrastructure documentation + runbooks

Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs
(PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR,
OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST,
and 12 runbooks (first deployment, normal deploy, rollback, Postgres
restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner
recovery, stuck Lab cleanup, failed migration, backup failure, disk
failure, Lab Host resource pressure).

PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine
architectural gap this phase surfaced: the Gateway's existing
env.internal_ip mechanism for reaching a Lab container assumes Gateway and
Lab share a Docker daemon, which breaks once Labs run on a genuinely
separate rn-lab-host-01 VM. The network-policy half is built (a firewall
allowance for Gateway->Lab-Host on a published port range); the
application half (Provisioner/Gateway address handling) is an explicit,
reviewed, NOT-YET-BUILT follow-up flagged there rather than silently
decided.

Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and
platform/README.md with Phase 4 pointers, consistent with how every prior
phase updated these.

Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
This commit is contained in:
2026-08-18 17:13:51 -05:00
parent 1d3afc736b
commit bf1e1b5c76
24 changed files with 1049 additions and 0 deletions
+59
View File
@@ -0,0 +1,59 @@
# Capacity Planning
How many concurrent student Labs `rn-lab-host-01` can actually run — determined from measured
data after deployment, not guessed or marketed now.
## The model
`infra/scripts/capacity-model.js`:
```
usable = hostTotal * (1 - overheadPct/100) - platformReserved
withMargin = usable * (1 - marginPct/100)
perLabLimit = floor(withMargin / perLabUsage)
recommended = min(cpuBoundLimit, memoryBoundLimit) # whichever runs out first
warning = floor(recommended * 0.8)
hardAdmission = recommended
```
Computed once for CPU, once for memory; the tighter (smaller) of the two constrains the actual
recommendation — a host can be memory-starved long before its CPU threads are exhausted, or
vice versa, and the model doesn't pretend otherwise.
**Inputs** (`infra/environments/production/capacity.yml.example` — explicitly placeholder data
until real numbers exist):
- `host_cpu_threads`, `host_memory_mb` — from the real Lab Host VM, once sized
- `overhead_pct` — Proxmox/hypervisor + OS overhead reserved off the top
- `platform_reserved_cpu`/`platform_reserved_memory_mb` — headroom for the egress proxy,
node-exporter, cadvisor, promtail themselves (they run on the Lab Host too)
- `per_lab_cpu`/`per_lab_memory_mb` — **measured**, from `infra/scripts/loadtest/`, not assumed
- `safety_margin_pct` — extra headroom below the raw math, for burst/variance
**Outputs**: recommended concurrent Labs, an 80%-of-recommended warning threshold (for
alerting — `LabHostHighMemory` in `docs/OBSERVABILITY.md`), and a hard admission-control
threshold the Provisioner's resource policy should enforce so it refuses to boot a Lab past that
point rather than degrading every running Lab.
## Load-test tooling that produces the measured inputs
`infra/scripts/loadtest/`:
- `boot-labs.js` — reuses `platform/test/session-helper.js` (built in Phase 3A) to drive the
real admin Prepare-Labs endpoint, booting N Labs and recording boot time/failure rate per Lab
into a JSON report.
- `sample-resources.sh` — samples real `docker stats` for every `rootnode-lab-*` container every
5 seconds to CSV (timestamp, name, CPU%, mem usage, mem%, net I/O, block I/O, PIDs).
Together these produce the `per_lab_cpu`/`per_lab_memory_mb` this model needs, plus data on how
boot time and failure rate change as concurrency rises — the dimensions this phase was told to
cover: single Lab boot, multiple simultaneous boots, steady-state active Labs, terminal
WebSocket load, scenario validation load, Gateway throughput, API request volume, Postgres
connection activity.
## What this document deliberately does not claim
No concurrency number in this repo (`capacity.yml.example`'s numbers included) is a real,
validated figure — every one is a placeholder until run against the real Lab Host VM with real
per-Lab measurements. Treating a placeholder as a marketing/sales number would be actively
misleading; the whole point of the model is that the real number gets filled in once, from
measurement, after deployment (`docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md` step 4 is the
trigger to re-measure and update it if the per-Lab footprint later changes).