docs(phase4): production infrastructure documentation + runbooks
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs (PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR, OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST, and 12 runbooks (first deployment, normal deploy, rollback, Postgres restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure). PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine architectural gap this phase surfaced: the Gateway's existing env.internal_ip mechanism for reaching a Lab container assumes Gateway and Lab share a Docker daemon, which breaks once Labs run on a genuinely separate rn-lab-host-01 VM. The network-policy half is built (a firewall allowance for Gateway->Lab-Host on a published port range); the application half (Provisioner/Gateway address handling) is an explicit, reviewed, NOT-YET-BUILT follow-up flagged there rather than silently decided. Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and platform/README.md with Phase 4 pointers, consistent with how every prior phase updated these. Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
This commit is contained in:
@@ -0,0 +1,59 @@
|
||||
# Capacity Planning
|
||||
|
||||
How many concurrent student Labs `rn-lab-host-01` can actually run — determined from measured
|
||||
data after deployment, not guessed or marketed now.
|
||||
|
||||
## The model
|
||||
|
||||
`infra/scripts/capacity-model.js`:
|
||||
|
||||
```
|
||||
usable = hostTotal * (1 - overheadPct/100) - platformReserved
|
||||
withMargin = usable * (1 - marginPct/100)
|
||||
perLabLimit = floor(withMargin / perLabUsage)
|
||||
recommended = min(cpuBoundLimit, memoryBoundLimit) # whichever runs out first
|
||||
warning = floor(recommended * 0.8)
|
||||
hardAdmission = recommended
|
||||
```
|
||||
|
||||
Computed once for CPU, once for memory; the tighter (smaller) of the two constrains the actual
|
||||
recommendation — a host can be memory-starved long before its CPU threads are exhausted, or
|
||||
vice versa, and the model doesn't pretend otherwise.
|
||||
|
||||
**Inputs** (`infra/environments/production/capacity.yml.example` — explicitly placeholder data
|
||||
until real numbers exist):
|
||||
- `host_cpu_threads`, `host_memory_mb` — from the real Lab Host VM, once sized
|
||||
- `overhead_pct` — Proxmox/hypervisor + OS overhead reserved off the top
|
||||
- `platform_reserved_cpu`/`platform_reserved_memory_mb` — headroom for the egress proxy,
|
||||
node-exporter, cadvisor, promtail themselves (they run on the Lab Host too)
|
||||
- `per_lab_cpu`/`per_lab_memory_mb` — **measured**, from `infra/scripts/loadtest/`, not assumed
|
||||
- `safety_margin_pct` — extra headroom below the raw math, for burst/variance
|
||||
|
||||
**Outputs**: recommended concurrent Labs, an 80%-of-recommended warning threshold (for
|
||||
alerting — `LabHostHighMemory` in `docs/OBSERVABILITY.md`), and a hard admission-control
|
||||
threshold the Provisioner's resource policy should enforce so it refuses to boot a Lab past that
|
||||
point rather than degrading every running Lab.
|
||||
|
||||
## Load-test tooling that produces the measured inputs
|
||||
|
||||
`infra/scripts/loadtest/`:
|
||||
- `boot-labs.js` — reuses `platform/test/session-helper.js` (built in Phase 3A) to drive the
|
||||
real admin Prepare-Labs endpoint, booting N Labs and recording boot time/failure rate per Lab
|
||||
into a JSON report.
|
||||
- `sample-resources.sh` — samples real `docker stats` for every `rootnode-lab-*` container every
|
||||
5 seconds to CSV (timestamp, name, CPU%, mem usage, mem%, net I/O, block I/O, PIDs).
|
||||
|
||||
Together these produce the `per_lab_cpu`/`per_lab_memory_mb` this model needs, plus data on how
|
||||
boot time and failure rate change as concurrency rises — the dimensions this phase was told to
|
||||
cover: single Lab boot, multiple simultaneous boots, steady-state active Labs, terminal
|
||||
WebSocket load, scenario validation load, Gateway throughput, API request volume, Postgres
|
||||
connection activity.
|
||||
|
||||
## What this document deliberately does not claim
|
||||
|
||||
No concurrency number in this repo (`capacity.yml.example`'s numbers included) is a real,
|
||||
validated figure — every one is a placeholder until run against the real Lab Host VM with real
|
||||
per-Lab measurements. Treating a placeholder as a marketing/sales number would be actively
|
||||
misleading; the whole point of the model is that the real number gets filled in once, from
|
||||
measurement, after deployment (`docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md` step 4 is the
|
||||
trigger to re-measure and update it if the per-Lab footprint later changes).
|
||||
Reference in New Issue
Block a user