ced4568
bf1e1b5c76
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs (PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR, OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST, and 12 runbooks (first deployment, normal deploy, rollback, Postgres restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure). PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine architectural gap this phase surfaced: the Gateway's existing env.internal_ip mechanism for reaching a Lab container assumes Gateway and Lab share a Docker daemon, which breaks once Labs run on a genuinely separate rn-lab-host-01 VM. The network-policy half is built (a firewall allowance for Gateway->Lab-Host on a published port range); the application half (Provisioner/Gateway address handling) is an explicit, reviewed, NOT-YET-BUILT follow-up flagged there rather than silently decided. Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and platform/README.md with Phase 4 pointers, consistent with how every prior phase updated these. Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
3.1 KiB
Capacity Planning
How many concurrent student Labs rn-lab-host-01 can actually run — determined from measured
data after deployment, not guessed or marketed now.
The model
infra/scripts/capacity-model.js:
usable = hostTotal * (1 - overheadPct/100) - platformReserved
withMargin = usable * (1 - marginPct/100)
perLabLimit = floor(withMargin / perLabUsage)
recommended = min(cpuBoundLimit, memoryBoundLimit) # whichever runs out first
warning = floor(recommended * 0.8)
hardAdmission = recommended
Computed once for CPU, once for memory; the tighter (smaller) of the two constrains the actual recommendation — a host can be memory-starved long before its CPU threads are exhausted, or vice versa, and the model doesn't pretend otherwise.
Inputs (infra/environments/production/capacity.yml.example — explicitly placeholder data
until real numbers exist):
host_cpu_threads,host_memory_mb— from the real Lab Host VM, once sizedoverhead_pct— Proxmox/hypervisor + OS overhead reserved off the topplatform_reserved_cpu/platform_reserved_memory_mb— headroom for the egress proxy, node-exporter, cadvisor, promtail themselves (they run on the Lab Host too)per_lab_cpu/per_lab_memory_mb— measured, frominfra/scripts/loadtest/, not assumedsafety_margin_pct— extra headroom below the raw math, for burst/variance
Outputs: recommended concurrent Labs, an 80%-of-recommended warning threshold (for
alerting — LabHostHighMemory in docs/OBSERVABILITY.md), and a hard admission-control
threshold the Provisioner's resource policy should enforce so it refuses to boot a Lab past that
point rather than degrading every running Lab.
Load-test tooling that produces the measured inputs
infra/scripts/loadtest/:
boot-labs.js— reusesplatform/test/session-helper.js(built in Phase 3A) to drive the real admin Prepare-Labs endpoint, booting N Labs and recording boot time/failure rate per Lab into a JSON report.sample-resources.sh— samples realdocker statsfor everyrootnode-lab-*container every 5 seconds to CSV (timestamp, name, CPU%, mem usage, mem%, net I/O, block I/O, PIDs).
Together these produce the per_lab_cpu/per_lab_memory_mb this model needs, plus data on how
boot time and failure rate change as concurrency rises — the dimensions this phase was told to
cover: single Lab boot, multiple simultaneous boots, steady-state active Labs, terminal
WebSocket load, scenario validation load, Gateway throughput, API request volume, Postgres
connection activity.
What this document deliberately does not claim
No concurrency number in this repo (capacity.yml.example's numbers included) is a real,
validated figure — every one is a placeholder until run against the real Lab Host VM with real
per-Lab measurements. Treating a placeholder as a marketing/sales number would be actively
misleading; the whole point of the model is that the real number gets filled in once, from
measurement, after deployment (docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md step 4 is the
trigger to re-measure and update it if the per-Lab footprint later changes).