ced4568
bf1e1b5c76
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs (PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR, OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST, and 12 runbooks (first deployment, normal deploy, rollback, Postgres restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure). PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine architectural gap this phase surfaced: the Gateway's existing env.internal_ip mechanism for reaching a Lab container assumes Gateway and Lab share a Docker daemon, which breaks once Labs run on a genuinely separate rn-lab-host-01 VM. The network-policy half is built (a firewall allowance for Gateway->Lab-Host on a published port range); the application half (Provisioner/Gateway address handling) is an explicit, reviewed, NOT-YET-BUILT follow-up flagged there rather than silently decided. Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and platform/README.md with Phase 4 pointers, consistent with how every prior phase updated these. Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
30 lines
1.6 KiB
Markdown
30 lines
1.6 KiB
Markdown
# Runbook: High Lab Host Resource Pressure
|
|
|
|
Triggered by the `LabHostHighMemory`/`HighDiskUsage` warning alerts.
|
|
|
|
```bash
|
|
# 1. How many Labs are actually running, and against what admission-control threshold?
|
|
curl -s http://localhost:4100/environments | jq '[.[] | select(.status=="running")] | length'
|
|
node infra/scripts/capacity-model.js # recommended/warning/hard thresholds
|
|
|
|
# 2. If genuinely over capacity: this is expected pressure from real usage, not a bug.
|
|
# - Destroy idle/abandoned Labs early via /admin (per-cohort "Destroy Labs" after a class
|
|
# session ends, rather than waiting for the idle reaper's default window)
|
|
# - If a specific cohort's class just ended, use /admin -> Cohorts -> [cohort] -> Destroy
|
|
# Cohort Labs immediately rather than waiting
|
|
|
|
# 3. If pressure is high with FEWER Labs running than capacity-model.js's threshold: something
|
|
# is consuming more per-Lab resources than the model assumes (a scenario doing something
|
|
# unusually heavy, a memory leak in a long-running Lab, zombie processes from a crashed
|
|
# reset). Check per-container usage:
|
|
docker stats --no-stream | grep rootnode-lab-
|
|
|
|
# 4. Update infra/environments/production/capacity.yml with fresh measured numbers
|
|
# (infra/scripts/loadtest/) if the per-Lab footprint has genuinely changed, so the
|
|
# admission-control threshold reflects reality going forward.
|
|
```
|
|
|
|
Do not silently raise resource limits in `provisioner-config/resource-policy.yaml` to make an
|
|
alert go away without understanding why usage is higher than modeled — that just moves the
|
|
same pressure onto host RAM/CPU directly instead of onto individual Lab limits.
|