# Runbook: High Lab Host Resource Pressure

Triggered by the `LabHostHighMemory`/`HighDiskUsage` warning alerts.

```bash
# 1. How many Labs are actually running, and against what admission-control threshold?
curl -s http://localhost:4100/environments | jq '[.[] | select(.status=="running")] | length'
node infra/scripts/capacity-model.js   # recommended/warning/hard thresholds

# 2. If genuinely over capacity: this is expected pressure from real usage, not a bug.
#    - Destroy idle/abandoned Labs early via /admin (per-cohort "Destroy Labs" after a class
#      session ends, rather than waiting for the idle reaper's default window)
#    - If a specific cohort's class just ended, use /admin -> Cohorts -> [cohort] -> Destroy
#      Cohort Labs immediately rather than waiting

# 3. If pressure is high with FEWER Labs running than capacity-model.js's threshold: something
#    is consuming more per-Lab resources than the model assumes (a scenario doing something
#    unusually heavy, a memory leak in a long-running Lab, zombie processes from a crashed
#    reset). Check per-container usage:
docker stats --no-stream | grep rootnode-lab-

# 4. Update infra/environments/production/capacity.yml with fresh measured numbers
#    (infra/scripts/loadtest/) if the per-Lab footprint has genuinely changed, so the
#    admission-control threshold reflects reality going forward.
```

Do not silently raise resource limits in `provisioner-config/resource-policy.yaml` to make an
alert go away without understanding why usage is higher than modeled — that just moves the
same pressure onto host RAM/CPU directly instead of onto individual Lab limits.