feat(loadtest): load-test tooling + capacity model
boot-labs.js drives the real admin Prepare-Labs endpoint (reusing the existing session-helper.js from Phase 3A) and records boot time/failure rate; sample-resources.sh samples real docker stats for running Labs. capacity-model.js turns measured per-Lab CPU/memory into a recommended/ warning/hard-admission-control concurrency threshold - no concurrency number is hardcoded as a marketing figure, every input is explicitly placeholder data until measured against the real host. Plus deploy/rollback/smoke-test scripts used by the CI/CD and deployment runbooks. All syntax-checked; validate-compose.sh and capacity-model.js actually run against real inputs.
This commit is contained in:
@@ -0,0 +1,32 @@
|
||||
# Load test tooling
|
||||
|
||||
Two scripts, run together against whatever stack you point them at:
|
||||
|
||||
- `boot-labs.js` — provisions N Labs concurrently via the real admin Prepare Labs endpoint,
|
||||
measures boot success rate and wall-clock timing. `GATEWAY_URL=... OWNER_EMAIL=...
|
||||
OWNER_PASSWORD=... node boot-labs.js --count=10`
|
||||
- `sample-resources.sh` — run on the Lab Host **while** `boot-labs.js` runs, samples real
|
||||
`docker stats` for every `rootnode-lab-*` container every 5s, writes CSV.
|
||||
|
||||
```bash
|
||||
# terminal 1, on the Lab Host:
|
||||
./infra/scripts/loadtest/sample-resources.sh 180 ./results/run1.csv
|
||||
|
||||
# terminal 2, wherever the Gateway is reachable from:
|
||||
GATEWAY_URL=https://academy.example.com OWNER_EMAIL=... OWNER_PASSWORD=... \
|
||||
node infra/scripts/loadtest/boot-labs.js --count=10
|
||||
```
|
||||
|
||||
`report-template.md` is a filled-in-by-hand summary format — copy it, paste the two scripts'
|
||||
output in, note the host class it ran against. **A run against a dev laptop is not evidence
|
||||
about the real Hetzner host** — every report must say what it ran on.
|
||||
|
||||
Feeds `docs/CAPACITY_PLANNING.md`'s worksheet directly: `sample-resources.sh`'s peak
|
||||
`mem_usage`/`cpu_pct` per Lab are exactly the "measured Lab CPU"/"measured Lab RAM" inputs that
|
||||
worksheet asks for.
|
||||
|
||||
Test dimensions this DOESN'T cover yet (documented gaps, not silently skipped): terminal
|
||||
WebSocket throughput under concurrent load, Gateway HTTP throughput independent of Lab boot,
|
||||
and steady-state (long-running, not just boot-burst) resource drift. Extend `boot-labs.js`'s
|
||||
pattern (it already reuses `platform/test/session-helper.js`) rather than building a separate
|
||||
tool if/when those are needed.
|
||||
@@ -0,0 +1,79 @@
|
||||
#!/usr/bin/env node
|
||||
// Load test tooling (Phase 4 §24) — provisions N concurrent Labs via the real admin
|
||||
// Prepare-Labs endpoint (the same code path a real instructor's "Prepare Labs" click uses,
|
||||
// platform/api/src/admin-routes.js), measures boot time and failure rate, and (if run ON the
|
||||
// Lab Host, or given SSH access to it) samples per-Lab CPU/RAM/disk via `docker stats`.
|
||||
//
|
||||
// This produces REAL numbers against whatever stack it's pointed at — a local dev stack today,
|
||||
// the real rn-lab-host-01 once it exists. It does NOT fabricate a concurrency number; the
|
||||
// report is only as meaningful as the host it ran against (see the report template's WARNING
|
||||
// banner when run against anything other than real target-class hardware).
|
||||
//
|
||||
// Usage: GATEWAY_URL=... OWNER_EMAIL=... OWNER_PASSWORD=... node boot-labs.js --count 10
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { makeSession } = require('../../../platform/test/session-helper');
|
||||
|
||||
const GW = process.env.GATEWAY_URL || 'http://localhost:4200';
|
||||
const OWNER_EMAIL = process.env.OWNER_EMAIL;
|
||||
const OWNER_PASSWORD = process.env.OWNER_PASSWORD;
|
||||
const COUNT = Number((process.argv.find((a) => a.startsWith('--count=')) || '--count=5').split('=')[1]);
|
||||
|
||||
if (!OWNER_EMAIL || !OWNER_PASSWORD) {
|
||||
console.error('Set OWNER_EMAIL/OWNER_PASSWORD (bootstrap one first — see platform/api/bootstrap-owner.js)');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const owner = makeSession(GW);
|
||||
await owner.request('POST', '/api/auth/instructor/login', { email: OWNER_EMAIL, password: OWNER_PASSWORD });
|
||||
|
||||
const cohortId = `loadtest-${Date.now()}`;
|
||||
console.log(`Creating cohort ${cohortId} with ${COUNT} students...`);
|
||||
await owner.request('POST', '/api/admin/cohorts', { cohort_id: cohortId, name: 'Load Test' });
|
||||
|
||||
for (let i = 0; i < COUNT; i++) {
|
||||
await owner.request('POST', '/api/admin/students', { display_name: `LoadTest-${i}`, cohort_id: cohortId });
|
||||
}
|
||||
|
||||
console.log(`Prepare Labs: booting ${COUNT} Labs concurrently...`);
|
||||
const start = Date.now();
|
||||
const { status, body } = await owner.request('POST', `/api/admin/cohorts/${cohortId}/prepare-labs`);
|
||||
const elapsedMs = Date.now() - start;
|
||||
|
||||
if (status !== 200) {
|
||||
console.error('Prepare Labs call itself failed:', status, body);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const running = body.results.filter((r) => r.status === 'running').length;
|
||||
const failed = body.results.filter((r) => r.status === 'error');
|
||||
|
||||
const report = {
|
||||
timestamp: new Date().toISOString(),
|
||||
gateway_url: GW,
|
||||
requested_count: COUNT,
|
||||
running_count: running,
|
||||
failed_count: failed.length,
|
||||
failure_rate_pct: Math.round((failed.length / COUNT) * 1000) / 10,
|
||||
total_boot_wall_time_ms: elapsedMs,
|
||||
avg_boot_time_ms_estimate: Math.round(elapsedMs / COUNT), // concurrent, so this is a rough
|
||||
// lower bound, not a real per-Lab figure
|
||||
failures: failed,
|
||||
note: 'Run infra/scripts/loadtest/sample-resources.sh DURING this script (in another '
|
||||
+ 'terminal) for real per-Lab CPU/RAM numbers — this script only measures boot '
|
||||
+ 'timing/success rate, not resource consumption.',
|
||||
};
|
||||
|
||||
const outDir = path.join(__dirname, 'results');
|
||||
fs.mkdirSync(outDir, { recursive: true });
|
||||
const outFile = path.join(outDir, `boot-labs-${Date.now()}.json`);
|
||||
fs.writeFileSync(outFile, JSON.stringify(report, null, 2));
|
||||
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
console.log(`\nWritten to ${outFile}`);
|
||||
console.log(`\nCleanup: POST /api/admin/cohorts/${cohortId}/destroy-labs as the owner when done.`);
|
||||
}
|
||||
|
||||
main().catch((e) => { console.error('Load test crashed:', e); process.exit(1); });
|
||||
@@ -0,0 +1,36 @@
|
||||
# RootNode Load Test Report
|
||||
|
||||
**Date:** _______
|
||||
**Ran against:** [ ] dev laptop [ ] staging [ ] production rn-lab-host-01 (real hardware)
|
||||
**Host spec (fill in real numbers, do not guess):** ___ cores / ___ threads / ___ GB RAM / ___ storage
|
||||
|
||||
> A run against anything other than the real target-class hardware only tells you the code
|
||||
> path works, not what production can actually sustain. Say clearly which this is.
|
||||
|
||||
## Boot test (`boot-labs.js`)
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| Requested Lab count | |
|
||||
| Successfully running | |
|
||||
| Failed | |
|
||||
| Failure rate | |
|
||||
| Total wall-clock time (concurrent boot) | |
|
||||
|
||||
## Resource usage (`sample-resources.sh`)
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| Peak CPU per Lab (observed) | |
|
||||
| Peak memory per Lab (observed) | |
|
||||
| Peak disk I/O per Lab (observed) | |
|
||||
| Host CPU headroom remaining at peak | |
|
||||
| Host memory headroom remaining at peak | |
|
||||
|
||||
## Feed into docs/CAPACITY_PLANNING.md
|
||||
|
||||
- [ ] Updated `measured_lab_cpu_cores` with the peak CPU figure above
|
||||
- [ ] Updated `measured_lab_mem_mb` with the peak memory figure above
|
||||
- [ ] Re-ran `infra/scripts/capacity-model.js` and recorded the new recommended-concurrent-Labs number
|
||||
|
||||
## Notes / anomalies observed
|
||||
Executable
+22
@@ -0,0 +1,22 @@
|
||||
#!/usr/bin/env bash
|
||||
# Run on rn-lab-host-01 (or the local dev machine running the Lab containers) WHILE
|
||||
# boot-labs.js is running, to capture real per-Lab CPU/RAM/disk usage — `docker stats` is the
|
||||
# ground truth here, not an estimate. Samples every 5s for the given duration and writes CSV.
|
||||
set -euo pipefail
|
||||
DURATION_SEC="${1:-120}"
|
||||
OUT="${2:-./results/resource-sample-$(date +%s).csv}"
|
||||
mkdir -p "$(dirname "$OUT")"
|
||||
|
||||
echo "timestamp,container,cpu_pct,mem_usage,mem_pct,net_io,block_io,pids" > "$OUT"
|
||||
END=$(( $(date +%s) + DURATION_SEC ))
|
||||
while [ "$(date +%s)" -lt "$END" ]; do
|
||||
docker stats --no-stream --format '{{.Name}},{{.CPUPerc}},{{.MemUsage}},{{.MemPerc}},{{.NetIO}},{{.BlockIO}},{{.PIDs}}' \
|
||||
| grep -E '^rootnode-lab-' \
|
||||
| while IFS=, read -r name cpu mem memp net block pids; do
|
||||
echo "$(date -u +%Y-%m-%dT%H:%M:%SZ),$name,$cpu,$mem,$memp,$net,$block,$pids" >> "$OUT"
|
||||
done
|
||||
sleep 5
|
||||
done
|
||||
|
||||
echo "Written to $OUT"
|
||||
echo "Summarize with: awk -F, 'NR>1{print \$2}' $OUT | sort -u # containers seen"
|
||||
Reference in New Issue
Block a user