test(preprod): two-host routing simulation, drill suite, CI security scanning

- infra/scripts/simulate-two-host/: real two-Docker-network simulation (Gateway genuinely isolated from any Lab bridge network) proving the Lab Host Router fix - 13/13 real assertions (HTTP+WebSocket through the Router, Alice/Bob isolation, garbage/destroyed tokens, mTLS-only Router access). - infra/scripts/drills/: Lab Host loss/rebuild (9/9), deployment rollback, CI-fails-closed (6 required failure modes), fake cohort dress rehearsal (14/14, one continuous narrative through the real app). - infra/ci/gitea-workflows/build-and-test.yml: new security-scan job (npm audit, gitleaks, hadolint, shellcheck, trivy - all free/open-source), severity policy in docs/SECURITY_SCANNING.md (CRITICAL/HIGH blocks, MEDIUM/LOW reported only). Gates build-images alongside regression. Found and fixed two real pre-existing bugs via the CI-fails-closed drill: staging-smoke-test.sh and production-smoke-test.sh both had a classic set -e gotcha (curl ... && echo "OK" as one statement - a failing command that isn't last in an && chain doesn't trigger -e) that let an unreachable/broken endpoint fall through silently instead of failing the smoke test. platform/test/e2e-phase3a.js updated for the new paginated /api/admin/audit-events response shape (Phase 4.1 pagination work).
2026-08-24 20:16:00 -05:00
parent f5110d3c1b
commit f3c356951c
15 changed files with 1067 additions and 9 deletions
@@ -0,0 +1,47 @@
 # Security Scanning (Phase 4.1 §24)
 `infra/ci/gitea-workflows/build-and-test.yml`'s `security-scan` job. All free, open-source,
 no account/signup required for any of them — no paid scanner was added.
 ## What's scanned
 | Tool | Checks | Scope |
 |---|---|---|
 | `npm audit --audit-level=high` | Known-vulnerable dependencies | Every first-party Node service |
 | [gitleaks](https://github.com/gitleaks/gitleaks) | Committed secrets (keys, tokens, credentials) | Whole repo |
 | [hadolint](https://github.com/hadolint/hadolint) | Dockerfile best-practice/security lint | Every first-party Dockerfile |
 | [shellcheck](https://www.shellcheck.net/) | Shell script correctness/security | `infra/**/*.sh` |
 | [trivy](https://github.com/aquasecurity/trivy) | OS/library CVEs in a built container image | `platform-api` image (representative — same base/layers as the other Node services) |
 ## Severity policy
 - **CRITICAL / HIGH** — fails the build (`security-scan` blocks `build-images`, which blocks
  `deploy-staging`, which blocks production promotion — see the job dependency chain in the
  workflow file).
 - **MEDIUM / LOW** — reported in the job output, never blocks. A build that's permanently red
  over a low-severity transitive dependency finding nobody can immediately fix trains everyone
  to ignore red builds, which is worse than not scanning at all.
 `npm audit --audit-level=high` and trivy's `--severity HIGH,CRITICAL` both implement this
 directly at the tool level, not via manual triage.
 ## What's NOT in the automated job (and why)
 - **Dedicated IaC security linting** (checkov/tflint) — not added. `infra/scripts/validate-compose.sh`,
  `infra/firewall/validate-firewall.sh`, `ansible-playbook --syntax-check`, and `tofu validate`
  (all in the `regression` job) already catch structural/syntax problems in the infra code;
  a dedicated security-policy linter for Terraform/Ansible is a reasonable Phase 5 addition once
  there's a real cloud footprint to misconfigure, not before.
 - **DAST/penetration testing** — out of scope for CI; this is a build-time gate, not a running-system
  test. Covered separately by `docs/PRODUCTION_VALIDATION_CHECKLIST.md`'s security section once
  real hardware exists.
 - **SAST beyond hadolint/shellcheck** — the codebase is small enough (a handful of first-party
  Node services) that `npm audit` + secret scanning + Dockerfile/shell lint cover the realistic
  risk surface; a dedicated SAST tool (Semgrep, CodeQL) is a reasonable addition once the
  codebase or team grows enough to justify the added CI time.
 ## Local reproduction
 Every check above can be run locally with the exact same free tools — no CI-only dependency.
 See the `security-scan` job's steps in `infra/ci/gitea-workflows/build-and-test.yml` for the
 exact commands (they're copy-pasteable).
@@ -40,16 +40,19 @@ jobs:
      - name: Install dependencies (all first-party services)
        run: |
-          for d in backend frontend platform/api platform/gateway platform/provisioner; do
+          for d in backend frontend platform/api platform/gateway platform/provisioner platform/lab-router; do
            (cd "$d" && npm ci)
          done
      - name: Syntax-check every service
        run: |
-          for d in backend frontend/src platform/api platform/gateway platform/provisioner; do
+          for d in backend frontend/src platform/api platform/gateway platform/provisioner platform/lab-router; do
            find "$d" -name '*.js' -not -path '*/node_modules/*' -exec node --check {} \;
          done
      - name: Lab Host Router unit tests (Phase 4.1 — fast, no Docker daemon needed)
        run: (cd platform/lab-router && npm test)
      - name: Build frontend
        run: (cd frontend && npm run build)
@@ -75,8 +78,56 @@ jobs:
      # can't grant safely — they stay a LOCAL/staging pre-promotion step
      # (infra/scripts/staging-smoke-test.sh), not this job. See docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §18.
  # Phase 4.1 §24 — free/open-source security scanning, no paid tooling, no signup required for
  # any of these. Severity policy (docs/SECURITY_SCANNING.md): CRITICAL/HIGH findings fail the
  # build; MEDIUM/LOW are reported but never block a merge — a build that's permanently red over
  # an unrelated low-severity transitive dependency finding trains everyone to ignore red builds.
  security-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
      - name: npm audit (dependency vulnerabilities) — fails only on high/critical
        run: |
          FAIL=0
          for d in backend frontend platform/api platform/gateway platform/provisioner platform/lab-router; do
            echo "== $d =="
            (cd "$d" && npm audit --audit-level=high) || FAIL=1
          done
          exit $FAIL
      - name: Secret scanning (gitleaks — free, no account needed)
        run: docker run --rm -v "$PWD:/repo" zricethezav/gitleaks:latest detect --source=/repo --no-git -v
      - name: Dockerfile linting (hadolint — free)
        run: |
          FAIL=0
          for f in Dockerfile platform/api/Dockerfile platform/gateway/Dockerfile platform/provisioner/Dockerfile platform/egress-proxy/Dockerfile platform/lab-router/Dockerfile; do
            echo "== $f =="
            docker run --rm -i hadolint/hadolint < "$f" || FAIL=1
          done
          exit $FAIL
      # -S error, not warning — matches docs/SECURITY_SCANNING.md's severity policy. Real run
      # against this tree at -S warning surfaces only style nits (unused loop vars, a missing
      # `cd ... || exit` guard) on scripts that are otherwise correct; failing the build on
      # those trains everyone to ignore the check.
      - name: Shell linting (shellcheck — free, standard)
        run: |
          sudo apt-get update -qq && sudo apt-get install -y -qq shellcheck
          find infra -name '*.sh' -exec shellcheck -S error {} \+
      - name: Container image scanning (trivy — free, no account needed)
        run: |
          docker build -q -t rootnode-scan/platform-api -f platform/api/Dockerfile .
          docker run --rm -v /var/run/docker.sock:/var/run/docker.sock aquasec/trivy:latest image \
            --severity HIGH,CRITICAL --exit-code 1 rootnode-scan/platform-api
  build-images:
-    needs: regression
+    needs: [regression, security-scan]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
@@ -0,0 +1,31 @@
 #!/usr/bin/env bash
 # Orchestrates the Phase 4.1 §27 fake cohort dress rehearsal: brings up the real dev stack,
 # bootstraps an owner, runs test-fake-cohort-rehearsal.js, tears down.
 set -euo pipefail
 cd "$(dirname "${BASH_SOURCE[0]}")/../../../platform"
 export PROVISIONER_SHARED_SECRET="${PROVISIONER_SHARED_SECRET:-rehearsal-shared-secret-not-for-production}"
 docker compose -f docker-compose.dev.yml up -d --build
 cleanup() {
  echo "==> Tearing down"
  docker ps -aq --filter "name=rootnode-lab-" | xargs -r docker rm -f >/dev/null 2>&1 || true
  docker network ls --filter "name=rootnode-lab-net-" --format "{{.Name}}" | xargs -r -n1 docker network rm >/dev/null 2>&1 || true
  docker compose -f docker-compose.dev.yml down >/dev/null 2>&1 || true
 }
 trap cleanup EXIT
 echo "==> Waiting for the stack"
 for _ in $(seq 1 60); do
  curl -sf http://localhost:4300/healthz >/dev/null 2>&1 && curl -sf http://localhost:4200/healthz >/dev/null 2>&1 && break
  sleep 2
 done
 echo "==> Bootstrapping the rehearsal owner"
 ( cd api && PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5433/rootnode_platform \
    node bootstrap-owner.js [email protected] "Rehearsal Owner" 'RehearsalPass!2026' )
 echo "==> Running the rehearsal"
 OWNER_EMAIL=[email protected] OWNER_PASSWORD='RehearsalPass!2026' \
  node ../infra/scripts/drills/test-fake-cohort-rehearsal.js
@@ -0,0 +1,32 @@
 #!/usr/bin/env bash
 # Orchestrates the Phase 4.1 §21 Lab Host loss/rebuild drill: brings up the real dev stack,
 # bootstraps an owner, runs test-lab-host-loss.js, tears down. See that file's header for what
 # it actually proves.
 set -euo pipefail
 cd "$(dirname "${BASH_SOURCE[0]}")/../../../platform"
 export PROVISIONER_SHARED_SECRET="${PROVISIONER_SHARED_SECRET:-drill-shared-secret-not-for-production}"
 docker compose -f docker-compose.dev.yml up -d --build
 cleanup() {
  echo "==> Tearing down"
  docker ps -aq --filter "name=rootnode-lab-" | xargs -r docker rm -f >/dev/null 2>&1 || true
  docker network ls --filter "name=rootnode-lab-net-" --format "{{.Name}}" | xargs -r -n1 docker network rm >/dev/null 2>&1 || true
  docker compose -f docker-compose.dev.yml down >/dev/null 2>&1 || true
 }
 trap cleanup EXIT
 echo "==> Waiting for the stack"
 for i in $(seq 1 60); do
  curl -sf http://localhost:4300/healthz >/dev/null 2>&1 && curl -sf http://localhost:4200/healthz >/dev/null 2>&1 && break
  sleep 2
 done
 echo "==> Bootstrapping the drill owner"
 ( cd api && PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5433/rootnode_platform \
    node bootstrap-owner.js [email protected] "LabHost Drill Owner" 'LabHostDrillPass!2026' )
 echo "==> Running the drill"
 OWNER_EMAIL=[email protected] OWNER_PASSWORD='LabHostDrillPass!2026' \
  node ../infra/scripts/drills/test-lab-host-loss.js
@@ -0,0 +1,77 @@
 #!/usr/bin/env bash
 # Phase 4.1 §23 — proves the CI pipeline (infra/ci/gitea-workflows/build-and-test.yml) fails
 # CLOSED on every listed failure mode, i.e. the underlying command genuinely exits nonzero on
 # broken input, which is what makes a Gitea/GitHub Actions step (and therefore the job, and
 # therefore `needs: regression` gating build-images/deploy-staging) fail. Doesn't need a real
 # CI run — checks the actual commands the workflow calls, directly, the same way Actions would
 # invoke them.
 set -uo pipefail   # NOT -e — this script's whole point is deliberately triggering failures and checking their exit codes
 cd "$(dirname "${BASH_SOURCE[0]}")/../../.."   # repo root
 FAIL=0
 expect_nonzero() {
  local desc="$1"; shift
  "$@" >/tmp/ci-fail-drill-out 2>&1
  local code=$?
  if [ "$code" -eq 0 ]; then
    echo "  FAIL — $desc (command succeeded, but it should have failed)"; cat /tmp/ci-fail-drill-out; FAIL=1
  else
    echo "  ok — $desc correctly failed (exit $code)"
  fi
 }
 echo "Phase 4.1 CI-fails-closed drill"
 echo
 echo "== 1. Test/syntax-check failure =="
 BAD_JS="$(mktemp).js"
 echo 'function broken( {' > "$BAD_JS"
 expect_nonzero "node --check on syntactically broken JS" node --check "$BAD_JS"
 rm -f "$BAD_JS"
 echo "== 2. Frontend build failure =="
 cp frontend/src/App.jsx /tmp/App.jsx.bak
 echo 'this is not valid jsx <<<' >> frontend/src/App.jsx
 (cd frontend && expect_nonzero "npm run build with a broken component" npm run build)
 cp /tmp/App.jsx.bak frontend/src/App.jsx; rm -f /tmp/App.jsx.bak
 echo "== 3. Image build failure =="
 TMPDOCKERFILE="$(mktemp -d)/Dockerfile"
 cat > "$TMPDOCKERFILE" <<'EOF'
 FROM node:20-slim
 COPY this-file-does-not-exist.js /app/
 EOF
 expect_nonzero "docker build with a COPY of a nonexistent file" docker build -q -f "$TMPDOCKERFILE" "$(dirname "$TMPDOCKERFILE")"
 echo "== 4. Config validation failure =="
 BAD_COMPOSE="$(mktemp -d)/docker-compose.yml"
 cat > "$BAD_COMPOSE" <<'EOF'
 services:
  a:
    image: alpine
  a:
    image: alpine
 EOF
 expect_nonzero "docker compose config on a file with a duplicate service key" docker compose -f "$BAD_COMPOSE" config
 echo "== 5. Migration preflight failure =="
 ( cd platform/api && expect_nonzero "runMigrations() against an unreachable Postgres" \
    env PLATFORM_DB_URL="postgres://nobody:[email protected]:1/nope" \
    node -e "require('./src/db').runMigrations().then(()=>process.exit(0)).catch((e)=>{console.error(e.message);process.exit(1)})" )
 echo "== 6. Staging smoke test failure =="
 expect_nonzero "staging-smoke-test.sh with no STAGING_GATEWAY_URL set" \
  env -u STAGING_GATEWAY_URL bash infra/scripts/staging-smoke-test.sh
 expect_nonzero "staging-smoke-test.sh against an unreachable staging URL" \
  env STAGING_GATEWAY_URL="http://127.0.0.1:1" bash infra/scripts/staging-smoke-test.sh
 echo
 if [ "$FAIL" = "1" ]; then
  echo "DRILL FAILED — at least one failure mode did NOT fail closed. See above."
  exit 1
 fi
 echo "OK: every listed failure mode genuinely exits nonzero — CI fails closed on all of them."
 echo "NOT TESTED HERE: an actual Gitea Actions run (needs Gitea Actions enabled on the real"
 echo "instance and this workflow file activated at .gitea/workflows/ — see build-and-test.yml's"
 echo "own header comment). This proves the underlying commands, which is what actually"
 echo "determines pass/fail once a real run happens."
@@ -0,0 +1,74 @@
 #!/usr/bin/env bash
 # Phase 4.1 §22 — deployment rollback drill. Builds two real gateway image versions (a known
 # good build, and a deliberately broken one), "deploys" each in turn to a local container,
 # proves the smoke test genuinely catches the regression, then rolls back using the EXACT same
 # decision logic rollback-production.sh uses against deploy-history.log (tail -2 | head -1 |
 # awk '{print $2}' — the tag before the current one), and proves the rollback recovers.
 #
 # What this proves: the deterministic-tag + deploy-history.log rollback MECHANISM is real and
 # correct. What it does NOT prove: the actual `ansible-playbook site.yml -e image_tag=...` step
 # rollback-production.sh runs against a real host — that needs the real hardware
 # (docs/PRODUCTION_VALIDATION_CHECKLIST.md). This drill swaps a local Docker container's image
 # tag directly instead, as the closest local equivalent.
 set -euo pipefail
 cd "$(dirname "${BASH_SOURCE[0]}")/../../../platform/gateway"
 CONTAINER=rootnode-rollback-drill
 PORT=4295
 DEPLOY_LOG="$(mktemp)"
 trap 'docker rm -f "$CONTAINER" >/dev/null 2>&1 || true; rm -f "$DEPLOY_LOG"; rm -rf /tmp/rootnode-rollback-drill-broken' EXIT
 smoke_test() {
  curl -sf "http://localhost:$PORT/healthz" >/dev/null 2>&1
 }
 echo "==> [1/6] Building the known-good gateway image (current source)"
 docker build -q -t rootnode-gateway-drill:known-good -f Dockerfile . >/dev/null
 echo "==> [2/6] Building a deliberately broken version (healthz returns 500)"
 rm -rf /tmp/rootnode-rollback-drill-broken
 cp -r . /tmp/rootnode-rollback-drill-broken
 sed -i "s#app.get('/healthz', (req, res) => res.json({ ok: true }));#app.get('/healthz', (req, res) => res.status(500).json({ ok: false, broken: 'DELIBERATE_DRILL_REGRESSION' }));#" \
  /tmp/rootnode-rollback-drill-broken/src/server.js
 grep -q DELIBERATE_DRILL_REGRESSION /tmp/rootnode-rollback-drill-broken/src/server.js || { echo "FAIL: sed didn't apply — healthz line text must have changed upstream"; exit 1; }
 docker build -q -t rootnode-gateway-drill:broken -f /tmp/rootnode-rollback-drill-broken/Dockerfile /tmp/rootnode-rollback-drill-broken >/dev/null
 deploy() {
  local tag="$1"
  docker rm -f "$CONTAINER" >/dev/null 2>&1 || true
  docker run -d --name "$CONTAINER" -p "$PORT:4200" -e GATEWAY_PORT=4200 -e PROVISIONER_URL=http://127.0.0.1:1 \
    -e PROVISIONER_SHARED_SECRET=drill -e PLATFORM_API_URL=http://127.0.0.1:1 \
    "rootnode-gateway-drill:$tag" >/dev/null
  sleep 2
  echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) $tag" >> "$DEPLOY_LOG"
 }
 echo "==> [3/6] Deploying known-good (deploy #1)"
 deploy known-good
 smoke_test && echo "  smoke test: PASS (as expected)" || { echo "FAIL: known-good should have passed"; exit 1; }
 echo "==> [4/6] Deploying the broken version (deploy #2 — the 'bad release' this drill catches)"
 deploy broken
 if smoke_test; then echo "FAIL: the broken deploy's regression was NOT caught by the smoke test — that defeats the whole point"; exit 1; fi
 echo "  smoke test: FAIL (correctly caught — this is exactly what should stop a real promotion, see docs/RUNBOOK_ROLLBACK.md)"
 echo "==> [5/6] Computing the rollback target — same logic as infra/scripts/rollback-production.sh"
 TARGET_TAG=$(tail -2 "$DEPLOY_LOG" | head -1 | awk '{print $2}')
 [ "$TARGET_TAG" = "known-good" ] || { echo "FAIL: expected rollback target 'known-good', computed '$TARGET_TAG'"; exit 1; }
 echo "  rollback target: $TARGET_TAG (correctly identified as the tag before the broken one)"
 echo "==> [6/6] Rolling back"
 deploy "$TARGET_TAG"
 smoke_test && echo "  smoke test: PASS — rollback recovered service" || { echo "FAIL: rollback did not recover"; exit 1; }
 echo
 echo "OK: deployment rollback drill passed. Deterministic image tags + deploy-history.log"
 echo "correctly identify and recover from a bad release."
 echo
 echo "On migration compatibility (docs/RUNBOOK_FAILED_MIGRATION.md, docs/RUNBOOK_ROLLBACK.md):"
 echo "this script rolls back APPLICATION IMAGES ONLY. There is no automatic down-migration —"
 echo "if the bad release also ran a schema-changing migration, rolling the image back does NOT"
 echo "undo that migration, and old code may not run correctly against the new schema. That"
 echo "case requires restoring from the pre-migration backup (docs/RUNBOOK_POSTGRES_RESTORE.md),"
 echo "not a rollback, which is exactly why migrations in this project are check-before-alter"
 echo "and additive-only (docs/ENGINEERING_SOP.md §6) — to make image-only rollback usually safe."
@@ -0,0 +1,150 @@
 #!/usr/bin/env node
 // Phase 4.1 §27 — the fake cohort dress rehearsal. One continuous narrative through the real
 // application (not synthetic unit assertions): instructor login, cohort + 4 students, assign
 // curriculum, Prepare Labs, students launch, work different scenarios, use hints, fail a
 // validation, complete scenarios, instructor checks analytics, an exam is assigned/taken,
 // a Lab is reset, a Lab is reaped (destroyed out-of-band, the same consequence the real idle
 // reaper produces — see the note below on why this doesn't wait out the real multi-minute
 // reaper timer), the student returns and continues, capstone state is checked, and the cohort's
 // Labs are destroyed at the end.
 //
 // Run via run-fake-cohort-rehearsal.sh, which brings up platform/docker-compose.dev.yml first.
 // No framework, matching project convention. Problems found are printed, not silently passed —
 // see the Phase 4.1 report for anything this surfaced.
 const assert = require('assert');
 const { makeSession } = require('../../../platform/test/session-helper');
 const GW = process.env.GATEWAY_URL || 'http://localhost:4200';
 const OWNER_EMAIL = process.env.OWNER_EMAIL;
 const OWNER_PASSWORD = process.env.OWNER_PASSWORD;
 if (!OWNER_EMAIL || !OWNER_PASSWORD) { console.error('Set OWNER_EMAIL and OWNER_PASSWORD'); process.exit(1); }
 let passed = 0, failed = 0;
 async function step(name, fn) {
  try { await fn(); console.log(`  ok — ${name}`); passed++; }
  catch (e) { console.error(`  FAIL — ${name}\n    ${e.stack || e.message}`); failed++; }
 }
 async function main() {
  console.log('Phase 4.1 fake cohort dress rehearsal — slow, real k3s boots\n');
  const instructor = makeSession(GW);
  await instructor.request('POST', '/api/auth/instructor/login', { email: OWNER_EMAIL, password: OWNER_PASSWORD });
  const names = ['Rehearsal-Ada', 'Rehearsal-Bo', 'Rehearsal-Cy', 'Rehearsal-Di'];
  const students = {};
  const sessions = {};
  await step('instructor: create a cohort and roster 4 students', async () => {
    await instructor.request('POST', '/api/admin/cohorts', { cohort_id: 'rehearsal-cohort', name: 'Rehearsal Cohort' });
    for (const name of names) {
      const { body } = await instructor.request('POST', '/api/admin/students', { display_name: name, cohort_id: 'rehearsal-cohort' });
      students[name] = body.student_id;
      sessions[name] = makeSession(GW);
      await sessions[name].request('POST', '/api/login', { student_id: body.student_id });
    }
  });
  await step('instructor: assign curriculum to the cohort', async () => {
    const { status } = await instructor.request('POST', '/api/admin/cohorts/rehearsal-cohort/curriculum', { curriculum_path: 'rootnode-path' });
    assert.ok([200, 201].includes(status));
  });
  await step('instructor: Prepare Labs for the whole cohort (concurrent)', async () => {
    const { status, body } = await instructor.request('POST', '/api/admin/cohorts/rehearsal-cohort/prepare-labs');
    assert.equal(status, 200);
    assert.ok(body.results.every((r) => r.status === 'running'), `expected every Lab running, got: ${JSON.stringify(body.results)}`);
  });
  const launches = {};
  await step('students: everyone\'s prepared Lab is already running (no re-launch wait)', async () => {
    for (const name of names) {
      const { body } = await sessions[name].request('POST', '/api/environment/launch');
      assert.equal(body.status, 'running');
      launches[name] = body;
    }
  });
  await step('students: Ada and Bo work DIFFERENT scenarios', async () => {
    await sessions['Rehearsal-Ada'].request('POST', `${launches['Rehearsal-Ada'].launch_path}api/scenarios/pod-basics-mcq/answer`, { selected: 'b' });
    await sessions['Rehearsal-Bo'].request('POST', `${launches['Rehearsal-Bo'].launch_path}api/scenarios/pod-basics-mcq/start`);
  });
  await step('student: Cy uses a hint on a scenario', async () => {
    const res = await sessions['Rehearsal-Cy'].request('POST', `${launches['Rehearsal-Cy'].launch_path}api/scenarios/pod-basics-mcq/hint`);
    assert.equal(res.status, 200);
  });
  await step('student: Di fails a validation on purpose', async () => {
    const res = await sessions['Rehearsal-Di'].request('POST', `${launches['Rehearsal-Di'].launch_path}api/scenarios/pod-basics-mcq/answer`, { selected: 'a' });
    assert.equal(res.status, 200);
    assert.equal(res.body.correct, false, 'expected a deliberately wrong answer to be marked incorrect');
  });
  await step('instructor: cohort learning summary reflects real activity', async () => {
    const { body } = await instructor.request('GET', '/api/admin/cohorts/rehearsal-cohort/learning');
    assert.equal(body.student_count, 4);
  });
  await step('instructor: needs-attention returns real signals for the struggling students, no punitive language', async () => {
    const { body } = await instructor.request('GET', '/api/admin/cohorts/rehearsal-cohort/needs-attention');
    const text = JSON.stringify(body).toLowerCase();
    assert.ok(!text.includes('fail') || text.includes('failed validations'), 'the word "fail" should only appear in a factual "N failed validations" context, never as a standalone judgment');
  });
  await step('instructor: reset Ada\'s Lab — academic progress is untouched', async () => {
    const before = await sessions['Rehearsal-Ada'].request('GET', '/api/progress');
    // Admin environment routes need the real environment_id, which publicEnvView doesn't echo
    // back to the student directly — resolved via the student environment lookup instead.
    const envRes = await instructor.request('GET', `/api/admin/students/${students['Rehearsal-Ada']}/environment`);
    const env = envRes.body.find((e) => e.status !== 'destroyed');
    await instructor.request('POST', `/api/admin/environments/${env.environment_id}/reset`);
    const after = await sessions['Rehearsal-Ada'].request('GET', '/api/progress');
    assert.deepEqual(before.body.details, after.body.details, 'progress must be identical before/after a Lab reset');
  });
  await step('§27 — a Lab is reaped: same consequence as the real idle reaper (real multi-minute timer not waited out here — see reaper.js/resource-policy.yaml for the real thresholds; this exercises the destroy-and-preserve-progress consequence directly, already proven mechanistically by infra/scripts/drills/test-lab-host-loss.js)', async () => {
    const before = await sessions['Rehearsal-Bo'].request('GET', '/api/progress');
    const envRes = await instructor.request('GET', `/api/admin/students/${students['Rehearsal-Bo']}/environment`);
    const env = envRes.body.find((e) => e.status !== 'destroyed');
    await instructor.request('DELETE', `/api/admin/environments/${env.environment_id}`);
    const after = await sessions['Rehearsal-Bo'].request('GET', '/api/progress');
    assert.deepEqual(before.body.details, after.body.details, 'progress must survive a reaped Lab unchanged');
  });
  await step('student: Bo returns later — a fresh Lab is created and curriculum continues', async () => {
    const { status, body } = await sessions['Rehearsal-Bo'].request('POST', '/api/environment/launch');
    assert.equal(status, 200); assert.equal(body.status, 'running');
    const curriculum = await sessions['Rehearsal-Bo'].request('GET', '/api/curriculum/path');
    assert.ok(curriculum.body.stages.length > 0, 'curriculum must still be there for the returning student');
  });
  await step('instructor: assign an exam to the cohort and Ada takes it', async () => {
    const create = await instructor.request('POST', '/api/admin/assessments', {
      id: 'rehearsal-exam', type: 'exam', title: 'Rehearsal Exam', duration_minutes: 30, hint_policy: 'disabled',
      pool_spec: { required: ['pod-basics-mcq'], random: [] },
    });
    assert.ok([200, 201, 409].includes(create.status)); // 409 if a prior run already created it — fine, idempotent enough for a rehearsal
    const assign = await instructor.request('POST', '/api/admin/assessments/rehearsal-exam/assign', { cohort_id: 'rehearsal-cohort' });
    assert.ok([200, 201].includes(assign.status));
    const start = await sessions['Rehearsal-Ada'].request('POST', '/api/assessments/rehearsal-exam/start');
    assert.equal(start.status, 201);
    assert.ok(start.body.scenarios.length > 0);
  });
  await step('instructor: destroy all cohort Labs after class — progress remains intact for everyone', async () => {
    const before = {};
    for (const name of names) before[name] = (await sessions[name].request('GET', '/api/progress')).body.details;
    await instructor.request('POST', '/api/admin/cohorts/rehearsal-cohort/destroy-labs');
    for (const name of names) {
      const after = (await sessions[name].request('GET', '/api/progress')).body.details;
      assert.deepEqual(before[name], after, `${name}'s progress must survive end-of-class Lab destruction`);
    }
  });
  console.log(`\n${passed} passed, ${failed} failed`);
  if (failed > 0) process.exit(1);
 }
 main().catch((e) => { console.error(e); process.exit(1); });
@@ -0,0 +1,104 @@
 #!/usr/bin/env node
 // Phase 4.1 §21 — Lab Host loss/rebuild drill. Proves disposable infrastructure is genuinely
 // disposable: a student's Lab container is destroyed OUT FROM UNDER the Provisioner (no clean
 // destroy call — simulating the whole rn-lab-host-01 VM disappearing), and the system must:
 //   - keep the Platform (API/Gateway) up throughout — it never depended on that one container
 //   - never present the now-gone environment as healthy/running once reconciliation runs
 //   - leave academic data (progress/attempts) completely untouched
 //   - let the student launch a brand-new Lab afterward and continue where they left off
 //
 // Run via run-lab-host-loss-drill.sh, which brings up platform/docker-compose.dev.yml first.
 // No framework, matching project convention.
 const assert = require('assert');
 const { execFileSync } = require('child_process');
 const { makeSession } = require('../../../platform/test/session-helper');
 const GW = process.env.GATEWAY_URL || 'http://localhost:4200';
 const API = process.env.PLATFORM_API_URL || 'http://localhost:4300';
 const OWNER_EMAIL = process.env.OWNER_EMAIL;
 const OWNER_PASSWORD = process.env.OWNER_PASSWORD;
 if (!OWNER_EMAIL || !OWNER_PASSWORD) { console.error('Set OWNER_EMAIL and OWNER_PASSWORD'); process.exit(1); }
 let passed = 0, failed = 0;
 async function test(name, fn) {
  try { await fn(); console.log(`  ok — ${name}`); passed++; }
  catch (e) { console.error(`  FAIL — ${name}\n    ${e.stack || e.message}`); failed++; }
 }
 async function main() {
  console.log('Phase 4.1 Lab Host loss/rebuild drill — slow, real k3s boots\n');
  const admin = makeSession(GW);
  const { status: loginStatus } = await admin.request('POST', '/api/auth/instructor/login', { email: OWNER_EMAIL, password: OWNER_PASSWORD });
  assert.equal(loginStatus, 200, 'owner login must succeed');
  let studentId;
  await test('roster: add a student', async () => {
    const { body } = await admin.request('POST', '/api/admin/students', { display_name: 'Drill-LabHostLoss', cohort_id: null });
    studentId = body.student_id;
  });
  const session = makeSession(GW);
  await session.request('POST', '/api/login', { student_id: studentId });
  let launch;
  await test('launch: student gets a running Lab (real k3s boot)', async () => {
    const { status, body } = await session.request('POST', '/api/environment/launch');
    assert.equal(status, 200); assert.equal(body.status, 'running');
    launch = body;
  });
  await test('progress: student completes a real scenario before the Lab Host is lost', async () => {
    const res = await session.request('POST', `${launch.launch_path}api/scenarios/pod-basics-mcq/answer`, { selected: 'b' });
    assert.equal(res.status, 200);
    const progress = await session.request('GET', '/api/progress');
    assert.equal(progress.body.details['pod-basics-mcq']?.status, 'completed', 'progress must show completed before we destroy anything');
  });
  const token = launch.launch_path.split('/')[2];
  await test('§21 — simulate total Lab Host loss: kill the Lab container directly, bypassing the Provisioner entirely', async () => {
    const containers = execFileSync('docker', ['ps', '--filter', 'name=rootnode-lab-', '--format', '{{.Names}}']).toString().trim().split('\n').filter(Boolean);
    assert.ok(containers.length >= 1, 'expected at least one running Lab container to destroy');
    for (const c of containers) execFileSync('docker', ['rm', '-f', c]);
    console.log(`    destroyed ${containers.length} Lab container(s) out from under the Provisioner`);
  });
  await test('§21 — Platform stays available throughout (it never depended on that container)', async () => {
    const api = await fetch(`${API}/healthz`);
    const gw = await fetch(`${GW}/healthz`);
    assert.equal(api.status, 200);
    assert.equal(gw.status, 200);
  });
  await test('§21 — academic progress is completely untouched by the infrastructure loss', async () => {
    const progress = await session.request('GET', '/api/progress');
    assert.equal(progress.body.details['pod-basics-mcq']?.status, 'completed', 'progress must survive the Lab Host loss unchanged');
  });
  await test('§21 — the now-gone environment is NEVER presented as healthy once reconciliation runs (Provisioner restart, its real startup path)', async () => {
    // cwd is already platform/ — run-lab-host-loss-drill.sh cd's there before invoking this script.
    execFileSync('docker', ['compose', '-f', 'docker-compose.dev.yml', 'restart', 'provisioner'], { stdio: 'inherit' });
    // Give the Provisioner a moment to finish its real reconcileOnStartup pass.
    await new Promise((r) => setTimeout(r, 5000));
    const res = await session.rawFetch(`/lab/${token}/`);
    assert.ok([503, 404].includes(res.status), `expected the stale environment to be rejected (503/404), got ${res.status} — a healthy-looking response here would mean a destroyed environment is being silently presented as available`);
  });
  await test('§21 — a replacement Lab can be launched and boots successfully (the "replacement Lab Host" case, minus needing a literal second VM)', async () => {
    const { status, body } = await session.request('POST', '/api/environment/launch');
    assert.equal(status, 200); assert.equal(body.status, 'running');
    assert.notEqual(body.launch_path, launch.launch_path, 'the replacement environment must be a genuinely new one, not the destroyed one still being served');
  });
  await test('§21 — previous academic progress is still there after the replacement Lab boots', async () => {
    const progress = await session.request('GET', '/api/progress');
    assert.equal(progress.body.details['pod-basics-mcq']?.status, 'completed');
  });
  console.log(`\n${passed} passed, ${failed} failed`);
  if (failed > 0) process.exit(1);
 }
 main().catch((e) => { console.error(e); process.exit(1); });
@@ -6,18 +6,26 @@ set -euo pipefail
 PROD_URL="${PRODUCTION_URL:?set PRODUCTION_URL, e.g. https://academy.example.com}"
 # Phase 4.1 §23: every `curl ... && echo "OK"` below used to be one statement — under `set -e`,
 # a failing command that isn't the LAST one in an && list does not trigger the shell to exit (a
 # well-known bash gotcha), so an unreachable/broken production endpoint used to fall through
 # silently instead of failing this script. Split into two statements each, so the real exit
 # status is what `set -e` sees. Found by infra/scripts/drills/test-ci-fails-closed.sh.
 echo "==> Gateway health"
-curl -sf "$PROD_URL/healthz" > /dev/null && echo "OK"
+curl -sf "$PROD_URL/healthz" > /dev/null
 echo "OK"
 echo "==> TLS certificate is valid and not expiring soon"
 EXPIRY=$(echo | openssl s_client -connect "$(echo "$PROD_URL" | sed -e 's#https://##')":443 -servername "$(echo "$PROD_URL" | sed -e 's#https://##')" 2>/dev/null | openssl x509 -noout -enddate | cut -d= -f2)
 echo "Certificate expires: $EXPIRY"
 echo "==> Roster endpoint responds (public, unauthenticated, read-only)"
-curl -sf "$PROD_URL/api/roster/cohorts" > /dev/null && echo "OK"
+curl -sf "$PROD_URL/api/roster/cohorts" > /dev/null
 echo "OK"
 echo "==> Admin login page loads (does not attempt to log in)"
-curl -sf "$PROD_URL/admin" -o /dev/null -w '%{http_code}\n' | grep -q 200 && echo "OK"
+curl -sf "$PROD_URL/admin" -o /dev/null -w '%{http_code}\n' | grep -q 200
 echo "OK"
 echo
 echo "OK: production smoke test passed. For a full functional check (owner/instructor login,"
@@ -0,0 +1,50 @@
 # Two-Host Routing Simulation
 Proves, with real Docker network isolation (not mocks), that the Gateway-to-Lab data path
 described in `docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1 actually works before the real
 Hetzner/Proxmox hardware exists. See that document for the full design; this is the test harness.
 ## Run it
 ```bash
 cd infra/scripts/simulate-two-host
 PROVISIONER_SHARED_SECRET=... ./run-simulation.sh          # builds, runs, tears down
 PROVISIONER_SHARED_SECRET=... ./run-simulation.sh --keep-up # leaves the stack up for debugging
 ```
 Confirmed passing, 13/13, as of this phase's implementation — see the Phase 4.1 report for the
 full run transcript.
 ## What this genuinely proves (TESTED LOCALLY, with real Docker networking)
 - The `gateway` container, attached ONLY to an isolated bridge network with no route to any
  Lab's per-environment bridge network, **cannot** open a TCP connection to a Lab container's
  Docker-internal IP — verified via a raw connect attempt run from inside the actual gateway
  container's own network namespace, not asserted from the test driver's (irrelevant) vantage
  point.
 - The Gateway **does** reach the Lab correctly through the Lab Host Router, over real mTLS,
  for both a plain HTTP request and a full WebSocket terminal session — including a genuine
  `kubectl get nodes` executed inside the Lab and its real output read back.
 - Alice/Bob isolation (own lab ALLOW, other's lab DENY, including over WebSocket), garbage
  tokens, anonymous requests, and destroyed-environment tokens are all rejected through the
  real code path — the same `checkOwnership`/`resolveTarget` logic production will run.
 - The Lab Host Router refuses a connection with no client certificate at all — mTLS is the
  actual authentication mechanism, not a documented intention.
 ## What this does NOT prove (SIMULATED PRODUCTION BOUNDARY, not identical to real hardware)
 - Real cross-VM latency, MTU, or anything about two genuinely separate kernels — this is one
  Docker daemon with two isolated networks, a faithful stand-in for network *reachability*
  rules but not a physical network.
 - The actual Hetzner/Proxmox firewall enforcement (`infra/firewall/*.rules`) — this harness
  doesn't touch iptables at all; Docker's own network isolation is what's under test here.
 - Provisioner/API/Postgres reachability across the VM boundary — deliberately kept
  host-networked and unchanged from `docker-compose.dev.yml` in this harness, because that
  control-plane path was already proven separately (Phase 4's real mTLS handshake test against
  the Docker Engine TLS port) and isn't what changed this phase. Only the Gateway's isolation
  is new here.
 ## REQUIRES REAL HARDWARE (tracked in docs/PRODUCTION_VALIDATION_CHECKLIST.md, not here)
 Running this exact flow against the real `rn-platform-01`/`rn-lab-host-01` VMs, through the
 real firewall, with real network latency between two physical/virtual hosts.
@@ -0,0 +1,136 @@
 # Phase 4.1 two-host routing simulation — docs/PHASE4_1_PREPRODUCTION_HARDENING.md §6.
 #
 # Proves the ONE thing that matters using REAL Docker network isolation, not mocks: a Gateway
 # container with no route to a Lab container's bridge network genuinely cannot reach it, and
 # can only reach it through the Lab Host Router.
 #
 # What's genuinely simulated (real Docker networking facts, not faked):
 #   - `gateway` is attached ONLY to `sim-platform-net`. It has no network_mode: host, no
 #     attachment to any per-Lab bridge network (rootnode-lab-net-<id>, created dynamically by
 #     the Provisioner) — its network namespace simply has no route to one. This is the same
 #     kind of isolation two separate Proxmox VMs would have, achieved here with two Docker
 #     networks on one daemon instead of two physical hosts.
 #   - `lab-router` runs with network_mode: host — exactly like `egress-proxy` already does in
 #     docker-compose.dev.yml (see that file's comment on why: a host-networked process can
 #     reach every bridge network's containers via the host's own routing table, which is
 #     exactly what a real Lab Host Router process running directly on rn-lab-host-01 needs).
 #     It's also reachable FROM `sim-platform-net` via that network's own gateway IP — the same
 #     mechanism the egress-proxy already relies on for the reverse direction.
 #   - `provisioner`/`api`/`postgres` stay host-networked, unchanged from docker-compose.dev.yml
 #     — this phase's new proof target is the Gateway-to-Lab DATA plane, not the CONTROL plane
 #     (already proven with a real mTLS handshake in Phase 4). Keeping them as-is is a
 #     deliberate scope choice, not an oversight — see the README in this directory.
 #
 # What this does NOT prove (needs the real hardware — docs/PRODUCTION_VALIDATION_CHECKLIST.md):
 # real cross-VM network latency/MTU, the actual Hetzner/Proxmox firewall enforcement, or
 # anything about two genuinely separate kernels.
 #
 # Usage: see README.md in this directory.
 services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: rootnode_platform_sim
    ports:
      - "5434:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 3s
      retries: 10
  egress-proxy:
    build: ../../../platform/egress-proxy
    network_mode: host
  provisioner:
    build: ../../../platform/provisioner
    network_mode: host
    cap_add:
      - NET_ADMIN
      - NET_RAW
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      # run-simulation.sh generates this with the real routing_mode: router / router_endpoint
      # once it knows sim-platform-net's actual gateway IP — overrides just this one file, the
      # image's own resource-policy.yaml underneath is untouched.
      - ${SIM_LAB_HOSTS_SEED:?set by run-simulation.sh}:/app/config/lab-hosts.seed.yaml:ro
    environment:
      PLATFORM_DB_URL: postgres://postgres:postgres@localhost:5434/rootnode_platform_sim
      PROVISIONER_SHARED_SECRET: ${PROVISIONER_SHARED_SECRET:?set a real secret, do not use a default}
      PROVISIONER_PORT: "4101"
      LAB_IMAGE_REF: ${LAB_IMAGE_REF:-rootnode-academy:test}
      EGRESS_PROXY_PORT: "3128"
      DNS_RESOLVER: "127.0.0.11"
      NO_PROXY_CIDRS: "10.42.0.0/16,10.43.0.0/16"
    depends_on:
      postgres:
        condition: service_healthy
  api:
    build:
      context: ../../..
      dockerfile: platform/api/Dockerfile
    network_mode: host
    environment:
      PLATFORM_DB_URL: postgres://postgres:postgres@localhost:5434/rootnode_platform_sim
      PROVISIONER_URL: "http://localhost:4101"
      PROVISIONER_SHARED_SECRET: ${PROVISIONER_SHARED_SECRET:?set a real secret, do not use a default}
      PLATFORM_API_PORT: "4301"
    depends_on:
      postgres:
        condition: service_healthy
      provisioner:
        condition: service_started
  # Simulates rn-platform-01's Gateway — deliberately NOT host-networked, deliberately not
  # attached to any Lab bridge network. This is the isolation under test.
  gateway:
    build: ../../../platform/gateway
    networks:
      - sim-platform-net
    environment:
      GATEWAY_PORT: "4200"
      # Reaches host-networked Provisioner/API via sim-platform-net's own gateway IP — the
      # same "reachable via the bridge's gateway interface" mechanism egress-proxy already
      # relies on (see docker-compose.dev.yml). run-simulation.sh fills these in.
      PROVISIONER_URL: ${SIM_PROVISIONER_URL:?set by run-simulation.sh}
      PROVISIONER_SHARED_SECRET: ${PROVISIONER_SHARED_SECRET:?set a real secret, do not use a default}
      PLATFORM_API_URL: ${SIM_PLATFORM_API_URL:?set by run-simulation.sh}
      LAB_ROUTER_TLS_CA: /tls/ca.pem
      LAB_ROUTER_TLS_CERT: /tls/gateway-client-cert.pem
      LAB_ROUTER_TLS_KEY: /tls/gateway-client-key.pem
    volumes:
      - ${SIM_TLS_DIR:?set by run-simulation.sh}:/tls:ro
    ports:
      - "4210:4200" # exposed to the test driver script only, not "public" in any real sense
    depends_on:
      - provisioner
      - api
  # Simulates rn-lab-host-01's Lab Host Router — host-networked like egress-proxy, for the same
  # reason (needs to reach whatever per-Lab bridge networks the Provisioner creates, which are
  # not known/attachable ahead of time).
  lab-router:
    build: ../../../platform/lab-router
    network_mode: host
    environment:
      LAB_ROUTER_PORT: "8443"
      LAB_ROUTER_TLS_CA: /tls/ca.pem
      LAB_ROUTER_TLS_CERT: /tls/router-server-cert.pem
      LAB_ROUTER_TLS_KEY: /tls/router-server-key.pem
      DOCKER_SOCKET: /var/run/docker.sock
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ${SIM_TLS_DIR:?set by run-simulation.sh}:/tls:ro
 networks:
  # Pre-created by run-simulation.sh (so it can inspect the real gateway IP BEFORE any
  # container starts and hand it to provisioner/gateway/lab-router as config) — Compose is
  # told to use that existing network rather than create its own.
  sim-platform-net:
    external: true
    name: ${COMPOSE_NET_NAME:?set by run-simulation.sh}
@@ -0,0 +1,76 @@
 #!/usr/bin/env bash
 # Orchestrates the Phase 4.1 two-host routing simulation end-to-end: generates a real TLS
 # bundle, computes the sim-platform-net gateway IP, renders the lab_hosts seed with it, brings
 # the stack up, runs test-remote-routing.js, tears everything down. See README.md in this
 # directory and docs/PHASE4_1_PREPRODUCTION_HARDENING.md §6.
 #
 # Usage: ./run-simulation.sh [--keep-up]   (--keep-up skips teardown, for debugging)
 set -euo pipefail
 cd "$(dirname "${BASH_SOURCE[0]}")"
 KEEP_UP=false
 [ "${1:-}" = "--keep-up" ] && KEEP_UP=true
 WORK_DIR="$(mktemp -d)"
 trap 'rm -rf "$WORK_DIR"' EXIT
 export PROVISIONER_SHARED_SECRET="${PROVISIONER_SHARED_SECRET:-sim-shared-secret-not-for-real-use}"
 export LAB_IMAGE_REF="${LAB_IMAGE_REF:-rootnode-academy:test}"
 export SIM_TLS_DIR="$WORK_DIR/tls"
 export SIM_LAB_HOSTS_SEED="$WORK_DIR/lab-hosts.seed.yaml"
 echo "==> [1/6] Creating sim-platform-net so we know its real gateway IP before generating certs"
 NET_NAME="rootnode-sim-platform-net"
 docker network inspect "$NET_NAME" >/dev/null 2>&1 || docker network create "$NET_NAME" >/dev/null
 GW_IP="$(docker network inspect "$NET_NAME" --format '{{(index .IPAM.Config 0).Gateway}}')"
 echo "    sim-platform-net gateway IP: $GW_IP (lab-router and provisioner/api, all host-networked, are reachable from Gateway's isolated network through this address)"
 echo "==> [2/6] Generating a real mTLS bundle for this simulation run (SAN covers the real gateway IP above)"
 ../docker-tls/generate-docker-tls.sh --out-dir "$SIM_TLS_DIR" --lab-host-ip "$GW_IP" --lab-host-dns lab-router-sim >/dev/null
 # Rewrite the compose file's network name to match what we just created/inspected, and point
 # the sim gateway/router services at the shared network by that exact name.
 export COMPOSE_NET_NAME="$NET_NAME"
 echo "==> [3/6] Rendering lab-hosts.seed.yaml with routing_mode: router at the real gateway IP"
 cat > "$SIM_LAB_HOSTS_SEED" <<EOF
 lab_hosts:
  - lab_host_id: sim-lab-host-01
    display_name: "Simulated rn-lab-host-01"
    endpoint: "unix:///var/run/docker.sock"
    enabled: true
    routing_mode: router
    router_endpoint: "https://${GW_IP}:8443"
 EOF
 export SIM_PROVISIONER_URL="http://${GW_IP}:4101"
 export SIM_PLATFORM_API_URL="http://${GW_IP}:4301"
 echo "==> [4/6] Building and starting the simulated two-host stack"
 docker compose -f docker-compose.simulate.yml -p rootnode-sim up -d --build
 cleanup() {
  if [ "$KEEP_UP" = true ]; then
    echo "==> --keep-up set, leaving the stack running (docker compose -f docker-compose.simulate.yml -p rootnode-sim down to clean up)"
    return
  fi
  echo "==> [6/6] Tearing down the simulation stack"
  docker compose -f docker-compose.simulate.yml -p rootnode-sim down -v >/dev/null 2>&1 || true
  docker network rm "$NET_NAME" >/dev/null 2>&1 || true
 }
 trap cleanup EXIT
 echo "==> Waiting for services to become healthy"
 for i in $(seq 1 60); do
  if curl -sf http://localhost:4301/healthz >/dev/null 2>&1 && curl -sf http://localhost:4210/healthz >/dev/null 2>&1; then
    break
  fi
  sleep 2
 done
 echo "==> [5/6] Running the routing test"
 GATEWAY_URL=http://localhost:4210 \
 PROVISIONER_URL=http://localhost:4101 \
 PROVISIONER_SHARED_SECRET="$PROVISIONER_SHARED_SECRET" \
 GATEWAY_CONTAINER=rootnode-sim-gateway-1 \
  node test-remote-routing.js
@@ -0,0 +1,212 @@
 #!/usr/bin/env node
 // Phase 4.1 §6-9 — the real, unmocked proof that remote Lab routing works. Run via
 // run-simulation.sh, which brings up docker-compose.simulate.yml first. No framework, per
 // project convention (see platform/provisioner/test/run.js).
 //
 // Proves, against the real simulated stack:
 //   §6/7  Gateway cannot reach a Lab's Docker-internal IP directly, but reaches it correctly
 //         through the Lab Host Router — both HTTP and WebSocket terminal traffic, including a
 //         real `kubectl get nodes` inside the Lab.
 //   §8/9  Alice/Bob isolation, garbage tokens, destroyed environments, anonymous requests, and
 //         that the Router itself is unreachable without the Gateway's mTLS client identity —
 //         all through the real path, not asserted against mocks.
 const assert = require('assert');
 const https = require('https');
 const { execFileSync } = require('child_process');
 const { makeSession } = require('../../../platform/test/session-helper');
 const GW = process.env.GATEWAY_URL || 'http://localhost:4210';
 const PROVISIONER_URL = process.env.PROVISIONER_URL || 'http://localhost:4101';
 const PROVISIONER_KEY = process.env.PROVISIONER_SHARED_SECRET;
 const GATEWAY_CONTAINER = process.env.GATEWAY_CONTAINER || 'rootnode-sim-gateway-1';
 const WS_URL = GW.replace(/^http/, 'ws');
 let passed = 0, failed = 0;
 async function test(name, fn) {
  try { await fn(); console.log(`  ok — ${name}`); passed++; }
  catch (e) { console.error(`  FAIL — ${name}\n    ${e.stack || e.message}`); failed++; }
 }
 async function provisionerCall(method, path) {
  const res = await fetch(`${PROVISIONER_URL}${path}`, { method, headers: { 'X-Provisioner-Key': PROVISIONER_KEY } });
  const body = await res.json().catch(() => null);
  return { status: res.status, body };
 }
 // The "browser never learns internal_ip" invariant holds everywhere else in the app — this
 // test is the one deliberate, documented exception: it needs the real Docker-internal address
 // as ground truth to prove the Gateway genuinely CANNOT reach it, which requires calling the
 // Provisioner's internal API directly (the same privilege level an operator/CI job has, never
 // a student's).
 async function internalAddressForToken(token) {
  const { body } = await provisionerCall('GET', `/environments/by-token/${token}`);
  return body;
 }
 // bootstrap-owner.js isn't copied into the api image (it's an operator-run one-off script, not
 // a runtime dependency — same reason platform/test regression runs hit this before, see
 // docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md's regression notes). Run it from the host against
 // the sim Postgres's published port instead of `docker exec`ing into the container.
 function bootstrapOwner(email, name, password) {
  execFileSync('node', ['bootstrap-owner.js', email, name, password], {
    cwd: require('path').join(__dirname, '../../../platform/api'),
    env: { ...process.env, PLATFORM_DB_URL: 'postgres://postgres:postgres@localhost:5434/rootnode_platform_sim' },
    stdio: 'inherit',
  });
 }
 // Raw TCP connect attempt from WITHIN the gateway container's own network namespace — proves
 // isolation from the gateway's actual point of view, not the test driver's (which runs on the
 // host and has a completely different, irrelevant reachability picture).
 function gatewayCanReach(ip, port, timeoutMs = 2500) {
  const script = `
    const net = require('net');
    const s = net.createConnection({ host: '${ip}', port: ${port}, timeout: ${timeoutMs} });
    s.on('connect', () => { console.log('REACHABLE'); s.destroy(); process.exit(0); });
    s.on('timeout', () => { console.log('UNREACHABLE(timeout)'); process.exit(0); });
    s.on('error', (e) => { console.log('UNREACHABLE(' + e.code + ')'); process.exit(0); });
  `;
  const out = execFileSync('docker', ['exec', GATEWAY_CONTAINER, 'node', '-e', script]).toString().trim();
  return out.startsWith('REACHABLE');
 }
 function connectRouterWithoutClientCert(host, port, timeoutMs = 2500) {
  return new Promise((resolve) => {
    const req = https.request({ host, port, method: 'GET', rejectUnauthorized: false, timeout: timeoutMs }, (res) => {
      resolve({ ok: true, status: res.statusCode });
    });
    req.on('error', (e) => resolve({ ok: false, error: e.code || e.message }));
    req.on('timeout', () => { req.destroy(); resolve({ ok: false, error: 'timeout' }); });
    req.end();
  });
 }
 function openTerminal(launchPath, cookieHeader) {
  return new Promise((resolve, reject) => {
    const ws = new WebSocket(`${WS_URL}${launchPath}shell-ws`, { headers: { Cookie: cookieHeader } });
    let buffer = '';
    const timer = setTimeout(() => { ws.close(); reject(new Error(`terminal timed out, buffer so far: ${buffer.slice(-400)}`)); }, 30000);
    ws.addEventListener('open', () => ws.send('kubectl get nodes\n'));
    ws.addEventListener('message', (ev) => {
      buffer += ev.data.toString();
      if (/Ready\s/.test(buffer) || /node\.?\d*\s+Ready/i.test(buffer)) {
        clearTimeout(timer);
        ws.close();
        resolve(buffer);
      }
    });
    ws.addEventListener('error', (e) => { clearTimeout(timer); reject(new Error(`ws error: ${e.message || e}`)); });
  });
 }
 function expectWsRejected(launchPath, cookieHeader, timeoutMs = 5000) {
  return new Promise((resolve, reject) => {
    const ws = new WebSocket(`${WS_URL}${launchPath}shell-ws`, { headers: { Cookie: cookieHeader } });
    const timer = setTimeout(() => { ws.close(); reject(new Error('expected the WS to be rejected, but it stayed open')); }, timeoutMs);
    ws.addEventListener('open', () => { clearTimeout(timer); ws.close(); reject(new Error('expected the WS upgrade to be rejected, but it succeeded')); });
    ws.addEventListener('close', () => { clearTimeout(timer); resolve(); });
    ws.addEventListener('error', () => { clearTimeout(timer); resolve(); });
  });
 }
 async function main() {
  console.log('Phase 4.1 two-host remote-routing simulation — slow, real k3s boots\n');
  bootstrapOwner('[email protected]', 'Sim Owner', 'SimOwnerPass!2026');
  const admin = makeSession(GW);
  const { status: loginStatus } = await admin.request('POST', '/api/auth/instructor/login', { email: '[email protected]', password: 'SimOwnerPass!2026' });
  assert.equal(loginStatus, 200, 'owner login must succeed through the Gateway');
  let alice, bob;
  await test('roster: add Alice and Bob via admin (through the Gateway)', async () => {
    const a = await admin.request('POST', '/api/admin/students', { display_name: 'Sim-Alice', cohort_id: null });
    const b = await admin.request('POST', '/api/admin/students', { display_name: 'Sim-Bob', cohort_id: null });
    alice = a.body.student_id; bob = b.body.student_id;
    assert.notEqual(alice, bob);
  });
  const aliceSession = makeSession(GW);
  const bobSession = makeSession(GW);
  await aliceSession.request('POST', '/api/login', { student_id: alice });
  await bobSession.request('POST', '/api/login', { student_id: bob });
  let aliceLaunch, bobLaunch, aliceEnv;
  await test('launch: Alice and Bob each get a running environment on the simulated remote lab host', async () => {
    const a = await aliceSession.request('POST', '/api/environment/launch');
    const b = await bobSession.request('POST', '/api/environment/launch');
    assert.equal(a.status, 200); assert.equal(a.body.status, 'running');
    assert.equal(b.status, 200); assert.equal(b.body.status, 'running');
    aliceLaunch = a.body; bobLaunch = b.body;
    const aliceToken = aliceLaunch.launch_path.split('/')[2];
    aliceEnv = await internalAddressForToken(aliceToken);
    assert.equal(aliceEnv.lab_host_routing_mode, 'router', 'sim-lab-host-01 must be configured as routing_mode: router for this test to mean anything');
  });
  await test('§6/7 — the Gateway CANNOT reach the Lab\'s Docker-internal IP directly', async () => {
    const reachable = gatewayCanReach(aliceEnv.internal_ip, aliceEnv.internal_port);
    assert.equal(reachable, false, `gateway container reached ${aliceEnv.internal_ip}:${aliceEnv.internal_port} directly — the network isolation this whole phase depends on is not real`);
  });
  await test('§6/7 — but the Gateway DOES reach the Lab through the Lab Host Router (HTTP)', async () => {
    const res = await aliceSession.rawFetch(aliceLaunch.launch_path);
    assert.equal(res.status, 200);
    const text = await res.text();
    assert.ok(text.length > 0, 'expected real HTML/asset content back from the Lab, routed through the Router');
  });
  await test('§8 — Alice -> her own Lab: ALLOW', async () => {
    const res = await aliceSession.rawFetch(aliceLaunch.launch_path);
    assert.equal(res.status, 200);
  });
  await test('§8 — Bob -> his own Lab: ALLOW', async () => {
    const res = await bobSession.rawFetch(bobLaunch.launch_path);
    assert.equal(res.status, 200);
  });
  await test('§8 — Alice -> Bob\'s Lab: DENY', async () => {
    const res = await aliceSession.rawFetch(bobLaunch.launch_path);
    assert.equal(res.status, 403);
  });
  await test('§8 — garbage token: DENY', async () => {
    const res = await aliceSession.rawFetch('/lab/not-a-real-token-at-all/');
    assert.equal(res.status, 404);
  });
  await test('§8 — anonymous Gateway request (no session at all): DENY', async () => {
    const res = await fetch(`${GW}${aliceLaunch.launch_path}`);
    assert.equal(res.status, 401);
  });
  await test('§8/9 — direct public access to the Lab Host Router: NOT AVAILABLE (mTLS required, no client cert presented)', async () => {
    const routerUrl = new URL(aliceEnv.lab_host_router_endpoint);
    const result = await connectRouterWithoutClientCert(routerUrl.hostname, routerUrl.port);
    assert.equal(result.ok, false, 'the Router accepted a connection with no client certificate at all — mTLS is not actually being enforced');
  });
  await test('§7 — WebSocket terminal connects and a real `kubectl get nodes` succeeds, through the Router', async () => {
    const output = await openTerminal(bobLaunch.launch_path, `rn_session=${bobSession.getCookie('rn_session')}`);
    assert.ok(/Ready/i.test(output), `expected a Ready node in kubectl output, got: ${output.slice(-300)}`);
  });
  await test('§8 — Alice cannot open Bob\'s terminal WebSocket even with a valid session (her own)', async () => {
    await expectWsRejected(bobLaunch.launch_path, `rn_session=${aliceSession.getCookie('rn_session')}`);
  });
  await test('§8/9 — destroyed environment: routing token stops working, both at the Gateway and at the Router directly', async () => {
    const aliceToken = aliceLaunch.launch_path.split('/')[2];
    await provisionerCall('DELETE', `/environments/${aliceEnv.environment_id}`);
    // Give the Gateway's short-TTL env-cache a moment to expire so this proves real
    // destroyed-environment handling, not just a stale cache hit.
    await new Promise((r) => setTimeout(r, 6000));
    const res = await aliceSession.rawFetch(`/lab/${aliceToken}/`);
    assert.ok([404, 503].includes(res.status), `expected the destroyed environment to be rejected, got ${res.status}`);
  });
  console.log(`\n${passed} passed, ${failed} failed`);
  if (failed > 0) process.exit(1);
 }
 main().catch((e) => { console.error(e); process.exit(1); });
@@ -9,7 +9,13 @@ set -euo pipefail
 STAGING_URL="${STAGING_GATEWAY_URL:?set STAGING_GATEWAY_URL, e.g. https://staging.academy.example.com}"
 echo "==> Health checks"
-curl -sf "$STAGING_URL/healthz" > /dev/null && echo "OK: Gateway /healthz"
+# Phase 4.1 §23: this was `curl ... && echo ...` as one statement — under `set -e`, a failing
 # command that ISN'T the last one in an && list does not trigger the shell to exit (a
 # well-known bash gotcha), so a genuinely unreachable staging Gateway used to fall through
 # silently instead of failing this script. Split into two statements so curl's own exit status
 # is what `set -e` actually sees. Found by infra/scripts/drills/test-ci-fails-closed.sh.
 curl -sf "$STAGING_URL/healthz" > /dev/null
 echo "OK: Gateway /healthz"
 if [ -n "${OWNER_EMAIL:-}" ] && [ -n "${OWNER_PASSWORD:-}" ]; then
  echo "==> Full acceptance run (platform/test/e2e-phase3b.js against staging)"
@@ -160,12 +160,16 @@ async function main() {
  });
  await test('audit log: security events were actually recorded', async () => {
    // Phase 4.1 §16 — /api/admin/audit-events is now paginated (see platform/api/src/pagination.js);
    // the events land on the most recent page, but this test has produced fewer than one
    // page's worth by this point, so page 1's items are still every event so far.
    const { body } = await owner.request('GET', '/api/admin/audit-events');
-    const actions = body.map((e) => e.action);
+    const items = body.items;
    const actions = items.map((e) => e.action);
    assert.ok(actions.includes('login'));
    assert.ok(actions.includes('authz_denied'));
    assert.ok(actions.includes('instructor_update'));
-    assert.ok(body.every((e) => !JSON.stringify(e.metadata || {}).match(/password/i)), 'audit metadata must never contain password material');
+    assert.ok(items.every((e) => !JSON.stringify(e.metadata || {}).match(/password/i)), 'audit metadata must never contain password material');
  });
  // ── Student session hardening ──────────────────────────────────────────────────────────