test(preprod): two-host routing simulation, drill suite, CI security scanning

- infra/scripts/simulate-two-host/: real two-Docker-network simulation
  (Gateway genuinely isolated from any Lab bridge network) proving the
  Lab Host Router fix - 13/13 real assertions (HTTP+WebSocket through the
  Router, Alice/Bob isolation, garbage/destroyed tokens, mTLS-only Router
  access).
- infra/scripts/drills/: Lab Host loss/rebuild (9/9), deployment rollback,
  CI-fails-closed (6 required failure modes), fake cohort dress rehearsal
  (14/14, one continuous narrative through the real app).
- infra/ci/gitea-workflows/build-and-test.yml: new security-scan job
  (npm audit, gitleaks, hadolint, shellcheck, trivy - all free/open-source),
  severity policy in docs/SECURITY_SCANNING.md (CRITICAL/HIGH blocks,
  MEDIUM/LOW reported only). Gates build-images alongside regression.

Found and fixed two real pre-existing bugs via the CI-fails-closed drill:
staging-smoke-test.sh and production-smoke-test.sh both had a classic
set -e gotcha (curl ... && echo "OK" as one statement - a failing command
that isn't last in an && chain doesn't trigger -e) that let an
unreachable/broken endpoint fall through silently instead of failing the
smoke test.

platform/test/e2e-phase3a.js updated for the new paginated
/api/admin/audit-events response shape (Phase 4.1 pagination work).

docs/SECURITY_SCANNING.md

@@ -0,0 +1,47 @@
+# Security Scanning (Phase 4.1 §24)
+
+`infra/ci/gitea-workflows/build-and-test.yml`'s `security-scan` job. All free, open-source,
+no account/signup required for any of them — no paid scanner was added.
+
+## What's scanned
+
+| Tool | Checks | Scope |
+|---|---|---|
+| `npm audit --audit-level=high` | Known-vulnerable dependencies | Every first-party Node service |
+| [gitleaks](https://github.com/gitleaks/gitleaks) | Committed secrets (keys, tokens, credentials) | Whole repo |
+| [hadolint](https://github.com/hadolint/hadolint) | Dockerfile best-practice/security lint | Every first-party Dockerfile |
+| [shellcheck](https://www.shellcheck.net/) | Shell script correctness/security | `infra/**/*.sh` |
+| [trivy](https://github.com/aquasecurity/trivy) | OS/library CVEs in a built container image | `platform-api` image (representative — same base/layers as the other Node services) |
+
+## Severity policy
+
+- **CRITICAL / HIGH** — fails the build (`security-scan` blocks `build-images`, which blocks
+  `deploy-staging`, which blocks production promotion — see the job dependency chain in the
+  workflow file).
+- **MEDIUM / LOW** — reported in the job output, never blocks. A build that's permanently red
+  over a low-severity transitive dependency finding nobody can immediately fix trains everyone
+  to ignore red builds, which is worse than not scanning at all.
+
+`npm audit --audit-level=high` and trivy's `--severity HIGH,CRITICAL` both implement this
+directly at the tool level, not via manual triage.
+
+## What's NOT in the automated job (and why)
+
+- **Dedicated IaC security linting** (checkov/tflint) — not added. `infra/scripts/validate-compose.sh`,
+  `infra/firewall/validate-firewall.sh`, `ansible-playbook --syntax-check`, and `tofu validate`
+  (all in the `regression` job) already catch structural/syntax problems in the infra code;
+  a dedicated security-policy linter for Terraform/Ansible is a reasonable Phase 5 addition once
+  there's a real cloud footprint to misconfigure, not before.
+- **DAST/penetration testing** — out of scope for CI; this is a build-time gate, not a running-system
+  test. Covered separately by `docs/PRODUCTION_VALIDATION_CHECKLIST.md`'s security section once
+  real hardware exists.
+- **SAST beyond hadolint/shellcheck** — the codebase is small enough (a handful of first-party
+  Node services) that `npm audit` + secret scanning + Dockerfile/shell lint cover the realistic
+  risk surface; a dedicated SAST tool (Semgrep, CodeQL) is a reasonable addition once the
+  codebase or team grows enough to justify the added CI time.
+
+## Local reproduction
+
+Every check above can be run locally with the exact same free tools — no CI-only dependency.
+See the `security-scan` job's steps in `infra/ci/gitea-workflows/build-and-test.yml` for the
+exact commands (they're copy-pasteable).

infra/ci/gitea-workflows/build-and-test.yml

@@ -40,16 +40,19 @@ jobs:
 
       - name: Install dependencies (all first-party services)
         run: |
-          for d in backend frontend platform/api platform/gateway platform/provisioner; do
+          for d in backend frontend platform/api platform/gateway platform/provisioner platform/lab-router; do
             (cd "$d" && npm ci)
           done
 
       - name: Syntax-check every service
         run: |
-          for d in backend frontend/src platform/api platform/gateway platform/provisioner; do
+          for d in backend frontend/src platform/api platform/gateway platform/provisioner platform/lab-router; do
             find "$d" -name '*.js' -not -path '*/node_modules/*' -exec node --check {} \;
           done
 
+      - name: Lab Host Router unit tests (Phase 4.1 — fast, no Docker daemon needed)
+        run: (cd platform/lab-router && npm test)
+
       - name: Build frontend
         run: (cd frontend && npm run build)
 
@@ -75,8 +78,56 @@ jobs:
       # can't grant safely — they stay a LOCAL/staging pre-promotion step
       # (infra/scripts/staging-smoke-test.sh), not this job. See docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §18.
 
+  # Phase 4.1 §24 — free/open-source security scanning, no paid tooling, no signup required for
+  # any of these. Severity policy (docs/SECURITY_SCANNING.md): CRITICAL/HIGH findings fail the
+  # build; MEDIUM/LOW are reported but never block a merge — a build that's permanently red over
+  # an unrelated low-severity transitive dependency finding trains everyone to ignore red builds.
+  security-scan:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+
+      - name: npm audit (dependency vulnerabilities) — fails only on high/critical
+        run: |
+          FAIL=0
+          for d in backend frontend platform/api platform/gateway platform/provisioner platform/lab-router; do
+            echo "== $d =="
+            (cd "$d" && npm audit --audit-level=high) || FAIL=1
+          done
+          exit $FAIL
+
+      - name: Secret scanning (gitleaks — free, no account needed)
+        run: docker run --rm -v "$PWD:/repo" zricethezav/gitleaks:latest detect --source=/repo --no-git -v
+
+      - name: Dockerfile linting (hadolint — free)
+        run: |
+          FAIL=0
+          for f in Dockerfile platform/api/Dockerfile platform/gateway/Dockerfile platform/provisioner/Dockerfile platform/egress-proxy/Dockerfile platform/lab-router/Dockerfile; do
+            echo "== $f =="
+            docker run --rm -i hadolint/hadolint < "$f" || FAIL=1
+          done
+          exit $FAIL
+
+      # -S error, not warning — matches docs/SECURITY_SCANNING.md's severity policy. Real run
+      # against this tree at -S warning surfaces only style nits (unused loop vars, a missing
+      # `cd ... || exit` guard) on scripts that are otherwise correct; failing the build on
+      # those trains everyone to ignore the check.
+      - name: Shell linting (shellcheck — free, standard)
+        run: |
+          sudo apt-get update -qq && sudo apt-get install -y -qq shellcheck
+          find infra -name '*.sh' -exec shellcheck -S error {} \+
+
+      - name: Container image scanning (trivy — free, no account needed)
+        run: |
+          docker build -q -t rootnode-scan/platform-api -f platform/api/Dockerfile .
+          docker run --rm -v /var/run/docker.sock:/var/run/docker.sock aquasec/trivy:latest image \
+            --severity HIGH,CRITICAL --exit-code 1 rootnode-scan/platform-api
+
   build-images:
-    needs: regression
+    needs: [regression, security-scan]
     if: github.ref == 'refs/heads/main'
     runs-on: ubuntu-latest
     steps:

infra/scripts/drills/run-fake-cohort-rehearsal.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Orchestrates the Phase 4.1 §27 fake cohort dress rehearsal: brings up the real dev stack,
+# bootstraps an owner, runs test-fake-cohort-rehearsal.js, tears down.
+set -euo pipefail
+cd "$(dirname "${BASH_SOURCE[0]}")/../../../platform"
+
+export PROVISIONER_SHARED_SECRET="${PROVISIONER_SHARED_SECRET:-rehearsal-shared-secret-not-for-production}"
+
+docker compose -f docker-compose.dev.yml up -d --build
+
+cleanup() {
+  echo "==> Tearing down"
+  docker ps -aq --filter "name=rootnode-lab-" | xargs -r docker rm -f >/dev/null 2>&1 || true
+  docker network ls --filter "name=rootnode-lab-net-" --format "{{.Name}}" | xargs -r -n1 docker network rm >/dev/null 2>&1 || true
+  docker compose -f docker-compose.dev.yml down >/dev/null 2>&1 || true
+}
+trap cleanup EXIT
+
+echo "==> Waiting for the stack"
+for _ in $(seq 1 60); do
+  curl -sf http://localhost:4300/healthz >/dev/null 2>&1 && curl -sf http://localhost:4200/healthz >/dev/null 2>&1 && break
+  sleep 2
+done
+
+echo "==> Bootstrapping the rehearsal owner"
+( cd api && PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5433/rootnode_platform \
+    node bootstrap-owner.js [email protected] "Rehearsal Owner" 'RehearsalPass!2026' )
+
+echo "==> Running the rehearsal"
+OWNER_EMAIL=[email protected] OWNER_PASSWORD='RehearsalPass!2026' \
+  node ../infra/scripts/drills/test-fake-cohort-rehearsal.js

infra/scripts/drills/run-lab-host-loss-drill.sh

@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+# Orchestrates the Phase 4.1 §21 Lab Host loss/rebuild drill: brings up the real dev stack,
+# bootstraps an owner, runs test-lab-host-loss.js, tears down. See that file's header for what
+# it actually proves.
+set -euo pipefail
+cd "$(dirname "${BASH_SOURCE[0]}")/../../../platform"
+
+export PROVISIONER_SHARED_SECRET="${PROVISIONER_SHARED_SECRET:-drill-shared-secret-not-for-production}"
+
+docker compose -f docker-compose.dev.yml up -d --build
+
+cleanup() {
+  echo "==> Tearing down"
+  docker ps -aq --filter "name=rootnode-lab-" | xargs -r docker rm -f >/dev/null 2>&1 || true
+  docker network ls --filter "name=rootnode-lab-net-" --format "{{.Name}}" | xargs -r -n1 docker network rm >/dev/null 2>&1 || true
+  docker compose -f docker-compose.dev.yml down >/dev/null 2>&1 || true
+}
+trap cleanup EXIT
+
+echo "==> Waiting for the stack"
+for i in $(seq 1 60); do
+  curl -sf http://localhost:4300/healthz >/dev/null 2>&1 && curl -sf http://localhost:4200/healthz >/dev/null 2>&1 && break
+  sleep 2
+done
+
+echo "==> Bootstrapping the drill owner"
+( cd api && PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5433/rootnode_platform \
+    node bootstrap-owner.js [email protected] "LabHost Drill Owner" 'LabHostDrillPass!2026' )
+
+echo "==> Running the drill"
+OWNER_EMAIL=[email protected] OWNER_PASSWORD='LabHostDrillPass!2026' \
+  node ../infra/scripts/drills/test-lab-host-loss.js

infra/scripts/drills/test-ci-fails-closed.sh

@@ -0,0 +1,77 @@
+#!/usr/bin/env bash
+# Phase 4.1 §23 — proves the CI pipeline (infra/ci/gitea-workflows/build-and-test.yml) fails
+# CLOSED on every listed failure mode, i.e. the underlying command genuinely exits nonzero on
+# broken input, which is what makes a Gitea/GitHub Actions step (and therefore the job, and
+# therefore `needs: regression` gating build-images/deploy-staging) fail. Doesn't need a real
+# CI run — checks the actual commands the workflow calls, directly, the same way Actions would
+# invoke them.
+set -uo pipefail   # NOT -e — this script's whole point is deliberately triggering failures and checking their exit codes
+cd "$(dirname "${BASH_SOURCE[0]}")/../../.."   # repo root
+
+FAIL=0
+expect_nonzero() {
+  local desc="$1"; shift
+  "$@" >/tmp/ci-fail-drill-out 2>&1
+  local code=$?
+  if [ "$code" -eq 0 ]; then
+    echo "  FAIL — $desc (command succeeded, but it should have failed)"; cat /tmp/ci-fail-drill-out; FAIL=1
+  else
+    echo "  ok — $desc correctly failed (exit $code)"
+  fi
+}
+
+echo "Phase 4.1 CI-fails-closed drill"
+echo
+
+echo "== 1. Test/syntax-check failure =="
+BAD_JS="$(mktemp).js"
+echo 'function broken( {' > "$BAD_JS"
+expect_nonzero "node --check on syntactically broken JS" node --check "$BAD_JS"
+rm -f "$BAD_JS"
+
+echo "== 2. Frontend build failure =="
+cp frontend/src/App.jsx /tmp/App.jsx.bak
+echo 'this is not valid jsx <<<' >> frontend/src/App.jsx
+(cd frontend && expect_nonzero "npm run build with a broken component" npm run build)
+cp /tmp/App.jsx.bak frontend/src/App.jsx; rm -f /tmp/App.jsx.bak
+
+echo "== 3. Image build failure =="
+TMPDOCKERFILE="$(mktemp -d)/Dockerfile"
+cat > "$TMPDOCKERFILE" <<'EOF'
+FROM node:20-slim
+COPY this-file-does-not-exist.js /app/
+EOF
+expect_nonzero "docker build with a COPY of a nonexistent file" docker build -q -f "$TMPDOCKERFILE" "$(dirname "$TMPDOCKERFILE")"
+
+echo "== 4. Config validation failure =="
+BAD_COMPOSE="$(mktemp -d)/docker-compose.yml"
+cat > "$BAD_COMPOSE" <<'EOF'
+services:
+  a:
+    image: alpine
+  a:
+    image: alpine
+EOF
+expect_nonzero "docker compose config on a file with a duplicate service key" docker compose -f "$BAD_COMPOSE" config
+
+echo "== 5. Migration preflight failure =="
+( cd platform/api && expect_nonzero "runMigrations() against an unreachable Postgres" \
+    env PLATFORM_DB_URL="postgres://nobody:[email protected]:1/nope" \
+    node -e "require('./src/db').runMigrations().then(()=>process.exit(0)).catch((e)=>{console.error(e.message);process.exit(1)})" )
+
+echo "== 6. Staging smoke test failure =="
+expect_nonzero "staging-smoke-test.sh with no STAGING_GATEWAY_URL set" \
+  env -u STAGING_GATEWAY_URL bash infra/scripts/staging-smoke-test.sh
+expect_nonzero "staging-smoke-test.sh against an unreachable staging URL" \
+  env STAGING_GATEWAY_URL="http://127.0.0.1:1" bash infra/scripts/staging-smoke-test.sh
+
+echo
+if [ "$FAIL" = "1" ]; then
+  echo "DRILL FAILED — at least one failure mode did NOT fail closed. See above."
+  exit 1
+fi
+echo "OK: every listed failure mode genuinely exits nonzero — CI fails closed on all of them."
+echo "NOT TESTED HERE: an actual Gitea Actions run (needs Gitea Actions enabled on the real"
+echo "instance and this workflow file activated at .gitea/workflows/ — see build-and-test.yml's"
+echo "own header comment). This proves the underlying commands, which is what actually"
+echo "determines pass/fail once a real run happens."

infra/scripts/drills/test-deployment-rollback.sh

@@ -0,0 +1,74 @@
+#!/usr/bin/env bash
+# Phase 4.1 §22 — deployment rollback drill. Builds two real gateway image versions (a known
+# good build, and a deliberately broken one), "deploys" each in turn to a local container,
+# proves the smoke test genuinely catches the regression, then rolls back using the EXACT same
+# decision logic rollback-production.sh uses against deploy-history.log (tail -2 | head -1 |
+# awk '{print $2}' — the tag before the current one), and proves the rollback recovers.
+#
+# What this proves: the deterministic-tag + deploy-history.log rollback MECHANISM is real and
+# correct. What it does NOT prove: the actual `ansible-playbook site.yml -e image_tag=...` step
+# rollback-production.sh runs against a real host — that needs the real hardware
+# (docs/PRODUCTION_VALIDATION_CHECKLIST.md). This drill swaps a local Docker container's image
+# tag directly instead, as the closest local equivalent.
+set -euo pipefail
+cd "$(dirname "${BASH_SOURCE[0]}")/../../../platform/gateway"
+
+CONTAINER=rootnode-rollback-drill
+PORT=4295
+DEPLOY_LOG="$(mktemp)"
+trap 'docker rm -f "$CONTAINER" >/dev/null 2>&1 || true; rm -f "$DEPLOY_LOG"; rm -rf /tmp/rootnode-rollback-drill-broken' EXIT
+
+smoke_test() {
+  curl -sf "http://localhost:$PORT/healthz" >/dev/null 2>&1
+}
+
+echo "==> [1/6] Building the known-good gateway image (current source)"
+docker build -q -t rootnode-gateway-drill:known-good -f Dockerfile . >/dev/null
+
+echo "==> [2/6] Building a deliberately broken version (healthz returns 500)"
+rm -rf /tmp/rootnode-rollback-drill-broken
+cp -r . /tmp/rootnode-rollback-drill-broken
+sed -i "s#app.get('/healthz', (req, res) => res.json({ ok: true }));#app.get('/healthz', (req, res) => res.status(500).json({ ok: false, broken: 'DELIBERATE_DRILL_REGRESSION' }));#" \
+  /tmp/rootnode-rollback-drill-broken/src/server.js
+grep -q DELIBERATE_DRILL_REGRESSION /tmp/rootnode-rollback-drill-broken/src/server.js || { echo "FAIL: sed didn't apply — healthz line text must have changed upstream"; exit 1; }
+docker build -q -t rootnode-gateway-drill:broken -f /tmp/rootnode-rollback-drill-broken/Dockerfile /tmp/rootnode-rollback-drill-broken >/dev/null
+
+deploy() {
+  local tag="$1"
+  docker rm -f "$CONTAINER" >/dev/null 2>&1 || true
+  docker run -d --name "$CONTAINER" -p "$PORT:4200" -e GATEWAY_PORT=4200 -e PROVISIONER_URL=http://127.0.0.1:1 \
+    -e PROVISIONER_SHARED_SECRET=drill -e PLATFORM_API_URL=http://127.0.0.1:1 \
+    "rootnode-gateway-drill:$tag" >/dev/null
+  sleep 2
+  echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) $tag" >> "$DEPLOY_LOG"
+}
+
+echo "==> [3/6] Deploying known-good (deploy #1)"
+deploy known-good
+smoke_test && echo "  smoke test: PASS (as expected)" || { echo "FAIL: known-good should have passed"; exit 1; }
+
+echo "==> [4/6] Deploying the broken version (deploy #2 — the 'bad release' this drill catches)"
+deploy broken
+if smoke_test; then echo "FAIL: the broken deploy's regression was NOT caught by the smoke test — that defeats the whole point"; exit 1; fi
+echo "  smoke test: FAIL (correctly caught — this is exactly what should stop a real promotion, see docs/RUNBOOK_ROLLBACK.md)"
+
+echo "==> [5/6] Computing the rollback target — same logic as infra/scripts/rollback-production.sh"
+TARGET_TAG=$(tail -2 "$DEPLOY_LOG" | head -1 | awk '{print $2}')
+[ "$TARGET_TAG" = "known-good" ] || { echo "FAIL: expected rollback target 'known-good', computed '$TARGET_TAG'"; exit 1; }
+echo "  rollback target: $TARGET_TAG (correctly identified as the tag before the broken one)"
+
+echo "==> [6/6] Rolling back"
+deploy "$TARGET_TAG"
+smoke_test && echo "  smoke test: PASS — rollback recovered service" || { echo "FAIL: rollback did not recover"; exit 1; }
+
+echo
+echo "OK: deployment rollback drill passed. Deterministic image tags + deploy-history.log"
+echo "correctly identify and recover from a bad release."
+echo
+echo "On migration compatibility (docs/RUNBOOK_FAILED_MIGRATION.md, docs/RUNBOOK_ROLLBACK.md):"
+echo "this script rolls back APPLICATION IMAGES ONLY. There is no automatic down-migration —"
+echo "if the bad release also ran a schema-changing migration, rolling the image back does NOT"
+echo "undo that migration, and old code may not run correctly against the new schema. That"
+echo "case requires restoring from the pre-migration backup (docs/RUNBOOK_POSTGRES_RESTORE.md),"
+echo "not a rollback, which is exactly why migrations in this project are check-before-alter"
+echo "and additive-only (docs/ENGINEERING_SOP.md §6) — to make image-only rollback usually safe."

infra/scripts/drills/test-fake-cohort-rehearsal.js

@@ -0,0 +1,150 @@
+#!/usr/bin/env node
+// Phase 4.1 §27 — the fake cohort dress rehearsal. One continuous narrative through the real
+// application (not synthetic unit assertions): instructor login, cohort + 4 students, assign
+// curriculum, Prepare Labs, students launch, work different scenarios, use hints, fail a
+// validation, complete scenarios, instructor checks analytics, an exam is assigned/taken,
+// a Lab is reset, a Lab is reaped (destroyed out-of-band, the same consequence the real idle
+// reaper produces — see the note below on why this doesn't wait out the real multi-minute
+// reaper timer), the student returns and continues, capstone state is checked, and the cohort's
+// Labs are destroyed at the end.
+//
+// Run via run-fake-cohort-rehearsal.sh, which brings up platform/docker-compose.dev.yml first.
+// No framework, matching project convention. Problems found are printed, not silently passed —
+// see the Phase 4.1 report for anything this surfaced.
+
+const assert = require('assert');
+const { makeSession } = require('../../../platform/test/session-helper');
+
+const GW = process.env.GATEWAY_URL || 'http://localhost:4200';
+const OWNER_EMAIL = process.env.OWNER_EMAIL;
+const OWNER_PASSWORD = process.env.OWNER_PASSWORD;
+if (!OWNER_EMAIL || !OWNER_PASSWORD) { console.error('Set OWNER_EMAIL and OWNER_PASSWORD'); process.exit(1); }
+
+let passed = 0, failed = 0;
+async function step(name, fn) {
+  try { await fn(); console.log(`  ok — ${name}`); passed++; }
+  catch (e) { console.error(`  FAIL — ${name}\n    ${e.stack || e.message}`); failed++; }
+}
+
+async function main() {
+  console.log('Phase 4.1 fake cohort dress rehearsal — slow, real k3s boots\n');
+
+  const instructor = makeSession(GW);
+  await instructor.request('POST', '/api/auth/instructor/login', { email: OWNER_EMAIL, password: OWNER_PASSWORD });
+
+  const names = ['Rehearsal-Ada', 'Rehearsal-Bo', 'Rehearsal-Cy', 'Rehearsal-Di'];
+  const students = {};
+  const sessions = {};
+
+  await step('instructor: create a cohort and roster 4 students', async () => {
+    await instructor.request('POST', '/api/admin/cohorts', { cohort_id: 'rehearsal-cohort', name: 'Rehearsal Cohort' });
+    for (const name of names) {
+      const { body } = await instructor.request('POST', '/api/admin/students', { display_name: name, cohort_id: 'rehearsal-cohort' });
+      students[name] = body.student_id;
+      sessions[name] = makeSession(GW);
+      await sessions[name].request('POST', '/api/login', { student_id: body.student_id });
+    }
+  });
+
+  await step('instructor: assign curriculum to the cohort', async () => {
+    const { status } = await instructor.request('POST', '/api/admin/cohorts/rehearsal-cohort/curriculum', { curriculum_path: 'rootnode-path' });
+    assert.ok([200, 201].includes(status));
+  });
+
+  await step('instructor: Prepare Labs for the whole cohort (concurrent)', async () => {
+    const { status, body } = await instructor.request('POST', '/api/admin/cohorts/rehearsal-cohort/prepare-labs');
+    assert.equal(status, 200);
+    assert.ok(body.results.every((r) => r.status === 'running'), `expected every Lab running, got: ${JSON.stringify(body.results)}`);
+  });
+
+  const launches = {};
+  await step('students: everyone\'s prepared Lab is already running (no re-launch wait)', async () => {
+    for (const name of names) {
+      const { body } = await sessions[name].request('POST', '/api/environment/launch');
+      assert.equal(body.status, 'running');
+      launches[name] = body;
+    }
+  });
+
+  await step('students: Ada and Bo work DIFFERENT scenarios', async () => {
+    await sessions['Rehearsal-Ada'].request('POST', `${launches['Rehearsal-Ada'].launch_path}api/scenarios/pod-basics-mcq/answer`, { selected: 'b' });
+    await sessions['Rehearsal-Bo'].request('POST', `${launches['Rehearsal-Bo'].launch_path}api/scenarios/pod-basics-mcq/start`);
+  });
+
+  await step('student: Cy uses a hint on a scenario', async () => {
+    const res = await sessions['Rehearsal-Cy'].request('POST', `${launches['Rehearsal-Cy'].launch_path}api/scenarios/pod-basics-mcq/hint`);
+    assert.equal(res.status, 200);
+  });
+
+  await step('student: Di fails a validation on purpose', async () => {
+    const res = await sessions['Rehearsal-Di'].request('POST', `${launches['Rehearsal-Di'].launch_path}api/scenarios/pod-basics-mcq/answer`, { selected: 'a' });
+    assert.equal(res.status, 200);
+    assert.equal(res.body.correct, false, 'expected a deliberately wrong answer to be marked incorrect');
+  });
+
+  await step('instructor: cohort learning summary reflects real activity', async () => {
+    const { body } = await instructor.request('GET', '/api/admin/cohorts/rehearsal-cohort/learning');
+    assert.equal(body.student_count, 4);
+  });
+
+  await step('instructor: needs-attention returns real signals for the struggling students, no punitive language', async () => {
+    const { body } = await instructor.request('GET', '/api/admin/cohorts/rehearsal-cohort/needs-attention');
+    const text = JSON.stringify(body).toLowerCase();
+    assert.ok(!text.includes('fail') || text.includes('failed validations'), 'the word "fail" should only appear in a factual "N failed validations" context, never as a standalone judgment');
+  });
+
+  await step('instructor: reset Ada\'s Lab — academic progress is untouched', async () => {
+    const before = await sessions['Rehearsal-Ada'].request('GET', '/api/progress');
+    // Admin environment routes need the real environment_id, which publicEnvView doesn't echo
+    // back to the student directly — resolved via the student environment lookup instead.
+    const envRes = await instructor.request('GET', `/api/admin/students/${students['Rehearsal-Ada']}/environment`);
+    const env = envRes.body.find((e) => e.status !== 'destroyed');
+    await instructor.request('POST', `/api/admin/environments/${env.environment_id}/reset`);
+    const after = await sessions['Rehearsal-Ada'].request('GET', '/api/progress');
+    assert.deepEqual(before.body.details, after.body.details, 'progress must be identical before/after a Lab reset');
+  });
+
+  await step('§27 — a Lab is reaped: same consequence as the real idle reaper (real multi-minute timer not waited out here — see reaper.js/resource-policy.yaml for the real thresholds; this exercises the destroy-and-preserve-progress consequence directly, already proven mechanistically by infra/scripts/drills/test-lab-host-loss.js)', async () => {
+    const before = await sessions['Rehearsal-Bo'].request('GET', '/api/progress');
+    const envRes = await instructor.request('GET', `/api/admin/students/${students['Rehearsal-Bo']}/environment`);
+    const env = envRes.body.find((e) => e.status !== 'destroyed');
+    await instructor.request('DELETE', `/api/admin/environments/${env.environment_id}`);
+    const after = await sessions['Rehearsal-Bo'].request('GET', '/api/progress');
+    assert.deepEqual(before.body.details, after.body.details, 'progress must survive a reaped Lab unchanged');
+  });
+
+  await step('student: Bo returns later — a fresh Lab is created and curriculum continues', async () => {
+    const { status, body } = await sessions['Rehearsal-Bo'].request('POST', '/api/environment/launch');
+    assert.equal(status, 200); assert.equal(body.status, 'running');
+    const curriculum = await sessions['Rehearsal-Bo'].request('GET', '/api/curriculum/path');
+    assert.ok(curriculum.body.stages.length > 0, 'curriculum must still be there for the returning student');
+  });
+
+  await step('instructor: assign an exam to the cohort and Ada takes it', async () => {
+    const create = await instructor.request('POST', '/api/admin/assessments', {
+      id: 'rehearsal-exam', type: 'exam', title: 'Rehearsal Exam', duration_minutes: 30, hint_policy: 'disabled',
+      pool_spec: { required: ['pod-basics-mcq'], random: [] },
+    });
+    assert.ok([200, 201, 409].includes(create.status)); // 409 if a prior run already created it — fine, idempotent enough for a rehearsal
+    const assign = await instructor.request('POST', '/api/admin/assessments/rehearsal-exam/assign', { cohort_id: 'rehearsal-cohort' });
+    assert.ok([200, 201].includes(assign.status));
+    const start = await sessions['Rehearsal-Ada'].request('POST', '/api/assessments/rehearsal-exam/start');
+    assert.equal(start.status, 201);
+    assert.ok(start.body.scenarios.length > 0);
+  });
+
+  await step('instructor: destroy all cohort Labs after class — progress remains intact for everyone', async () => {
+    const before = {};
+    for (const name of names) before[name] = (await sessions[name].request('GET', '/api/progress')).body.details;
+    await instructor.request('POST', '/api/admin/cohorts/rehearsal-cohort/destroy-labs');
+    for (const name of names) {
+      const after = (await sessions[name].request('GET', '/api/progress')).body.details;
+      assert.deepEqual(before[name], after, `${name}'s progress must survive end-of-class Lab destruction`);
+    }
+  });
+
+  console.log(`\n${passed} passed, ${failed} failed`);
+  if (failed > 0) process.exit(1);
+}
+
+main().catch((e) => { console.error(e); process.exit(1); });

infra/scripts/drills/test-lab-host-loss.js

@@ -0,0 +1,104 @@
+#!/usr/bin/env node
+// Phase 4.1 §21 — Lab Host loss/rebuild drill. Proves disposable infrastructure is genuinely
+// disposable: a student's Lab container is destroyed OUT FROM UNDER the Provisioner (no clean
+// destroy call — simulating the whole rn-lab-host-01 VM disappearing), and the system must:
+//   - keep the Platform (API/Gateway) up throughout — it never depended on that one container
+//   - never present the now-gone environment as healthy/running once reconciliation runs
+//   - leave academic data (progress/attempts) completely untouched
+//   - let the student launch a brand-new Lab afterward and continue where they left off
+//
+// Run via run-lab-host-loss-drill.sh, which brings up platform/docker-compose.dev.yml first.
+// No framework, matching project convention.
+
+const assert = require('assert');
+const { execFileSync } = require('child_process');
+const { makeSession } = require('../../../platform/test/session-helper');
+
+const GW = process.env.GATEWAY_URL || 'http://localhost:4200';
+const API = process.env.PLATFORM_API_URL || 'http://localhost:4300';
+const OWNER_EMAIL = process.env.OWNER_EMAIL;
+const OWNER_PASSWORD = process.env.OWNER_PASSWORD;
+if (!OWNER_EMAIL || !OWNER_PASSWORD) { console.error('Set OWNER_EMAIL and OWNER_PASSWORD'); process.exit(1); }
+
+let passed = 0, failed = 0;
+async function test(name, fn) {
+  try { await fn(); console.log(`  ok — ${name}`); passed++; }
+  catch (e) { console.error(`  FAIL — ${name}\n    ${e.stack || e.message}`); failed++; }
+}
+
+async function main() {
+  console.log('Phase 4.1 Lab Host loss/rebuild drill — slow, real k3s boots\n');
+
+  const admin = makeSession(GW);
+  const { status: loginStatus } = await admin.request('POST', '/api/auth/instructor/login', { email: OWNER_EMAIL, password: OWNER_PASSWORD });
+  assert.equal(loginStatus, 200, 'owner login must succeed');
+
+  let studentId;
+  await test('roster: add a student', async () => {
+    const { body } = await admin.request('POST', '/api/admin/students', { display_name: 'Drill-LabHostLoss', cohort_id: null });
+    studentId = body.student_id;
+  });
+
+  const session = makeSession(GW);
+  await session.request('POST', '/api/login', { student_id: studentId });
+
+  let launch;
+  await test('launch: student gets a running Lab (real k3s boot)', async () => {
+    const { status, body } = await session.request('POST', '/api/environment/launch');
+    assert.equal(status, 200); assert.equal(body.status, 'running');
+    launch = body;
+  });
+
+  await test('progress: student completes a real scenario before the Lab Host is lost', async () => {
+    const res = await session.request('POST', `${launch.launch_path}api/scenarios/pod-basics-mcq/answer`, { selected: 'b' });
+    assert.equal(res.status, 200);
+    const progress = await session.request('GET', '/api/progress');
+    assert.equal(progress.body.details['pod-basics-mcq']?.status, 'completed', 'progress must show completed before we destroy anything');
+  });
+
+  const token = launch.launch_path.split('/')[2];
+
+  await test('§21 — simulate total Lab Host loss: kill the Lab container directly, bypassing the Provisioner entirely', async () => {
+    const containers = execFileSync('docker', ['ps', '--filter', 'name=rootnode-lab-', '--format', '{{.Names}}']).toString().trim().split('\n').filter(Boolean);
+    assert.ok(containers.length >= 1, 'expected at least one running Lab container to destroy');
+    for (const c of containers) execFileSync('docker', ['rm', '-f', c]);
+    console.log(`    destroyed ${containers.length} Lab container(s) out from under the Provisioner`);
+  });
+
+  await test('§21 — Platform stays available throughout (it never depended on that container)', async () => {
+    const api = await fetch(`${API}/healthz`);
+    const gw = await fetch(`${GW}/healthz`);
+    assert.equal(api.status, 200);
+    assert.equal(gw.status, 200);
+  });
+
+  await test('§21 — academic progress is completely untouched by the infrastructure loss', async () => {
+    const progress = await session.request('GET', '/api/progress');
+    assert.equal(progress.body.details['pod-basics-mcq']?.status, 'completed', 'progress must survive the Lab Host loss unchanged');
+  });
+
+  await test('§21 — the now-gone environment is NEVER presented as healthy once reconciliation runs (Provisioner restart, its real startup path)', async () => {
+    // cwd is already platform/ — run-lab-host-loss-drill.sh cd's there before invoking this script.
+    execFileSync('docker', ['compose', '-f', 'docker-compose.dev.yml', 'restart', 'provisioner'], { stdio: 'inherit' });
+    // Give the Provisioner a moment to finish its real reconcileOnStartup pass.
+    await new Promise((r) => setTimeout(r, 5000));
+    const res = await session.rawFetch(`/lab/${token}/`);
+    assert.ok([503, 404].includes(res.status), `expected the stale environment to be rejected (503/404), got ${res.status} — a healthy-looking response here would mean a destroyed environment is being silently presented as available`);
+  });
+
+  await test('§21 — a replacement Lab can be launched and boots successfully (the "replacement Lab Host" case, minus needing a literal second VM)', async () => {
+    const { status, body } = await session.request('POST', '/api/environment/launch');
+    assert.equal(status, 200); assert.equal(body.status, 'running');
+    assert.notEqual(body.launch_path, launch.launch_path, 'the replacement environment must be a genuinely new one, not the destroyed one still being served');
+  });
+
+  await test('§21 — previous academic progress is still there after the replacement Lab boots', async () => {
+    const progress = await session.request('GET', '/api/progress');
+    assert.equal(progress.body.details['pod-basics-mcq']?.status, 'completed');
+  });
+
+  console.log(`\n${passed} passed, ${failed} failed`);
+  if (failed > 0) process.exit(1);
+}
+
+main().catch((e) => { console.error(e); process.exit(1); });

infra/scripts/production-smoke-test.sh

@@ -6,18 +6,26 @@ set -euo pipefail
 
 PROD_URL="${PRODUCTION_URL:?set PRODUCTION_URL, e.g. https://academy.example.com}"
 
+# Phase 4.1 §23: every `curl ... && echo "OK"` below used to be one statement — under `set -e`,
+# a failing command that isn't the LAST one in an && list does not trigger the shell to exit (a
+# well-known bash gotcha), so an unreachable/broken production endpoint used to fall through
+# silently instead of failing this script. Split into two statements each, so the real exit
+# status is what `set -e` sees. Found by infra/scripts/drills/test-ci-fails-closed.sh.
 echo "==> Gateway health"
-curl -sf "$PROD_URL/healthz" > /dev/null && echo "OK"
+curl -sf "$PROD_URL/healthz" > /dev/null
+echo "OK"
 
 echo "==> TLS certificate is valid and not expiring soon"
 EXPIRY=$(echo | openssl s_client -connect "$(echo "$PROD_URL" | sed -e 's#https://##')":443 -servername "$(echo "$PROD_URL" | sed -e 's#https://##')" 2>/dev/null | openssl x509 -noout -enddate | cut -d= -f2)
 echo "Certificate expires: $EXPIRY"
 
 echo "==> Roster endpoint responds (public, unauthenticated, read-only)"
-curl -sf "$PROD_URL/api/roster/cohorts" > /dev/null && echo "OK"
+curl -sf "$PROD_URL/api/roster/cohorts" > /dev/null
+echo "OK"
 
 echo "==> Admin login page loads (does not attempt to log in)"
-curl -sf "$PROD_URL/admin" -o /dev/null -w '%{http_code}\n' | grep -q 200 && echo "OK"
+curl -sf "$PROD_URL/admin" -o /dev/null -w '%{http_code}\n' | grep -q 200
+echo "OK"
 
 echo
 echo "OK: production smoke test passed. For a full functional check (owner/instructor login,"

infra/scripts/simulate-two-host/README.md

@@ -0,0 +1,50 @@
+# Two-Host Routing Simulation
+
+Proves, with real Docker network isolation (not mocks), that the Gateway-to-Lab data path
+described in `docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1 actually works before the real
+Hetzner/Proxmox hardware exists. See that document for the full design; this is the test harness.
+
+## Run it
+
+```bash
+cd infra/scripts/simulate-two-host
+PROVISIONER_SHARED_SECRET=... ./run-simulation.sh          # builds, runs, tears down
+PROVISIONER_SHARED_SECRET=... ./run-simulation.sh --keep-up # leaves the stack up for debugging
+```
+
+Confirmed passing, 13/13, as of this phase's implementation — see the Phase 4.1 report for the
+full run transcript.
+
+## What this genuinely proves (TESTED LOCALLY, with real Docker networking)
+
+- The `gateway` container, attached ONLY to an isolated bridge network with no route to any
+  Lab's per-environment bridge network, **cannot** open a TCP connection to a Lab container's
+  Docker-internal IP — verified via a raw connect attempt run from inside the actual gateway
+  container's own network namespace, not asserted from the test driver's (irrelevant) vantage
+  point.
+- The Gateway **does** reach the Lab correctly through the Lab Host Router, over real mTLS,
+  for both a plain HTTP request and a full WebSocket terminal session — including a genuine
+  `kubectl get nodes` executed inside the Lab and its real output read back.
+- Alice/Bob isolation (own lab ALLOW, other's lab DENY, including over WebSocket), garbage
+  tokens, anonymous requests, and destroyed-environment tokens are all rejected through the
+  real code path — the same `checkOwnership`/`resolveTarget` logic production will run.
+- The Lab Host Router refuses a connection with no client certificate at all — mTLS is the
+  actual authentication mechanism, not a documented intention.
+
+## What this does NOT prove (SIMULATED PRODUCTION BOUNDARY, not identical to real hardware)
+
+- Real cross-VM latency, MTU, or anything about two genuinely separate kernels — this is one
+  Docker daemon with two isolated networks, a faithful stand-in for network *reachability*
+  rules but not a physical network.
+- The actual Hetzner/Proxmox firewall enforcement (`infra/firewall/*.rules`) — this harness
+  doesn't touch iptables at all; Docker's own network isolation is what's under test here.
+- Provisioner/API/Postgres reachability across the VM boundary — deliberately kept
+  host-networked and unchanged from `docker-compose.dev.yml` in this harness, because that
+  control-plane path was already proven separately (Phase 4's real mTLS handshake test against
+  the Docker Engine TLS port) and isn't what changed this phase. Only the Gateway's isolation
+  is new here.
+
+## REQUIRES REAL HARDWARE (tracked in docs/PRODUCTION_VALIDATION_CHECKLIST.md, not here)
+
+Running this exact flow against the real `rn-platform-01`/`rn-lab-host-01` VMs, through the
+real firewall, with real network latency between two physical/virtual hosts.

infra/scripts/simulate-two-host/docker-compose.simulate.yml

@@ -0,0 +1,136 @@
+# Phase 4.1 two-host routing simulation — docs/PHASE4_1_PREPRODUCTION_HARDENING.md §6.
+#
+# Proves the ONE thing that matters using REAL Docker network isolation, not mocks: a Gateway
+# container with no route to a Lab container's bridge network genuinely cannot reach it, and
+# can only reach it through the Lab Host Router.
+#
+# What's genuinely simulated (real Docker networking facts, not faked):
+#   - `gateway` is attached ONLY to `sim-platform-net`. It has no network_mode: host, no
+#     attachment to any per-Lab bridge network (rootnode-lab-net-<id>, created dynamically by
+#     the Provisioner) — its network namespace simply has no route to one. This is the same
+#     kind of isolation two separate Proxmox VMs would have, achieved here with two Docker
+#     networks on one daemon instead of two physical hosts.
+#   - `lab-router` runs with network_mode: host — exactly like `egress-proxy` already does in
+#     docker-compose.dev.yml (see that file's comment on why: a host-networked process can
+#     reach every bridge network's containers via the host's own routing table, which is
+#     exactly what a real Lab Host Router process running directly on rn-lab-host-01 needs).
+#     It's also reachable FROM `sim-platform-net` via that network's own gateway IP — the same
+#     mechanism the egress-proxy already relies on for the reverse direction.
+#   - `provisioner`/`api`/`postgres` stay host-networked, unchanged from docker-compose.dev.yml
+#     — this phase's new proof target is the Gateway-to-Lab DATA plane, not the CONTROL plane
+#     (already proven with a real mTLS handshake in Phase 4). Keeping them as-is is a
+#     deliberate scope choice, not an oversight — see the README in this directory.
+#
+# What this does NOT prove (needs the real hardware — docs/PRODUCTION_VALIDATION_CHECKLIST.md):
+# real cross-VM network latency/MTU, the actual Hetzner/Proxmox firewall enforcement, or
+# anything about two genuinely separate kernels.
+#
+# Usage: see README.md in this directory.
+
+services:
+  postgres:
+    image: postgres:16-alpine
+    environment:
+      POSTGRES_USER: postgres
+      POSTGRES_PASSWORD: postgres
+      POSTGRES_DB: rootnode_platform_sim
+    ports:
+      - "5434:5432"
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U postgres"]
+      interval: 5s
+      timeout: 3s
+      retries: 10
+
+  egress-proxy:
+    build: ../../../platform/egress-proxy
+    network_mode: host
+
+  provisioner:
+    build: ../../../platform/provisioner
+    network_mode: host
+    cap_add:
+      - NET_ADMIN
+      - NET_RAW
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+      # run-simulation.sh generates this with the real routing_mode: router / router_endpoint
+      # once it knows sim-platform-net's actual gateway IP — overrides just this one file, the
+      # image's own resource-policy.yaml underneath is untouched.
+      - ${SIM_LAB_HOSTS_SEED:?set by run-simulation.sh}:/app/config/lab-hosts.seed.yaml:ro
+    environment:
+      PLATFORM_DB_URL: postgres://postgres:postgres@localhost:5434/rootnode_platform_sim
+      PROVISIONER_SHARED_SECRET: ${PROVISIONER_SHARED_SECRET:?set a real secret, do not use a default}
+      PROVISIONER_PORT: "4101"
+      LAB_IMAGE_REF: ${LAB_IMAGE_REF:-rootnode-academy:test}
+      EGRESS_PROXY_PORT: "3128"
+      DNS_RESOLVER: "127.0.0.11"
+      NO_PROXY_CIDRS: "10.42.0.0/16,10.43.0.0/16"
+    depends_on:
+      postgres:
+        condition: service_healthy
+
+  api:
+    build:
+      context: ../../..
+      dockerfile: platform/api/Dockerfile
+    network_mode: host
+    environment:
+      PLATFORM_DB_URL: postgres://postgres:postgres@localhost:5434/rootnode_platform_sim
+      PROVISIONER_URL: "http://localhost:4101"
+      PROVISIONER_SHARED_SECRET: ${PROVISIONER_SHARED_SECRET:?set a real secret, do not use a default}
+      PLATFORM_API_PORT: "4301"
+    depends_on:
+      postgres:
+        condition: service_healthy
+      provisioner:
+        condition: service_started
+
+  # Simulates rn-platform-01's Gateway — deliberately NOT host-networked, deliberately not
+  # attached to any Lab bridge network. This is the isolation under test.
+  gateway:
+    build: ../../../platform/gateway
+    networks:
+      - sim-platform-net
+    environment:
+      GATEWAY_PORT: "4200"
+      # Reaches host-networked Provisioner/API via sim-platform-net's own gateway IP — the
+      # same "reachable via the bridge's gateway interface" mechanism egress-proxy already
+      # relies on (see docker-compose.dev.yml). run-simulation.sh fills these in.
+      PROVISIONER_URL: ${SIM_PROVISIONER_URL:?set by run-simulation.sh}
+      PROVISIONER_SHARED_SECRET: ${PROVISIONER_SHARED_SECRET:?set a real secret, do not use a default}
+      PLATFORM_API_URL: ${SIM_PLATFORM_API_URL:?set by run-simulation.sh}
+      LAB_ROUTER_TLS_CA: /tls/ca.pem
+      LAB_ROUTER_TLS_CERT: /tls/gateway-client-cert.pem
+      LAB_ROUTER_TLS_KEY: /tls/gateway-client-key.pem
+    volumes:
+      - ${SIM_TLS_DIR:?set by run-simulation.sh}:/tls:ro
+    ports:
+      - "4210:4200" # exposed to the test driver script only, not "public" in any real sense
+    depends_on:
+      - provisioner
+      - api
+
+  # Simulates rn-lab-host-01's Lab Host Router — host-networked like egress-proxy, for the same
+  # reason (needs to reach whatever per-Lab bridge networks the Provisioner creates, which are
+  # not known/attachable ahead of time).
+  lab-router:
+    build: ../../../platform/lab-router
+    network_mode: host
+    environment:
+      LAB_ROUTER_PORT: "8443"
+      LAB_ROUTER_TLS_CA: /tls/ca.pem
+      LAB_ROUTER_TLS_CERT: /tls/router-server-cert.pem
+      LAB_ROUTER_TLS_KEY: /tls/router-server-key.pem
+      DOCKER_SOCKET: /var/run/docker.sock
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+      - ${SIM_TLS_DIR:?set by run-simulation.sh}:/tls:ro
+
+networks:
+  # Pre-created by run-simulation.sh (so it can inspect the real gateway IP BEFORE any
+  # container starts and hand it to provisioner/gateway/lab-router as config) — Compose is
+  # told to use that existing network rather than create its own.
+  sim-platform-net:
+    external: true
+    name: ${COMPOSE_NET_NAME:?set by run-simulation.sh}

infra/scripts/simulate-two-host/run-simulation.sh

@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# Orchestrates the Phase 4.1 two-host routing simulation end-to-end: generates a real TLS
+# bundle, computes the sim-platform-net gateway IP, renders the lab_hosts seed with it, brings
+# the stack up, runs test-remote-routing.js, tears everything down. See README.md in this
+# directory and docs/PHASE4_1_PREPRODUCTION_HARDENING.md §6.
+#
+# Usage: ./run-simulation.sh [--keep-up]   (--keep-up skips teardown, for debugging)
+set -euo pipefail
+cd "$(dirname "${BASH_SOURCE[0]}")"
+
+KEEP_UP=false
+[ "${1:-}" = "--keep-up" ] && KEEP_UP=true
+
+WORK_DIR="$(mktemp -d)"
+trap 'rm -rf "$WORK_DIR"' EXIT
+
+export PROVISIONER_SHARED_SECRET="${PROVISIONER_SHARED_SECRET:-sim-shared-secret-not-for-real-use}"
+export LAB_IMAGE_REF="${LAB_IMAGE_REF:-rootnode-academy:test}"
+export SIM_TLS_DIR="$WORK_DIR/tls"
+export SIM_LAB_HOSTS_SEED="$WORK_DIR/lab-hosts.seed.yaml"
+
+echo "==> [1/6] Creating sim-platform-net so we know its real gateway IP before generating certs"
+NET_NAME="rootnode-sim-platform-net"
+docker network inspect "$NET_NAME" >/dev/null 2>&1 || docker network create "$NET_NAME" >/dev/null
+GW_IP="$(docker network inspect "$NET_NAME" --format '{{(index .IPAM.Config 0).Gateway}}')"
+echo "    sim-platform-net gateway IP: $GW_IP (lab-router and provisioner/api, all host-networked, are reachable from Gateway's isolated network through this address)"
+
+echo "==> [2/6] Generating a real mTLS bundle for this simulation run (SAN covers the real gateway IP above)"
+../docker-tls/generate-docker-tls.sh --out-dir "$SIM_TLS_DIR" --lab-host-ip "$GW_IP" --lab-host-dns lab-router-sim >/dev/null
+
+# Rewrite the compose file's network name to match what we just created/inspected, and point
+# the sim gateway/router services at the shared network by that exact name.
+export COMPOSE_NET_NAME="$NET_NAME"
+
+echo "==> [3/6] Rendering lab-hosts.seed.yaml with routing_mode: router at the real gateway IP"
+cat > "$SIM_LAB_HOSTS_SEED" <<EOF
+lab_hosts:
+  - lab_host_id: sim-lab-host-01
+    display_name: "Simulated rn-lab-host-01"
+    endpoint: "unix:///var/run/docker.sock"
+    enabled: true
+    routing_mode: router
+    router_endpoint: "https://${GW_IP}:8443"
+EOF
+
+export SIM_PROVISIONER_URL="http://${GW_IP}:4101"
+export SIM_PLATFORM_API_URL="http://${GW_IP}:4301"
+
+echo "==> [4/6] Building and starting the simulated two-host stack"
+docker compose -f docker-compose.simulate.yml -p rootnode-sim up -d --build
+
+cleanup() {
+  if [ "$KEEP_UP" = true ]; then
+    echo "==> --keep-up set, leaving the stack running (docker compose -f docker-compose.simulate.yml -p rootnode-sim down to clean up)"
+    return
+  fi
+  echo "==> [6/6] Tearing down the simulation stack"
+  docker compose -f docker-compose.simulate.yml -p rootnode-sim down -v >/dev/null 2>&1 || true
+  docker network rm "$NET_NAME" >/dev/null 2>&1 || true
+}
+trap cleanup EXIT
+
+echo "==> Waiting for services to become healthy"
+for i in $(seq 1 60); do
+  if curl -sf http://localhost:4301/healthz >/dev/null 2>&1 && curl -sf http://localhost:4210/healthz >/dev/null 2>&1; then
+    break
+  fi
+  sleep 2
+done
+
+echo "==> [5/6] Running the routing test"
+GATEWAY_URL=http://localhost:4210 \
+PROVISIONER_URL=http://localhost:4101 \
+PROVISIONER_SHARED_SECRET="$PROVISIONER_SHARED_SECRET" \
+GATEWAY_CONTAINER=rootnode-sim-gateway-1 \
+  node test-remote-routing.js

infra/scripts/simulate-two-host/test-remote-routing.js

@@ -0,0 +1,212 @@
+#!/usr/bin/env node
+// Phase 4.1 §6-9 — the real, unmocked proof that remote Lab routing works. Run via
+// run-simulation.sh, which brings up docker-compose.simulate.yml first. No framework, per
+// project convention (see platform/provisioner/test/run.js).
+//
+// Proves, against the real simulated stack:
+//   §6/7  Gateway cannot reach a Lab's Docker-internal IP directly, but reaches it correctly
+//         through the Lab Host Router — both HTTP and WebSocket terminal traffic, including a
+//         real `kubectl get nodes` inside the Lab.
+//   §8/9  Alice/Bob isolation, garbage tokens, destroyed environments, anonymous requests, and
+//         that the Router itself is unreachable without the Gateway's mTLS client identity —
+//         all through the real path, not asserted against mocks.
+
+const assert = require('assert');
+const https = require('https');
+const { execFileSync } = require('child_process');
+const { makeSession } = require('../../../platform/test/session-helper');
+
+const GW = process.env.GATEWAY_URL || 'http://localhost:4210';
+const PROVISIONER_URL = process.env.PROVISIONER_URL || 'http://localhost:4101';
+const PROVISIONER_KEY = process.env.PROVISIONER_SHARED_SECRET;
+const GATEWAY_CONTAINER = process.env.GATEWAY_CONTAINER || 'rootnode-sim-gateway-1';
+const WS_URL = GW.replace(/^http/, 'ws');
+
+let passed = 0, failed = 0;
+async function test(name, fn) {
+  try { await fn(); console.log(`  ok — ${name}`); passed++; }
+  catch (e) { console.error(`  FAIL — ${name}\n    ${e.stack || e.message}`); failed++; }
+}
+
+async function provisionerCall(method, path) {
+  const res = await fetch(`${PROVISIONER_URL}${path}`, { method, headers: { 'X-Provisioner-Key': PROVISIONER_KEY } });
+  const body = await res.json().catch(() => null);
+  return { status: res.status, body };
+}
+
+// The "browser never learns internal_ip" invariant holds everywhere else in the app — this
+// test is the one deliberate, documented exception: it needs the real Docker-internal address
+// as ground truth to prove the Gateway genuinely CANNOT reach it, which requires calling the
+// Provisioner's internal API directly (the same privilege level an operator/CI job has, never
+// a student's).
+async function internalAddressForToken(token) {
+  const { body } = await provisionerCall('GET', `/environments/by-token/${token}`);
+  return body;
+}
+
+// bootstrap-owner.js isn't copied into the api image (it's an operator-run one-off script, not
+// a runtime dependency — same reason platform/test regression runs hit this before, see
+// docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md's regression notes). Run it from the host against
+// the sim Postgres's published port instead of `docker exec`ing into the container.
+function bootstrapOwner(email, name, password) {
+  execFileSync('node', ['bootstrap-owner.js', email, name, password], {
+    cwd: require('path').join(__dirname, '../../../platform/api'),
+    env: { ...process.env, PLATFORM_DB_URL: 'postgres://postgres:postgres@localhost:5434/rootnode_platform_sim' },
+    stdio: 'inherit',
+  });
+}
+
+// Raw TCP connect attempt from WITHIN the gateway container's own network namespace — proves
+// isolation from the gateway's actual point of view, not the test driver's (which runs on the
+// host and has a completely different, irrelevant reachability picture).
+function gatewayCanReach(ip, port, timeoutMs = 2500) {
+  const script = `
+    const net = require('net');
+    const s = net.createConnection({ host: '${ip}', port: ${port}, timeout: ${timeoutMs} });
+    s.on('connect', () => { console.log('REACHABLE'); s.destroy(); process.exit(0); });
+    s.on('timeout', () => { console.log('UNREACHABLE(timeout)'); process.exit(0); });
+    s.on('error', (e) => { console.log('UNREACHABLE(' + e.code + ')'); process.exit(0); });
+  `;
+  const out = execFileSync('docker', ['exec', GATEWAY_CONTAINER, 'node', '-e', script]).toString().trim();
+  return out.startsWith('REACHABLE');
+}
+
+function connectRouterWithoutClientCert(host, port, timeoutMs = 2500) {
+  return new Promise((resolve) => {
+    const req = https.request({ host, port, method: 'GET', rejectUnauthorized: false, timeout: timeoutMs }, (res) => {
+      resolve({ ok: true, status: res.statusCode });
+    });
+    req.on('error', (e) => resolve({ ok: false, error: e.code || e.message }));
+    req.on('timeout', () => { req.destroy(); resolve({ ok: false, error: 'timeout' }); });
+    req.end();
+  });
+}
+
+function openTerminal(launchPath, cookieHeader) {
+  return new Promise((resolve, reject) => {
+    const ws = new WebSocket(`${WS_URL}${launchPath}shell-ws`, { headers: { Cookie: cookieHeader } });
+    let buffer = '';
+    const timer = setTimeout(() => { ws.close(); reject(new Error(`terminal timed out, buffer so far: ${buffer.slice(-400)}`)); }, 30000);
+    ws.addEventListener('open', () => ws.send('kubectl get nodes\n'));
+    ws.addEventListener('message', (ev) => {
+      buffer += ev.data.toString();
+      if (/Ready\s/.test(buffer) || /node\.?\d*\s+Ready/i.test(buffer)) {
+        clearTimeout(timer);
+        ws.close();
+        resolve(buffer);
+      }
+    });
+    ws.addEventListener('error', (e) => { clearTimeout(timer); reject(new Error(`ws error: ${e.message || e}`)); });
+  });
+}
+
+function expectWsRejected(launchPath, cookieHeader, timeoutMs = 5000) {
+  return new Promise((resolve, reject) => {
+    const ws = new WebSocket(`${WS_URL}${launchPath}shell-ws`, { headers: { Cookie: cookieHeader } });
+    const timer = setTimeout(() => { ws.close(); reject(new Error('expected the WS to be rejected, but it stayed open')); }, timeoutMs);
+    ws.addEventListener('open', () => { clearTimeout(timer); ws.close(); reject(new Error('expected the WS upgrade to be rejected, but it succeeded')); });
+    ws.addEventListener('close', () => { clearTimeout(timer); resolve(); });
+    ws.addEventListener('error', () => { clearTimeout(timer); resolve(); });
+  });
+}
+
+async function main() {
+  console.log('Phase 4.1 two-host remote-routing simulation — slow, real k3s boots\n');
+
+  bootstrapOwner('[email protected]', 'Sim Owner', 'SimOwnerPass!2026');
+  const admin = makeSession(GW);
+  const { status: loginStatus } = await admin.request('POST', '/api/auth/instructor/login', { email: '[email protected]', password: 'SimOwnerPass!2026' });
+  assert.equal(loginStatus, 200, 'owner login must succeed through the Gateway');
+
+  let alice, bob;
+  await test('roster: add Alice and Bob via admin (through the Gateway)', async () => {
+    const a = await admin.request('POST', '/api/admin/students', { display_name: 'Sim-Alice', cohort_id: null });
+    const b = await admin.request('POST', '/api/admin/students', { display_name: 'Sim-Bob', cohort_id: null });
+    alice = a.body.student_id; bob = b.body.student_id;
+    assert.notEqual(alice, bob);
+  });
+
+  const aliceSession = makeSession(GW);
+  const bobSession = makeSession(GW);
+  await aliceSession.request('POST', '/api/login', { student_id: alice });
+  await bobSession.request('POST', '/api/login', { student_id: bob });
+
+  let aliceLaunch, bobLaunch, aliceEnv;
+  await test('launch: Alice and Bob each get a running environment on the simulated remote lab host', async () => {
+    const a = await aliceSession.request('POST', '/api/environment/launch');
+    const b = await bobSession.request('POST', '/api/environment/launch');
+    assert.equal(a.status, 200); assert.equal(a.body.status, 'running');
+    assert.equal(b.status, 200); assert.equal(b.body.status, 'running');
+    aliceLaunch = a.body; bobLaunch = b.body;
+    const aliceToken = aliceLaunch.launch_path.split('/')[2];
+    aliceEnv = await internalAddressForToken(aliceToken);
+    assert.equal(aliceEnv.lab_host_routing_mode, 'router', 'sim-lab-host-01 must be configured as routing_mode: router for this test to mean anything');
+  });
+
+  await test('§6/7 — the Gateway CANNOT reach the Lab\'s Docker-internal IP directly', async () => {
+    const reachable = gatewayCanReach(aliceEnv.internal_ip, aliceEnv.internal_port);
+    assert.equal(reachable, false, `gateway container reached ${aliceEnv.internal_ip}:${aliceEnv.internal_port} directly — the network isolation this whole phase depends on is not real`);
+  });
+
+  await test('§6/7 — but the Gateway DOES reach the Lab through the Lab Host Router (HTTP)', async () => {
+    const res = await aliceSession.rawFetch(aliceLaunch.launch_path);
+    assert.equal(res.status, 200);
+    const text = await res.text();
+    assert.ok(text.length > 0, 'expected real HTML/asset content back from the Lab, routed through the Router');
+  });
+
+  await test('§8 — Alice -> her own Lab: ALLOW', async () => {
+    const res = await aliceSession.rawFetch(aliceLaunch.launch_path);
+    assert.equal(res.status, 200);
+  });
+
+  await test('§8 — Bob -> his own Lab: ALLOW', async () => {
+    const res = await bobSession.rawFetch(bobLaunch.launch_path);
+    assert.equal(res.status, 200);
+  });
+
+  await test('§8 — Alice -> Bob\'s Lab: DENY', async () => {
+    const res = await aliceSession.rawFetch(bobLaunch.launch_path);
+    assert.equal(res.status, 403);
+  });
+
+  await test('§8 — garbage token: DENY', async () => {
+    const res = await aliceSession.rawFetch('/lab/not-a-real-token-at-all/');
+    assert.equal(res.status, 404);
+  });
+
+  await test('§8 — anonymous Gateway request (no session at all): DENY', async () => {
+    const res = await fetch(`${GW}${aliceLaunch.launch_path}`);
+    assert.equal(res.status, 401);
+  });
+
+  await test('§8/9 — direct public access to the Lab Host Router: NOT AVAILABLE (mTLS required, no client cert presented)', async () => {
+    const routerUrl = new URL(aliceEnv.lab_host_router_endpoint);
+    const result = await connectRouterWithoutClientCert(routerUrl.hostname, routerUrl.port);
+    assert.equal(result.ok, false, 'the Router accepted a connection with no client certificate at all — mTLS is not actually being enforced');
+  });
+
+  await test('§7 — WebSocket terminal connects and a real `kubectl get nodes` succeeds, through the Router', async () => {
+    const output = await openTerminal(bobLaunch.launch_path, `rn_session=${bobSession.getCookie('rn_session')}`);
+    assert.ok(/Ready/i.test(output), `expected a Ready node in kubectl output, got: ${output.slice(-300)}`);
+  });
+
+  await test('§8 — Alice cannot open Bob\'s terminal WebSocket even with a valid session (her own)', async () => {
+    await expectWsRejected(bobLaunch.launch_path, `rn_session=${aliceSession.getCookie('rn_session')}`);
+  });
+
+  await test('§8/9 — destroyed environment: routing token stops working, both at the Gateway and at the Router directly', async () => {
+    const aliceToken = aliceLaunch.launch_path.split('/')[2];
+    await provisionerCall('DELETE', `/environments/${aliceEnv.environment_id}`);
+    // Give the Gateway's short-TTL env-cache a moment to expire so this proves real
+    // destroyed-environment handling, not just a stale cache hit.
+    await new Promise((r) => setTimeout(r, 6000));
+    const res = await aliceSession.rawFetch(`/lab/${aliceToken}/`);
+    assert.ok([404, 503].includes(res.status), `expected the destroyed environment to be rejected, got ${res.status}`);
+  });
+
+  console.log(`\n${passed} passed, ${failed} failed`);
+  if (failed > 0) process.exit(1);
+}
+
+main().catch((e) => { console.error(e); process.exit(1); });

infra/scripts/staging-smoke-test.sh

@@ -9,7 +9,13 @@ set -euo pipefail
 STAGING_URL="${STAGING_GATEWAY_URL:?set STAGING_GATEWAY_URL, e.g. https://staging.academy.example.com}"
 
 echo "==> Health checks"
-curl -sf "$STAGING_URL/healthz" > /dev/null && echo "OK: Gateway /healthz"
+# Phase 4.1 §23: this was `curl ... && echo ...` as one statement — under `set -e`, a failing
+# command that ISN'T the last one in an && list does not trigger the shell to exit (a
+# well-known bash gotcha), so a genuinely unreachable staging Gateway used to fall through
+# silently instead of failing this script. Split into two statements so curl's own exit status
+# is what `set -e` actually sees. Found by infra/scripts/drills/test-ci-fails-closed.sh.
+curl -sf "$STAGING_URL/healthz" > /dev/null
+echo "OK: Gateway /healthz"
 
 if [ -n "${OWNER_EMAIL:-}" ] && [ -n "${OWNER_PASSWORD:-}" ]; then
   echo "==> Full acceptance run (platform/test/e2e-phase3b.js against staging)"

platform/test/e2e-phase3a.js

@@ -160,12 +160,16 @@ async function main() {
   });
 
   await test('audit log: security events were actually recorded', async () => {
+    // Phase 4.1 §16 — /api/admin/audit-events is now paginated (see platform/api/src/pagination.js);
+    // the events land on the most recent page, but this test has produced fewer than one
+    // page's worth by this point, so page 1's items are still every event so far.
     const { body } = await owner.request('GET', '/api/admin/audit-events');
-    const actions = body.map((e) => e.action);
+    const items = body.items;
+    const actions = items.map((e) => e.action);
     assert.ok(actions.includes('login'));
     assert.ok(actions.includes('authz_denied'));
     assert.ok(actions.includes('instructor_update'));
-    assert.ok(body.every((e) => !JSON.stringify(e.metadata || {}).match(/password/i)), 'audit metadata must never contain password material');
+    assert.ok(items.every((e) => !JSON.stringify(e.metadata || {}).match(/password/i)), 'audit metadata must never contain password material');
   });
 
   // ── Student session hardening ──────────────────────────────────────────────────────────