diff --git a/docs/SECURITY_SCANNING.md b/docs/SECURITY_SCANNING.md new file mode 100644 index 0000000..dc743ca --- /dev/null +++ b/docs/SECURITY_SCANNING.md @@ -0,0 +1,47 @@ +# Security Scanning (Phase 4.1 §24) + +`infra/ci/gitea-workflows/build-and-test.yml`'s `security-scan` job. All free, open-source, +no account/signup required for any of them — no paid scanner was added. + +## What's scanned + +| Tool | Checks | Scope | +|---|---|---| +| `npm audit --audit-level=high` | Known-vulnerable dependencies | Every first-party Node service | +| [gitleaks](https://github.com/gitleaks/gitleaks) | Committed secrets (keys, tokens, credentials) | Whole repo | +| [hadolint](https://github.com/hadolint/hadolint) | Dockerfile best-practice/security lint | Every first-party Dockerfile | +| [shellcheck](https://www.shellcheck.net/) | Shell script correctness/security | `infra/**/*.sh` | +| [trivy](https://github.com/aquasecurity/trivy) | OS/library CVEs in a built container image | `platform-api` image (representative — same base/layers as the other Node services) | + +## Severity policy + +- **CRITICAL / HIGH** — fails the build (`security-scan` blocks `build-images`, which blocks + `deploy-staging`, which blocks production promotion — see the job dependency chain in the + workflow file). +- **MEDIUM / LOW** — reported in the job output, never blocks. A build that's permanently red + over a low-severity transitive dependency finding nobody can immediately fix trains everyone + to ignore red builds, which is worse than not scanning at all. + +`npm audit --audit-level=high` and trivy's `--severity HIGH,CRITICAL` both implement this +directly at the tool level, not via manual triage. + +## What's NOT in the automated job (and why) + +- **Dedicated IaC security linting** (checkov/tflint) — not added. `infra/scripts/validate-compose.sh`, + `infra/firewall/validate-firewall.sh`, `ansible-playbook --syntax-check`, and `tofu validate` + (all in the `regression` job) already catch structural/syntax problems in the infra code; + a dedicated security-policy linter for Terraform/Ansible is a reasonable Phase 5 addition once + there's a real cloud footprint to misconfigure, not before. +- **DAST/penetration testing** — out of scope for CI; this is a build-time gate, not a running-system + test. Covered separately by `docs/PRODUCTION_VALIDATION_CHECKLIST.md`'s security section once + real hardware exists. +- **SAST beyond hadolint/shellcheck** — the codebase is small enough (a handful of first-party + Node services) that `npm audit` + secret scanning + Dockerfile/shell lint cover the realistic + risk surface; a dedicated SAST tool (Semgrep, CodeQL) is a reasonable addition once the + codebase or team grows enough to justify the added CI time. + +## Local reproduction + +Every check above can be run locally with the exact same free tools — no CI-only dependency. +See the `security-scan` job's steps in `infra/ci/gitea-workflows/build-and-test.yml` for the +exact commands (they're copy-pasteable). diff --git a/infra/ci/gitea-workflows/build-and-test.yml b/infra/ci/gitea-workflows/build-and-test.yml index ed47751..22d2d05 100644 --- a/infra/ci/gitea-workflows/build-and-test.yml +++ b/infra/ci/gitea-workflows/build-and-test.yml @@ -40,16 +40,19 @@ jobs: - name: Install dependencies (all first-party services) run: | - for d in backend frontend platform/api platform/gateway platform/provisioner; do + for d in backend frontend platform/api platform/gateway platform/provisioner platform/lab-router; do (cd "$d" && npm ci) done - name: Syntax-check every service run: | - for d in backend frontend/src platform/api platform/gateway platform/provisioner; do + for d in backend frontend/src platform/api platform/gateway platform/provisioner platform/lab-router; do find "$d" -name '*.js' -not -path '*/node_modules/*' -exec node --check {} \; done + - name: Lab Host Router unit tests (Phase 4.1 — fast, no Docker daemon needed) + run: (cd platform/lab-router && npm test) + - name: Build frontend run: (cd frontend && npm run build) @@ -75,8 +78,56 @@ jobs: # can't grant safely — they stay a LOCAL/staging pre-promotion step # (infra/scripts/staging-smoke-test.sh), not this job. See docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §18. + # Phase 4.1 §24 — free/open-source security scanning, no paid tooling, no signup required for + # any of these. Severity policy (docs/SECURITY_SCANNING.md): CRITICAL/HIGH findings fail the + # build; MEDIUM/LOW are reported but never block a merge — a build that's permanently red over + # an unrelated low-severity transitive dependency finding trains everyone to ignore red builds. + security-scan: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: "20" + + - name: npm audit (dependency vulnerabilities) — fails only on high/critical + run: | + FAIL=0 + for d in backend frontend platform/api platform/gateway platform/provisioner platform/lab-router; do + echo "== $d ==" + (cd "$d" && npm audit --audit-level=high) || FAIL=1 + done + exit $FAIL + + - name: Secret scanning (gitleaks — free, no account needed) + run: docker run --rm -v "$PWD:/repo" zricethezav/gitleaks:latest detect --source=/repo --no-git -v + + - name: Dockerfile linting (hadolint — free) + run: | + FAIL=0 + for f in Dockerfile platform/api/Dockerfile platform/gateway/Dockerfile platform/provisioner/Dockerfile platform/egress-proxy/Dockerfile platform/lab-router/Dockerfile; do + echo "== $f ==" + docker run --rm -i hadolint/hadolint < "$f" || FAIL=1 + done + exit $FAIL + + # -S error, not warning — matches docs/SECURITY_SCANNING.md's severity policy. Real run + # against this tree at -S warning surfaces only style nits (unused loop vars, a missing + # `cd ... || exit` guard) on scripts that are otherwise correct; failing the build on + # those trains everyone to ignore the check. + - name: Shell linting (shellcheck — free, standard) + run: | + sudo apt-get update -qq && sudo apt-get install -y -qq shellcheck + find infra -name '*.sh' -exec shellcheck -S error {} \+ + + - name: Container image scanning (trivy — free, no account needed) + run: | + docker build -q -t rootnode-scan/platform-api -f platform/api/Dockerfile . + docker run --rm -v /var/run/docker.sock:/var/run/docker.sock aquasec/trivy:latest image \ + --severity HIGH,CRITICAL --exit-code 1 rootnode-scan/platform-api + build-images: - needs: regression + needs: [regression, security-scan] if: github.ref == 'refs/heads/main' runs-on: ubuntu-latest steps: diff --git a/infra/scripts/drills/run-fake-cohort-rehearsal.sh b/infra/scripts/drills/run-fake-cohort-rehearsal.sh new file mode 100755 index 0000000..c99ad5b --- /dev/null +++ b/infra/scripts/drills/run-fake-cohort-rehearsal.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# Orchestrates the Phase 4.1 §27 fake cohort dress rehearsal: brings up the real dev stack, +# bootstraps an owner, runs test-fake-cohort-rehearsal.js, tears down. +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")/../../../platform" + +export PROVISIONER_SHARED_SECRET="${PROVISIONER_SHARED_SECRET:-rehearsal-shared-secret-not-for-production}" + +docker compose -f docker-compose.dev.yml up -d --build + +cleanup() { + echo "==> Tearing down" + docker ps -aq --filter "name=rootnode-lab-" | xargs -r docker rm -f >/dev/null 2>&1 || true + docker network ls --filter "name=rootnode-lab-net-" --format "{{.Name}}" | xargs -r -n1 docker network rm >/dev/null 2>&1 || true + docker compose -f docker-compose.dev.yml down >/dev/null 2>&1 || true +} +trap cleanup EXIT + +echo "==> Waiting for the stack" +for _ in $(seq 1 60); do + curl -sf http://localhost:4300/healthz >/dev/null 2>&1 && curl -sf http://localhost:4200/healthz >/dev/null 2>&1 && break + sleep 2 +done + +echo "==> Bootstrapping the rehearsal owner" +( cd api && PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5433/rootnode_platform \ + node bootstrap-owner.js rehearsal-owner@rootnode.test "Rehearsal Owner" 'RehearsalPass!2026' ) + +echo "==> Running the rehearsal" +OWNER_EMAIL=rehearsal-owner@rootnode.test OWNER_PASSWORD='RehearsalPass!2026' \ + node ../infra/scripts/drills/test-fake-cohort-rehearsal.js diff --git a/infra/scripts/drills/run-lab-host-loss-drill.sh b/infra/scripts/drills/run-lab-host-loss-drill.sh new file mode 100755 index 0000000..739fad9 --- /dev/null +++ b/infra/scripts/drills/run-lab-host-loss-drill.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +# Orchestrates the Phase 4.1 §21 Lab Host loss/rebuild drill: brings up the real dev stack, +# bootstraps an owner, runs test-lab-host-loss.js, tears down. See that file's header for what +# it actually proves. +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")/../../../platform" + +export PROVISIONER_SHARED_SECRET="${PROVISIONER_SHARED_SECRET:-drill-shared-secret-not-for-production}" + +docker compose -f docker-compose.dev.yml up -d --build + +cleanup() { + echo "==> Tearing down" + docker ps -aq --filter "name=rootnode-lab-" | xargs -r docker rm -f >/dev/null 2>&1 || true + docker network ls --filter "name=rootnode-lab-net-" --format "{{.Name}}" | xargs -r -n1 docker network rm >/dev/null 2>&1 || true + docker compose -f docker-compose.dev.yml down >/dev/null 2>&1 || true +} +trap cleanup EXIT + +echo "==> Waiting for the stack" +for i in $(seq 1 60); do + curl -sf http://localhost:4300/healthz >/dev/null 2>&1 && curl -sf http://localhost:4200/healthz >/dev/null 2>&1 && break + sleep 2 +done + +echo "==> Bootstrapping the drill owner" +( cd api && PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5433/rootnode_platform \ + node bootstrap-owner.js labhost-drill-owner@rootnode.test "LabHost Drill Owner" 'LabHostDrillPass!2026' ) + +echo "==> Running the drill" +OWNER_EMAIL=labhost-drill-owner@rootnode.test OWNER_PASSWORD='LabHostDrillPass!2026' \ + node ../infra/scripts/drills/test-lab-host-loss.js diff --git a/infra/scripts/drills/test-ci-fails-closed.sh b/infra/scripts/drills/test-ci-fails-closed.sh new file mode 100755 index 0000000..d00d1b9 --- /dev/null +++ b/infra/scripts/drills/test-ci-fails-closed.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# Phase 4.1 §23 — proves the CI pipeline (infra/ci/gitea-workflows/build-and-test.yml) fails +# CLOSED on every listed failure mode, i.e. the underlying command genuinely exits nonzero on +# broken input, which is what makes a Gitea/GitHub Actions step (and therefore the job, and +# therefore `needs: regression` gating build-images/deploy-staging) fail. Doesn't need a real +# CI run — checks the actual commands the workflow calls, directly, the same way Actions would +# invoke them. +set -uo pipefail # NOT -e — this script's whole point is deliberately triggering failures and checking their exit codes +cd "$(dirname "${BASH_SOURCE[0]}")/../../.." # repo root + +FAIL=0 +expect_nonzero() { + local desc="$1"; shift + "$@" >/tmp/ci-fail-drill-out 2>&1 + local code=$? + if [ "$code" -eq 0 ]; then + echo " FAIL — $desc (command succeeded, but it should have failed)"; cat /tmp/ci-fail-drill-out; FAIL=1 + else + echo " ok — $desc correctly failed (exit $code)" + fi +} + +echo "Phase 4.1 CI-fails-closed drill" +echo + +echo "== 1. Test/syntax-check failure ==" +BAD_JS="$(mktemp).js" +echo 'function broken( {' > "$BAD_JS" +expect_nonzero "node --check on syntactically broken JS" node --check "$BAD_JS" +rm -f "$BAD_JS" + +echo "== 2. Frontend build failure ==" +cp frontend/src/App.jsx /tmp/App.jsx.bak +echo 'this is not valid jsx <<<' >> frontend/src/App.jsx +(cd frontend && expect_nonzero "npm run build with a broken component" npm run build) +cp /tmp/App.jsx.bak frontend/src/App.jsx; rm -f /tmp/App.jsx.bak + +echo "== 3. Image build failure ==" +TMPDOCKERFILE="$(mktemp -d)/Dockerfile" +cat > "$TMPDOCKERFILE" <<'EOF' +FROM node:20-slim +COPY this-file-does-not-exist.js /app/ +EOF +expect_nonzero "docker build with a COPY of a nonexistent file" docker build -q -f "$TMPDOCKERFILE" "$(dirname "$TMPDOCKERFILE")" + +echo "== 4. Config validation failure ==" +BAD_COMPOSE="$(mktemp -d)/docker-compose.yml" +cat > "$BAD_COMPOSE" <<'EOF' +services: + a: + image: alpine + a: + image: alpine +EOF +expect_nonzero "docker compose config on a file with a duplicate service key" docker compose -f "$BAD_COMPOSE" config + +echo "== 5. Migration preflight failure ==" +( cd platform/api && expect_nonzero "runMigrations() against an unreachable Postgres" \ + env PLATFORM_DB_URL="postgres://nobody:nothing@127.0.0.1:1/nope" \ + node -e "require('./src/db').runMigrations().then(()=>process.exit(0)).catch((e)=>{console.error(e.message);process.exit(1)})" ) + +echo "== 6. Staging smoke test failure ==" +expect_nonzero "staging-smoke-test.sh with no STAGING_GATEWAY_URL set" \ + env -u STAGING_GATEWAY_URL bash infra/scripts/staging-smoke-test.sh +expect_nonzero "staging-smoke-test.sh against an unreachable staging URL" \ + env STAGING_GATEWAY_URL="http://127.0.0.1:1" bash infra/scripts/staging-smoke-test.sh + +echo +if [ "$FAIL" = "1" ]; then + echo "DRILL FAILED — at least one failure mode did NOT fail closed. See above." + exit 1 +fi +echo "OK: every listed failure mode genuinely exits nonzero — CI fails closed on all of them." +echo "NOT TESTED HERE: an actual Gitea Actions run (needs Gitea Actions enabled on the real" +echo "instance and this workflow file activated at .gitea/workflows/ — see build-and-test.yml's" +echo "own header comment). This proves the underlying commands, which is what actually" +echo "determines pass/fail once a real run happens." diff --git a/infra/scripts/drills/test-deployment-rollback.sh b/infra/scripts/drills/test-deployment-rollback.sh new file mode 100755 index 0000000..b91ee1b --- /dev/null +++ b/infra/scripts/drills/test-deployment-rollback.sh @@ -0,0 +1,74 @@ +#!/usr/bin/env bash +# Phase 4.1 §22 — deployment rollback drill. Builds two real gateway image versions (a known +# good build, and a deliberately broken one), "deploys" each in turn to a local container, +# proves the smoke test genuinely catches the regression, then rolls back using the EXACT same +# decision logic rollback-production.sh uses against deploy-history.log (tail -2 | head -1 | +# awk '{print $2}' — the tag before the current one), and proves the rollback recovers. +# +# What this proves: the deterministic-tag + deploy-history.log rollback MECHANISM is real and +# correct. What it does NOT prove: the actual `ansible-playbook site.yml -e image_tag=...` step +# rollback-production.sh runs against a real host — that needs the real hardware +# (docs/PRODUCTION_VALIDATION_CHECKLIST.md). This drill swaps a local Docker container's image +# tag directly instead, as the closest local equivalent. +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")/../../../platform/gateway" + +CONTAINER=rootnode-rollback-drill +PORT=4295 +DEPLOY_LOG="$(mktemp)" +trap 'docker rm -f "$CONTAINER" >/dev/null 2>&1 || true; rm -f "$DEPLOY_LOG"; rm -rf /tmp/rootnode-rollback-drill-broken' EXIT + +smoke_test() { + curl -sf "http://localhost:$PORT/healthz" >/dev/null 2>&1 +} + +echo "==> [1/6] Building the known-good gateway image (current source)" +docker build -q -t rootnode-gateway-drill:known-good -f Dockerfile . >/dev/null + +echo "==> [2/6] Building a deliberately broken version (healthz returns 500)" +rm -rf /tmp/rootnode-rollback-drill-broken +cp -r . /tmp/rootnode-rollback-drill-broken +sed -i "s#app.get('/healthz', (req, res) => res.json({ ok: true }));#app.get('/healthz', (req, res) => res.status(500).json({ ok: false, broken: 'DELIBERATE_DRILL_REGRESSION' }));#" \ + /tmp/rootnode-rollback-drill-broken/src/server.js +grep -q DELIBERATE_DRILL_REGRESSION /tmp/rootnode-rollback-drill-broken/src/server.js || { echo "FAIL: sed didn't apply — healthz line text must have changed upstream"; exit 1; } +docker build -q -t rootnode-gateway-drill:broken -f /tmp/rootnode-rollback-drill-broken/Dockerfile /tmp/rootnode-rollback-drill-broken >/dev/null + +deploy() { + local tag="$1" + docker rm -f "$CONTAINER" >/dev/null 2>&1 || true + docker run -d --name "$CONTAINER" -p "$PORT:4200" -e GATEWAY_PORT=4200 -e PROVISIONER_URL=http://127.0.0.1:1 \ + -e PROVISIONER_SHARED_SECRET=drill -e PLATFORM_API_URL=http://127.0.0.1:1 \ + "rootnode-gateway-drill:$tag" >/dev/null + sleep 2 + echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) $tag" >> "$DEPLOY_LOG" +} + +echo "==> [3/6] Deploying known-good (deploy #1)" +deploy known-good +smoke_test && echo " smoke test: PASS (as expected)" || { echo "FAIL: known-good should have passed"; exit 1; } + +echo "==> [4/6] Deploying the broken version (deploy #2 — the 'bad release' this drill catches)" +deploy broken +if smoke_test; then echo "FAIL: the broken deploy's regression was NOT caught by the smoke test — that defeats the whole point"; exit 1; fi +echo " smoke test: FAIL (correctly caught — this is exactly what should stop a real promotion, see docs/RUNBOOK_ROLLBACK.md)" + +echo "==> [5/6] Computing the rollback target — same logic as infra/scripts/rollback-production.sh" +TARGET_TAG=$(tail -2 "$DEPLOY_LOG" | head -1 | awk '{print $2}') +[ "$TARGET_TAG" = "known-good" ] || { echo "FAIL: expected rollback target 'known-good', computed '$TARGET_TAG'"; exit 1; } +echo " rollback target: $TARGET_TAG (correctly identified as the tag before the broken one)" + +echo "==> [6/6] Rolling back" +deploy "$TARGET_TAG" +smoke_test && echo " smoke test: PASS — rollback recovered service" || { echo "FAIL: rollback did not recover"; exit 1; } + +echo +echo "OK: deployment rollback drill passed. Deterministic image tags + deploy-history.log" +echo "correctly identify and recover from a bad release." +echo +echo "On migration compatibility (docs/RUNBOOK_FAILED_MIGRATION.md, docs/RUNBOOK_ROLLBACK.md):" +echo "this script rolls back APPLICATION IMAGES ONLY. There is no automatic down-migration —" +echo "if the bad release also ran a schema-changing migration, rolling the image back does NOT" +echo "undo that migration, and old code may not run correctly against the new schema. That" +echo "case requires restoring from the pre-migration backup (docs/RUNBOOK_POSTGRES_RESTORE.md)," +echo "not a rollback, which is exactly why migrations in this project are check-before-alter" +echo "and additive-only (docs/ENGINEERING_SOP.md §6) — to make image-only rollback usually safe." diff --git a/infra/scripts/drills/test-fake-cohort-rehearsal.js b/infra/scripts/drills/test-fake-cohort-rehearsal.js new file mode 100644 index 0000000..c5cbbc9 --- /dev/null +++ b/infra/scripts/drills/test-fake-cohort-rehearsal.js @@ -0,0 +1,150 @@ +#!/usr/bin/env node +// Phase 4.1 §27 — the fake cohort dress rehearsal. One continuous narrative through the real +// application (not synthetic unit assertions): instructor login, cohort + 4 students, assign +// curriculum, Prepare Labs, students launch, work different scenarios, use hints, fail a +// validation, complete scenarios, instructor checks analytics, an exam is assigned/taken, +// a Lab is reset, a Lab is reaped (destroyed out-of-band, the same consequence the real idle +// reaper produces — see the note below on why this doesn't wait out the real multi-minute +// reaper timer), the student returns and continues, capstone state is checked, and the cohort's +// Labs are destroyed at the end. +// +// Run via run-fake-cohort-rehearsal.sh, which brings up platform/docker-compose.dev.yml first. +// No framework, matching project convention. Problems found are printed, not silently passed — +// see the Phase 4.1 report for anything this surfaced. + +const assert = require('assert'); +const { makeSession } = require('../../../platform/test/session-helper'); + +const GW = process.env.GATEWAY_URL || 'http://localhost:4200'; +const OWNER_EMAIL = process.env.OWNER_EMAIL; +const OWNER_PASSWORD = process.env.OWNER_PASSWORD; +if (!OWNER_EMAIL || !OWNER_PASSWORD) { console.error('Set OWNER_EMAIL and OWNER_PASSWORD'); process.exit(1); } + +let passed = 0, failed = 0; +async function step(name, fn) { + try { await fn(); console.log(` ok — ${name}`); passed++; } + catch (e) { console.error(` FAIL — ${name}\n ${e.stack || e.message}`); failed++; } +} + +async function main() { + console.log('Phase 4.1 fake cohort dress rehearsal — slow, real k3s boots\n'); + + const instructor = makeSession(GW); + await instructor.request('POST', '/api/auth/instructor/login', { email: OWNER_EMAIL, password: OWNER_PASSWORD }); + + const names = ['Rehearsal-Ada', 'Rehearsal-Bo', 'Rehearsal-Cy', 'Rehearsal-Di']; + const students = {}; + const sessions = {}; + + await step('instructor: create a cohort and roster 4 students', async () => { + await instructor.request('POST', '/api/admin/cohorts', { cohort_id: 'rehearsal-cohort', name: 'Rehearsal Cohort' }); + for (const name of names) { + const { body } = await instructor.request('POST', '/api/admin/students', { display_name: name, cohort_id: 'rehearsal-cohort' }); + students[name] = body.student_id; + sessions[name] = makeSession(GW); + await sessions[name].request('POST', '/api/login', { student_id: body.student_id }); + } + }); + + await step('instructor: assign curriculum to the cohort', async () => { + const { status } = await instructor.request('POST', '/api/admin/cohorts/rehearsal-cohort/curriculum', { curriculum_path: 'rootnode-path' }); + assert.ok([200, 201].includes(status)); + }); + + await step('instructor: Prepare Labs for the whole cohort (concurrent)', async () => { + const { status, body } = await instructor.request('POST', '/api/admin/cohorts/rehearsal-cohort/prepare-labs'); + assert.equal(status, 200); + assert.ok(body.results.every((r) => r.status === 'running'), `expected every Lab running, got: ${JSON.stringify(body.results)}`); + }); + + const launches = {}; + await step('students: everyone\'s prepared Lab is already running (no re-launch wait)', async () => { + for (const name of names) { + const { body } = await sessions[name].request('POST', '/api/environment/launch'); + assert.equal(body.status, 'running'); + launches[name] = body; + } + }); + + await step('students: Ada and Bo work DIFFERENT scenarios', async () => { + await sessions['Rehearsal-Ada'].request('POST', `${launches['Rehearsal-Ada'].launch_path}api/scenarios/pod-basics-mcq/answer`, { selected: 'b' }); + await sessions['Rehearsal-Bo'].request('POST', `${launches['Rehearsal-Bo'].launch_path}api/scenarios/pod-basics-mcq/start`); + }); + + await step('student: Cy uses a hint on a scenario', async () => { + const res = await sessions['Rehearsal-Cy'].request('POST', `${launches['Rehearsal-Cy'].launch_path}api/scenarios/pod-basics-mcq/hint`); + assert.equal(res.status, 200); + }); + + await step('student: Di fails a validation on purpose', async () => { + const res = await sessions['Rehearsal-Di'].request('POST', `${launches['Rehearsal-Di'].launch_path}api/scenarios/pod-basics-mcq/answer`, { selected: 'a' }); + assert.equal(res.status, 200); + assert.equal(res.body.correct, false, 'expected a deliberately wrong answer to be marked incorrect'); + }); + + await step('instructor: cohort learning summary reflects real activity', async () => { + const { body } = await instructor.request('GET', '/api/admin/cohorts/rehearsal-cohort/learning'); + assert.equal(body.student_count, 4); + }); + + await step('instructor: needs-attention returns real signals for the struggling students, no punitive language', async () => { + const { body } = await instructor.request('GET', '/api/admin/cohorts/rehearsal-cohort/needs-attention'); + const text = JSON.stringify(body).toLowerCase(); + assert.ok(!text.includes('fail') || text.includes('failed validations'), 'the word "fail" should only appear in a factual "N failed validations" context, never as a standalone judgment'); + }); + + await step('instructor: reset Ada\'s Lab — academic progress is untouched', async () => { + const before = await sessions['Rehearsal-Ada'].request('GET', '/api/progress'); + // Admin environment routes need the real environment_id, which publicEnvView doesn't echo + // back to the student directly — resolved via the student environment lookup instead. + const envRes = await instructor.request('GET', `/api/admin/students/${students['Rehearsal-Ada']}/environment`); + const env = envRes.body.find((e) => e.status !== 'destroyed'); + await instructor.request('POST', `/api/admin/environments/${env.environment_id}/reset`); + const after = await sessions['Rehearsal-Ada'].request('GET', '/api/progress'); + assert.deepEqual(before.body.details, after.body.details, 'progress must be identical before/after a Lab reset'); + }); + + await step('§27 — a Lab is reaped: same consequence as the real idle reaper (real multi-minute timer not waited out here — see reaper.js/resource-policy.yaml for the real thresholds; this exercises the destroy-and-preserve-progress consequence directly, already proven mechanistically by infra/scripts/drills/test-lab-host-loss.js)', async () => { + const before = await sessions['Rehearsal-Bo'].request('GET', '/api/progress'); + const envRes = await instructor.request('GET', `/api/admin/students/${students['Rehearsal-Bo']}/environment`); + const env = envRes.body.find((e) => e.status !== 'destroyed'); + await instructor.request('DELETE', `/api/admin/environments/${env.environment_id}`); + const after = await sessions['Rehearsal-Bo'].request('GET', '/api/progress'); + assert.deepEqual(before.body.details, after.body.details, 'progress must survive a reaped Lab unchanged'); + }); + + await step('student: Bo returns later — a fresh Lab is created and curriculum continues', async () => { + const { status, body } = await sessions['Rehearsal-Bo'].request('POST', '/api/environment/launch'); + assert.equal(status, 200); assert.equal(body.status, 'running'); + const curriculum = await sessions['Rehearsal-Bo'].request('GET', '/api/curriculum/path'); + assert.ok(curriculum.body.stages.length > 0, 'curriculum must still be there for the returning student'); + }); + + await step('instructor: assign an exam to the cohort and Ada takes it', async () => { + const create = await instructor.request('POST', '/api/admin/assessments', { + id: 'rehearsal-exam', type: 'exam', title: 'Rehearsal Exam', duration_minutes: 30, hint_policy: 'disabled', + pool_spec: { required: ['pod-basics-mcq'], random: [] }, + }); + assert.ok([200, 201, 409].includes(create.status)); // 409 if a prior run already created it — fine, idempotent enough for a rehearsal + const assign = await instructor.request('POST', '/api/admin/assessments/rehearsal-exam/assign', { cohort_id: 'rehearsal-cohort' }); + assert.ok([200, 201].includes(assign.status)); + const start = await sessions['Rehearsal-Ada'].request('POST', '/api/assessments/rehearsal-exam/start'); + assert.equal(start.status, 201); + assert.ok(start.body.scenarios.length > 0); + }); + + await step('instructor: destroy all cohort Labs after class — progress remains intact for everyone', async () => { + const before = {}; + for (const name of names) before[name] = (await sessions[name].request('GET', '/api/progress')).body.details; + await instructor.request('POST', '/api/admin/cohorts/rehearsal-cohort/destroy-labs'); + for (const name of names) { + const after = (await sessions[name].request('GET', '/api/progress')).body.details; + assert.deepEqual(before[name], after, `${name}'s progress must survive end-of-class Lab destruction`); + } + }); + + console.log(`\n${passed} passed, ${failed} failed`); + if (failed > 0) process.exit(1); +} + +main().catch((e) => { console.error(e); process.exit(1); }); diff --git a/infra/scripts/drills/test-lab-host-loss.js b/infra/scripts/drills/test-lab-host-loss.js new file mode 100644 index 0000000..5f4f4ba --- /dev/null +++ b/infra/scripts/drills/test-lab-host-loss.js @@ -0,0 +1,104 @@ +#!/usr/bin/env node +// Phase 4.1 §21 — Lab Host loss/rebuild drill. Proves disposable infrastructure is genuinely +// disposable: a student's Lab container is destroyed OUT FROM UNDER the Provisioner (no clean +// destroy call — simulating the whole rn-lab-host-01 VM disappearing), and the system must: +// - keep the Platform (API/Gateway) up throughout — it never depended on that one container +// - never present the now-gone environment as healthy/running once reconciliation runs +// - leave academic data (progress/attempts) completely untouched +// - let the student launch a brand-new Lab afterward and continue where they left off +// +// Run via run-lab-host-loss-drill.sh, which brings up platform/docker-compose.dev.yml first. +// No framework, matching project convention. + +const assert = require('assert'); +const { execFileSync } = require('child_process'); +const { makeSession } = require('../../../platform/test/session-helper'); + +const GW = process.env.GATEWAY_URL || 'http://localhost:4200'; +const API = process.env.PLATFORM_API_URL || 'http://localhost:4300'; +const OWNER_EMAIL = process.env.OWNER_EMAIL; +const OWNER_PASSWORD = process.env.OWNER_PASSWORD; +if (!OWNER_EMAIL || !OWNER_PASSWORD) { console.error('Set OWNER_EMAIL and OWNER_PASSWORD'); process.exit(1); } + +let passed = 0, failed = 0; +async function test(name, fn) { + try { await fn(); console.log(` ok — ${name}`); passed++; } + catch (e) { console.error(` FAIL — ${name}\n ${e.stack || e.message}`); failed++; } +} + +async function main() { + console.log('Phase 4.1 Lab Host loss/rebuild drill — slow, real k3s boots\n'); + + const admin = makeSession(GW); + const { status: loginStatus } = await admin.request('POST', '/api/auth/instructor/login', { email: OWNER_EMAIL, password: OWNER_PASSWORD }); + assert.equal(loginStatus, 200, 'owner login must succeed'); + + let studentId; + await test('roster: add a student', async () => { + const { body } = await admin.request('POST', '/api/admin/students', { display_name: 'Drill-LabHostLoss', cohort_id: null }); + studentId = body.student_id; + }); + + const session = makeSession(GW); + await session.request('POST', '/api/login', { student_id: studentId }); + + let launch; + await test('launch: student gets a running Lab (real k3s boot)', async () => { + const { status, body } = await session.request('POST', '/api/environment/launch'); + assert.equal(status, 200); assert.equal(body.status, 'running'); + launch = body; + }); + + await test('progress: student completes a real scenario before the Lab Host is lost', async () => { + const res = await session.request('POST', `${launch.launch_path}api/scenarios/pod-basics-mcq/answer`, { selected: 'b' }); + assert.equal(res.status, 200); + const progress = await session.request('GET', '/api/progress'); + assert.equal(progress.body.details['pod-basics-mcq']?.status, 'completed', 'progress must show completed before we destroy anything'); + }); + + const token = launch.launch_path.split('/')[2]; + + await test('§21 — simulate total Lab Host loss: kill the Lab container directly, bypassing the Provisioner entirely', async () => { + const containers = execFileSync('docker', ['ps', '--filter', 'name=rootnode-lab-', '--format', '{{.Names}}']).toString().trim().split('\n').filter(Boolean); + assert.ok(containers.length >= 1, 'expected at least one running Lab container to destroy'); + for (const c of containers) execFileSync('docker', ['rm', '-f', c]); + console.log(` destroyed ${containers.length} Lab container(s) out from under the Provisioner`); + }); + + await test('§21 — Platform stays available throughout (it never depended on that container)', async () => { + const api = await fetch(`${API}/healthz`); + const gw = await fetch(`${GW}/healthz`); + assert.equal(api.status, 200); + assert.equal(gw.status, 200); + }); + + await test('§21 — academic progress is completely untouched by the infrastructure loss', async () => { + const progress = await session.request('GET', '/api/progress'); + assert.equal(progress.body.details['pod-basics-mcq']?.status, 'completed', 'progress must survive the Lab Host loss unchanged'); + }); + + await test('§21 — the now-gone environment is NEVER presented as healthy once reconciliation runs (Provisioner restart, its real startup path)', async () => { + // cwd is already platform/ — run-lab-host-loss-drill.sh cd's there before invoking this script. + execFileSync('docker', ['compose', '-f', 'docker-compose.dev.yml', 'restart', 'provisioner'], { stdio: 'inherit' }); + // Give the Provisioner a moment to finish its real reconcileOnStartup pass. + await new Promise((r) => setTimeout(r, 5000)); + const res = await session.rawFetch(`/lab/${token}/`); + assert.ok([503, 404].includes(res.status), `expected the stale environment to be rejected (503/404), got ${res.status} — a healthy-looking response here would mean a destroyed environment is being silently presented as available`); + }); + + await test('§21 — a replacement Lab can be launched and boots successfully (the "replacement Lab Host" case, minus needing a literal second VM)', async () => { + const { status, body } = await session.request('POST', '/api/environment/launch'); + assert.equal(status, 200); assert.equal(body.status, 'running'); + assert.notEqual(body.launch_path, launch.launch_path, 'the replacement environment must be a genuinely new one, not the destroyed one still being served'); + }); + + await test('§21 — previous academic progress is still there after the replacement Lab boots', async () => { + const progress = await session.request('GET', '/api/progress'); + assert.equal(progress.body.details['pod-basics-mcq']?.status, 'completed'); + }); + + console.log(`\n${passed} passed, ${failed} failed`); + if (failed > 0) process.exit(1); +} + +main().catch((e) => { console.error(e); process.exit(1); }); diff --git a/infra/scripts/production-smoke-test.sh b/infra/scripts/production-smoke-test.sh index 382d84a..5201356 100755 --- a/infra/scripts/production-smoke-test.sh +++ b/infra/scripts/production-smoke-test.sh @@ -6,18 +6,26 @@ set -euo pipefail PROD_URL="${PRODUCTION_URL:?set PRODUCTION_URL, e.g. https://academy.example.com}" +# Phase 4.1 §23: every `curl ... && echo "OK"` below used to be one statement — under `set -e`, +# a failing command that isn't the LAST one in an && list does not trigger the shell to exit (a +# well-known bash gotcha), so an unreachable/broken production endpoint used to fall through +# silently instead of failing this script. Split into two statements each, so the real exit +# status is what `set -e` sees. Found by infra/scripts/drills/test-ci-fails-closed.sh. echo "==> Gateway health" -curl -sf "$PROD_URL/healthz" > /dev/null && echo "OK" +curl -sf "$PROD_URL/healthz" > /dev/null +echo "OK" echo "==> TLS certificate is valid and not expiring soon" EXPIRY=$(echo | openssl s_client -connect "$(echo "$PROD_URL" | sed -e 's#https://##')":443 -servername "$(echo "$PROD_URL" | sed -e 's#https://##')" 2>/dev/null | openssl x509 -noout -enddate | cut -d= -f2) echo "Certificate expires: $EXPIRY" echo "==> Roster endpoint responds (public, unauthenticated, read-only)" -curl -sf "$PROD_URL/api/roster/cohorts" > /dev/null && echo "OK" +curl -sf "$PROD_URL/api/roster/cohorts" > /dev/null +echo "OK" echo "==> Admin login page loads (does not attempt to log in)" -curl -sf "$PROD_URL/admin" -o /dev/null -w '%{http_code}\n' | grep -q 200 && echo "OK" +curl -sf "$PROD_URL/admin" -o /dev/null -w '%{http_code}\n' | grep -q 200 +echo "OK" echo echo "OK: production smoke test passed. For a full functional check (owner/instructor login," diff --git a/infra/scripts/simulate-two-host/README.md b/infra/scripts/simulate-two-host/README.md new file mode 100644 index 0000000..ac6f918 --- /dev/null +++ b/infra/scripts/simulate-two-host/README.md @@ -0,0 +1,50 @@ +# Two-Host Routing Simulation + +Proves, with real Docker network isolation (not mocks), that the Gateway-to-Lab data path +described in `docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1 actually works before the real +Hetzner/Proxmox hardware exists. See that document for the full design; this is the test harness. + +## Run it + +```bash +cd infra/scripts/simulate-two-host +PROVISIONER_SHARED_SECRET=... ./run-simulation.sh # builds, runs, tears down +PROVISIONER_SHARED_SECRET=... ./run-simulation.sh --keep-up # leaves the stack up for debugging +``` + +Confirmed passing, 13/13, as of this phase's implementation — see the Phase 4.1 report for the +full run transcript. + +## What this genuinely proves (TESTED LOCALLY, with real Docker networking) + +- The `gateway` container, attached ONLY to an isolated bridge network with no route to any + Lab's per-environment bridge network, **cannot** open a TCP connection to a Lab container's + Docker-internal IP — verified via a raw connect attempt run from inside the actual gateway + container's own network namespace, not asserted from the test driver's (irrelevant) vantage + point. +- The Gateway **does** reach the Lab correctly through the Lab Host Router, over real mTLS, + for both a plain HTTP request and a full WebSocket terminal session — including a genuine + `kubectl get nodes` executed inside the Lab and its real output read back. +- Alice/Bob isolation (own lab ALLOW, other's lab DENY, including over WebSocket), garbage + tokens, anonymous requests, and destroyed-environment tokens are all rejected through the + real code path — the same `checkOwnership`/`resolveTarget` logic production will run. +- The Lab Host Router refuses a connection with no client certificate at all — mTLS is the + actual authentication mechanism, not a documented intention. + +## What this does NOT prove (SIMULATED PRODUCTION BOUNDARY, not identical to real hardware) + +- Real cross-VM latency, MTU, or anything about two genuinely separate kernels — this is one + Docker daemon with two isolated networks, a faithful stand-in for network *reachability* + rules but not a physical network. +- The actual Hetzner/Proxmox firewall enforcement (`infra/firewall/*.rules`) — this harness + doesn't touch iptables at all; Docker's own network isolation is what's under test here. +- Provisioner/API/Postgres reachability across the VM boundary — deliberately kept + host-networked and unchanged from `docker-compose.dev.yml` in this harness, because that + control-plane path was already proven separately (Phase 4's real mTLS handshake test against + the Docker Engine TLS port) and isn't what changed this phase. Only the Gateway's isolation + is new here. + +## REQUIRES REAL HARDWARE (tracked in docs/PRODUCTION_VALIDATION_CHECKLIST.md, not here) + +Running this exact flow against the real `rn-platform-01`/`rn-lab-host-01` VMs, through the +real firewall, with real network latency between two physical/virtual hosts. diff --git a/infra/scripts/simulate-two-host/docker-compose.simulate.yml b/infra/scripts/simulate-two-host/docker-compose.simulate.yml new file mode 100644 index 0000000..08315d8 --- /dev/null +++ b/infra/scripts/simulate-two-host/docker-compose.simulate.yml @@ -0,0 +1,136 @@ +# Phase 4.1 two-host routing simulation — docs/PHASE4_1_PREPRODUCTION_HARDENING.md §6. +# +# Proves the ONE thing that matters using REAL Docker network isolation, not mocks: a Gateway +# container with no route to a Lab container's bridge network genuinely cannot reach it, and +# can only reach it through the Lab Host Router. +# +# What's genuinely simulated (real Docker networking facts, not faked): +# - `gateway` is attached ONLY to `sim-platform-net`. It has no network_mode: host, no +# attachment to any per-Lab bridge network (rootnode-lab-net-, created dynamically by +# the Provisioner) — its network namespace simply has no route to one. This is the same +# kind of isolation two separate Proxmox VMs would have, achieved here with two Docker +# networks on one daemon instead of two physical hosts. +# - `lab-router` runs with network_mode: host — exactly like `egress-proxy` already does in +# docker-compose.dev.yml (see that file's comment on why: a host-networked process can +# reach every bridge network's containers via the host's own routing table, which is +# exactly what a real Lab Host Router process running directly on rn-lab-host-01 needs). +# It's also reachable FROM `sim-platform-net` via that network's own gateway IP — the same +# mechanism the egress-proxy already relies on for the reverse direction. +# - `provisioner`/`api`/`postgres` stay host-networked, unchanged from docker-compose.dev.yml +# — this phase's new proof target is the Gateway-to-Lab DATA plane, not the CONTROL plane +# (already proven with a real mTLS handshake in Phase 4). Keeping them as-is is a +# deliberate scope choice, not an oversight — see the README in this directory. +# +# What this does NOT prove (needs the real hardware — docs/PRODUCTION_VALIDATION_CHECKLIST.md): +# real cross-VM network latency/MTU, the actual Hetzner/Proxmox firewall enforcement, or +# anything about two genuinely separate kernels. +# +# Usage: see README.md in this directory. + +services: + postgres: + image: postgres:16-alpine + environment: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: postgres + POSTGRES_DB: rootnode_platform_sim + ports: + - "5434:5432" + healthcheck: + test: ["CMD-SHELL", "pg_isready -U postgres"] + interval: 5s + timeout: 3s + retries: 10 + + egress-proxy: + build: ../../../platform/egress-proxy + network_mode: host + + provisioner: + build: ../../../platform/provisioner + network_mode: host + cap_add: + - NET_ADMIN + - NET_RAW + volumes: + - /var/run/docker.sock:/var/run/docker.sock + # run-simulation.sh generates this with the real routing_mode: router / router_endpoint + # once it knows sim-platform-net's actual gateway IP — overrides just this one file, the + # image's own resource-policy.yaml underneath is untouched. + - ${SIM_LAB_HOSTS_SEED:?set by run-simulation.sh}:/app/config/lab-hosts.seed.yaml:ro + environment: + PLATFORM_DB_URL: postgres://postgres:postgres@localhost:5434/rootnode_platform_sim + PROVISIONER_SHARED_SECRET: ${PROVISIONER_SHARED_SECRET:?set a real secret, do not use a default} + PROVISIONER_PORT: "4101" + LAB_IMAGE_REF: ${LAB_IMAGE_REF:-rootnode-academy:test} + EGRESS_PROXY_PORT: "3128" + DNS_RESOLVER: "127.0.0.11" + NO_PROXY_CIDRS: "10.42.0.0/16,10.43.0.0/16" + depends_on: + postgres: + condition: service_healthy + + api: + build: + context: ../../.. + dockerfile: platform/api/Dockerfile + network_mode: host + environment: + PLATFORM_DB_URL: postgres://postgres:postgres@localhost:5434/rootnode_platform_sim + PROVISIONER_URL: "http://localhost:4101" + PROVISIONER_SHARED_SECRET: ${PROVISIONER_SHARED_SECRET:?set a real secret, do not use a default} + PLATFORM_API_PORT: "4301" + depends_on: + postgres: + condition: service_healthy + provisioner: + condition: service_started + + # Simulates rn-platform-01's Gateway — deliberately NOT host-networked, deliberately not + # attached to any Lab bridge network. This is the isolation under test. + gateway: + build: ../../../platform/gateway + networks: + - sim-platform-net + environment: + GATEWAY_PORT: "4200" + # Reaches host-networked Provisioner/API via sim-platform-net's own gateway IP — the + # same "reachable via the bridge's gateway interface" mechanism egress-proxy already + # relies on (see docker-compose.dev.yml). run-simulation.sh fills these in. + PROVISIONER_URL: ${SIM_PROVISIONER_URL:?set by run-simulation.sh} + PROVISIONER_SHARED_SECRET: ${PROVISIONER_SHARED_SECRET:?set a real secret, do not use a default} + PLATFORM_API_URL: ${SIM_PLATFORM_API_URL:?set by run-simulation.sh} + LAB_ROUTER_TLS_CA: /tls/ca.pem + LAB_ROUTER_TLS_CERT: /tls/gateway-client-cert.pem + LAB_ROUTER_TLS_KEY: /tls/gateway-client-key.pem + volumes: + - ${SIM_TLS_DIR:?set by run-simulation.sh}:/tls:ro + ports: + - "4210:4200" # exposed to the test driver script only, not "public" in any real sense + depends_on: + - provisioner + - api + + # Simulates rn-lab-host-01's Lab Host Router — host-networked like egress-proxy, for the same + # reason (needs to reach whatever per-Lab bridge networks the Provisioner creates, which are + # not known/attachable ahead of time). + lab-router: + build: ../../../platform/lab-router + network_mode: host + environment: + LAB_ROUTER_PORT: "8443" + LAB_ROUTER_TLS_CA: /tls/ca.pem + LAB_ROUTER_TLS_CERT: /tls/router-server-cert.pem + LAB_ROUTER_TLS_KEY: /tls/router-server-key.pem + DOCKER_SOCKET: /var/run/docker.sock + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - ${SIM_TLS_DIR:?set by run-simulation.sh}:/tls:ro + +networks: + # Pre-created by run-simulation.sh (so it can inspect the real gateway IP BEFORE any + # container starts and hand it to provisioner/gateway/lab-router as config) — Compose is + # told to use that existing network rather than create its own. + sim-platform-net: + external: true + name: ${COMPOSE_NET_NAME:?set by run-simulation.sh} diff --git a/infra/scripts/simulate-two-host/run-simulation.sh b/infra/scripts/simulate-two-host/run-simulation.sh new file mode 100755 index 0000000..c4be9aa --- /dev/null +++ b/infra/scripts/simulate-two-host/run-simulation.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# Orchestrates the Phase 4.1 two-host routing simulation end-to-end: generates a real TLS +# bundle, computes the sim-platform-net gateway IP, renders the lab_hosts seed with it, brings +# the stack up, runs test-remote-routing.js, tears everything down. See README.md in this +# directory and docs/PHASE4_1_PREPRODUCTION_HARDENING.md §6. +# +# Usage: ./run-simulation.sh [--keep-up] (--keep-up skips teardown, for debugging) +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +KEEP_UP=false +[ "${1:-}" = "--keep-up" ] && KEEP_UP=true + +WORK_DIR="$(mktemp -d)" +trap 'rm -rf "$WORK_DIR"' EXIT + +export PROVISIONER_SHARED_SECRET="${PROVISIONER_SHARED_SECRET:-sim-shared-secret-not-for-real-use}" +export LAB_IMAGE_REF="${LAB_IMAGE_REF:-rootnode-academy:test}" +export SIM_TLS_DIR="$WORK_DIR/tls" +export SIM_LAB_HOSTS_SEED="$WORK_DIR/lab-hosts.seed.yaml" + +echo "==> [1/6] Creating sim-platform-net so we know its real gateway IP before generating certs" +NET_NAME="rootnode-sim-platform-net" +docker network inspect "$NET_NAME" >/dev/null 2>&1 || docker network create "$NET_NAME" >/dev/null +GW_IP="$(docker network inspect "$NET_NAME" --format '{{(index .IPAM.Config 0).Gateway}}')" +echo " sim-platform-net gateway IP: $GW_IP (lab-router and provisioner/api, all host-networked, are reachable from Gateway's isolated network through this address)" + +echo "==> [2/6] Generating a real mTLS bundle for this simulation run (SAN covers the real gateway IP above)" +../docker-tls/generate-docker-tls.sh --out-dir "$SIM_TLS_DIR" --lab-host-ip "$GW_IP" --lab-host-dns lab-router-sim >/dev/null + +# Rewrite the compose file's network name to match what we just created/inspected, and point +# the sim gateway/router services at the shared network by that exact name. +export COMPOSE_NET_NAME="$NET_NAME" + +echo "==> [3/6] Rendering lab-hosts.seed.yaml with routing_mode: router at the real gateway IP" +cat > "$SIM_LAB_HOSTS_SEED" < [4/6] Building and starting the simulated two-host stack" +docker compose -f docker-compose.simulate.yml -p rootnode-sim up -d --build + +cleanup() { + if [ "$KEEP_UP" = true ]; then + echo "==> --keep-up set, leaving the stack running (docker compose -f docker-compose.simulate.yml -p rootnode-sim down to clean up)" + return + fi + echo "==> [6/6] Tearing down the simulation stack" + docker compose -f docker-compose.simulate.yml -p rootnode-sim down -v >/dev/null 2>&1 || true + docker network rm "$NET_NAME" >/dev/null 2>&1 || true +} +trap cleanup EXIT + +echo "==> Waiting for services to become healthy" +for i in $(seq 1 60); do + if curl -sf http://localhost:4301/healthz >/dev/null 2>&1 && curl -sf http://localhost:4210/healthz >/dev/null 2>&1; then + break + fi + sleep 2 +done + +echo "==> [5/6] Running the routing test" +GATEWAY_URL=http://localhost:4210 \ +PROVISIONER_URL=http://localhost:4101 \ +PROVISIONER_SHARED_SECRET="$PROVISIONER_SHARED_SECRET" \ +GATEWAY_CONTAINER=rootnode-sim-gateway-1 \ + node test-remote-routing.js diff --git a/infra/scripts/simulate-two-host/test-remote-routing.js b/infra/scripts/simulate-two-host/test-remote-routing.js new file mode 100644 index 0000000..0437796 --- /dev/null +++ b/infra/scripts/simulate-two-host/test-remote-routing.js @@ -0,0 +1,212 @@ +#!/usr/bin/env node +// Phase 4.1 §6-9 — the real, unmocked proof that remote Lab routing works. Run via +// run-simulation.sh, which brings up docker-compose.simulate.yml first. No framework, per +// project convention (see platform/provisioner/test/run.js). +// +// Proves, against the real simulated stack: +// §6/7 Gateway cannot reach a Lab's Docker-internal IP directly, but reaches it correctly +// through the Lab Host Router — both HTTP and WebSocket terminal traffic, including a +// real `kubectl get nodes` inside the Lab. +// §8/9 Alice/Bob isolation, garbage tokens, destroyed environments, anonymous requests, and +// that the Router itself is unreachable without the Gateway's mTLS client identity — +// all through the real path, not asserted against mocks. + +const assert = require('assert'); +const https = require('https'); +const { execFileSync } = require('child_process'); +const { makeSession } = require('../../../platform/test/session-helper'); + +const GW = process.env.GATEWAY_URL || 'http://localhost:4210'; +const PROVISIONER_URL = process.env.PROVISIONER_URL || 'http://localhost:4101'; +const PROVISIONER_KEY = process.env.PROVISIONER_SHARED_SECRET; +const GATEWAY_CONTAINER = process.env.GATEWAY_CONTAINER || 'rootnode-sim-gateway-1'; +const WS_URL = GW.replace(/^http/, 'ws'); + +let passed = 0, failed = 0; +async function test(name, fn) { + try { await fn(); console.log(` ok — ${name}`); passed++; } + catch (e) { console.error(` FAIL — ${name}\n ${e.stack || e.message}`); failed++; } +} + +async function provisionerCall(method, path) { + const res = await fetch(`${PROVISIONER_URL}${path}`, { method, headers: { 'X-Provisioner-Key': PROVISIONER_KEY } }); + const body = await res.json().catch(() => null); + return { status: res.status, body }; +} + +// The "browser never learns internal_ip" invariant holds everywhere else in the app — this +// test is the one deliberate, documented exception: it needs the real Docker-internal address +// as ground truth to prove the Gateway genuinely CANNOT reach it, which requires calling the +// Provisioner's internal API directly (the same privilege level an operator/CI job has, never +// a student's). +async function internalAddressForToken(token) { + const { body } = await provisionerCall('GET', `/environments/by-token/${token}`); + return body; +} + +// bootstrap-owner.js isn't copied into the api image (it's an operator-run one-off script, not +// a runtime dependency — same reason platform/test regression runs hit this before, see +// docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md's regression notes). Run it from the host against +// the sim Postgres's published port instead of `docker exec`ing into the container. +function bootstrapOwner(email, name, password) { + execFileSync('node', ['bootstrap-owner.js', email, name, password], { + cwd: require('path').join(__dirname, '../../../platform/api'), + env: { ...process.env, PLATFORM_DB_URL: 'postgres://postgres:postgres@localhost:5434/rootnode_platform_sim' }, + stdio: 'inherit', + }); +} + +// Raw TCP connect attempt from WITHIN the gateway container's own network namespace — proves +// isolation from the gateway's actual point of view, not the test driver's (which runs on the +// host and has a completely different, irrelevant reachability picture). +function gatewayCanReach(ip, port, timeoutMs = 2500) { + const script = ` + const net = require('net'); + const s = net.createConnection({ host: '${ip}', port: ${port}, timeout: ${timeoutMs} }); + s.on('connect', () => { console.log('REACHABLE'); s.destroy(); process.exit(0); }); + s.on('timeout', () => { console.log('UNREACHABLE(timeout)'); process.exit(0); }); + s.on('error', (e) => { console.log('UNREACHABLE(' + e.code + ')'); process.exit(0); }); + `; + const out = execFileSync('docker', ['exec', GATEWAY_CONTAINER, 'node', '-e', script]).toString().trim(); + return out.startsWith('REACHABLE'); +} + +function connectRouterWithoutClientCert(host, port, timeoutMs = 2500) { + return new Promise((resolve) => { + const req = https.request({ host, port, method: 'GET', rejectUnauthorized: false, timeout: timeoutMs }, (res) => { + resolve({ ok: true, status: res.statusCode }); + }); + req.on('error', (e) => resolve({ ok: false, error: e.code || e.message })); + req.on('timeout', () => { req.destroy(); resolve({ ok: false, error: 'timeout' }); }); + req.end(); + }); +} + +function openTerminal(launchPath, cookieHeader) { + return new Promise((resolve, reject) => { + const ws = new WebSocket(`${WS_URL}${launchPath}shell-ws`, { headers: { Cookie: cookieHeader } }); + let buffer = ''; + const timer = setTimeout(() => { ws.close(); reject(new Error(`terminal timed out, buffer so far: ${buffer.slice(-400)}`)); }, 30000); + ws.addEventListener('open', () => ws.send('kubectl get nodes\n')); + ws.addEventListener('message', (ev) => { + buffer += ev.data.toString(); + if (/Ready\s/.test(buffer) || /node\.?\d*\s+Ready/i.test(buffer)) { + clearTimeout(timer); + ws.close(); + resolve(buffer); + } + }); + ws.addEventListener('error', (e) => { clearTimeout(timer); reject(new Error(`ws error: ${e.message || e}`)); }); + }); +} + +function expectWsRejected(launchPath, cookieHeader, timeoutMs = 5000) { + return new Promise((resolve, reject) => { + const ws = new WebSocket(`${WS_URL}${launchPath}shell-ws`, { headers: { Cookie: cookieHeader } }); + const timer = setTimeout(() => { ws.close(); reject(new Error('expected the WS to be rejected, but it stayed open')); }, timeoutMs); + ws.addEventListener('open', () => { clearTimeout(timer); ws.close(); reject(new Error('expected the WS upgrade to be rejected, but it succeeded')); }); + ws.addEventListener('close', () => { clearTimeout(timer); resolve(); }); + ws.addEventListener('error', () => { clearTimeout(timer); resolve(); }); + }); +} + +async function main() { + console.log('Phase 4.1 two-host remote-routing simulation — slow, real k3s boots\n'); + + bootstrapOwner('sim-owner@rootnode.test', 'Sim Owner', 'SimOwnerPass!2026'); + const admin = makeSession(GW); + const { status: loginStatus } = await admin.request('POST', '/api/auth/instructor/login', { email: 'sim-owner@rootnode.test', password: 'SimOwnerPass!2026' }); + assert.equal(loginStatus, 200, 'owner login must succeed through the Gateway'); + + let alice, bob; + await test('roster: add Alice and Bob via admin (through the Gateway)', async () => { + const a = await admin.request('POST', '/api/admin/students', { display_name: 'Sim-Alice', cohort_id: null }); + const b = await admin.request('POST', '/api/admin/students', { display_name: 'Sim-Bob', cohort_id: null }); + alice = a.body.student_id; bob = b.body.student_id; + assert.notEqual(alice, bob); + }); + + const aliceSession = makeSession(GW); + const bobSession = makeSession(GW); + await aliceSession.request('POST', '/api/login', { student_id: alice }); + await bobSession.request('POST', '/api/login', { student_id: bob }); + + let aliceLaunch, bobLaunch, aliceEnv; + await test('launch: Alice and Bob each get a running environment on the simulated remote lab host', async () => { + const a = await aliceSession.request('POST', '/api/environment/launch'); + const b = await bobSession.request('POST', '/api/environment/launch'); + assert.equal(a.status, 200); assert.equal(a.body.status, 'running'); + assert.equal(b.status, 200); assert.equal(b.body.status, 'running'); + aliceLaunch = a.body; bobLaunch = b.body; + const aliceToken = aliceLaunch.launch_path.split('/')[2]; + aliceEnv = await internalAddressForToken(aliceToken); + assert.equal(aliceEnv.lab_host_routing_mode, 'router', 'sim-lab-host-01 must be configured as routing_mode: router for this test to mean anything'); + }); + + await test('§6/7 — the Gateway CANNOT reach the Lab\'s Docker-internal IP directly', async () => { + const reachable = gatewayCanReach(aliceEnv.internal_ip, aliceEnv.internal_port); + assert.equal(reachable, false, `gateway container reached ${aliceEnv.internal_ip}:${aliceEnv.internal_port} directly — the network isolation this whole phase depends on is not real`); + }); + + await test('§6/7 — but the Gateway DOES reach the Lab through the Lab Host Router (HTTP)', async () => { + const res = await aliceSession.rawFetch(aliceLaunch.launch_path); + assert.equal(res.status, 200); + const text = await res.text(); + assert.ok(text.length > 0, 'expected real HTML/asset content back from the Lab, routed through the Router'); + }); + + await test('§8 — Alice -> her own Lab: ALLOW', async () => { + const res = await aliceSession.rawFetch(aliceLaunch.launch_path); + assert.equal(res.status, 200); + }); + + await test('§8 — Bob -> his own Lab: ALLOW', async () => { + const res = await bobSession.rawFetch(bobLaunch.launch_path); + assert.equal(res.status, 200); + }); + + await test('§8 — Alice -> Bob\'s Lab: DENY', async () => { + const res = await aliceSession.rawFetch(bobLaunch.launch_path); + assert.equal(res.status, 403); + }); + + await test('§8 — garbage token: DENY', async () => { + const res = await aliceSession.rawFetch('/lab/not-a-real-token-at-all/'); + assert.equal(res.status, 404); + }); + + await test('§8 — anonymous Gateway request (no session at all): DENY', async () => { + const res = await fetch(`${GW}${aliceLaunch.launch_path}`); + assert.equal(res.status, 401); + }); + + await test('§8/9 — direct public access to the Lab Host Router: NOT AVAILABLE (mTLS required, no client cert presented)', async () => { + const routerUrl = new URL(aliceEnv.lab_host_router_endpoint); + const result = await connectRouterWithoutClientCert(routerUrl.hostname, routerUrl.port); + assert.equal(result.ok, false, 'the Router accepted a connection with no client certificate at all — mTLS is not actually being enforced'); + }); + + await test('§7 — WebSocket terminal connects and a real `kubectl get nodes` succeeds, through the Router', async () => { + const output = await openTerminal(bobLaunch.launch_path, `rn_session=${bobSession.getCookie('rn_session')}`); + assert.ok(/Ready/i.test(output), `expected a Ready node in kubectl output, got: ${output.slice(-300)}`); + }); + + await test('§8 — Alice cannot open Bob\'s terminal WebSocket even with a valid session (her own)', async () => { + await expectWsRejected(bobLaunch.launch_path, `rn_session=${aliceSession.getCookie('rn_session')}`); + }); + + await test('§8/9 — destroyed environment: routing token stops working, both at the Gateway and at the Router directly', async () => { + const aliceToken = aliceLaunch.launch_path.split('/')[2]; + await provisionerCall('DELETE', `/environments/${aliceEnv.environment_id}`); + // Give the Gateway's short-TTL env-cache a moment to expire so this proves real + // destroyed-environment handling, not just a stale cache hit. + await new Promise((r) => setTimeout(r, 6000)); + const res = await aliceSession.rawFetch(`/lab/${aliceToken}/`); + assert.ok([404, 503].includes(res.status), `expected the destroyed environment to be rejected, got ${res.status}`); + }); + + console.log(`\n${passed} passed, ${failed} failed`); + if (failed > 0) process.exit(1); +} + +main().catch((e) => { console.error(e); process.exit(1); }); diff --git a/infra/scripts/staging-smoke-test.sh b/infra/scripts/staging-smoke-test.sh index 9ab240f..9f53847 100755 --- a/infra/scripts/staging-smoke-test.sh +++ b/infra/scripts/staging-smoke-test.sh @@ -9,7 +9,13 @@ set -euo pipefail STAGING_URL="${STAGING_GATEWAY_URL:?set STAGING_GATEWAY_URL, e.g. https://staging.academy.example.com}" echo "==> Health checks" -curl -sf "$STAGING_URL/healthz" > /dev/null && echo "OK: Gateway /healthz" +# Phase 4.1 §23: this was `curl ... && echo ...` as one statement — under `set -e`, a failing +# command that ISN'T the last one in an && list does not trigger the shell to exit (a +# well-known bash gotcha), so a genuinely unreachable staging Gateway used to fall through +# silently instead of failing this script. Split into two statements so curl's own exit status +# is what `set -e` actually sees. Found by infra/scripts/drills/test-ci-fails-closed.sh. +curl -sf "$STAGING_URL/healthz" > /dev/null +echo "OK: Gateway /healthz" if [ -n "${OWNER_EMAIL:-}" ] && [ -n "${OWNER_PASSWORD:-}" ]; then echo "==> Full acceptance run (platform/test/e2e-phase3b.js against staging)" diff --git a/platform/test/e2e-phase3a.js b/platform/test/e2e-phase3a.js index 49edebd..c8e305a 100644 --- a/platform/test/e2e-phase3a.js +++ b/platform/test/e2e-phase3a.js @@ -160,12 +160,16 @@ async function main() { }); await test('audit log: security events were actually recorded', async () => { + // Phase 4.1 §16 — /api/admin/audit-events is now paginated (see platform/api/src/pagination.js); + // the events land on the most recent page, but this test has produced fewer than one + // page's worth by this point, so page 1's items are still every event so far. const { body } = await owner.request('GET', '/api/admin/audit-events'); - const actions = body.map((e) => e.action); + const items = body.items; + const actions = items.map((e) => e.action); assert.ok(actions.includes('login')); assert.ok(actions.includes('authz_denied')); assert.ok(actions.includes('instructor_update')); - assert.ok(body.every((e) => !JSON.stringify(e.metadata || {}).match(/password/i)), 'audit metadata must never contain password material'); + assert.ok(items.every((e) => !JSON.stringify(e.metadata || {}).match(/password/i)), 'audit metadata must never contain password material'); }); // ── Student session hardening ──────────────────────────────────────────────────────────