7 Commits
Author SHA1 Message Date
ced4568 5bc0743b08 docs(phase4.1): pre-production hardening documentation
Release to Docker Hub / Build & Push Docker Image (push) Canceled after 0s
Release to Docker Hub / Create GitHub Release (push) Canceled after 0s
Master overview (PHASE4_1_PREPRODUCTION_HARDENING.md) covering all 27
substantive items, plus REMOTE_AND_SELF_PACED_ARCHITECTURE.md (reviews
what already works for remote/self-paced delivery - student-driven
provisioning, cohort-less students, idle-reaper-destroy-then-continue -
and the one real gap: student authentication, explicitly Phase 5 scope,
not built here).

Updated PRODUCTION_NETWORK_SECURITY.md (the Gateway-to-Lab gap is now
RESOLVED, with the full design writeup and a new control-plane-vs-data-plane
section), PRODUCTION_DEPLOYMENT.md (four-cert TLS bundle),
PRODUCTION_VALIDATION_CHECKLIST.md (Router mTLS check, admission control
checks), PHASE4_PRODUCTION_INFRASTRUCTURE.md and ENGINEERING_SOP.md and
LAB_ARCHITECTURE.md (stale 'known gap' pointers updated to point at the
resolution), BACKUP_DR.md (destructive drill), README.md and
platform/README.md (Phase 4.1 pointers, lab-router component, updated
known-limitations).

Also fixes a real pre-existing bug found while auditing what gets
committed: infra/.gitignore's patterns were written repo-root-relative
(e.g. "infra/terraform/...") but the file lives in infra/, so git resolved
them relative to THAT directory and none of them ever matched anything -
a real .terraform/ provider cache showed up as untracked when it should
have been ignored. Fixed by dropping the redundant leading "infra/" from
every pattern.

Status: PHASE 4.1 COMPLETE / PRE-PRODUCTION HARDENING COMPLETE / READY FOR
HARDWARE DEPLOYMENT.
2026-08-24 20:16:20 -05:00
ced4568 f3c356951c test(preprod): two-host routing simulation, drill suite, CI security scanning
- infra/scripts/simulate-two-host/: real two-Docker-network simulation
  (Gateway genuinely isolated from any Lab bridge network) proving the
  Lab Host Router fix - 13/13 real assertions (HTTP+WebSocket through the
  Router, Alice/Bob isolation, garbage/destroyed tokens, mTLS-only Router
  access).
- infra/scripts/drills/: Lab Host loss/rebuild (9/9), deployment rollback,
  CI-fails-closed (6 required failure modes), fake cohort dress rehearsal
  (14/14, one continuous narrative through the real app).
- infra/ci/gitea-workflows/build-and-test.yml: new security-scan job
  (npm audit, gitleaks, hadolint, shellcheck, trivy - all free/open-source),
  severity policy in docs/SECURITY_SCANNING.md (CRITICAL/HIGH blocks,
  MEDIUM/LOW reported only). Gates build-images alongside regression.

Found and fixed two real pre-existing bugs via the CI-fails-closed drill:
staging-smoke-test.sh and production-smoke-test.sh both had a classic
set -e gotcha (curl ... && echo "OK" as one statement - a failing command
that isn't last in an && chain doesn't trigger -e) that let an
unreachable/broken endpoint fall through silently instead of failing the
smoke test.

platform/test/e2e-phase3a.js updated for the new paginated
/api/admin/audit-events response shape (Phase 4.1 pagination work).
2026-08-24 20:16:00 -05:00
ced4568 f5110d3c1b test(dr): destructive Postgres restore drill
infra/backup/test-destructive-restore-drill.sh - realistic academy data
(owner, admin instructor, cohort, 3 students, progress, attempts, an
assessment definition, achievements) seeded through the REAL Platform API,
encrypted-backed-up, the source Postgres container genuinely destroyed
(docker rm, not just left running alongside a second one), a fresh
Postgres created from nothing, restored, and the real Platform API booted
again against the restored database and used to log in as the restored
admin instructor over HTTP. Never touches the developer's real database.
Confirmed passing.
2026-08-24 20:15:45 -05:00
ced4568 a8e543234c feat(platform): pagination, admission control, cohort roles, version endpoint, structured assessment editor
- Real server-side pagination (platform/api/src/pagination.js) on the audit
  log (was a flat LIMIT 200), student attempt history (was LIMIT 100), and
  assessment session history (was fully unbounded - the one genuine growth
  risk found). Frontend updated (AuditView, AssessmentsView, StudentsView).
- Admission control: resource-policy.yaml's new (disabled-by-default)
  admission_control.max_concurrent_labs, checked before touching Docker at
  all, returns a clean 503 CAPACITY_EXCEEDED instead of failing mid-provision.
  Relayed as a clean student-facing message, never a raw 500. New
  GET /capacity endpoint feeds the instructor dashboard.
- cohort_instructors.role_in_cohort (existed since Phase 2B, never read) is
  now enforced: teacher vs viewer, rbac.js's
  requireCohortWriteAccess/requireStudentCohortWriteAccess.
- GET /api/version - app version, deployed Git SHA, latest migration as a
  schema-version stand-in.
- Structured assessment authoring UI (AssessmentsView.jsx's PoolSpecEditor)
  replaces the raw-JSON pool_spec textarea, backed by a new
  GET /api/admin/scenario-catalog endpoint. Owner-only raw-JSON fallback
  remains for anything the structured editor doesn't cover.
- NEEDS_ATTENTION_FAIL_THRESHOLD/STALE_DAYS moved to env-var configuration.

Full regression suite (e2e-phase2a/2b/3a/3b, 67/67) re-run clean after
these changes; frontend production build confirmed clean.
2026-08-24 20:15:39 -05:00
ced4568 addf474b70 feat(security): Postgres-backed login rate limiting
Replaces the in-memory sliding-window Map (reset on every restart, not
shareable across instances) with a login_attempts table (migration 016).
No Redis added - login is not a hot path, and this is a single indexed
range-scan query. Restart behavior confirmed for real:
platform/test/e2e-phase3a.js's rate-limiting test passes after a genuine
Platform API restart in the same run.
2026-08-24 20:15:03 -05:00
ced4568 6dda13b837 feat(learning): abandoned attempt handling
scenario_attempts.status = 'abandoned' (a valid value since Phase 3B,
never previously set by anything) now gets set on: environment
destroy/reset, logout, session expiry (piggybacked on the existing hourly
session sweeper - no new background job), and switching to a different
scenario while one is still open. Never an academic failure marker -
result stays NULL, history is preserved. Call-site wiring in
server.js/admin-routes.js lands in the next commit alongside other
platform hardening in those same files.
2026-08-24 20:14:52 -05:00
ced4568 eae8e48d43 feat(routing): Lab Host Router closes the Gateway-to-Lab cross-VM gap
Adds platform/lab-router/ — a narrow service that runs on the Lab Host,
authenticates the Gateway via mTLS, and resolves an opaque routing token to
a local container purely from Docker label state (no Postgres access, no
callback to the Provisioner). Replaces the Gateway's env.internal_ip
assumption (only valid when Gateway and Lab share a Docker daemon) with a
data-driven direct/router decision per lab_host
(lab_hosts.routing_mode/router_endpoint, migration 004), resolved in one
place (platform/gateway/src/lab-endpoint.js's labTarget()) that every
proxy/fetch call site in server.js now goes through.

Extends the existing Docker TLS CA (infra/scripts/docker-tls/generate-docker-tls.sh)
with a second mTLS leaf pair (router-server/gateway-client) for this new
data-plane connection, deployed by the docker-tls Ansible role into a
separate directory from the control-plane pair. Firewall
(rn-lab-host-01.rules) replaces the old speculative wide port-range
allowance with one narrow rule: platform_ip -> the Router's port only.

dev-local stays routing_mode: direct — zero behavior change for local dev,
confirmed by the full existing regression suite passing unmodified.
2026-08-24 20:14:39 -05:00
75 changed files with 3775 additions and 232 deletions
+5 -2
View File
@@ -196,8 +196,11 @@ Production topology and what's built vs. not yet: `docs/LAB_ARCHITECTURE.md`,
The production infrastructure package (Proxmox/Ansible/Terraform/firewall/monitoring/backup/CI)
that runs all of this on real hardware lives in `infra/` — see
`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`. Built and locally validated now; deployment is
pending purchase of the target Hetzner dedicated server.
`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` and `docs/PHASE4_1_PREPRODUCTION_HARDENING.md` (the
Gateway-to-Lab cross-VM routing fix, plus application hardening — real pagination, abandonment
handling, a Postgres-backed rate limiter, a structured assessment editor, admission control).
Built and locally validated now; deployment is pending purchase of the target Hetzner dedicated
server.
---
+10 -1
View File
@@ -37,10 +37,19 @@ requires typing `RESTORE` to confirm, then decrypts and pipes into `psql`.
**Genuinely tested**, not just written: `infra/backup/test-backup-restore.sh` runs the full
cycle against two disposable local Postgres containers, with the actual production migrations
applied (all 17 — 3 Provisioner + 14 Platform API, in the correct cross-service order), a
applied (3 Provisioner + Platform API, in the correct cross-service order), a
sample row inserted, backed up, restored into the second container, and verified byte-for-byte.
Confirmed passing, exit 0.
**Phase 4.1's destructive drill goes one step further**
(`infra/backup/test-destructive-restore-drill.sh`): realistic data (owner, admin instructor,
cohort, 3 students, progress, attempts, an assessment definition, achievements) seeded through
the REAL Platform API, backed up, the source Postgres **container genuinely destroyed**
(`docker rm`, not just left running alongside a second one), a fresh Postgres created from
nothing, restored, and the real Platform API booted again against the restored database and
used to log in as the restored admin instructor over HTTP — proving the whole stack recovers,
not just the SQL. Never touches the developer's real persistent database. Confirmed passing.
## Disaster recovery scenarios
| Scenario | RPO | RTO (estimate) | What happens |
+32 -4
View File
@@ -296,12 +296,40 @@ Compose, firewall, monitoring, backup, CI, load-test tooling).
- **Where the boundary held:** the Platform VM / Lab Host VM security split from
`docs/LAB_ARCHITECTURE.md` is codified in firewall rulesets and mutual-TLS Docker config, not
weakened anywhere.
- **Known gap surfaced, not silently fixed:** the Gateway's Lab-reachability mechanism doesn't
survive the move to two real separate VMs — see `docs/PRODUCTION_NETWORK_SECURITY.md`'s
"Known gap" section for what's built (firewall policy) vs. what's an explicit, reviewed,
not-yet-built application follow-up (Provisioner/Gateway address handling).
- **Known gap surfaced, then resolved in Phase 4.1:** the Gateway's Lab-reachability mechanism
didn't survive the move to two real separate VMs — flagged explicitly rather than silently
fixed, then actually built and simulation-tested (a Lab Host Router,
`platform/lab-router/`) in Phase 4.1. See §13 below.
- **Validated without the hardware:** real `tofu validate`, `ansible-playbook --syntax-check`,
`docker compose config`, `caddy validate`, `promtool`/`amtool`, and a genuine end-to-end
backup→encrypt→restore→verify cycle against disposable local Postgres with the real
production migrations applied. None of this proves the real Hetzner network boundary — that
needs `docs/PRODUCTION_VALIDATION_CHECKLIST.md` walked against the real server.
## 13. Pre-Production Hardening (Phase 4.1)
Full design: `docs/PHASE4_1_PREPRODUCTION_HARDENING.md`, plus
`docs/PRODUCTION_NETWORK_SECURITY.md` (routing resolution), `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`,
`docs/SECURITY_SCANNING.md`.
- **What it is:** closes the Gateway-to-Lab cross-VM gap Phase 4 flagged (a Lab Host Router,
`platform/lab-router/`, plus a routing abstraction in the Gateway,
`platform/gateway/src/lab-endpoint.js`), then hardens the application layer found along the
way: abandoned-attempt handling (`scenario_attempts.status = 'abandoned'`, never a punitive
failure), a Postgres-backed login rate limiter (survives a restart, ready to be
shared/multi-instance), real pagination on audit/attempt/assessment-session history, a
structured assessment authoring UI (owner-only raw JSON remains as a fallback), a minimal
`role_in_cohort` enforcement (`teacher`/`viewer`), a `/api/version` endpoint, and admission
control (a clean 503 "capacity full" instead of a 500 once a configured cap is reached).
- **Tested, not just designed:** a real two-Docker-network routing simulation
(`infra/scripts/simulate-two-host/`, 13/13), a destructive Postgres restore drill
(`infra/backup/test-destructive-restore-drill.sh`), a Lab Host loss/rebuild drill
(`infra/scripts/drills/test-lab-host-loss.js`), a deployment rollback drill, a CI-fails-closed
drill (which found and fixed a real pre-existing `set -e` bug in both smoke-test scripts —
see that drill's own output), and a full fake-cohort dress rehearsal
(`infra/scripts/drills/test-fake-cohort-rehearsal.js`) — all passing, all in
`infra/scripts/drills/` or `infra/backup/`.
- **Known limitation carried forward:** real remote/self-paced delivery needs actual student
authentication (today's `student_id`-only login relies on instructors controlling who gets a
valid id) — see `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md` for the full accounting of what
already works vs. what's genuinely Phase 5 scope.
+7 -5
View File
@@ -491,8 +491,10 @@ networking, egress, resource policy, reaper) against these decisions.
Phase 4 (`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`) codifies the Platform/Lab-Host split this
document designed as two real, separately-firewalled Proxmox VMs (`rn-platform-01`,
`rn-lab-host-01`) connected only by mutual-TLS Docker and a tightly scoped firewall allowance —
see `docs/PRODUCTION_NETWORK_SECURITY.md`. That document also records a gap Phase 4 surfaced but
did not resolve: the Gateway's `env.internal_ip` mechanism for reaching a Lab container assumes
Gateway and Lab share a Docker daemon, which breaks once they're genuinely on separate VMs. Read
that document's "Known gap" section before treating two-VM production as fully wired.
`rn-lab-host-01`) connected by two separate mTLS channels (Provisioner↔dockerd CONTROL plane,
Gateway↔Lab Host Router DATA plane) and a tightly scoped firewall allowance — see
`docs/PRODUCTION_NETWORK_SECURITY.md`. Phase 4 surfaced a gap in the Gateway's `env.internal_ip`
mechanism for reaching a Lab container (it assumes Gateway and Lab share a Docker daemon, which
breaks once they're genuinely on separate VMs); Phase 4.1
(`docs/PHASE4_1_PREPRODUCTION_HARDENING.md`) built and simulation-tested the fix, a Lab Host
Router (`platform/lab-router/`).
+254
View File
@@ -0,0 +1,254 @@
# Phase 4.1 — Pre-Production Hardening
**Status: PHASE 4.1 COMPLETE. PRE-PRODUCTION HARDENING COMPLETE. READY FOR HARDWARE DEPLOYMENT.**
Phase 4 built the production infrastructure package with the real Hetzner/Proxmox hardware not
yet purchased, and flagged one genuine architectural gap: the Gateway's mechanism for reaching
a student Lab assumed Gateway and Lab shared a Docker daemon, which is false the moment Labs run
on a genuinely separate `rn-lab-host-01` VM. Phase 4.1's job was to close that gap and harden
everything else discovered along the way, still without the real hardware.
## 1. The core fix: a Lab Host Router
**Before:** Gateway → `http://${env.internal_ip}:${env.internal_port}` directly — a Docker
bridge-local address, unreachable across a real VM boundary.
**After:** Gateway → (mode-aware) → either `direct` (unchanged, dev/single-host) or `router`
(production): a small new service, `platform/lab-router/`, runs on the Lab Host itself,
authenticates the Gateway via mTLS (the same private CA the Docker TLS control plane already
used, extended with two new leaf certs), resolves an opaque routing token to a local container
purely from `docker ps` label state (no Postgres access, no call back to the Provisioner), and
reverse-proxies HTTP and WebSocket traffic to it.
The decision of which mode a given `lab_host` uses is data-driven
(`lab_hosts.routing_mode`/`router_endpoint`, `platform/provisioner/migrations/004_lab_hosts_routing.sql`),
resolved once per request in `platform/gateway/src/lab-endpoint.js`'s `labTarget()` — the only
place in the Gateway that knows this distinction exists. Every proxy/fetch call site in
`platform/gateway/src/server.js` goes through it.
**Exact `env.internal_ip` replacement:** `labTarget(env, token)` returns `{ url, headers, agent,
mode }`. In `direct` mode, `url` is exactly the old `http://internal_ip:internal_port`. In
`router` mode, `url` is the lab_host's `router_endpoint`, `headers` carries
`X-RootNode-Route-Token: <token>` (computed server-side from the Provisioner-verified
environment row — never trusted from the browser), and `agent` is an `https.Agent` configured
with the Gateway's mTLS client identity.
## 2. No per-student host port
Deliberately not solved by publishing a host port per Lab (the originally-flagged, rejected
approach). The Router resolves entirely from Docker's own label state on its own host — no
port-per-student anything, no dynamic firewall rule per Lab.
## 3. Lab Host Router — design
See `platform/lab-router/src/server.js`'s header comment for the full "narrow by design" list:
one route (proxy-by-token), requires mTLS (no bearer-token fallback), never exposes Docker API/
container IDs/network info to its caller, never provides student-to-student routing (each
request resolves to exactly the one container matching its own token).
## 4. Multi-Lab-Host ready
Routing is resolved per `lab_host_id` — adding `rn-lab-host-02` later is one new `lab_hosts` row
(its own `routing_mode`/`router_endpoint`/TLS material), no Gateway code change. No scheduler
was built (explicitly out of scope) — the Provisioner still picks "the one enabled lab_host"
(`selectLabHost`), unchanged from Phase 1.
## 5. Local development compatibility
`dev-local` (the Unix-socket lab_host used by `docker-compose.dev.yml`) stays `routing_mode:
direct` — zero behavior change for the T480/any local dev setup, confirmed by the full existing
regression suite passing unmodified (§29 below) and by every new drill in this phase running
against the same dev stack. The standalone single-container Lab image build/boot is unaffected
(it never touches the Gateway or Provisioner at all).
## 6-9. Simulation, HTTP/WS validation, security tests, routing token security
`infra/scripts/simulate-two-host/` — a real two-Docker-network simulation (not mocks): Gateway
attached ONLY to an isolated bridge with no route to any Lab's bridge network, Lab Host Router
host-networked (mirrors `egress-proxy`'s existing pattern) so it can reach whatever per-Lab
bridge the Provisioner creates. 13/13 real assertions pass, including:
- Gateway genuinely cannot open a TCP connection to a Lab's Docker-internal IP (proven from
inside the Gateway container's own network namespace via a raw connect attempt).
- Gateway reaches the Lab correctly through the Router — HTTP AND a real WebSocket terminal
session running a genuine `kubectl get nodes` inside the Lab.
- Alice/Bob isolation (own lab ALLOW, other's DENY — HTTP and WebSocket both), garbage tokens,
anonymous requests, and destroyed-environment tokens are all rejected through the real code
path.
- The Router refuses any connection with no client certificate — mTLS is the real
authentication mechanism.
Routing token security reviewed: already CSPRNG since Phase 1 (`generateRoutingToken`), never
logged in plaintext (the Router's `tokenFingerprint` truncates), redacted in every API response
except the two narrow internal reads that legitimately need it, and — critically — the header
the Router trusts is computed server-side from the Provisioner-verified environment row, never
read from the browser's own request, so a browser cannot forge routing by supplying its own
`X-RootNode-Route-Token`.
## 10. Control plane vs data plane
Kept strictly separate — see `docs/PRODUCTION_NETWORK_SECURITY.md`'s dedicated section. Student
browser traffic is never proxied through the Docker API at any point.
## 11. Firewall update
`infra/firewall/rn-lab-host-01.rules`'s old speculative wide port-range allowance replaced with
one narrow rule: `platform_ip` → the Router's port only (`8443` default, configurable).
## 12. Self-paced / remote delivery architecture check
`docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`. Short version: student-driven provisioning,
cohort-less students, and "idle reaper destroys → progress survives → student returns → fresh
Lab → curriculum continues" all already worked before this phase and are now proven end-to-end
by two new drills. The one real gap for actual remote/self-paced delivery is student
authentication (today's bare `student_id` login assumes an instructor-controlled roster) — not
built here, explicitly Phase 5 scope.
## 13. Abandoned attempt cleanup
`scenario_attempts.status = 'abandoned'` (already a valid value since Phase 3B, never
previously set) now gets set on: environment destroy/reset (student- or instructor-triggered),
logout, session expiry (piggybacked on the existing hourly session sweeper — no new background
job), and switching to a different scenario while one is still open. Never a punitive marker —
`result` stays NULL, history is preserved, instructors see an honest "abandoned (reason)"
instead of a phantom "still in progress."
## 14. Gateway session cache
Reviewed — already a tight 2-second TTL since Phase 3A, specifically to bound the
revocation-propagation window. Decision: **keep it, don't add push-based invalidation** — the
added coupling (Platform API would need to know how to reach the Gateway) isn't justified by
shrinking an already-2-second window further. Documented reasoning, not a silently-accepted gap.
## 15. Rate limiting
Login rate limiting moved from an in-memory `Map` (reset on every restart, not shareable across
instances) to Postgres (`login_attempts` table, `migrations/016_login_attempts.sql`) —
`platform/api/src/security.js`'s `tooManyAttempts()`. No Redis added. Restart behavior
confirmed for real: `platform/test/e2e-phase3a.js`'s rate-limiting test passes after a genuine
Platform API restart in the same run.
## 16. Pagination
Real server-side pagination (`platform/api/src/pagination.js`) on: the audit log (was a flat
`LIMIT 200`), student attempt history (was `LIMIT 100`), and assessment session history (was
**fully unbounded** — the one genuine growth risk found). Scenario analytics was deliberately
left as-is — grouped by scenario id, naturally bounded around ~100 rows, no real growth risk to
paginate against. Frontend (`AuditView.jsx`, `AssessmentsView.jsx`, `StudentsView.jsx`) updated
to match; `frontend` build confirmed clean.
## 17. Needs-attention configuration
`NEEDS_ATTENTION_FAIL_THRESHOLD`/`NEEDS_ATTENTION_STALE_DAYS` moved from hardcoded constants to
env-var-configurable (matching the existing session-TTL pattern), sensible defaults unchanged.
## 18. Assessment authoring UX
Raw-JSON `pool_spec` textarea replaced with a structured editor
(`frontend/src/admin/AssessmentsView.jsx`'s `PoolSpecEditor`): required scenarios and random
pools picked from a real scenario catalog (`GET /api/admin/scenario-catalog`, new), grouped by
curriculum stage, via native multi-select — no new UI framework. An owner-only "advanced" raw
JSON view remains for anything the structured editor doesn't cover.
## 19. Cohort role-in-cohort
`cohort_instructors.role_in_cohort` existed since Phase 2B but was never read. Implemented
minimally: `teacher` (unchanged full capability) vs `viewer` (read-only — dashboards/analytics
still visible, but Prepare/Destroy Labs and provisioning a student's environment are now 403'd
server-side, `rbac.js`'s `requireCohortWriteAccess`/`requireStudentCohortWriteAccess`).
## 20. Backup / restore destructive drill
`infra/backup/test-destructive-restore-drill.sh` — real owner/admin/cohort/3-students/progress/
attempts/assessment/achievements seeded through the real Platform API, encrypted-backed-up, the
disposable Postgres container **genuinely destroyed** (not just restored into a second
container), a fresh Postgres created from nothing, restored, and the real Platform API booted
again against the restored database and used to log in as the restored admin over HTTP. All
data verified present; never touches the developer's real database.
## 21. Lab Host rebuild drill
`infra/scripts/drills/test-lab-host-loss.js` — a student's Lab container is destroyed out from
under the Provisioner (no clean destroy call, simulating total Lab Host loss). Proves: Platform
stays up throughout, the stale environment is never presented as healthy once the Provisioner's
real `reconcileOnStartup` reconciliation runs, academic progress is untouched, and the student
can launch a genuinely new Lab and continue.
## 22. Deployment rollback test
`infra/scripts/drills/test-deployment-rollback.sh` — builds a known-good and a deliberately
broken gateway image, "deploys" each via the real deterministic-tag mechanism, proves the smoke
test catches the regression, computes the rollback target with the exact same
`deploy-history.log` logic `rollback-production.sh` uses, and proves the rollback recovers.
Documents explicitly that image rollback never implies a DB migration rollback.
## 23. CI/CD failure tests
`infra/scripts/drills/test-ci-fails-closed.sh` — proves all six required failure modes (test/
syntax failure, frontend build failure, image build failure, config validation failure,
migration preflight failure, staging smoke test failure) genuinely exit nonzero. **Found and
fixed two real pre-existing bugs**: `staging-smoke-test.sh` and `production-smoke-test.sh` both
had a classic `set -e` gotcha (`curl ... && echo "OK"` as one statement — a failing command
that isn't the last in an `&&` chain doesn't trigger `-e`) that let an unreachable/broken
endpoint fall through silently instead of failing the smoke test. Fixed in both files.
## 24. Security scanning
`infra/ci/gitea-workflows/build-and-test.yml`'s new `security-scan` job — npm audit, gitleaks
(secrets), hadolint (Dockerfiles), shellcheck (shell scripts), trivy (container image CVEs).
All free/open-source. Severity policy documented in `docs/SECURITY_SCANNING.md`: CRITICAL/HIGH
blocks the build, MEDIUM/LOW is reported only. `security-scan` gates `build-images` the same
way `regression` does.
## 25. Version / release visibility
`GET /api/version` (Platform API) — app version (`package.json`), deployed Git SHA
(`GIT_SHA` env var, set to the same value as the image tag in the production compose file), and
the most recently applied migration as a schema-version stand-in. Unauthenticated, minimal, no
infrastructure detail.
## 26. Admission control failure mode
`resource-policy.yaml`'s new (disabled-by-default) `admission_control.max_concurrent_labs` —
when set, the Provisioner checks it BEFORE touching Docker at all, so a full host fails cleanly
(503, `CAPACITY_EXCEEDED`) instead of failing mid-provision. Relayed by the Platform API as a
clean "Lab capacity is temporarily full. Try again shortly." — never a raw 500. A new
`GET /capacity` Provisioner endpoint feeds the instructor dashboard's capacity display.
## 27. Fake cohort dress rehearsal
`infra/scripts/drills/test-fake-cohort-rehearsal.js` — one continuous narrative: instructor
login, cohort + 4 students, curriculum assignment, Prepare Labs, students working different
scenarios, hints, a deliberate failed validation, instructor analytics, a Lab reset, a Lab
reaped (same consequence as the real idle reaper — see the drill's own comment on why the real
multi-minute timer isn't waited out), the student returning and continuing, an exam
assigned/taken, and end-of-class Lab destruction — progress verified intact after every
destructive step. 14/14 passing.
## 28. Documentation
This file, `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`, `docs/SECURITY_SCANNING.md` (new).
Updated: `docs/LAB_ARCHITECTURE.md`, `docs/PRODUCTION_NETWORK_SECURITY.md`,
`docs/PRODUCTION_DEPLOYMENT.md`, `docs/PRODUCTION_VALIDATION_CHECKLIST.md`, `docs/BACKUP_DR.md`,
`docs/ENGINEERING_SOP.md`, `README.md`, `platform/README.md`.
## 29. Regression
Re-ran the full existing suite after every application-code change in this phase:
`e2e-phase2a.js` (10/10), `e2e-phase2b.js` (12/12), `e2e-phase3a.js` (21/21 — including a real
audit-log pagination-shape fix and confirming rate-limit persistence across a genuine restart),
`e2e-phase3b.js` (24/24). Frontend production build clean. Real Lab image builds/boots (both
dev-compose and standalone single-container). `tofu validate`, `ansible-playbook
--syntax-check`, production Compose config (both platform and lab-host), `caddy validate`,
`promtool`/`amtool`, and the backup/restore mechanism all re-validated clean after the
infrastructure changes in this phase (new `lab-router` service, new firewall rule, new Ansible
tasks). No knowingly-unexplained regression was left in this codebase at the end of this phase.
## What Phase 4.1 deliberately did NOT build
Production deployment itself, Hetzner/Proxmox installation, public course signup, payments,
subscriptions, parent portal, marketing site, full LMS, video hosting, email automation,
certificates, course marketplace, mobile app, a multi-host scheduler, an AI tutor, invasive
proctoring, held scenario content, student email/password authentication (documented as the one
real gap for self-paced delivery, not built). All explicit non-goals per this phase's
authorization.
+11 -8
View File
@@ -170,15 +170,18 @@ unchanged. No parent portal, billing, held scenarios, or production OIDC provide
RootNode production infrastructure is dedicated to RootNode Academy — never folded into
existing Synthos production servers, and canonical source stays Gitea, never GitHub `kubekosh`.
## Known gap surfaced during this phase (not silently papered over)
## Known gap surfaced during this phase (not silently papered over) — RESOLVED in Phase 4.1
Building the real production Compose/network config surfaced that the Gateway's existing
mechanism for reaching a Lab container (`env.internal_ip`, a Docker-bridge-local address) only
works when Gateway and Lab share a Docker daemon — true in every dev/single-host setup so far,
**not** true once Labs run on a genuinely separate `rn-lab-host-01`. The network-policy half is
built (a firewall allowance for Gateway→Lab-Host on a published port range,
`infra/firewall/rn-lab-host-01.rules`); the application half (the Provisioner recording a
reachable external address for each Lab, and the Gateway using it) is an explicit, reviewed,
NOT-YET-BUILT follow-up — see `docs/PRODUCTION_NETWORK_SECURITY.md` for the two candidate
designs and why neither was silently chosen here. This must land before real two-VM go-live;
flagging it clearly was judged better than guessing at an architecture change mid-infra-phase.
**not** true once Labs run on a genuinely separate `rn-lab-host-01`. At the time this phase
closed, the network-policy half was built but the application half was flagged as an explicit,
not-yet-built follow-up rather than guessed at mid-infra-phase.
**Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1) built and proved the application
half** — a Lab Host Router (`platform/lab-router/`) plus a `labTarget()` routing abstraction in
the Gateway, real-tested against a genuine two-network Docker simulation
(`infra/scripts/simulate-two-host/`), not just designed. See
`docs/PRODUCTION_NETWORK_SECURITY.md`'s "Gateway-to-Lab reachability" section for the full
resolution. This item no longer blocks real two-VM go-live.
+9 -3
View File
@@ -31,7 +31,7 @@ tofu apply
Creates `rn-platform-01` and `rn-lab-host-01` from the cloud-init template. Confirm both boot
and are SSH-reachable from the management network before continuing.
## 2. Generate Docker TLS certificates (offline, once)
## 2. Generate TLS certificates (offline, once)
```bash
export ROOTNODE_TLS_BUNDLE_DIR=/secure/offline/path # never inside the repo
@@ -39,8 +39,14 @@ export ROOTNODE_TLS_BUNDLE_DIR=/secure/offline/path # never inside the repo
./infra/scripts/docker-tls/verify-docker-tls.sh
```
The CA private key stays in `ROOTNODE_TLS_BUNDLE_DIR` and never touches either VM — only the CA
cert (public) and the relevant leaf cert+key go to each host.
Generates FOUR leaf certs under one CA (Phase 4.1 extended this from two): the CONTROL-plane
pair (`server-*`/`provisioner-client-*`, for Provisioner↔dockerd) and the DATA-plane pair
(`router-server-*`/`gateway-client-*`, for Gateway↔Lab Host Router — see
`docs/PRODUCTION_NETWORK_SECURITY.md`'s "control plane vs data plane" section). The CA private
key stays in `ROOTNODE_TLS_BUNDLE_DIR` and never touches either VM — only the CA cert (public)
and each host's own leaf cert+key go to each host, both pairs deployed by the same `docker-tls`
Ansible role into two separate directories (`/etc/rootnode/docker-tls/` and
`/etc/rootnode/lab-router-tls/`).
## 3. Configure both hosts
+62 -25
View File
@@ -12,7 +12,8 @@ management_cidr ──SSH──▶ rn-platform-01 │
management_cidr ──SSH──▶ rn-lab-host-01 │
proxmox_mgmt_cidr ──22/8006──▶ Proxmox host │
│
rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd)
rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd) [CONTROL plane]
rn-platform-01 (Gateway) ──mTLS:8443──▶ rn-lab-host-01 (Lab Host Router) [DATA plane]
rn-lab-host-01 (promtail) ──3100──▶ rn-platform-01 (Loki)
rn-lab-host-01 (node-exporter/cadvisor) ◀──scrape── rn-platform-01 (Prometheus)
```
@@ -37,6 +38,25 @@ Firewall source of truth: `infra/firewall/{proxmox-host,rn-platform-01,rn-lab-ho
(Grafana is reachable only via Caddy's `/grafana/*` path, itself restricted to
`management_cidr` by a `caddy` matcher — see `infra/docker/platform/Caddyfile.j2`).
## Control plane vs data plane (Phase 4.1 §10)
Two separate, never-conflated mTLS connections between the two VMs, deliberately kept apart
both conceptually and operationally:
| | Control plane | Data plane |
|---|---|---|
| What | Provisioner manages Lab containers | Gateway serves student browser traffic |
| Path | Provisioner → dockerd (port 2376) | Gateway → Lab Host Router (port 8443) |
| Who initiates | Provisioner only | Gateway only |
| What it can do | Create/start/stop/remove containers, full Docker API | Reverse-proxy one HTTP/WS request to one already-running container by opaque token — nothing else |
| Certs | `provisioner-client-*` / `server-*` | `gateway-client-*` / `router-server-*` |
| Exposes Docker API to caller? | Yes (that's its job — Provisioner IS a Docker client) | **Never** — the Router never exposes Docker API, container IDs, or network info to its caller |
The student browser's traffic is proxied at the Gateway/Router layer and never touches the
Docker API at any point — the Router deliberately has no route that echoes back anything
Docker-internal (see `platform/lab-router/src/server.js`'s header comment for the full "narrow
by design" list).
## Docker TLS (Provisioner ↔ Lab Host)
Mutual TLS on port 2376, restricted at the firewall to `platform_ip` only. Application support
@@ -57,34 +77,51 @@ front this with Cloudflare or expose the origin directly is left as a deployment
being able to see plaintext unless "Full (strict)" mode is used) — not decided here since it
doesn't affect anything else in this document.
## Known gap: Gateway-to-Lab reachability across two real VMs
## Gateway-to-Lab reachability across two real VMs — RESOLVED in Phase 4.1
**This is the most important item in this document.**
Phase 4 flagged this as the single most important open item; Phase 4.1
(`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1) closed it. Kept here as the historical record of
what the gap was and how it was closed, since the reasoning matters for anyone extending this
later (e.g. adding a second Lab Host).
The Gateway currently reaches a running Lab container via `env.internal_ip` — a Docker
bridge-local address recorded by the Provisioner when it creates the container
(`platform/gateway/src/server.js`, `platform/provisioner`, built in Phase 2A). This works when
Gateway and the Lab share a Docker daemon, which has been true in every setup so far, including
dev and any single-host deployment. **It stops being true once Labs run on a genuinely separate
`rn-lab-host-01`** — a Docker bridge-internal IP on one host is not routable from a different
host by default.
**The gap, as originally found:** the Gateway used to reach a running Lab container via
`env.internal_ip` — a Docker bridge-local address recorded by the Provisioner when it creates
the container. That only works when Gateway and the Lab share a Docker daemon, true in dev and
any single-host setup, false the moment Labs run on a genuinely separate `rn-lab-host-01`.
What's built now (network-policy half only): a firewall allowance in
`infra/firewall/rn-lab-host-01.rules` for `platform_ip` → `lab_gateway_port_range`
(`30000:32767` by default, configurable), so that once Lab containers publish ports on the Lab
Host's own interface, the Gateway VM is permitted to reach them.
**The fix — a Lab Host Router (`platform/lab-router/`):** a small, narrow service that runs on
rn-lab-host-01 (host-networked, like `egress-proxy`, so it can reach whatever per-Lab bridge
network the Provisioner creates) and does exactly one thing: given an opaque routing token over
mTLS, find the ONE local container labeled with that token (`docker ps --filter
label=rootnode.routing-token=<token>` — resolved purely from local Docker state, no Postgres,
no call back to the Provisioner) and reverse-proxy the request to it, HTTP and WebSocket both.
What's **not yet built** (application half — explicitly flagged, not silently decided here):
the Provisioner needs to either (a) publish each Lab container's relevant port(s) on the Lab
Host's interface and record a reachable `external_ip`/`external_port` alongside (or instead of)
`internal_ip`/`internal_port`, or (b) route Gateway→Lab traffic through a proxy/tunnel that
lives on the Lab Host and forwards to the container's internal address. Option (a) is simpler
and consistent with how the egress proxy and cadvisor are already handled in this phase's
compose files; option (b) avoids publishing a wide port range but adds a moving part. This is a
small, reviewable, application-level change — not a network-policy change — and should land as
its own reviewed commit before real two-VM production traffic depends on it. Until it lands,
Gateway↔Lab terminal/scenario traffic will not work correctly across two separate VMs, even
though every other production networking, firewall, and TLS boundary is in place and correct.
The Gateway now decides `direct` vs `router` per `lab_host` (`lab_hosts.routing_mode` +
`router_endpoint`, `platform/provisioner/migrations/004_lab_hosts_routing.sql`) via
`platform/gateway/src/lab-endpoint.js` — a single function, `labTarget(env, token)`, that every
proxy/fetch call site in `server.js` goes through instead of building
`http://${env.internal_ip}:${env.internal_port}` itself. `dev-local` stays `direct` (T480/local
dev unaffected, zero behavior change there); a real `rn-lab-host-01` is configured `router`.
**Firewall (`infra/firewall/rn-lab-host-01.rules`):** the old speculative
`lab_gateway_port_range` allowance (which assumed option (a), publishing per-Lab ports) was
replaced with a single, narrow rule: `platform_ip` → port `{{ lab_router_port | default('8443')
}}` only. No wide port range, no per-Lab port publishing at all — the Router is the only thing
the Gateway ever talks to on rn-lab-host-01 for Lab traffic.
**mTLS:** the Gateway↔Router pair uses the SAME private CA as the Provisioner↔dockerd pair
(`infra/scripts/docker-tls/generate-docker-tls.sh`, extended in Phase 4.1 to also issue
`router-server`/`gateway-client` leaf certs), deployed to a separate directory
(`/etc/rootnode/lab-router-tls/`) by the same `docker-tls` Ansible role.
**Proven, not just designed:** `infra/scripts/simulate-two-host/` builds a real two-network
Docker simulation (Gateway genuinely isolated on its own bridge, no route to any Lab's bridge)
and runs 13 real assertions against it — Gateway cannot reach a Lab's Docker-internal IP
directly, but reaches it correctly through the Router (HTTP and a real WebSocket
`kubectl get nodes`), Alice/Bob isolation holds, garbage/destroyed tokens are rejected, and the
Router itself refuses any connection without the Gateway's client certificate. See that
directory's README for exactly what is and isn't proven by a same-daemon simulation versus real
separate hardware.
## Cookie / auth production settings
+10 -2
View File
@@ -19,8 +19,9 @@ can prove in advance.
## Lab Host VM
- [ ] Docker daemon TLS verified: `docker --tlsverify --tlscacert=... -H tcp://<lab_host_ip>:2376 info` succeeds from the Platform VM, fails from anywhere else
- [ ] Lab Host Router mTLS verified: a request to `https://<lab_host_ip>:8443` with no client cert is rejected; with the Gateway's real client cert it isn't (mirrors `infra/scripts/simulate-two-host/test-remote-routing.js`'s "direct public access: NOT AVAILABLE" case, now against the real host)
- [ ] Provisioner successfully creates a real Lab against the real Lab Host
- [ ] Gateway can actually reach that Lab's terminal — **this depends on the not-yet-built application change documented in `docs/PRODUCTION_NETWORK_SECURITY.md`'s "Known gap" section; do not mark this item done until that change has landed and been verified for real**
- [ ] Gateway can actually reach that Lab's terminal (HTTP UI load + WebSocket `kubectl` command) — application mechanism built and simulation-tested in Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1/§6/§7); this line is the real-hardware confirmation that the same result holds across the actual two VMs, not two Docker networks on one daemon
- [ ] Egress proxy correctly restricts a Lab's outbound traffic (Provisioner's per-Lab `DOCKER-USER` chain)
## Security
@@ -38,7 +39,7 @@ can prove in advance.
## Backup
- [ ] Manual `backup-postgres.sh` run succeeds against real Postgres, real off-host target reachable
- [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness
- [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness — Phase 4.1's `infra/backup/test-destructive-restore-drill.sh` proved the full mechanism (realistic data, real storage destruction, restore, real Platform API boot) locally; this line is the real-hardware confirmation
- [ ] Cron job confirmed scheduled (`crontab -l`)
## Observability
@@ -46,5 +47,12 @@ can prove in advance.
- [ ] Trigger one alert deliberately (e.g. stop a service) and confirm it fires and reaches the webhook
- [ ] Loki has logs flowing from both VMs
## Pre-production hardening (Phase 4.1 — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md)
- [ ] `resource-policy.yaml`'s `admission_control.max_concurrent_labs` set from a real measured
`infra/scripts/capacity-model.js` run against the real Lab Host, not left disabled
- [ ] A deliberate over-capacity attempt returns the clean 503 "Lab capacity is temporarily
full" message, not a 500 — confirms the real host's admission control matches the local test
- [ ] `docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md`'s steps still make sense once capacity numbers are real
Only once every box above is checked against the real server does the deployment graduate from
"infrastructure package built" to "actually in production."
@@ -0,0 +1,85 @@
# Remote & Self-Paced Delivery Architecture
Phase 4.1 §12. This is an architecture review, not a new product — no self-registration,
payments, or public enrollment is built here (explicit non-goal). The question this document
answers: does the infrastructure and application assume "an instructor always provisions the
Lab," and if so, where, and what would need to change for self-paced delivery to work.
## The short answer: less of that assumption exists than expected
RootNode Academy's Lab lifecycle was built student-driven from Phase 1 — `POST
/api/environment/launch` is called by the STUDENT'S OWN session, not by an instructor action.
"Prepare Labs" (`POST /api/admin/cohorts/:id/prepare-labs`) is a classroom **convenience** that
front-loads several students' launches at once before a synchronous class session, not the only
path to a running Lab. This was true before Phase 4.1 and confirmed still true by this phase's
own regression suites (`platform/test/e2e-phase2a.js` roster's two students with `cohort_id:
null` and takes them through login → curriculum → launch → scenario work with zero cohort
involvement) and by two new Phase 4.1 drills built specifically to exercise the remote/returning
-student narrative:
- `infra/scripts/drills/test-lab-host-loss.js` — student launches, makes progress, their Lab is
destroyed out from under them (standing in for an idle-reaper destroy), Platform stays up,
progress is untouched, they launch again unprompted and continue.
- `infra/scripts/drills/test-fake-cohort-rehearsal.js` — the same "returns later, fresh Lab,
curriculum continues" narrative as part of a fuller classroom rehearsal.
Both pass today, with zero application code changes needed for the self-paced-relevant parts of
this flow specifically — see the Phase 4.1 report for the full run results.
## What already works, unmodified
- **Student-initiated provisioning** — no instructor step is required between "student wants to
work" and "Lab exists." (`platform/api/src/server.js`'s `/api/environment/launch`.)
- **Cohort-less students** — `cohort_id` is nullable on student creation
(`POST /api/admin/students`); a student with no cohort logs in, sees the curriculum, launches
Labs, and makes progress identically to a cohorted one. Confirmed by inspection AND by the
existing regression suite, not just by schema reading.
- **Idle reaper → academic state survives → return → continue** — the exact lifecycle this
section describes as the target self-paced experience is what Phase 1's reaper +
Phase 2B's Postgres-backed academic data (separate from the disposable Lab) were already
designed to produce. Phase 4.1 proved it end-to-end, twice (see above).
- **Curriculum content itself is not cohort-gated** — `GET /api/curriculum/path` reads the one
curriculum path file directly (`curriculum.loadCurriculumPath()`); it never checks
`curriculum_assignments` or cohort membership at all. A self-paced student sees the full
curriculum with no instructor action required to "unlock" it.
## What's genuinely missing for a real self-paced product (Phase 5, not built here)
1. **No self-registration / public enrollment.** A student row is created only by an instructor
(`POST /api/admin/students`, admin/owner RBAC-gated). There is no public "sign up" surface.
This is almost certainly the right call to leave alone for now — it's explicitly a Phase 5+
product/business decision (course commerce, is it free, who's allowed to enroll), not an
infrastructure gap.
2. **No real student authentication.** A student session is created from a bare `student_id`
(`POST /api/login`, no password) — deliberately, since today every `student_id` is issued and
controlled by an instructor via the roster, not chosen or known by the public. A genuinely
remote/self-paced product where students access their account without an instructor in the
loop needs real credentials (email+password, or a magic link) — reusing the same
`password_hash`/`scrypt` machinery `instructors` already has (`src/auth/password.js`) would
be the natural extension, not a new auth system. **This is the one item on this list that
would actually block self-paced delivery today** — everything else already works or is a
product decision, not an infrastructure one.
3. **`curriculum_assignments.student_id` is schema-ready but unwired.** The table has supported
a direct per-student curriculum assignment (independent of any cohort) since Phase 2B
(`CONSTRAINT curriculum_assignment_target CHECK (cohort_id IS NOT NULL OR student_id IS NOT
NULL)`) — but nothing in the application ever reads or writes that column today; there's only
one curriculum path, so nothing has needed to differentiate yet. Not a gap that blocks
self-paced delivery now (every student already sees the one path with no gate), but the
moment a second curriculum path exists, this is where per-individual assignment lands — the
schema was designed for exactly this and doesn't need a migration to support it.
4. **No delivery-mode concept anywhere.** In-person, remote, and self-paced sessions aren't
distinguished in any data model today, and nothing currently needs them to be — capacity
planning, the reaper, and academic records all behave identically regardless of how a student
is attending. Worth being aware this is unmodeled, not worth building a taxonomy for before a
real product reason exists (YAGNI).
## What Phase 4.1 deliberately did NOT do here
No signup flow, no student password/email auth implementation, no per-student curriculum
assignment route, no payments, no public course catalog, no held/premium scenario content. All
explicit non-goals per this phase's authorization. This document exists so Phase 5 planning
starts from an accurate picture of the gap (one real item: student authentication) instead of a
guess.
+47
View File
@@ -0,0 +1,47 @@
# Security Scanning (Phase 4.1 §24)
`infra/ci/gitea-workflows/build-and-test.yml`'s `security-scan` job. All free, open-source,
no account/signup required for any of them — no paid scanner was added.
## What's scanned
| Tool | Checks | Scope |
|---|---|---|
| `npm audit --audit-level=high` | Known-vulnerable dependencies | Every first-party Node service |
| [gitleaks](https://github.com/gitleaks/gitleaks) | Committed secrets (keys, tokens, credentials) | Whole repo |
| [hadolint](https://github.com/hadolint/hadolint) | Dockerfile best-practice/security lint | Every first-party Dockerfile |
| [shellcheck](https://www.shellcheck.net/) | Shell script correctness/security | `infra/**/*.sh` |
| [trivy](https://github.com/aquasecurity/trivy) | OS/library CVEs in a built container image | `platform-api` image (representative — same base/layers as the other Node services) |
## Severity policy
- **CRITICAL / HIGH** — fails the build (`security-scan` blocks `build-images`, which blocks
`deploy-staging`, which blocks production promotion — see the job dependency chain in the
workflow file).
- **MEDIUM / LOW** — reported in the job output, never blocks. A build that's permanently red
over a low-severity transitive dependency finding nobody can immediately fix trains everyone
to ignore red builds, which is worse than not scanning at all.
`npm audit --audit-level=high` and trivy's `--severity HIGH,CRITICAL` both implement this
directly at the tool level, not via manual triage.
## What's NOT in the automated job (and why)
- **Dedicated IaC security linting** (checkov/tflint) — not added. `infra/scripts/validate-compose.sh`,
`infra/firewall/validate-firewall.sh`, `ansible-playbook --syntax-check`, and `tofu validate`
(all in the `regression` job) already catch structural/syntax problems in the infra code;
a dedicated security-policy linter for Terraform/Ansible is a reasonable Phase 5 addition once
there's a real cloud footprint to misconfigure, not before.
- **DAST/penetration testing** — out of scope for CI; this is a build-time gate, not a running-system
test. Covered separately by `docs/PRODUCTION_VALIDATION_CHECKLIST.md`'s security section once
real hardware exists.
- **SAST beyond hadolint/shellcheck** — the codebase is small enough (a handful of first-party
Node services) that `npm audit` + secret scanning + Dockerfile/shell lint cover the realistic
risk surface; a dedicated SAST tool (Semgrep, CodeQL) is a reasonable addition once the
codebase or team grows enough to justify the added CI time.
## Local reproduction
Every check above can be run locally with the exact same free tools — no CI-only dependency.
See the `security-scan` job's steps in `infra/ci/gitea-workflows/build-and-test.yml` for the
exact commands (they're copy-pasteable).
+3
View File
@@ -57,3 +57,6 @@ tr:hover td { background: var(--surface2); }
.linkBtn { background: none; border: none; color: var(--blue); cursor: pointer; font-size: 0.85rem; padding: 0; text-decoration: underline; }
.errorBox { background: var(--red-dim); color: var(--red); border: 1px solid var(--red); border-radius: var(--radius); padding: 0.6rem 0.8rem; font-size: 0.85rem; margin-bottom: 1rem; }
.muted { color: var(--text-2); font-size: 0.85rem; }
/* Phase 4.1 §16 — shared pagination control (AuditView, StudentsView attempts). */
.pager { display: flex; align-items: center; gap: 0.75rem; margin-top: 0.75rem; }
+107 -13
View File
@@ -29,7 +29,7 @@ export default function AssessmentsView({ role }) {
{canAuthor && (
<div className={styles.card}>
<h2>Create assessment</h2>
<CreateAssessmentForm onCreated={reload} />
<CreateAssessmentForm onCreated={reload} role={role} />
</div>
)}
<div className={styles.card}>
@@ -56,20 +56,86 @@ export default function AssessmentsView({ role }) {
)
}
function CreateAssessmentForm({ onCreated }) {
const [form, setForm] = useState({ id: '', type: 'exam', title: '', duration_minutes: 60, hint_policy: 'disabled', pool_spec: '' })
// Phase 4.1 §18 — pool_spec is built from structured picks (required scenarios + named random
// pools), not hand-typed JSON. `catalog` ({ stages: [{ id, name, scenarios: [{id,title}] }] })
// comes from GET /api/admin/scenario-catalog. An owner can still drop into a raw-JSON view for
// anything the structured editor doesn't cover — see the toggle below.
function PoolSpecEditor({ catalog, required, setRequired, pools, setPools }) {
const allScenarios = (catalog?.stages || []).flatMap((s) => s.scenarios.map((sc) => ({ ...sc, stageName: s.name })))
const titleFor = (id) => allScenarios.find((s) => s.id === id)?.title || id
const addPool = () => setPools([...pools, { count: 1, from: [] }])
const removePool = (i) => setPools(pools.filter((_, idx) => idx !== i))
const updatePool = (i, patch) => setPools(pools.map((p, idx) => (idx === i ? { ...p, ...patch } : p)))
return (
<div>
<label className={styles.label}>Required scenarios (always included)</label>
<select className={styles.select} multiple size={6} style={{ width: '100%' }}
value={required} onChange={(e) => setRequired([...e.target.selectedOptions].map((o) => o.value))}>
{catalog?.stages.map((stage) => (
<optgroup key={stage.id} label={stage.name}>
{stage.scenarios.map((sc) => <option key={sc.id} value={sc.id}>{sc.title}</option>)}
</optgroup>
))}
</select>
<p className={styles.muted}>{required.length ? `${required.length} selected: ${required.map(titleFor).join(', ')}` : 'None selected — cmd/ctrl-click to pick multiple.'}</p>
<label className={styles.label} style={{ marginTop: '1rem', display: 'block' }}>Random pools (draw N at random from a set, every attempt)</label>
{pools.map((pool, i) => (
<div key={i} className={styles.card} style={{ marginBottom: '0.5rem' }}>
<div className={styles.formRow}>
<div style={{ maxWidth: '100px' }}><label className={styles.label}>Draw count</label>
<input className={styles.input} type="number" min="1" value={pool.count}
onChange={(e) => updatePool(i, { count: Number(e.target.value) })} /></div>
<div><button type="button" className={styles.linkBtn} onClick={() => removePool(i)}>Remove pool</button></div>
</div>
<label className={styles.label}>Draw from</label>
<select className={styles.select} multiple size={5} style={{ width: '100%' }}
value={pool.from} onChange={(e) => updatePool(i, { from: [...e.target.selectedOptions].map((o) => o.value) })}>
{catalog?.stages.map((stage) => (
<optgroup key={stage.id} label={stage.name}>
{stage.scenarios.map((sc) => <option key={sc.id} value={sc.id}>{sc.title}</option>)}
</optgroup>
))}
</select>
{pool.count > pool.from.length && <p className={styles.muted} style={{ color: 'var(--amber, #b58900)' }}>Draw count ({pool.count}) exceeds the pool size ({pool.from.length}) — every draw will use every scenario, not a random subset.</p>}
</div>
))}
<button type="button" className={styles.linkBtn} onClick={addPool}>+ Add random pool</button>
</div>
)
}
function CreateAssessmentForm({ onCreated, role }) {
const [form, setForm] = useState({ id: '', type: 'exam', title: '', duration_minutes: 60, hint_policy: 'disabled' })
const [catalog, setCatalog] = useState(null)
const [required, setRequired] = useState([])
const [pools, setPools] = useState([])
const [advanced, setAdvanced] = useState(false)
const [rawPoolSpec, setRawPoolSpec] = useState('')
const [error, setError] = useState(null)
const [busy, setBusy] = useState(false)
useEffect(() => { adminApi.scenarioCatalog().then(setCatalog).catch((e) => setError(e.message)) }, [])
const submit = async (e) => {
e.preventDefault()
setBusy(true); setError(null)
try {
const pool_spec = JSON.parse(form.pool_spec || '{"required":[],"random":[]}')
// Owner's raw-JSON view, when open, is the source of truth — otherwise built from the
// structured pickers above.
const pool_spec = advanced
? JSON.parse(rawPoolSpec || '{"required":[],"random":[]}')
: { required, random: pools.filter((p) => p.count > 0 && p.from.length > 0) };
if (!advanced && !required.length && !pool_spec.random.length) {
throw new Error('Pick at least one required scenario or one random pool.')
}
await adminApi.createAssessment({ ...form, duration_minutes: Number(form.duration_minutes), pool_spec })
setForm({ id: '', type: 'exam', title: '', duration_minutes: 60, hint_policy: 'disabled', pool_spec: '' })
setForm({ id: '', type: 'exam', title: '', duration_minutes: 60, hint_policy: 'disabled' })
setRequired([]); setPools([]); setRawPoolSpec('')
onCreated()
} catch (e) { setError(e.message?.includes('JSON') ? 'pool_spec must be valid JSON' : e.message) } finally { setBusy(false) }
} catch (e) { setError(e.message?.includes('JSON') ? 'Raw pool_spec must be valid JSON' : e.message) } finally { setBusy(false) }
}
return (
@@ -91,26 +157,47 @@ function CreateAssessmentForm({ onCreated }) {
<option value="disabled">disabled</option><option value="flagged">flagged</option><option value="free">free</option>
</select></div>
</div>
<label className={styles.label}>Pool spec (JSON)</label>
<textarea className={styles.input} style={{ width: '100%', fontFamily: 'var(--mono)', minHeight: '4rem' }}
placeholder='{"required":["scenario-id"],"random":[{"count":1,"from":["a","b"]}]}'
value={form.pool_spec} onChange={(e) => setForm({ ...form, pool_spec: e.target.value })} required />
{role === 'owner' && (
<p className={styles.muted}>
<button type="button" className={styles.linkBtn} onClick={() => setAdvanced(!advanced)}>
{advanced ? '← Back to the structured editor' : 'Advanced: edit pool_spec as raw JSON →'}
</button>
</p>
)}
{advanced ? (
<>
<label className={styles.label}>Pool spec (JSON)</label>
<textarea className={styles.input} style={{ width: '100%', fontFamily: 'var(--mono)', minHeight: '4rem' }}
placeholder='{"required":["scenario-id"],"random":[{"count":1,"from":["a","b"]}]}'
value={rawPoolSpec} onChange={(e) => setRawPoolSpec(e.target.value)} required />
</>
) : !catalog ? <p className={styles.muted}>Loading scenario catalog…</p> : (
<PoolSpecEditor catalog={catalog} required={required} setRequired={setRequired} pools={pools} setPools={setPools} />
)}
<button className={styles.btn} style={{ marginTop: '0.75rem' }} disabled={busy}>{busy ? 'Creating…' : 'Create'}</button>
</form>
)
}
function AssessmentDetail({ definition, cohorts, onBack }) {
const [sessions, setSessions] = useState(null)
// Phase 4.1 §16 — assessmentSessions is now paginated (was unbounded); `result` holds the
// full { items, page, total, total_pages } envelope, `sessions` stays the plain array for
// the table below so its render logic doesn't need to change.
const [result, setResult] = useState(null)
const [page, setPage] = useState(1)
const [error, setError] = useState(null)
const [notice, setNotice] = useState(null)
const [cohortId, setCohortId] = useState(cohorts?.[0]?.cohort_id || '')
const [busy, setBusy] = useState(false)
const reload = useCallback(() => {
adminApi.assessmentSessions(definition.assessment_definition_id).then(setSessions).catch((e) => setError(e.message))
}, [definition])
adminApi.assessmentSessions(definition.assessment_definition_id, page).then(setResult).catch((e) => setError(e.message))
}, [definition, page])
useEffect(reload, [reload])
const sessions = result?.items || null
const assign = async () => {
setBusy(true); setError(null); setNotice(null)
@@ -158,6 +245,13 @@ function AssessmentDetail({ definition, cohorts, onBack }) {
</tbody>
</table>
)}
{result && result.total_pages > 1 && (
<div className={styles.pager}>
<button disabled={page <= 1} onClick={() => setPage((p) => p - 1)}>Previous</button>
<span className={styles.muted}>Page {result.page} of {result.total_pages} ({result.total} total)</span>
<button disabled={page >= result.total_pages} onClick={() => setPage((p) => p + 1)}>Next</button>
</div>
)}
</div>
</>
)
+32 -21
View File
@@ -2,36 +2,47 @@ import { useState, useEffect } from 'react'
import { adminApi } from './api'
import styles from './Admin.module.css'
// admin/owner only (enforced server-side too). Latest 200 events, no pagination UI — this is a
// pilot-scale audit log, not a SIEM; add pagination if that ever stops being true.
// admin/owner only (enforced server-side too). Phase 4.1 §16 — real server-side pagination
// (platform/api/src/pagination.js) replaces the old flat "latest 200, nothing older" cap.
export default function AuditView() {
const [events, setEvents] = useState(null)
const [page, setPage] = useState(1)
const [result, setResult] = useState(null)
const [error, setError] = useState(null)
useEffect(() => {
adminApi.auditEvents().then(setEvents).catch((e) => setError(e.message))
}, [])
setResult(null)
adminApi.auditEvents(page).then(setResult).catch((e) => setError(e.message))
}, [page])
const events = result?.items || []
return (
<div className={styles.card}>
<h2>Audit log</h2>
<p className={styles.muted}>Most recent 200 security/admin events.</p>
<p className={styles.muted}>{result ? `${result.total} total events` : 'Security/admin events, most recent first.'}</p>
{error && <div className={styles.errorBox}>{error}</div>}
{!events ? <p className={styles.muted}>Loading…</p> : (
<table>
<thead><tr><th>When</th><th>Actor</th><th>Action</th><th>Resource</th><th>Result</th></tr></thead>
<tbody>
{events.map((e) => (
<tr key={e.audit_event_id}>
<td>{new Date(e.created_at).toLocaleString()}</td>
<td>{e.actor_type}{e.actor_id ? ` · ${e.actor_id.slice(0, 8)}` : ''}</td>
<td>{e.action}</td>
<td>{e.resource_type ? `${e.resource_type}${e.resource_id ? ` (${e.resource_id})` : ''}` : '—'}</td>
<td><span className={`${styles.pill} ${e.result === 'success' ? styles.pillOk : styles.pillErr}`}>{e.result}</span></td>
</tr>
))}
</tbody>
</table>
{!result ? <p className={styles.muted}>Loading…</p> : (
<>
<table>
<thead><tr><th>When</th><th>Actor</th><th>Action</th><th>Resource</th><th>Result</th></tr></thead>
<tbody>
{events.map((e) => (
<tr key={e.audit_event_id}>
<td>{new Date(e.created_at).toLocaleString()}</td>
<td>{e.actor_type}{e.actor_id ? ` · ${e.actor_id.slice(0, 8)}` : ''}</td>
<td>{e.action}</td>
<td>{e.resource_type ? `${e.resource_type}${e.resource_id ? ` (${e.resource_id})` : ''}` : '—'}</td>
<td><span className={`${styles.pill} ${e.result === 'success' ? styles.pillOk : styles.pillErr}`}>{e.result}</span></td>
</tr>
))}
</tbody>
</table>
<div className={styles.pager}>
<button disabled={page <= 1} onClick={() => setPage((p) => p - 1)}>Previous</button>
<span className={styles.muted}>Page {result.page} of {result.total_pages}</span>
<button disabled={page >= result.total_pages} onClick={() => setPage((p) => p + 1)}>Next</button>
</div>
</>
)}
</div>
)
+4 -1
View File
@@ -49,7 +49,10 @@ function StudentDetail({ studentId, onBack }) {
const reload = useCallback(() => {
adminApi.studentProgress(studentId).then(setProgress).catch(() => {})
adminApi.studentAttempts(studentId).then(setAttempts).catch(() => {})
// Phase 4.1 §16 — studentAttempts is now paginated; this panel only ever shows the most
// recent page (labeled "Recent attempts," not a full history browser — see
// /api/admin/students/:id/attempts's own pagination for the underlying capability).
adminApi.studentAttempts(studentId).then((r) => setAttempts(r.items)).catch(() => {})
adminApi.studentEnvironment(studentId).then((rows) => setEnv(rows?.find((e) => e.status !== 'destroyed') || null)).catch(() => {})
}, [studentId])
useEffect(reload, [reload])
+4 -3
View File
@@ -52,7 +52,7 @@ export const adminApi = {
createStudent: (display_name, cohort_id) => post('/api/admin/students', { display_name, cohort_id }),
updateStudent: (id, fields) => patch(`/api/admin/students/${id}`, fields),
studentProgress: (id) => get(`/api/admin/students/${id}/progress`),
studentAttempts: (id) => get(`/api/admin/students/${id}/attempts`),
studentAttempts: (id, page = 1) => get(`/api/admin/students/${id}/attempts?page=${page}`), // Phase 4.1 §16 — paginated, see pagination.js
studentEnvironment: (id) => get(`/api/admin/students/${id}/environment`),
provisionStudentLab: (id) => post(`/api/admin/students/${id}/environment/provision`),
assignStudentCohort: (id, cohort_id) => post(`/api/admin/students/${id}/cohort`, { cohort_id }),
@@ -79,19 +79,20 @@ export const adminApi = {
revokeInstructorSessions: (id) => post(`/api/admin/instructors/${id}/revoke-sessions`),
resetInstructorPassword: (id, new_password) => post(`/api/admin/instructors/${id}/reset-password`, { new_password }),
auditEvents: () => get('/api/admin/audit-events'),
auditEvents: (page = 1) => get(`/api/admin/audit-events?page=${page}`), // Phase 4.1 §16 — paginated, see pagination.js
// ── Phase 3B: learning analytics ────────────────────────────────────────────────────────
cohortLearning: (id) => get(`/api/admin/cohorts/${id}/learning`),
studentLearning: (id) => get(`/api/admin/students/${id}/learning`),
scenarioAnalytics: (cohortId) => get(`/api/admin/scenario-analytics${cohortId ? `?cohort_id=${cohortId}` : ''}`),
needsAttention: (cohortId) => get(`/api/admin/cohorts/${cohortId}/needs-attention`),
scenarioCatalog: () => get('/api/admin/scenario-catalog'), // Phase 4.1 §18 — structured assessment editor
// ── Phase 3B: assessment (exam + capstone) management ───────────────────────────────────
assessments: () => get('/api/admin/assessments'),
createAssessment: (fields) => post('/api/admin/assessments', fields),
updateAssessment: (id, fields) => patch(`/api/admin/assessments/${id}`, fields),
assignAssessment: (id, cohort_id) => post(`/api/admin/assessments/${id}/assign`, { cohort_id }),
assessmentSessions: (id) => get(`/api/admin/assessments/${id}/sessions`),
assessmentSessions: (id, page = 1) => get(`/api/admin/assessments/${id}/sessions?page=${page}`), // Phase 4.1 §16 — paginated, see pagination.js
cancelAssessmentSession: (sessionId) => post(`/api/admin/assessment-sessions/${sessionId}/cancel`),
}
+20 -15
View File
@@ -4,26 +4,31 @@
# Docker TLS material — CA/server/client certs and keys, generated by
# infra/scripts/docker-tls/generate-ca.sh and friends. Templates/READMEs stay tracked.
infra/docker/tls/*.pem
infra/docker/tls/*.key
infra/docker/tls/*.crt
infra/docker/tls/*.srl
!infra/docker/tls/**/README.md
docker/tls/*.pem
docker/tls/*.key
docker/tls/*.crt
docker/tls/*.srl
!docker/tls/**/README.md
# Ansible Vault password files and any decrypted vars
infra/ansible/.vault_pass
infra/ansible/**/vault_plain.yml
ansible/.vault_pass
ansible/**/vault_plain.yml
# Terraform/OpenTofu state and generated vars (may contain resource IDs/IPs)
infra/terraform/**/*.tfstate
infra/terraform/**/*.tfstate.*
infra/terraform/**/.terraform/
infra/terraform/**/terraform.tfvars
!infra/terraform/**/terraform.tfvars.example
# Phase 4.1: these patterns were previously written repo-root-relative (e.g.
# "infra/terraform/...") inside THIS file, which git resolves relative to the .gitignore's own
# directory (infra/) — so they silently matched nothing at all. Found via `git status` showing
# a real .terraform/ provider cache as untracked when it should have been ignored. Fixed by
# dropping the redundant leading "infra/" from every pattern below.
terraform/**/*.tfstate
terraform/**/*.tfstate.*
terraform/**/.terraform/
terraform/**/terraform.tfvars
!terraform/**/terraform.tfvars.example
# Backup output and restore scratch space
infra/backup/output/
infra/backup/restore-test/
backup/output/
backup/restore-test/
# Load test results (local artifacts, not source)
infra/scripts/loadtest/results/
scripts/loadtest/results/
+1 -1
View File
@@ -13,7 +13,7 @@ lab_host_ip: __LAB_HOST_VM_IP__
management_cidr: __YOUR_ADMIN_ACCESS_CIDR__ # e.g. a VPN/WireGuard subnet, never 0.0.0.0/0
proxmox_mgmt_cidr: __PROXMOX_HOST_MGMT_CIDR__
lab_subnet_range: "10.42.0.0/16" # matches NO_PROXY_CIDRS already used by the app
lab_gateway_port_range: "30000:32767" # see docs/PRODUCTION_NETWORK_SECURITY.md — Gateway-to-Lab reachability follow-up
lab_router_port: "8443" # Gateway -> Lab Host Router (data plane) — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3
# ── Storage (Phase 4 §4 — a decision framework, not a final layout; see docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §4)
rootnode_data_root: /srv/rootnode-data # expected to be a ZFS dataset mountpoint on the real host
@@ -4,3 +4,19 @@
name: docker
state: restarted
daemon_reload: true
# Phase 4.1 — these restart the whole per-host compose wrapper unit (no per-container systemd
# units exist in this project's convention, see the `platform`/`lab-host` roles), which is
# heavier than a surgical single-container restart but consistent with how every other
# config-changed-so-restart-the-stack case is already handled here.
- name: restart lab-router
ansible.builtin.systemd:
name: rootnode-lab-host.service
state: restarted
when: inventory_hostname in groups['lab_host']
- name: restart gateway
ansible.builtin.systemd:
name: rootnode-platform.service
state: restarted
when: inventory_hostname in groups['platform']
@@ -87,3 +87,50 @@
mode: "0644"
when: inventory_hostname in groups['lab_host']
notify: restart docker with tls
# ── Phase 4.1: the DATA-plane mTLS pair (Gateway <-> Lab Host Router). Same CA as the
# CONTROL-plane pair above, separate directory (/etc/rootnode/lab-router-tls) so the two
# concerns stay bind-mount-separable — the `lab-router`/`gateway` containers only ever need
# their own pair mounted in, never the Docker TLS one. See
# docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3.
- name: Create /etc/rootnode/lab-router-tls
ansible.builtin.file:
path: /etc/rootnode/lab-router-tls
state: directory
owner: root
group: root
mode: "0700"
- name: Deploy CA cert for the data-plane pair (public, both hosts)
ansible.builtin.copy:
src: "{{ docker_tls_local_bundle_dir }}/ca.pem"
dest: /etc/rootnode/lab-router-tls/ca.pem
owner: root
group: root
mode: "0444"
- name: Deploy Lab Host Router server cert/key (rn-lab-host-01 only)
ansible.builtin.copy:
src: "{{ docker_tls_local_bundle_dir }}/{{ item.src }}"
dest: "/etc/rootnode/lab-router-tls/{{ item.dest }}"
owner: root
group: root
mode: "{{ item.mode }}"
loop:
- { src: router-server-cert.pem, dest: router-server-cert.pem, mode: "0444" }
- { src: router-server-key.pem, dest: router-server-key.pem, mode: "0400" }
when: inventory_hostname in groups['lab_host']
notify: restart lab-router
- name: Deploy Gateway client cert/key (rn-platform-01 only)
ansible.builtin.copy:
src: "{{ docker_tls_local_bundle_dir }}/{{ item.src }}"
dest: "/etc/rootnode/lab-router-tls/{{ item.dest }}"
owner: root
group: root
mode: "{{ item.mode }}"
loop:
- { src: gateway-client-cert.pem, dest: gateway-client-cert.pem, mode: "0444" }
- { src: gateway-client-key.pem, dest: gateway-client-key.pem, mode: "0400" }
when: inventory_hostname in groups['platform']
notify: restart gateway
@@ -3,3 +3,11 @@ PLATFORM_API_URL=http://api:4300
PROVISIONER_URL=http://provisioner:4100
PROVISIONER_SHARED_SECRET={{ vault_provisioner_shared_secret }}
GATEWAY_PORT=4200
# Phase 4.1 — mTLS client identity for the Lab Host Router (data plane). Only used when a
# lab_host's routing_mode is 'router' (see platform/gateway/src/lab-endpoint.js); harmless if
# every configured lab_host is still 'direct'. Paths are inside the container — mounted from
# /etc/rootnode/lab-router-tls on the host by the `docker-tls` role, same pattern as the
# Provisioner's Docker TLS client cert.
LAB_ROUTER_TLS_CA=/tls/ca.pem
LAB_ROUTER_TLS_CERT=/tls/gateway-client-cert.pem
LAB_ROUTER_TLS_KEY=/tls/gateway-client-key.pem
@@ -16,3 +16,9 @@ INSTRUCTOR_SESSION_IDLE_HOURS=2
COOKIE_SECURE=1
TRUST_PROXY=1
# Phase 4.1 §17 — instructor "needs attention" thresholds. Commented out (code defaults apply:
# 3 failures, 7 stale days) — uncomment and override in group_vars/{{ env }}.yml only if a
# cohort's real usage shows the defaults are wrong for it.
# NEEDS_ATTENTION_FAIL_THRESHOLD=3
# NEEDS_ATTENTION_STALE_DAYS=7
+94
View File
@@ -0,0 +1,94 @@
#!/usr/bin/env node
// Seeds realistic academy data for the Phase 4.1 §20 destructive restore drill — an owner, an
// admin instructor, a cohort, 3 students, progress, attempts, an assessment definition +
// session, and an achievement. Uses the REAL Platform API over HTTP for everything that
// doesn't need a live Provisioner/k3s Lab (owner login, instructor/cohort/student/assessment
// creation — genuinely exercises the real RBAC/creation code paths), and direct SQL inserts
// only for progress/attempts/achievements (those need real scenario activity in the full app;
// re-running the whole learning engine here would just re-prove what e2e-phase3b.js already
// proves — this drill is about DATA DURABILITY, not re-testing scenario validation).
//
// Usage: PLATFORM_API_URL=http://localhost:PORT PLATFORM_DB_URL=postgres://... node seed-realistic-data.js
const { Pool } = require('../../platform/api/node_modules/pg');
const API = process.env.PLATFORM_API_URL || 'http://localhost:4390';
const DB_URL = process.env.PLATFORM_DB_URL;
if (!DB_URL) { console.error('Set PLATFORM_DB_URL'); process.exit(1); }
function makeSession() {
const jar = new Map();
function cookieHeader() { return [...jar].map(([k, v]) => `${k}=${v}`).join('; '); }
function absorb(res) {
const cookies = typeof res.headers.getSetCookie === 'function' ? res.headers.getSetCookie() : [];
for (const c of cookies) {
const [pair] = c.split(';');
const eq = pair.indexOf('=');
if (eq === -1) continue;
jar.set(pair.slice(0, eq).trim(), pair.slice(eq + 1).trim());
}
}
return {
async req(method, path, body) {
const headers = { 'Content-Type': 'application/json' };
if (cookieHeader()) headers.Cookie = cookieHeader();
if (!['GET', 'HEAD'].includes(method)) headers['X-CSRF-Token'] = jar.get('rn_csrf_i') || '';
const res = await fetch(`${API}${path}`, { method, headers, body: body ? JSON.stringify(body) : undefined });
absorb(res);
const text = await res.text();
let json; try { json = JSON.parse(text); } catch { json = text; }
if (!res.ok) throw new Error(`${method} ${path} -> ${res.status}: ${JSON.stringify(json)}`);
return json;
},
};
}
async function main() {
const owner = makeSession();
await owner.req('POST', '/api/auth/instructor/login', { email: '[email protected]', password: 'DrillOwnerPass!2026' });
console.log(' owner login ok');
const admin = await owner.req('POST', '/api/admin/instructors', {
email: '[email protected]', display_name: 'Drill Admin', password: 'DrillAdminPass!2026', role: 'admin',
});
console.log(' admin instructor created:', admin.instructor_id);
const cohort = await owner.req('POST', '/api/admin/cohorts', { cohort_id: 'drill-cohort-01', name: 'Restore Drill Cohort' });
console.log(' cohort created');
const students = [];
for (const name of ['Drill Alice', 'Drill Bob', 'Drill Carlos']) {
const s = await owner.req('POST', '/api/admin/students', { display_name: name, cohort_id: 'drill-cohort-01' });
students.push(s.student_id);
}
console.log(' 3 students rostered and enrolled:', students.join(', '));
const assessment = await owner.req('POST', '/api/admin/assessments', {
id: 'drill-exam-01', type: 'exam', title: 'Restore Drill Exam', duration_minutes: 30, hint_policy: 'disabled',
pool_spec: { required: ['pod-basics-mcq'], random: [] },
});
console.log(' assessment definition created:', assessment.assessment_definition_id || 'drill-exam-01');
// Direct SQL for progress/attempts/achievements — see file header for why.
const pool = new Pool({ connectionString: DB_URL });
for (const studentId of students) {
// environment_id is a real FK to the Provisioner's environments table (nullable) — left
// NULL here rather than faking a row that doesn't exist, same as how a genuinely old
// attempt whose environment has long since been cleaned up would look in real data.
await pool.query(
`INSERT INTO scenario_attempts (student_id, scenario_id, status, result, started_at, completed_at, validation_attempt_count, validation_pass_count)
VALUES ($1, 'pod-basics-mcq', 'completed', 'passed', now() - interval '1 hour', now(), 1, 1)`,
[studentId]
);
await pool.query(
`INSERT INTO achievements (student_id, achievement_type, stage_id, achieved_at) VALUES ($1, 'stage_completed', 'foundation', now())`,
[studentId]
);
}
await pool.end();
console.log(' progress/attempts/achievements seeded for all 3 students');
console.log('\nSeed complete.');
}
main().catch((e) => { console.error('SEED FAILED:', e.message); process.exit(1); });
+149
View File
@@ -0,0 +1,149 @@
#!/usr/bin/env bash
# Phase 4.1 §20 — the DESTRUCTIVE disaster-recovery drill, one level up from
# test-backup-restore.sh's mechanism-only test: seeds realistic academy data through the REAL
# Platform API (owner, admin instructor, cohort, 3 students, progress, attempts, an assessment
# definition, achievements), backs it up, DESTROYS the disposable Postgres storage entirely
# (docker rm — not just "restore into a second container," the actual source is gone),
# recreates Postgres from nothing, restores, and boots the REAL Platform API again against the
# restored database to prove the whole stack — not just the SQL — comes back.
#
# Never touches the developer's real persistent database — every Postgres here is a disposable
# container on a throwaway port, created and destroyed entirely within this script.
set -euo pipefail
cd "$(dirname "$0")/../.." # repo root
CONTAINER=rootnode-drill-postgres
PORT=55434
API_PORT=4390
TESTDIR="$(mktemp -d)"
# curriculum.js defaults to platform/api/scenarios/{data,bundles} — only present inside the
# Docker image build (COPY scenarios/ ./scenarios/). Running src/server.js directly from
# source (as this drill does, to avoid needing a full container build) needs these pointed at
# the repo's real scenario content instead.
export SCENARIOS_DIR="$PWD/scenarios/data"
export BUNDLES_DIR="$PWD/scenarios/bundles"
export CURRICULUM_PATH_FILE="$PWD/scenarios/rootnode-path.json"
export BACKUP_ENCRYPTION_PASSPHRASE="drill-passphrase-not-for-production"
export PROVISIONER_SHARED_SECRET="drill-shared-secret-not-for-production"
API_PID=""
cleanup() {
[ -n "$API_PID" ] && kill "$API_PID" 2>/dev/null || true
docker rm -f "$CONTAINER" >/dev/null 2>&1 || true
rm -rf "$TESTDIR"
}
trap cleanup EXIT
if ! command -v psql >/dev/null 2>&1 || ! command -v pg_dump >/dev/null 2>&1; then
echo "==> psql/pg_dump not found locally — reusing the same docker-exec shim as test-backup-restore.sh"
SHIM_BIN="$TESTDIR/bin"; mkdir -p "$SHIM_BIN"
# Same shim as test-backup-restore.sh (see that file's comment for the full "why") — the URL
# is DROPPED, not kept, since the host-side port it names doesn't exist inside the
# container's own network namespace; -tAc counts as a self-contained call (needs no stdin)
# the same way -c does, or the admin-verification queries below would hang waiting on stdin.
cat > "$SHIM_BIN/psql" <<'EOF'
#!/usr/bin/env bash
args=(); skip_next=0
for a in "$@"; do
if [ "$skip_next" = "1" ]; then skip_next=0; continue; fi
if [ "$a" = "-p" ] || [ "$a" = "-h" ]; then skip_next=1; continue; fi
if [[ "$a" =~ ^postgres://[^:]+:[^@]+@[^:]+:([0-9]+)/ ]]; then continue; fi # drop the URL — reconnect locally inside the container instead
args+=("$a")
done
container="rootnode-drill-postgres"
needs_stdin=1
for a in "${args[@]}"; do { [ "$a" = "-c" ] || [ "$a" = "-tAc" ]; } && needs_stdin=0; done
if [ "$needs_stdin" = "1" ]; then
exec docker exec -e PGPASSWORD=postgres -i "$container" psql -h localhost -U postgres "${args[@]}"
else
exec docker exec -e PGPASSWORD=postgres "$container" psql -h localhost -U postgres "${args[@]}"
fi
EOF
cat > "$SHIM_BIN/pg_dump" <<'EOF'
#!/usr/bin/env bash
args=()
for a in "$@"; do
if [[ "$a" =~ ^postgres://[^:]+:[^@]+@[^:]+:([0-9]+)/(.*)$ ]]; then args+=("postgres://postgres:postgres@localhost:5432/${BASH_REMATCH[2]}"); continue; fi
args+=("$a")
done
exec docker exec -e PGPASSWORD=postgres rootnode-drill-postgres pg_dump "${args[@]}"
EOF
chmod +x "$SHIM_BIN/psql" "$SHIM_BIN/pg_dump"
export PATH="$SHIM_BIN:$PATH"
fi
DB_URL="postgres://postgres:postgres@localhost:$PORT/postgres"
echo "==> [1/9] Starting disposable Postgres"
docker run -d --name "$CONTAINER" -e POSTGRES_PASSWORD=postgres -p "$PORT:5432" postgres:16-alpine >/dev/null
for i in $(seq 1 30); do PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -c "SELECT 1" >/dev/null 2>&1 && break; sleep 1; done
echo "==> [2/9] Applying real Provisioner + Platform API migrations"
( cd platform/provisioner && [ -d node_modules ] || npm install --no-audit --no-fund >/dev/null
PLATFORM_DB_URL="$DB_URL" node -e "require('./src/db').runMigrations().then((m)=>console.log(' Provisioner:', m.join(', ')))" )
( cd platform/api && [ -d node_modules ] || npm install --no-audit --no-fund >/dev/null
PLATFORM_DB_URL="$DB_URL" node -e "require('./src/db').runMigrations().then((m)=>console.log(' Platform API:', m.join(', ')))" )
echo "==> [3/9] Bootstrapping the drill owner and starting a REAL Platform API"
( cd platform/api && PLATFORM_DB_URL="$DB_URL" node bootstrap-owner.js [email protected] "Drill Owner" 'DrillOwnerPass!2026' )
( cd platform/api && PLATFORM_DB_URL="$DB_URL" PROVISIONER_URL="http://localhost:1" PLATFORM_API_PORT="$API_PORT" node src/server.js > "$TESTDIR/api.log" 2>&1 & echo $! > "$TESTDIR/api.pid" )
API_PID=$(cat "$TESTDIR/api.pid")
for i in $(seq 1 30); do curl -sf "http://localhost:$API_PORT/healthz" >/dev/null 2>&1 && break; sleep 1; done
curl -sf "http://localhost:$API_PORT/healthz" >/dev/null || { echo "FAIL: Platform API never came up"; cat "$TESTDIR/api.log"; exit 1; }
echo " Platform API up on :$API_PORT"
echo "==> [4/9] Seeding realistic academy data through the real API (owner/admin/cohort/3 students/progress/attempts/assessment/achievements)"
( cd infra/backup && PLATFORM_API_URL="http://localhost:$API_PORT" PLATFORM_DB_URL="$DB_URL" node seed-realistic-data.js )
echo "==> [5/9] Backing up (real backup-postgres.sh)"
BACKUP_DIR="$TESTDIR/backups" PLATFORM_DB_URL="$DB_URL" ./infra/backup/backup-postgres.sh
ENC_FILE=$(ls "$TESTDIR"/backups/*.sql.gz.gpg | head -1)
[ -f "$ENC_FILE" ] || { echo "FAIL: no encrypted backup produced"; exit 1; }
echo "==> [6/9] DESTROYING the Postgres storage entirely (this is the destructive part — not restoring into a second container, the original is gone)"
kill "$API_PID" 2>/dev/null || true; API_PID=""
docker rm -f "$CONTAINER" >/dev/null
echo " source container + its storage are gone"
echo "==> [7/9] Recreating Postgres from nothing"
docker run -d --name "$CONTAINER" -e POSTGRES_PASSWORD=postgres -p "$PORT:5432" postgres:16-alpine >/dev/null
for i in $(seq 1 30); do PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -c "SELECT 1" >/dev/null 2>&1 && break; sleep 1; done
echo "==> [8/9] Restoring from the encrypted backup (schema + data both come from the dump — no separate migration step)"
echo "RESTORE" | BACKUP_ENCRYPTION_PASSPHRASE="$BACKUP_ENCRYPTION_PASSPHRASE" ./infra/backup/restore-postgres.sh "$ENC_FILE" "$DB_URL"
./infra/backup/verify-restore.sh "$DB_URL"
echo "==> [9/9] Booting the REAL Platform API again against the RESTORED database"
( cd platform/api && PLATFORM_DB_URL="$DB_URL" PROVISIONER_URL="http://localhost:1" PLATFORM_API_PORT="$API_PORT" node src/server.js > "$TESTDIR/api2.log" 2>&1 & echo $! > "$TESTDIR/api2.pid" )
API_PID=$(cat "$TESTDIR/api2.pid")
for i in $(seq 1 30); do curl -sf "http://localhost:$API_PORT/healthz" >/dev/null 2>&1 && break; sleep 1; done
curl -sf "http://localhost:$API_PORT/healthz" >/dev/null || { echo "FAIL: Platform API did not come back up against the restored DB"; cat "$TESTDIR/api2.log"; exit 1; }
echo " Platform API is back up, against the RESTORED database — no fresh migration run needed"
echo "==> Verifying accounts/RBAC/cohorts/students/progress/attempts/achievements survived"
FAIL=0
check() { local desc="$1" want="$2" got="$3"; if [ "$want" = "$got" ]; then echo " ok — $desc"; else echo " FAIL — $desc (want $want, got $got)"; FAIL=1; fi; }
check "owner account (role) restored" "owner" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT role FROM instructors WHERE email='[email protected]'" | tr -d '[:space:]')"
check "admin instructor restored" "admin" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT role FROM instructors WHERE email='[email protected]'" | tr -d '[:space:]')"
check "cohort restored" "1" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM cohorts WHERE cohort_id='drill-cohort-01'" | tr -d '[:space:]')"
check "3 students restored" "3" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM cohort_enrollments WHERE cohort_id='drill-cohort-01'" | tr -d '[:space:]')"
check "3 completed attempts restored" "3" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM scenario_attempts WHERE result='passed' AND scenario_id='pod-basics-mcq'" | tr -d '[:space:]')"
check "3 achievements restored" "3" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM achievements WHERE stage_id='foundation'" | tr -d '[:space:]')"
check "assessment definition restored" "1" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM assessment_definitions WHERE assessment_definition_id='drill-exam-01'" | tr -d '[:space:]')"
echo "==> Verifying RBAC still works against the restored DB (real login through the real API)"
LOGIN_STATUS=$(curl -s -o /dev/null -w '%{http_code}' -X POST "http://localhost:$API_PORT/api/auth/instructor/login" \
-H 'Content-Type: application/json' -d '{"email":"[email protected]","password":"DrillAdminPass!2026"}')
check "restored admin instructor can log in" "200" "$LOGIN_STATUS"
if [ "$FAIL" = "1" ]; then
echo; echo "DRILL FAILED — see above."; exit 1
fi
echo
echo "OK: full destructive restore drill passed — owner/admin/cohort/3 students/progress/attempts/"
echo "assessment/achievements all survived a real Postgres storage loss + recreation + restore, and"
echo "the real Platform API booted and authenticated against the restored database with no fresh"
echo "migration run required."
echo "NOT TESTED HERE (needs real credentials/hardware): the actual off-host Storage Box/S3 copy."
+54 -3
View File
@@ -40,16 +40,19 @@ jobs:
- name: Install dependencies (all first-party services)
run: |
for d in backend frontend platform/api platform/gateway platform/provisioner; do
for d in backend frontend platform/api platform/gateway platform/provisioner platform/lab-router; do
(cd "$d" && npm ci)
done
- name: Syntax-check every service
run: |
for d in backend frontend/src platform/api platform/gateway platform/provisioner; do
for d in backend frontend/src platform/api platform/gateway platform/provisioner platform/lab-router; do
find "$d" -name '*.js' -not -path '*/node_modules/*' -exec node --check {} \;
done
- name: Lab Host Router unit tests (Phase 4.1 — fast, no Docker daemon needed)
run: (cd platform/lab-router && npm test)
- name: Build frontend
run: (cd frontend && npm run build)
@@ -75,8 +78,56 @@ jobs:
# can't grant safely — they stay a LOCAL/staging pre-promotion step
# (infra/scripts/staging-smoke-test.sh), not this job. See docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §18.
# Phase 4.1 §24 — free/open-source security scanning, no paid tooling, no signup required for
# any of these. Severity policy (docs/SECURITY_SCANNING.md): CRITICAL/HIGH findings fail the
# build; MEDIUM/LOW are reported but never block a merge — a build that's permanently red over
# an unrelated low-severity transitive dependency finding trains everyone to ignore red builds.
security-scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"
- name: npm audit (dependency vulnerabilities) — fails only on high/critical
run: |
FAIL=0
for d in backend frontend platform/api platform/gateway platform/provisioner platform/lab-router; do
echo "== $d =="
(cd "$d" && npm audit --audit-level=high) || FAIL=1
done
exit $FAIL
- name: Secret scanning (gitleaks — free, no account needed)
run: docker run --rm -v "$PWD:/repo" zricethezav/gitleaks:latest detect --source=/repo --no-git -v
- name: Dockerfile linting (hadolint — free)
run: |
FAIL=0
for f in Dockerfile platform/api/Dockerfile platform/gateway/Dockerfile platform/provisioner/Dockerfile platform/egress-proxy/Dockerfile platform/lab-router/Dockerfile; do
echo "== $f =="
docker run --rm -i hadolint/hadolint < "$f" || FAIL=1
done
exit $FAIL
# -S error, not warning — matches docs/SECURITY_SCANNING.md's severity policy. Real run
# against this tree at -S warning surfaces only style nits (unused loop vars, a missing
# `cd ... || exit` guard) on scripts that are otherwise correct; failing the build on
# those trains everyone to ignore the check.
- name: Shell linting (shellcheck — free, standard)
run: |
sudo apt-get update -qq && sudo apt-get install -y -qq shellcheck
find infra -name '*.sh' -exec shellcheck -S error {} \+
- name: Container image scanning (trivy — free, no account needed)
run: |
docker build -q -t rootnode-scan/platform-api -f platform/api/Dockerfile .
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock aquasec/trivy:latest image \
--severity HIGH,CRITICAL --exit-code 1 rootnode-scan/platform-api
build-images:
needs: regression
needs: [regression, security-scan]
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
steps:
+1
View File
@@ -8,6 +8,7 @@
| `rootnode/gateway` | `platform/gateway/Dockerfile` | `{{ registry }}/rootnode/gateway` |
| `rootnode/provisioner` | `platform/provisioner/Dockerfile` | `{{ registry }}/rootnode/provisioner` |
| `rootnode/egress-proxy` | `platform/egress-proxy/Dockerfile` | `{{ registry }}/rootnode/egress-proxy` |
| `rootnode/lab-router` | `platform/lab-router/Dockerfile` (Phase 4.1) | `{{ registry }}/rootnode/lab-router` |
| `rootnode/lab` | root `Dockerfile` (the student Lab image) | `{{ registry }}/rootnode/lab` |
## Tagging
@@ -1,10 +1,13 @@
# Production compose for rn-lab-host-01. Deployed by infra/ansible/roles/lab-host/ to
# /opt/rootnode/lab-host/docker-compose.yml, run as a systemd unit
# (rootnode-lab-host.service.j2). Runs exactly one long-lived service: the shared egress proxy
# every student Lab's outbound traffic passes through (platform/egress-proxy/, unchanged since
# Phase 1). Student Lab containers themselves are NOT part of this compose file — they're
# created/destroyed individually by the Provisioner over the remote Docker TLS endpoint
# (dockerd itself, not compose), same mechanism as dev, just remote.
# (rootnode-lab-host.service.j2). Runs the shared egress proxy every student Lab's outbound
# traffic passes through (platform/egress-proxy/, unchanged since Phase 1) and, as of Phase
# 4.1, the Lab Host Router (platform/lab-router/) that gives the Gateway on rn-platform-01 a
# way to reach a running Lab without ever touching a Docker-internal address — see
# docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3. Student Lab containers themselves are NOT
# part of this compose file — they're created/destroyed individually by the Provisioner over
# the remote Docker TLS endpoint (dockerd itself, not compose), same mechanism as dev, just
# remote; the Router finds them by label, purely from local Docker state (see its own README).
networks:
rn-egress:
@@ -22,6 +25,26 @@ services:
# is the actual control, see platform/egress-proxy/allowlist.txt)
networks: [rn-egress]
lab-router:
image: "{{ registry }}/rootnode/lab-router:{{ image_tag }}"
restart: unless-stopped
# Needs the host's own routing table to reach whichever per-Lab bridge network
# (rootnode-lab-net-<id>) the Provisioner created for a given container — those networks
# aren't known ahead of time and aren't attachable one-by-one, the same reason egress-proxy
# would need this too if it initiated connections outward instead of just accepting them.
# Restricted to platform_ip on port {{ lab_router_port | default('8443') }} at the host
# firewall (infra/firewall/rn-lab-host-01.rules) — this is the ONLY thing that makes host
# networking here safe, same defense-in-depth pattern as the Docker TLS port.
network_mode: host
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /etc/rootnode/lab-router-tls:/tls:ro
environment:
LAB_ROUTER_PORT: "{{ lab_router_port | default('8443') }}"
LAB_ROUTER_TLS_CA: /tls/ca.pem
LAB_ROUTER_TLS_CERT: /tls/router-server-cert.pem
LAB_ROUTER_TLS_KEY: /tls/router-server-key.pem
node-exporter:
image: prom/node-exporter:latest
restart: unless-stopped
@@ -88,6 +88,11 @@ services:
image: "{{ registry }}/rootnode/platform-api:{{ image_tag }}"
restart: unless-stopped
env_file: /etc/rootnode/platform.env
environment:
# Phase 4.1 §25 — not a secret, just the deployed tag; set directly here rather than in
# the env template since it's the same value already used for the image tag above, not a
# separately-managed setting.
GIT_SHA: "{{ image_tag }}"
depends_on:
postgres:
condition: service_healthy
@@ -99,6 +104,10 @@ services:
image: "{{ registry }}/rootnode/gateway:{{ image_tag }}"
restart: unless-stopped
env_file: /etc/rootnode/gateway.env
volumes:
# mTLS client identity for the Lab Host Router (data plane) — see gateway.env.j2's
# LAB_ROUTER_TLS_* vars. Read-only; the private key never leaves this one bind mount.
- /etc/rootnode/lab-router-tls:/tls:ro
depends_on: [provisioner, api]
networks: [rn-platform-internal]
@@ -1,7 +1,12 @@
# Production lab_hosts seed — points at the real rn-lab-host-01 VM instead of Phase 1's dev
# Unix socket. Same mechanism the Provisioner has had since Phase 1
# (platform/provisioner/config/lab-hosts.seed.yaml), just pointed at a real remote endpoint —
# no application code changes required for this, see docs/PRODUCTION_NETWORK_SECURITY.md §7.
# (platform/provisioner/config/lab-hosts.seed.yaml), just pointed at real remote endpoints.
#
# Phase 4.1 correction: routing_mode/router_endpoint is REQUIRED here — Phase 4's original
# claim that "no application code changes required" was true for the Provisioner's CONTROL
# plane (Docker mTLS) but false for the Gateway's DATA plane, which used to assume
# env.internal_ip was directly reachable. It is not, across two real VMs. See
# docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1 and docs/PRODUCTION_NETWORK_SECURITY.md.
#
# {{ lab_host_ip }} is an Ansible template variable — see infra/ansible/group_vars/production.yml.
# Rendered by the `platform` role; the raw file here keeps the placeholder for readability.
@@ -9,8 +14,14 @@
lab_hosts:
- lab_host_id: labhost-01
display_name: "RootNode Lab Host VM 01 (production)"
# CONTROL plane: Provisioner -> dockerd, mTLS, port 2376.
endpoint: "tcp://{{ lab_host_ip }}:2376"
tls_ca_path: /etc/rootnode/docker-tls/ca.pem
tls_cert_path: /etc/rootnode/docker-tls/provisioner-client-cert.pem
tls_key_path: /etc/rootnode/docker-tls/provisioner-client-key.pem
enabled: true
# DATA plane: Gateway -> Lab Host Router, mTLS, router's own port (8443 by default —
# infra/docker/lab-host/docker-compose.production.yml). Gateway never touches a
# Docker-internal address once routing_mode is 'router'.
routing_mode: router
router_endpoint: "https://{{ lab_host_ip }}:8443"
@@ -1,16 +1,17 @@
# Production lab_hosts seed — points at the real rn-lab-host-01 VM instead of Phase 1's dev
# Unix socket. Same mechanism the Provisioner has had since Phase 1
# (platform/provisioner/config/lab-hosts.seed.yaml), just pointed at a real remote endpoint —
# no application code changes required for this, see docs/PRODUCTION_NETWORK_SECURITY.md §7.
#
# {{ lab_host_ip }} is an Ansible template variable — see infra/ansible/group_vars/production.yml.
# Rendered by the `platform` role; the raw file here keeps the placeholder for readability.
# Staging lab_hosts seed — see infra/environments/production/provisioner-config/lab-hosts.yaml
# for the full explanation; this is the same shape, lower stakes. Staging's `lab_host_ip` is
# distinct from production's own (infra/ansible/group_vars/staging.yml), so it needs the same
# router routing_mode as production once staging genuinely runs on two hosts. If staging is
# ever collapsed onto a single host instead, change routing_mode to `direct` and drop
# router_endpoint — that's the only edit needed either way.
lab_hosts:
- lab_host_id: labhost-01
display_name: "RootNode Lab Host VM 01 (production)"
display_name: "RootNode Lab Host VM 01 (staging)"
endpoint: "tcp://{{ lab_host_ip }}:2376"
tls_ca_path: /etc/rootnode/docker-tls/ca.pem
tls_cert_path: /etc/rootnode/docker-tls/provisioner-client-cert.pem
tls_key_path: /etc/rootnode/docker-tls/provisioner-client-key.pem
enabled: true
routing_mode: router
router_endpoint: "https://{{ lab_host_ip }}:8443"
+8 -7
View File
@@ -38,13 +38,14 @@
# endpoint: ALLOW" example in the Phase 4 spec, encoded literally here.
-A INPUT -p tcp --dport 2376 -s {{ platform_ip }} -m conntrack --ctstate NEW -j ACCEPT
# Gateway -> per-Lab published port range, rn-platform-01 only. REQUIRED so the Gateway can
# reach each student's Lab UI/terminal over the network once Labs run on a genuinely separate
# host — see docs/PRODUCTION_NETWORK_SECURITY.md §"Gateway-to-Lab cross-VM reachability" for
# why this exists and the small Provisioner/Gateway code change it depends on (NOT YET BUILT —
# this firewall rule is the network-policy half; the port-publishing half is an explicit,
# reviewed application-code follow-up, tracked there rather than silently assumed here).
-A INPUT -p tcp --dport {{ lab_gateway_port_range | default('30000:32767') }} -s {{ platform_ip }} -m conntrack --ctstate NEW -j ACCEPT
# Lab Host Router (8443) — rn-platform-01 (the Gateway) only. This is the DATA-plane
# equivalent of the Docker TLS rule above: the Gateway never reaches a Lab's Docker-internal
# address directly (no per-Lab port publishing, no wide port range needed) — it reaches this
# one mTLS-authenticated Router instead, which resolves the request to a local container on
# ITS OWN host. See docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3 and
# docs/PRODUCTION_NETWORK_SECURITY.md — this closes the "Gateway-to-Lab cross-VM reachability"
# gap Phase 4 flagged as a required follow-up; it is no longer open.
-A INPUT -p tcp --dport {{ lab_router_port | default('8443') }} -s {{ platform_ip }} -m conntrack --ctstate NEW -j ACCEPT
# Prometheus (rn-platform-01) scraping this host's node-exporter/cadvisor — metrics only,
# read-only exposition endpoints, still restricted to the one legitimate scraper.
+1 -1
View File
@@ -21,7 +21,7 @@ render() {
-e 's/{{ lab_host_ip }}/10.10.10.20/g' \
-e 's/{{ lab_subnet_range }}/10.42.0.0\/16/g' \
-e 's/{{ proxmox_mgmt_cidr }}/203.0.113.0\/24/g' \
-e "s/{{ lab_gateway_port_range | default('30000:32767') }}/30000:32767/g" \
-e "s/{{ lab_router_port | default('8443') }}/8443/g" \
"$1" > "$2"
}
+1
View File
@@ -22,6 +22,7 @@ declare -A IMAGES=(
[rootnode/gateway]="platform/gateway/Dockerfile platform/gateway"
[rootnode/provisioner]="platform/provisioner/Dockerfile platform/provisioner"
[rootnode/egress-proxy]="platform/egress-proxy/Dockerfile platform/egress-proxy"
[rootnode/lab-router]="platform/lab-router/Dockerfile platform/lab-router"
[rootnode/lab]="Dockerfile ."
)
+43 -12
View File
@@ -1,14 +1,21 @@
#!/usr/bin/env bash
# Generates a private CA and a mutual-TLS cert/key pair for the Provisioner <-> rn-lab-host-01
# Docker Engine connection (docs/PRODUCTION_NETWORK_SECURITY.md §7). The application already
# supports this (platform/provisioner/src/docker.js buildClient() has required mTLS since
# Phase 1) — this script only ever needed to exist for the CERTIFICATES, not new app code.
# Generates a private CA and TWO mutual-TLS cert/key pairs, both under that one CA:
# 1. Provisioner <-> rn-lab-host-01 Docker Engine (CONTROL plane, port 2376). The application
# already supports this (platform/provisioner/src/docker.js buildClient() has required
# mTLS since Phase 1) — this script only ever needed to exist for the CERTIFICATES.
# 2. Gateway <-> rn-lab-host-01 Lab Host Router (DATA plane, port 8443) — Phase 4.1, see
# docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3 and docs/PRODUCTION_NETWORK_SECURITY.md.
# One CA for both keeps the PKI surface small (one thing to protect, one rotation runbook) —
# see docs/RUNBOOK_DOCKER_TLS_ROTATION.md, which now covers all four leaf certs, not just two.
#
# Output goes to $OUT_DIR (default ./out, gitignored) — never into a tracked path. Deploy the
# resulting files with the `docker-tls` Ansible role, which places:
# rn-lab-host-01: ca.pem, server-cert.pem, server-key.pem -> dockerd's --tlsverify config
# rn-platform-01: ca.pem, provisioner-client-cert.pem, provisioner-client-key.pem
# rn-lab-host-01: ca.pem, server-cert.pem, server-key.pem -> dockerd --tlsverify
# ca.pem, router-server-cert.pem, router-server-key.pem -> lab-router (data plane)
# rn-platform-01: ca.pem, provisioner-client-cert.pem, provisioner-client-key.pem
# -> /etc/rootnode/docker-tls/ (referenced by lab-hosts.yaml)
# ca.pem, gateway-client-cert.pem, gateway-client-key.pem
# -> /etc/rootnode/lab-router-tls/ (referenced by gateway's LAB_ROUTER_TLS_*)
#
# Usage:
# ./generate-docker-tls.sh [--out-dir DIR] [--lab-host-ip IP] [--lab-host-dns NAME] [--days N]
@@ -61,13 +68,37 @@ EOF
openssl x509 -req -days "$DAYS" -sha256 -in client.csr -CA ca.pem -CAkey ca-key.pem \
-CAcreateserial -out provisioner-client-cert.pem -extfile client-extfile.cnf
rm -f server.csr client.csr server-extfile.cnf client-extfile.cnf
chmod 0400 ca-key.pem server-key.pem provisioner-client-key.pem
chmod 0444 ca.pem server-cert.pem provisioner-client-cert.pem
echo "==> Generating server cert for rn-lab-host-01 (Lab Host Router, data plane)"
openssl genrsa -out router-server-key.pem 4096
openssl req -subj "/CN=$LAB_HOST_DNS" -sha256 -new -key router-server-key.pem -out router-server.csr
cat > router-server-extfile.cnf <<EOF
subjectAltName = DNS:$LAB_HOST_DNS,IP:$LAB_HOST_IP
extendedKeyUsage = serverAuth
EOF
openssl x509 -req -days "$DAYS" -sha256 -in router-server.csr -CA ca.pem -CAkey ca-key.pem \
-CAcreateserial -out router-server-cert.pem -extfile router-server-extfile.cnf
echo "==> Generating client cert for the Gateway (data plane)"
openssl genrsa -out gateway-client-key.pem 4096
openssl req -subj "/CN=rootnode-gateway" -new -key gateway-client-key.pem -out gateway-client.csr
openssl x509 -req -days "$DAYS" -sha256 -in gateway-client.csr -CA ca.pem -CAkey ca-key.pem \
-CAcreateserial -out gateway-client-cert.pem -extfile client-extfile.cnf
rm -f server.csr client.csr router-server.csr gateway-client.csr server-extfile.cnf client-extfile.cnf router-server-extfile.cnf
chmod 0400 ca-key.pem server-key.pem provisioner-client-key.pem router-server-key.pem gateway-client-key.pem
chmod 0444 ca.pem server-cert.pem provisioner-client-cert.pem router-server-cert.pem gateway-client-cert.pem
echo "==> Done. Generated in $OUT_DIR:"
ls -la
echo
echo "Deploy server-{cert,key}.pem + ca.pem to rn-lab-host-01 (docker-tls Ansible role)."
echo "Deploy provisioner-client-{cert,key}.pem + ca.pem to rn-platform-01 (same role)."
echo "Verify with: ../verify-docker-tls.sh $OUT_DIR <lab-host-ip>"
echo "CONTROL plane (Provisioner -> dockerd, port 2376):"
echo " Deploy server-{cert,key}.pem + ca.pem to rn-lab-host-01 (docker-tls Ansible role)."
echo " Deploy provisioner-client-{cert,key}.pem + ca.pem to rn-platform-01 (same role)."
echo "DATA plane (Gateway -> Lab Host Router, port 8443):"
echo " Deploy router-server-{cert,key}.pem + ca.pem to rn-lab-host-01 (lab-router Ansible role)."
echo " Deploy gateway-client-{cert,key}.pem + ca.pem to rn-platform-01 (platform Ansible role)."
echo
echo "Verify Docker control-plane TLS with: ../verify-docker-tls.sh $OUT_DIR <lab-host-ip>"
+31
View File
@@ -0,0 +1,31 @@
#!/usr/bin/env bash
# Orchestrates the Phase 4.1 §27 fake cohort dress rehearsal: brings up the real dev stack,
# bootstraps an owner, runs test-fake-cohort-rehearsal.js, tears down.
set -euo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")/../../../platform"
export PROVISIONER_SHARED_SECRET="${PROVISIONER_SHARED_SECRET:-rehearsal-shared-secret-not-for-production}"
docker compose -f docker-compose.dev.yml up -d --build
cleanup() {
echo "==> Tearing down"
docker ps -aq --filter "name=rootnode-lab-" | xargs -r docker rm -f >/dev/null 2>&1 || true
docker network ls --filter "name=rootnode-lab-net-" --format "{{.Name}}" | xargs -r -n1 docker network rm >/dev/null 2>&1 || true
docker compose -f docker-compose.dev.yml down >/dev/null 2>&1 || true
}
trap cleanup EXIT
echo "==> Waiting for the stack"
for _ in $(seq 1 60); do
curl -sf http://localhost:4300/healthz >/dev/null 2>&1 && curl -sf http://localhost:4200/healthz >/dev/null 2>&1 && break
sleep 2
done
echo "==> Bootstrapping the rehearsal owner"
( cd api && PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5433/rootnode_platform \
node bootstrap-owner.js [email protected] "Rehearsal Owner" 'RehearsalPass!2026' )
echo "==> Running the rehearsal"
OWNER_EMAIL=[email protected] OWNER_PASSWORD='RehearsalPass!2026' \
node ../infra/scripts/drills/test-fake-cohort-rehearsal.js
+32
View File
@@ -0,0 +1,32 @@
#!/usr/bin/env bash
# Orchestrates the Phase 4.1 §21 Lab Host loss/rebuild drill: brings up the real dev stack,
# bootstraps an owner, runs test-lab-host-loss.js, tears down. See that file's header for what
# it actually proves.
set -euo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")/../../../platform"
export PROVISIONER_SHARED_SECRET="${PROVISIONER_SHARED_SECRET:-drill-shared-secret-not-for-production}"
docker compose -f docker-compose.dev.yml up -d --build
cleanup() {
echo "==> Tearing down"
docker ps -aq --filter "name=rootnode-lab-" | xargs -r docker rm -f >/dev/null 2>&1 || true
docker network ls --filter "name=rootnode-lab-net-" --format "{{.Name}}" | xargs -r -n1 docker network rm >/dev/null 2>&1 || true
docker compose -f docker-compose.dev.yml down >/dev/null 2>&1 || true
}
trap cleanup EXIT
echo "==> Waiting for the stack"
for i in $(seq 1 60); do
curl -sf http://localhost:4300/healthz >/dev/null 2>&1 && curl -sf http://localhost:4200/healthz >/dev/null 2>&1 && break
sleep 2
done
echo "==> Bootstrapping the drill owner"
( cd api && PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5433/rootnode_platform \
node bootstrap-owner.js [email protected] "LabHost Drill Owner" 'LabHostDrillPass!2026' )
echo "==> Running the drill"
OWNER_EMAIL=[email protected] OWNER_PASSWORD='LabHostDrillPass!2026' \
node ../infra/scripts/drills/test-lab-host-loss.js
+77
View File
@@ -0,0 +1,77 @@
#!/usr/bin/env bash
# Phase 4.1 §23 — proves the CI pipeline (infra/ci/gitea-workflows/build-and-test.yml) fails
# CLOSED on every listed failure mode, i.e. the underlying command genuinely exits nonzero on
# broken input, which is what makes a Gitea/GitHub Actions step (and therefore the job, and
# therefore `needs: regression` gating build-images/deploy-staging) fail. Doesn't need a real
# CI run — checks the actual commands the workflow calls, directly, the same way Actions would
# invoke them.
set -uo pipefail # NOT -e — this script's whole point is deliberately triggering failures and checking their exit codes
cd "$(dirname "${BASH_SOURCE[0]}")/../../.." # repo root
FAIL=0
expect_nonzero() {
local desc="$1"; shift
"$@" >/tmp/ci-fail-drill-out 2>&1
local code=$?
if [ "$code" -eq 0 ]; then
echo " FAIL — $desc (command succeeded, but it should have failed)"; cat /tmp/ci-fail-drill-out; FAIL=1
else
echo " ok — $desc correctly failed (exit $code)"
fi
}
echo "Phase 4.1 CI-fails-closed drill"
echo
echo "== 1. Test/syntax-check failure =="
BAD_JS="$(mktemp).js"
echo 'function broken( {' > "$BAD_JS"
expect_nonzero "node --check on syntactically broken JS" node --check "$BAD_JS"
rm -f "$BAD_JS"
echo "== 2. Frontend build failure =="
cp frontend/src/App.jsx /tmp/App.jsx.bak
echo 'this is not valid jsx <<<' >> frontend/src/App.jsx
(cd frontend && expect_nonzero "npm run build with a broken component" npm run build)
cp /tmp/App.jsx.bak frontend/src/App.jsx; rm -f /tmp/App.jsx.bak
echo "== 3. Image build failure =="
TMPDOCKERFILE="$(mktemp -d)/Dockerfile"
cat > "$TMPDOCKERFILE" <<'EOF'
FROM node:20-slim
COPY this-file-does-not-exist.js /app/
EOF
expect_nonzero "docker build with a COPY of a nonexistent file" docker build -q -f "$TMPDOCKERFILE" "$(dirname "$TMPDOCKERFILE")"
echo "== 4. Config validation failure =="
BAD_COMPOSE="$(mktemp -d)/docker-compose.yml"
cat > "$BAD_COMPOSE" <<'EOF'
services:
a:
image: alpine
a:
image: alpine
EOF
expect_nonzero "docker compose config on a file with a duplicate service key" docker compose -f "$BAD_COMPOSE" config
echo "== 5. Migration preflight failure =="
( cd platform/api && expect_nonzero "runMigrations() against an unreachable Postgres" \
env PLATFORM_DB_URL="postgres://nobody:[email protected]:1/nope" \
node -e "require('./src/db').runMigrations().then(()=>process.exit(0)).catch((e)=>{console.error(e.message);process.exit(1)})" )
echo "== 6. Staging smoke test failure =="
expect_nonzero "staging-smoke-test.sh with no STAGING_GATEWAY_URL set" \
env -u STAGING_GATEWAY_URL bash infra/scripts/staging-smoke-test.sh
expect_nonzero "staging-smoke-test.sh against an unreachable staging URL" \
env STAGING_GATEWAY_URL="http://127.0.0.1:1" bash infra/scripts/staging-smoke-test.sh
echo
if [ "$FAIL" = "1" ]; then
echo "DRILL FAILED — at least one failure mode did NOT fail closed. See above."
exit 1
fi
echo "OK: every listed failure mode genuinely exits nonzero — CI fails closed on all of them."
echo "NOT TESTED HERE: an actual Gitea Actions run (needs Gitea Actions enabled on the real"
echo "instance and this workflow file activated at .gitea/workflows/ — see build-and-test.yml's"
echo "own header comment). This proves the underlying commands, which is what actually"
echo "determines pass/fail once a real run happens."
+74
View File
@@ -0,0 +1,74 @@
#!/usr/bin/env bash
# Phase 4.1 §22 — deployment rollback drill. Builds two real gateway image versions (a known
# good build, and a deliberately broken one), "deploys" each in turn to a local container,
# proves the smoke test genuinely catches the regression, then rolls back using the EXACT same
# decision logic rollback-production.sh uses against deploy-history.log (tail -2 | head -1 |
# awk '{print $2}' — the tag before the current one), and proves the rollback recovers.
#
# What this proves: the deterministic-tag + deploy-history.log rollback MECHANISM is real and
# correct. What it does NOT prove: the actual `ansible-playbook site.yml -e image_tag=...` step
# rollback-production.sh runs against a real host — that needs the real hardware
# (docs/PRODUCTION_VALIDATION_CHECKLIST.md). This drill swaps a local Docker container's image
# tag directly instead, as the closest local equivalent.
set -euo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")/../../../platform/gateway"
CONTAINER=rootnode-rollback-drill
PORT=4295
DEPLOY_LOG="$(mktemp)"
trap 'docker rm -f "$CONTAINER" >/dev/null 2>&1 || true; rm -f "$DEPLOY_LOG"; rm -rf /tmp/rootnode-rollback-drill-broken' EXIT
smoke_test() {
curl -sf "http://localhost:$PORT/healthz" >/dev/null 2>&1
}
echo "==> [1/6] Building the known-good gateway image (current source)"
docker build -q -t rootnode-gateway-drill:known-good -f Dockerfile . >/dev/null
echo "==> [2/6] Building a deliberately broken version (healthz returns 500)"
rm -rf /tmp/rootnode-rollback-drill-broken
cp -r . /tmp/rootnode-rollback-drill-broken
sed -i "s#app.get('/healthz', (req, res) => res.json({ ok: true }));#app.get('/healthz', (req, res) => res.status(500).json({ ok: false, broken: 'DELIBERATE_DRILL_REGRESSION' }));#" \
/tmp/rootnode-rollback-drill-broken/src/server.js
grep -q DELIBERATE_DRILL_REGRESSION /tmp/rootnode-rollback-drill-broken/src/server.js || { echo "FAIL: sed didn't apply — healthz line text must have changed upstream"; exit 1; }
docker build -q -t rootnode-gateway-drill:broken -f /tmp/rootnode-rollback-drill-broken/Dockerfile /tmp/rootnode-rollback-drill-broken >/dev/null
deploy() {
local tag="$1"
docker rm -f "$CONTAINER" >/dev/null 2>&1 || true
docker run -d --name "$CONTAINER" -p "$PORT:4200" -e GATEWAY_PORT=4200 -e PROVISIONER_URL=http://127.0.0.1:1 \
-e PROVISIONER_SHARED_SECRET=drill -e PLATFORM_API_URL=http://127.0.0.1:1 \
"rootnode-gateway-drill:$tag" >/dev/null
sleep 2
echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) $tag" >> "$DEPLOY_LOG"
}
echo "==> [3/6] Deploying known-good (deploy #1)"
deploy known-good
smoke_test && echo " smoke test: PASS (as expected)" || { echo "FAIL: known-good should have passed"; exit 1; }
echo "==> [4/6] Deploying the broken version (deploy #2 — the 'bad release' this drill catches)"
deploy broken
if smoke_test; then echo "FAIL: the broken deploy's regression was NOT caught by the smoke test — that defeats the whole point"; exit 1; fi
echo " smoke test: FAIL (correctly caught — this is exactly what should stop a real promotion, see docs/RUNBOOK_ROLLBACK.md)"
echo "==> [5/6] Computing the rollback target — same logic as infra/scripts/rollback-production.sh"
TARGET_TAG=$(tail -2 "$DEPLOY_LOG" | head -1 | awk '{print $2}')
[ "$TARGET_TAG" = "known-good" ] || { echo "FAIL: expected rollback target 'known-good', computed '$TARGET_TAG'"; exit 1; }
echo " rollback target: $TARGET_TAG (correctly identified as the tag before the broken one)"
echo "==> [6/6] Rolling back"
deploy "$TARGET_TAG"
smoke_test && echo " smoke test: PASS — rollback recovered service" || { echo "FAIL: rollback did not recover"; exit 1; }
echo
echo "OK: deployment rollback drill passed. Deterministic image tags + deploy-history.log"
echo "correctly identify and recover from a bad release."
echo
echo "On migration compatibility (docs/RUNBOOK_FAILED_MIGRATION.md, docs/RUNBOOK_ROLLBACK.md):"
echo "this script rolls back APPLICATION IMAGES ONLY. There is no automatic down-migration —"
echo "if the bad release also ran a schema-changing migration, rolling the image back does NOT"
echo "undo that migration, and old code may not run correctly against the new schema. That"
echo "case requires restoring from the pre-migration backup (docs/RUNBOOK_POSTGRES_RESTORE.md),"
echo "not a rollback, which is exactly why migrations in this project are check-before-alter"
echo "and additive-only (docs/ENGINEERING_SOP.md §6) — to make image-only rollback usually safe."
@@ -0,0 +1,150 @@
#!/usr/bin/env node
// Phase 4.1 §27 — the fake cohort dress rehearsal. One continuous narrative through the real
// application (not synthetic unit assertions): instructor login, cohort + 4 students, assign
// curriculum, Prepare Labs, students launch, work different scenarios, use hints, fail a
// validation, complete scenarios, instructor checks analytics, an exam is assigned/taken,
// a Lab is reset, a Lab is reaped (destroyed out-of-band, the same consequence the real idle
// reaper produces — see the note below on why this doesn't wait out the real multi-minute
// reaper timer), the student returns and continues, capstone state is checked, and the cohort's
// Labs are destroyed at the end.
//
// Run via run-fake-cohort-rehearsal.sh, which brings up platform/docker-compose.dev.yml first.
// No framework, matching project convention. Problems found are printed, not silently passed —
// see the Phase 4.1 report for anything this surfaced.
const assert = require('assert');
const { makeSession } = require('../../../platform/test/session-helper');
const GW = process.env.GATEWAY_URL || 'http://localhost:4200';
const OWNER_EMAIL = process.env.OWNER_EMAIL;
const OWNER_PASSWORD = process.env.OWNER_PASSWORD;
if (!OWNER_EMAIL || !OWNER_PASSWORD) { console.error('Set OWNER_EMAIL and OWNER_PASSWORD'); process.exit(1); }
let passed = 0, failed = 0;
async function step(name, fn) {
try { await fn(); console.log(` ok — ${name}`); passed++; }
catch (e) { console.error(` FAIL — ${name}\n ${e.stack || e.message}`); failed++; }
}
async function main() {
console.log('Phase 4.1 fake cohort dress rehearsal — slow, real k3s boots\n');
const instructor = makeSession(GW);
await instructor.request('POST', '/api/auth/instructor/login', { email: OWNER_EMAIL, password: OWNER_PASSWORD });
const names = ['Rehearsal-Ada', 'Rehearsal-Bo', 'Rehearsal-Cy', 'Rehearsal-Di'];
const students = {};
const sessions = {};
await step('instructor: create a cohort and roster 4 students', async () => {
await instructor.request('POST', '/api/admin/cohorts', { cohort_id: 'rehearsal-cohort', name: 'Rehearsal Cohort' });
for (const name of names) {
const { body } = await instructor.request('POST', '/api/admin/students', { display_name: name, cohort_id: 'rehearsal-cohort' });
students[name] = body.student_id;
sessions[name] = makeSession(GW);
await sessions[name].request('POST', '/api/login', { student_id: body.student_id });
}
});
await step('instructor: assign curriculum to the cohort', async () => {
const { status } = await instructor.request('POST', '/api/admin/cohorts/rehearsal-cohort/curriculum', { curriculum_path: 'rootnode-path' });
assert.ok([200, 201].includes(status));
});
await step('instructor: Prepare Labs for the whole cohort (concurrent)', async () => {
const { status, body } = await instructor.request('POST', '/api/admin/cohorts/rehearsal-cohort/prepare-labs');
assert.equal(status, 200);
assert.ok(body.results.every((r) => r.status === 'running'), `expected every Lab running, got: ${JSON.stringify(body.results)}`);
});
const launches = {};
await step('students: everyone\'s prepared Lab is already running (no re-launch wait)', async () => {
for (const name of names) {
const { body } = await sessions[name].request('POST', '/api/environment/launch');
assert.equal(body.status, 'running');
launches[name] = body;
}
});
await step('students: Ada and Bo work DIFFERENT scenarios', async () => {
await sessions['Rehearsal-Ada'].request('POST', `${launches['Rehearsal-Ada'].launch_path}api/scenarios/pod-basics-mcq/answer`, { selected: 'b' });
await sessions['Rehearsal-Bo'].request('POST', `${launches['Rehearsal-Bo'].launch_path}api/scenarios/pod-basics-mcq/start`);
});
await step('student: Cy uses a hint on a scenario', async () => {
const res = await sessions['Rehearsal-Cy'].request('POST', `${launches['Rehearsal-Cy'].launch_path}api/scenarios/pod-basics-mcq/hint`);
assert.equal(res.status, 200);
});
await step('student: Di fails a validation on purpose', async () => {
const res = await sessions['Rehearsal-Di'].request('POST', `${launches['Rehearsal-Di'].launch_path}api/scenarios/pod-basics-mcq/answer`, { selected: 'a' });
assert.equal(res.status, 200);
assert.equal(res.body.correct, false, 'expected a deliberately wrong answer to be marked incorrect');
});
await step('instructor: cohort learning summary reflects real activity', async () => {
const { body } = await instructor.request('GET', '/api/admin/cohorts/rehearsal-cohort/learning');
assert.equal(body.student_count, 4);
});
await step('instructor: needs-attention returns real signals for the struggling students, no punitive language', async () => {
const { body } = await instructor.request('GET', '/api/admin/cohorts/rehearsal-cohort/needs-attention');
const text = JSON.stringify(body).toLowerCase();
assert.ok(!text.includes('fail') || text.includes('failed validations'), 'the word "fail" should only appear in a factual "N failed validations" context, never as a standalone judgment');
});
await step('instructor: reset Ada\'s Lab — academic progress is untouched', async () => {
const before = await sessions['Rehearsal-Ada'].request('GET', '/api/progress');
// Admin environment routes need the real environment_id, which publicEnvView doesn't echo
// back to the student directly — resolved via the student environment lookup instead.
const envRes = await instructor.request('GET', `/api/admin/students/${students['Rehearsal-Ada']}/environment`);
const env = envRes.body.find((e) => e.status !== 'destroyed');
await instructor.request('POST', `/api/admin/environments/${env.environment_id}/reset`);
const after = await sessions['Rehearsal-Ada'].request('GET', '/api/progress');
assert.deepEqual(before.body.details, after.body.details, 'progress must be identical before/after a Lab reset');
});
await step('§27 — a Lab is reaped: same consequence as the real idle reaper (real multi-minute timer not waited out here — see reaper.js/resource-policy.yaml for the real thresholds; this exercises the destroy-and-preserve-progress consequence directly, already proven mechanistically by infra/scripts/drills/test-lab-host-loss.js)', async () => {
const before = await sessions['Rehearsal-Bo'].request('GET', '/api/progress');
const envRes = await instructor.request('GET', `/api/admin/students/${students['Rehearsal-Bo']}/environment`);
const env = envRes.body.find((e) => e.status !== 'destroyed');
await instructor.request('DELETE', `/api/admin/environments/${env.environment_id}`);
const after = await sessions['Rehearsal-Bo'].request('GET', '/api/progress');
assert.deepEqual(before.body.details, after.body.details, 'progress must survive a reaped Lab unchanged');
});
await step('student: Bo returns later — a fresh Lab is created and curriculum continues', async () => {
const { status, body } = await sessions['Rehearsal-Bo'].request('POST', '/api/environment/launch');
assert.equal(status, 200); assert.equal(body.status, 'running');
const curriculum = await sessions['Rehearsal-Bo'].request('GET', '/api/curriculum/path');
assert.ok(curriculum.body.stages.length > 0, 'curriculum must still be there for the returning student');
});
await step('instructor: assign an exam to the cohort and Ada takes it', async () => {
const create = await instructor.request('POST', '/api/admin/assessments', {
id: 'rehearsal-exam', type: 'exam', title: 'Rehearsal Exam', duration_minutes: 30, hint_policy: 'disabled',
pool_spec: { required: ['pod-basics-mcq'], random: [] },
});
assert.ok([200, 201, 409].includes(create.status)); // 409 if a prior run already created it — fine, idempotent enough for a rehearsal
const assign = await instructor.request('POST', '/api/admin/assessments/rehearsal-exam/assign', { cohort_id: 'rehearsal-cohort' });
assert.ok([200, 201].includes(assign.status));
const start = await sessions['Rehearsal-Ada'].request('POST', '/api/assessments/rehearsal-exam/start');
assert.equal(start.status, 201);
assert.ok(start.body.scenarios.length > 0);
});
await step('instructor: destroy all cohort Labs after class — progress remains intact for everyone', async () => {
const before = {};
for (const name of names) before[name] = (await sessions[name].request('GET', '/api/progress')).body.details;
await instructor.request('POST', '/api/admin/cohorts/rehearsal-cohort/destroy-labs');
for (const name of names) {
const after = (await sessions[name].request('GET', '/api/progress')).body.details;
assert.deepEqual(before[name], after, `${name}'s progress must survive end-of-class Lab destruction`);
}
});
console.log(`\n${passed} passed, ${failed} failed`);
if (failed > 0) process.exit(1);
}
main().catch((e) => { console.error(e); process.exit(1); });
+104
View File
@@ -0,0 +1,104 @@
#!/usr/bin/env node
// Phase 4.1 §21 — Lab Host loss/rebuild drill. Proves disposable infrastructure is genuinely
// disposable: a student's Lab container is destroyed OUT FROM UNDER the Provisioner (no clean
// destroy call — simulating the whole rn-lab-host-01 VM disappearing), and the system must:
// - keep the Platform (API/Gateway) up throughout — it never depended on that one container
// - never present the now-gone environment as healthy/running once reconciliation runs
// - leave academic data (progress/attempts) completely untouched
// - let the student launch a brand-new Lab afterward and continue where they left off
//
// Run via run-lab-host-loss-drill.sh, which brings up platform/docker-compose.dev.yml first.
// No framework, matching project convention.
const assert = require('assert');
const { execFileSync } = require('child_process');
const { makeSession } = require('../../../platform/test/session-helper');
const GW = process.env.GATEWAY_URL || 'http://localhost:4200';
const API = process.env.PLATFORM_API_URL || 'http://localhost:4300';
const OWNER_EMAIL = process.env.OWNER_EMAIL;
const OWNER_PASSWORD = process.env.OWNER_PASSWORD;
if (!OWNER_EMAIL || !OWNER_PASSWORD) { console.error('Set OWNER_EMAIL and OWNER_PASSWORD'); process.exit(1); }
let passed = 0, failed = 0;
async function test(name, fn) {
try { await fn(); console.log(` ok — ${name}`); passed++; }
catch (e) { console.error(` FAIL — ${name}\n ${e.stack || e.message}`); failed++; }
}
async function main() {
console.log('Phase 4.1 Lab Host loss/rebuild drill — slow, real k3s boots\n');
const admin = makeSession(GW);
const { status: loginStatus } = await admin.request('POST', '/api/auth/instructor/login', { email: OWNER_EMAIL, password: OWNER_PASSWORD });
assert.equal(loginStatus, 200, 'owner login must succeed');
let studentId;
await test('roster: add a student', async () => {
const { body } = await admin.request('POST', '/api/admin/students', { display_name: 'Drill-LabHostLoss', cohort_id: null });
studentId = body.student_id;
});
const session = makeSession(GW);
await session.request('POST', '/api/login', { student_id: studentId });
let launch;
await test('launch: student gets a running Lab (real k3s boot)', async () => {
const { status, body } = await session.request('POST', '/api/environment/launch');
assert.equal(status, 200); assert.equal(body.status, 'running');
launch = body;
});
await test('progress: student completes a real scenario before the Lab Host is lost', async () => {
const res = await session.request('POST', `${launch.launch_path}api/scenarios/pod-basics-mcq/answer`, { selected: 'b' });
assert.equal(res.status, 200);
const progress = await session.request('GET', '/api/progress');
assert.equal(progress.body.details['pod-basics-mcq']?.status, 'completed', 'progress must show completed before we destroy anything');
});
const token = launch.launch_path.split('/')[2];
await test('§21 — simulate total Lab Host loss: kill the Lab container directly, bypassing the Provisioner entirely', async () => {
const containers = execFileSync('docker', ['ps', '--filter', 'name=rootnode-lab-', '--format', '{{.Names}}']).toString().trim().split('\n').filter(Boolean);
assert.ok(containers.length >= 1, 'expected at least one running Lab container to destroy');
for (const c of containers) execFileSync('docker', ['rm', '-f', c]);
console.log(` destroyed ${containers.length} Lab container(s) out from under the Provisioner`);
});
await test('§21 — Platform stays available throughout (it never depended on that container)', async () => {
const api = await fetch(`${API}/healthz`);
const gw = await fetch(`${GW}/healthz`);
assert.equal(api.status, 200);
assert.equal(gw.status, 200);
});
await test('§21 — academic progress is completely untouched by the infrastructure loss', async () => {
const progress = await session.request('GET', '/api/progress');
assert.equal(progress.body.details['pod-basics-mcq']?.status, 'completed', 'progress must survive the Lab Host loss unchanged');
});
await test('§21 — the now-gone environment is NEVER presented as healthy once reconciliation runs (Provisioner restart, its real startup path)', async () => {
// cwd is already platform/ — run-lab-host-loss-drill.sh cd's there before invoking this script.
execFileSync('docker', ['compose', '-f', 'docker-compose.dev.yml', 'restart', 'provisioner'], { stdio: 'inherit' });
// Give the Provisioner a moment to finish its real reconcileOnStartup pass.
await new Promise((r) => setTimeout(r, 5000));
const res = await session.rawFetch(`/lab/${token}/`);
assert.ok([503, 404].includes(res.status), `expected the stale environment to be rejected (503/404), got ${res.status} — a healthy-looking response here would mean a destroyed environment is being silently presented as available`);
});
await test('§21 — a replacement Lab can be launched and boots successfully (the "replacement Lab Host" case, minus needing a literal second VM)', async () => {
const { status, body } = await session.request('POST', '/api/environment/launch');
assert.equal(status, 200); assert.equal(body.status, 'running');
assert.notEqual(body.launch_path, launch.launch_path, 'the replacement environment must be a genuinely new one, not the destroyed one still being served');
});
await test('§21 — previous academic progress is still there after the replacement Lab boots', async () => {
const progress = await session.request('GET', '/api/progress');
assert.equal(progress.body.details['pod-basics-mcq']?.status, 'completed');
});
console.log(`\n${passed} passed, ${failed} failed`);
if (failed > 0) process.exit(1);
}
main().catch((e) => { console.error(e); process.exit(1); });
+11 -3
View File
@@ -6,18 +6,26 @@ set -euo pipefail
PROD_URL="${PRODUCTION_URL:?set PRODUCTION_URL, e.g. https://academy.example.com}"
# Phase 4.1 §23: every `curl ... && echo "OK"` below used to be one statement — under `set -e`,
# a failing command that isn't the LAST one in an && list does not trigger the shell to exit (a
# well-known bash gotcha), so an unreachable/broken production endpoint used to fall through
# silently instead of failing this script. Split into two statements each, so the real exit
# status is what `set -e` sees. Found by infra/scripts/drills/test-ci-fails-closed.sh.
echo "==> Gateway health"
curl -sf "$PROD_URL/healthz" > /dev/null && echo "OK"
curl -sf "$PROD_URL/healthz" > /dev/null
echo "OK"
echo "==> TLS certificate is valid and not expiring soon"
EXPIRY=$(echo | openssl s_client -connect "$(echo "$PROD_URL" | sed -e 's#https://##')":443 -servername "$(echo "$PROD_URL" | sed -e 's#https://##')" 2>/dev/null | openssl x509 -noout -enddate | cut -d= -f2)
echo "Certificate expires: $EXPIRY"
echo "==> Roster endpoint responds (public, unauthenticated, read-only)"
curl -sf "$PROD_URL/api/roster/cohorts" > /dev/null && echo "OK"
curl -sf "$PROD_URL/api/roster/cohorts" > /dev/null
echo "OK"
echo "==> Admin login page loads (does not attempt to log in)"
curl -sf "$PROD_URL/admin" -o /dev/null -w '%{http_code}\n' | grep -q 200 && echo "OK"
curl -sf "$PROD_URL/admin" -o /dev/null -w '%{http_code}\n' | grep -q 200
echo "OK"
echo
echo "OK: production smoke test passed. For a full functional check (owner/instructor login,"
+50
View File
@@ -0,0 +1,50 @@
# Two-Host Routing Simulation
Proves, with real Docker network isolation (not mocks), that the Gateway-to-Lab data path
described in `docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1 actually works before the real
Hetzner/Proxmox hardware exists. See that document for the full design; this is the test harness.
## Run it
```bash
cd infra/scripts/simulate-two-host
PROVISIONER_SHARED_SECRET=... ./run-simulation.sh # builds, runs, tears down
PROVISIONER_SHARED_SECRET=... ./run-simulation.sh --keep-up # leaves the stack up for debugging
```
Confirmed passing, 13/13, as of this phase's implementation — see the Phase 4.1 report for the
full run transcript.
## What this genuinely proves (TESTED LOCALLY, with real Docker networking)
- The `gateway` container, attached ONLY to an isolated bridge network with no route to any
Lab's per-environment bridge network, **cannot** open a TCP connection to a Lab container's
Docker-internal IP — verified via a raw connect attempt run from inside the actual gateway
container's own network namespace, not asserted from the test driver's (irrelevant) vantage
point.
- The Gateway **does** reach the Lab correctly through the Lab Host Router, over real mTLS,
for both a plain HTTP request and a full WebSocket terminal session — including a genuine
`kubectl get nodes` executed inside the Lab and its real output read back.
- Alice/Bob isolation (own lab ALLOW, other's lab DENY, including over WebSocket), garbage
tokens, anonymous requests, and destroyed-environment tokens are all rejected through the
real code path — the same `checkOwnership`/`resolveTarget` logic production will run.
- The Lab Host Router refuses a connection with no client certificate at all — mTLS is the
actual authentication mechanism, not a documented intention.
## What this does NOT prove (SIMULATED PRODUCTION BOUNDARY, not identical to real hardware)
- Real cross-VM latency, MTU, or anything about two genuinely separate kernels — this is one
Docker daemon with two isolated networks, a faithful stand-in for network *reachability*
rules but not a physical network.
- The actual Hetzner/Proxmox firewall enforcement (`infra/firewall/*.rules`) — this harness
doesn't touch iptables at all; Docker's own network isolation is what's under test here.
- Provisioner/API/Postgres reachability across the VM boundary — deliberately kept
host-networked and unchanged from `docker-compose.dev.yml` in this harness, because that
control-plane path was already proven separately (Phase 4's real mTLS handshake test against
the Docker Engine TLS port) and isn't what changed this phase. Only the Gateway's isolation
is new here.
## REQUIRES REAL HARDWARE (tracked in docs/PRODUCTION_VALIDATION_CHECKLIST.md, not here)
Running this exact flow against the real `rn-platform-01`/`rn-lab-host-01` VMs, through the
real firewall, with real network latency between two physical/virtual hosts.
@@ -0,0 +1,136 @@
# Phase 4.1 two-host routing simulation — docs/PHASE4_1_PREPRODUCTION_HARDENING.md §6.
#
# Proves the ONE thing that matters using REAL Docker network isolation, not mocks: a Gateway
# container with no route to a Lab container's bridge network genuinely cannot reach it, and
# can only reach it through the Lab Host Router.
#
# What's genuinely simulated (real Docker networking facts, not faked):
# - `gateway` is attached ONLY to `sim-platform-net`. It has no network_mode: host, no
# attachment to any per-Lab bridge network (rootnode-lab-net-<id>, created dynamically by
# the Provisioner) — its network namespace simply has no route to one. This is the same
# kind of isolation two separate Proxmox VMs would have, achieved here with two Docker
# networks on one daemon instead of two physical hosts.
# - `lab-router` runs with network_mode: host — exactly like `egress-proxy` already does in
# docker-compose.dev.yml (see that file's comment on why: a host-networked process can
# reach every bridge network's containers via the host's own routing table, which is
# exactly what a real Lab Host Router process running directly on rn-lab-host-01 needs).
# It's also reachable FROM `sim-platform-net` via that network's own gateway IP — the same
# mechanism the egress-proxy already relies on for the reverse direction.
# - `provisioner`/`api`/`postgres` stay host-networked, unchanged from docker-compose.dev.yml
# — this phase's new proof target is the Gateway-to-Lab DATA plane, not the CONTROL plane
# (already proven with a real mTLS handshake in Phase 4). Keeping them as-is is a
# deliberate scope choice, not an oversight — see the README in this directory.
#
# What this does NOT prove (needs the real hardware — docs/PRODUCTION_VALIDATION_CHECKLIST.md):
# real cross-VM network latency/MTU, the actual Hetzner/Proxmox firewall enforcement, or
# anything about two genuinely separate kernels.
#
# Usage: see README.md in this directory.
services:
postgres:
image: postgres:16-alpine
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: rootnode_platform_sim
ports:
- "5434:5432"
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 5s
timeout: 3s
retries: 10
egress-proxy:
build: ../../../platform/egress-proxy
network_mode: host
provisioner:
build: ../../../platform/provisioner
network_mode: host
cap_add:
- NET_ADMIN
- NET_RAW
volumes:
- /var/run/docker.sock:/var/run/docker.sock
# run-simulation.sh generates this with the real routing_mode: router / router_endpoint
# once it knows sim-platform-net's actual gateway IP — overrides just this one file, the
# image's own resource-policy.yaml underneath is untouched.
- ${SIM_LAB_HOSTS_SEED:?set by run-simulation.sh}:/app/config/lab-hosts.seed.yaml:ro
environment:
PLATFORM_DB_URL: postgres://postgres:postgres@localhost:5434/rootnode_platform_sim
PROVISIONER_SHARED_SECRET: ${PROVISIONER_SHARED_SECRET:?set a real secret, do not use a default}
PROVISIONER_PORT: "4101"
LAB_IMAGE_REF: ${LAB_IMAGE_REF:-rootnode-academy:test}
EGRESS_PROXY_PORT: "3128"
DNS_RESOLVER: "127.0.0.11"
NO_PROXY_CIDRS: "10.42.0.0/16,10.43.0.0/16"
depends_on:
postgres:
condition: service_healthy
api:
build:
context: ../../..
dockerfile: platform/api/Dockerfile
network_mode: host
environment:
PLATFORM_DB_URL: postgres://postgres:postgres@localhost:5434/rootnode_platform_sim
PROVISIONER_URL: "http://localhost:4101"
PROVISIONER_SHARED_SECRET: ${PROVISIONER_SHARED_SECRET:?set a real secret, do not use a default}
PLATFORM_API_PORT: "4301"
depends_on:
postgres:
condition: service_healthy
provisioner:
condition: service_started
# Simulates rn-platform-01's Gateway — deliberately NOT host-networked, deliberately not
# attached to any Lab bridge network. This is the isolation under test.
gateway:
build: ../../../platform/gateway
networks:
- sim-platform-net
environment:
GATEWAY_PORT: "4200"
# Reaches host-networked Provisioner/API via sim-platform-net's own gateway IP — the
# same "reachable via the bridge's gateway interface" mechanism egress-proxy already
# relies on (see docker-compose.dev.yml). run-simulation.sh fills these in.
PROVISIONER_URL: ${SIM_PROVISIONER_URL:?set by run-simulation.sh}
PROVISIONER_SHARED_SECRET: ${PROVISIONER_SHARED_SECRET:?set a real secret, do not use a default}
PLATFORM_API_URL: ${SIM_PLATFORM_API_URL:?set by run-simulation.sh}
LAB_ROUTER_TLS_CA: /tls/ca.pem
LAB_ROUTER_TLS_CERT: /tls/gateway-client-cert.pem
LAB_ROUTER_TLS_KEY: /tls/gateway-client-key.pem
volumes:
- ${SIM_TLS_DIR:?set by run-simulation.sh}:/tls:ro
ports:
- "4210:4200" # exposed to the test driver script only, not "public" in any real sense
depends_on:
- provisioner
- api
# Simulates rn-lab-host-01's Lab Host Router — host-networked like egress-proxy, for the same
# reason (needs to reach whatever per-Lab bridge networks the Provisioner creates, which are
# not known/attachable ahead of time).
lab-router:
build: ../../../platform/lab-router
network_mode: host
environment:
LAB_ROUTER_PORT: "8443"
LAB_ROUTER_TLS_CA: /tls/ca.pem
LAB_ROUTER_TLS_CERT: /tls/router-server-cert.pem
LAB_ROUTER_TLS_KEY: /tls/router-server-key.pem
DOCKER_SOCKET: /var/run/docker.sock
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ${SIM_TLS_DIR:?set by run-simulation.sh}:/tls:ro
networks:
# Pre-created by run-simulation.sh (so it can inspect the real gateway IP BEFORE any
# container starts and hand it to provisioner/gateway/lab-router as config) — Compose is
# told to use that existing network rather than create its own.
sim-platform-net:
external: true
name: ${COMPOSE_NET_NAME:?set by run-simulation.sh}
+76
View File
@@ -0,0 +1,76 @@
#!/usr/bin/env bash
# Orchestrates the Phase 4.1 two-host routing simulation end-to-end: generates a real TLS
# bundle, computes the sim-platform-net gateway IP, renders the lab_hosts seed with it, brings
# the stack up, runs test-remote-routing.js, tears everything down. See README.md in this
# directory and docs/PHASE4_1_PREPRODUCTION_HARDENING.md §6.
#
# Usage: ./run-simulation.sh [--keep-up] (--keep-up skips teardown, for debugging)
set -euo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")"
KEEP_UP=false
[ "${1:-}" = "--keep-up" ] && KEEP_UP=true
WORK_DIR="$(mktemp -d)"
trap 'rm -rf "$WORK_DIR"' EXIT
export PROVISIONER_SHARED_SECRET="${PROVISIONER_SHARED_SECRET:-sim-shared-secret-not-for-real-use}"
export LAB_IMAGE_REF="${LAB_IMAGE_REF:-rootnode-academy:test}"
export SIM_TLS_DIR="$WORK_DIR/tls"
export SIM_LAB_HOSTS_SEED="$WORK_DIR/lab-hosts.seed.yaml"
echo "==> [1/6] Creating sim-platform-net so we know its real gateway IP before generating certs"
NET_NAME="rootnode-sim-platform-net"
docker network inspect "$NET_NAME" >/dev/null 2>&1 || docker network create "$NET_NAME" >/dev/null
GW_IP="$(docker network inspect "$NET_NAME" --format '{{(index .IPAM.Config 0).Gateway}}')"
echo " sim-platform-net gateway IP: $GW_IP (lab-router and provisioner/api, all host-networked, are reachable from Gateway's isolated network through this address)"
echo "==> [2/6] Generating a real mTLS bundle for this simulation run (SAN covers the real gateway IP above)"
../docker-tls/generate-docker-tls.sh --out-dir "$SIM_TLS_DIR" --lab-host-ip "$GW_IP" --lab-host-dns lab-router-sim >/dev/null
# Rewrite the compose file's network name to match what we just created/inspected, and point
# the sim gateway/router services at the shared network by that exact name.
export COMPOSE_NET_NAME="$NET_NAME"
echo "==> [3/6] Rendering lab-hosts.seed.yaml with routing_mode: router at the real gateway IP"
cat > "$SIM_LAB_HOSTS_SEED" <<EOF
lab_hosts:
- lab_host_id: sim-lab-host-01
display_name: "Simulated rn-lab-host-01"
endpoint: "unix:///var/run/docker.sock"
enabled: true
routing_mode: router
router_endpoint: "https://${GW_IP}:8443"
EOF
export SIM_PROVISIONER_URL="http://${GW_IP}:4101"
export SIM_PLATFORM_API_URL="http://${GW_IP}:4301"
echo "==> [4/6] Building and starting the simulated two-host stack"
docker compose -f docker-compose.simulate.yml -p rootnode-sim up -d --build
cleanup() {
if [ "$KEEP_UP" = true ]; then
echo "==> --keep-up set, leaving the stack running (docker compose -f docker-compose.simulate.yml -p rootnode-sim down to clean up)"
return
fi
echo "==> [6/6] Tearing down the simulation stack"
docker compose -f docker-compose.simulate.yml -p rootnode-sim down -v >/dev/null 2>&1 || true
docker network rm "$NET_NAME" >/dev/null 2>&1 || true
}
trap cleanup EXIT
echo "==> Waiting for services to become healthy"
for i in $(seq 1 60); do
if curl -sf http://localhost:4301/healthz >/dev/null 2>&1 && curl -sf http://localhost:4210/healthz >/dev/null 2>&1; then
break
fi
sleep 2
done
echo "==> [5/6] Running the routing test"
GATEWAY_URL=http://localhost:4210 \
PROVISIONER_URL=http://localhost:4101 \
PROVISIONER_SHARED_SECRET="$PROVISIONER_SHARED_SECRET" \
GATEWAY_CONTAINER=rootnode-sim-gateway-1 \
node test-remote-routing.js
@@ -0,0 +1,212 @@
#!/usr/bin/env node
// Phase 4.1 §6-9 — the real, unmocked proof that remote Lab routing works. Run via
// run-simulation.sh, which brings up docker-compose.simulate.yml first. No framework, per
// project convention (see platform/provisioner/test/run.js).
//
// Proves, against the real simulated stack:
// §6/7 Gateway cannot reach a Lab's Docker-internal IP directly, but reaches it correctly
// through the Lab Host Router — both HTTP and WebSocket terminal traffic, including a
// real `kubectl get nodes` inside the Lab.
// §8/9 Alice/Bob isolation, garbage tokens, destroyed environments, anonymous requests, and
// that the Router itself is unreachable without the Gateway's mTLS client identity —
// all through the real path, not asserted against mocks.
const assert = require('assert');
const https = require('https');
const { execFileSync } = require('child_process');
const { makeSession } = require('../../../platform/test/session-helper');
const GW = process.env.GATEWAY_URL || 'http://localhost:4210';
const PROVISIONER_URL = process.env.PROVISIONER_URL || 'http://localhost:4101';
const PROVISIONER_KEY = process.env.PROVISIONER_SHARED_SECRET;
const GATEWAY_CONTAINER = process.env.GATEWAY_CONTAINER || 'rootnode-sim-gateway-1';
const WS_URL = GW.replace(/^http/, 'ws');
let passed = 0, failed = 0;
async function test(name, fn) {
try { await fn(); console.log(` ok — ${name}`); passed++; }
catch (e) { console.error(` FAIL — ${name}\n ${e.stack || e.message}`); failed++; }
}
async function provisionerCall(method, path) {
const res = await fetch(`${PROVISIONER_URL}${path}`, { method, headers: { 'X-Provisioner-Key': PROVISIONER_KEY } });
const body = await res.json().catch(() => null);
return { status: res.status, body };
}
// The "browser never learns internal_ip" invariant holds everywhere else in the app — this
// test is the one deliberate, documented exception: it needs the real Docker-internal address
// as ground truth to prove the Gateway genuinely CANNOT reach it, which requires calling the
// Provisioner's internal API directly (the same privilege level an operator/CI job has, never
// a student's).
async function internalAddressForToken(token) {
const { body } = await provisionerCall('GET', `/environments/by-token/${token}`);
return body;
}
// bootstrap-owner.js isn't copied into the api image (it's an operator-run one-off script, not
// a runtime dependency — same reason platform/test regression runs hit this before, see
// docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md's regression notes). Run it from the host against
// the sim Postgres's published port instead of `docker exec`ing into the container.
function bootstrapOwner(email, name, password) {
execFileSync('node', ['bootstrap-owner.js', email, name, password], {
cwd: require('path').join(__dirname, '../../../platform/api'),
env: { ...process.env, PLATFORM_DB_URL: 'postgres://postgres:postgres@localhost:5434/rootnode_platform_sim' },
stdio: 'inherit',
});
}
// Raw TCP connect attempt from WITHIN the gateway container's own network namespace — proves
// isolation from the gateway's actual point of view, not the test driver's (which runs on the
// host and has a completely different, irrelevant reachability picture).
function gatewayCanReach(ip, port, timeoutMs = 2500) {
const script = `
const net = require('net');
const s = net.createConnection({ host: '${ip}', port: ${port}, timeout: ${timeoutMs} });
s.on('connect', () => { console.log('REACHABLE'); s.destroy(); process.exit(0); });
s.on('timeout', () => { console.log('UNREACHABLE(timeout)'); process.exit(0); });
s.on('error', (e) => { console.log('UNREACHABLE(' + e.code + ')'); process.exit(0); });
`;
const out = execFileSync('docker', ['exec', GATEWAY_CONTAINER, 'node', '-e', script]).toString().trim();
return out.startsWith('REACHABLE');
}
function connectRouterWithoutClientCert(host, port, timeoutMs = 2500) {
return new Promise((resolve) => {
const req = https.request({ host, port, method: 'GET', rejectUnauthorized: false, timeout: timeoutMs }, (res) => {
resolve({ ok: true, status: res.statusCode });
});
req.on('error', (e) => resolve({ ok: false, error: e.code || e.message }));
req.on('timeout', () => { req.destroy(); resolve({ ok: false, error: 'timeout' }); });
req.end();
});
}
function openTerminal(launchPath, cookieHeader) {
return new Promise((resolve, reject) => {
const ws = new WebSocket(`${WS_URL}${launchPath}shell-ws`, { headers: { Cookie: cookieHeader } });
let buffer = '';
const timer = setTimeout(() => { ws.close(); reject(new Error(`terminal timed out, buffer so far: ${buffer.slice(-400)}`)); }, 30000);
ws.addEventListener('open', () => ws.send('kubectl get nodes\n'));
ws.addEventListener('message', (ev) => {
buffer += ev.data.toString();
if (/Ready\s/.test(buffer) || /node\.?\d*\s+Ready/i.test(buffer)) {
clearTimeout(timer);
ws.close();
resolve(buffer);
}
});
ws.addEventListener('error', (e) => { clearTimeout(timer); reject(new Error(`ws error: ${e.message || e}`)); });
});
}
function expectWsRejected(launchPath, cookieHeader, timeoutMs = 5000) {
return new Promise((resolve, reject) => {
const ws = new WebSocket(`${WS_URL}${launchPath}shell-ws`, { headers: { Cookie: cookieHeader } });
const timer = setTimeout(() => { ws.close(); reject(new Error('expected the WS to be rejected, but it stayed open')); }, timeoutMs);
ws.addEventListener('open', () => { clearTimeout(timer); ws.close(); reject(new Error('expected the WS upgrade to be rejected, but it succeeded')); });
ws.addEventListener('close', () => { clearTimeout(timer); resolve(); });
ws.addEventListener('error', () => { clearTimeout(timer); resolve(); });
});
}
async function main() {
console.log('Phase 4.1 two-host remote-routing simulation — slow, real k3s boots\n');
bootstrapOwner('[email protected]', 'Sim Owner', 'SimOwnerPass!2026');
const admin = makeSession(GW);
const { status: loginStatus } = await admin.request('POST', '/api/auth/instructor/login', { email: '[email protected]', password: 'SimOwnerPass!2026' });
assert.equal(loginStatus, 200, 'owner login must succeed through the Gateway');
let alice, bob;
await test('roster: add Alice and Bob via admin (through the Gateway)', async () => {
const a = await admin.request('POST', '/api/admin/students', { display_name: 'Sim-Alice', cohort_id: null });
const b = await admin.request('POST', '/api/admin/students', { display_name: 'Sim-Bob', cohort_id: null });
alice = a.body.student_id; bob = b.body.student_id;
assert.notEqual(alice, bob);
});
const aliceSession = makeSession(GW);
const bobSession = makeSession(GW);
await aliceSession.request('POST', '/api/login', { student_id: alice });
await bobSession.request('POST', '/api/login', { student_id: bob });
let aliceLaunch, bobLaunch, aliceEnv;
await test('launch: Alice and Bob each get a running environment on the simulated remote lab host', async () => {
const a = await aliceSession.request('POST', '/api/environment/launch');
const b = await bobSession.request('POST', '/api/environment/launch');
assert.equal(a.status, 200); assert.equal(a.body.status, 'running');
assert.equal(b.status, 200); assert.equal(b.body.status, 'running');
aliceLaunch = a.body; bobLaunch = b.body;
const aliceToken = aliceLaunch.launch_path.split('/')[2];
aliceEnv = await internalAddressForToken(aliceToken);
assert.equal(aliceEnv.lab_host_routing_mode, 'router', 'sim-lab-host-01 must be configured as routing_mode: router for this test to mean anything');
});
await test('§6/7 — the Gateway CANNOT reach the Lab\'s Docker-internal IP directly', async () => {
const reachable = gatewayCanReach(aliceEnv.internal_ip, aliceEnv.internal_port);
assert.equal(reachable, false, `gateway container reached ${aliceEnv.internal_ip}:${aliceEnv.internal_port} directly — the network isolation this whole phase depends on is not real`);
});
await test('§6/7 — but the Gateway DOES reach the Lab through the Lab Host Router (HTTP)', async () => {
const res = await aliceSession.rawFetch(aliceLaunch.launch_path);
assert.equal(res.status, 200);
const text = await res.text();
assert.ok(text.length > 0, 'expected real HTML/asset content back from the Lab, routed through the Router');
});
await test('§8 — Alice -> her own Lab: ALLOW', async () => {
const res = await aliceSession.rawFetch(aliceLaunch.launch_path);
assert.equal(res.status, 200);
});
await test('§8 — Bob -> his own Lab: ALLOW', async () => {
const res = await bobSession.rawFetch(bobLaunch.launch_path);
assert.equal(res.status, 200);
});
await test('§8 — Alice -> Bob\'s Lab: DENY', async () => {
const res = await aliceSession.rawFetch(bobLaunch.launch_path);
assert.equal(res.status, 403);
});
await test('§8 — garbage token: DENY', async () => {
const res = await aliceSession.rawFetch('/lab/not-a-real-token-at-all/');
assert.equal(res.status, 404);
});
await test('§8 — anonymous Gateway request (no session at all): DENY', async () => {
const res = await fetch(`${GW}${aliceLaunch.launch_path}`);
assert.equal(res.status, 401);
});
await test('§8/9 — direct public access to the Lab Host Router: NOT AVAILABLE (mTLS required, no client cert presented)', async () => {
const routerUrl = new URL(aliceEnv.lab_host_router_endpoint);
const result = await connectRouterWithoutClientCert(routerUrl.hostname, routerUrl.port);
assert.equal(result.ok, false, 'the Router accepted a connection with no client certificate at all — mTLS is not actually being enforced');
});
await test('§7 — WebSocket terminal connects and a real `kubectl get nodes` succeeds, through the Router', async () => {
const output = await openTerminal(bobLaunch.launch_path, `rn_session=${bobSession.getCookie('rn_session')}`);
assert.ok(/Ready/i.test(output), `expected a Ready node in kubectl output, got: ${output.slice(-300)}`);
});
await test('§8 — Alice cannot open Bob\'s terminal WebSocket even with a valid session (her own)', async () => {
await expectWsRejected(bobLaunch.launch_path, `rn_session=${aliceSession.getCookie('rn_session')}`);
});
await test('§8/9 — destroyed environment: routing token stops working, both at the Gateway and at the Router directly', async () => {
const aliceToken = aliceLaunch.launch_path.split('/')[2];
await provisionerCall('DELETE', `/environments/${aliceEnv.environment_id}`);
// Give the Gateway's short-TTL env-cache a moment to expire so this proves real
// destroyed-environment handling, not just a stale cache hit.
await new Promise((r) => setTimeout(r, 6000));
const res = await aliceSession.rawFetch(`/lab/${aliceToken}/`);
assert.ok([404, 503].includes(res.status), `expected the destroyed environment to be rejected, got ${res.status}`);
});
console.log(`\n${passed} passed, ${failed} failed`);
if (failed > 0) process.exit(1);
}
main().catch((e) => { console.error(e); process.exit(1); });
+7 -1
View File
@@ -9,7 +9,13 @@ set -euo pipefail
STAGING_URL="${STAGING_GATEWAY_URL:?set STAGING_GATEWAY_URL, e.g. https://staging.academy.example.com}"
echo "==> Health checks"
curl -sf "$STAGING_URL/healthz" > /dev/null && echo "OK: Gateway /healthz"
# Phase 4.1 §23: this was `curl ... && echo ...` as one statement — under `set -e`, a failing
# command that ISN'T the last one in an && list does not trigger the shell to exit (a
# well-known bash gotcha), so a genuinely unreachable staging Gateway used to fall through
# silently instead of failing this script. Split into two statements so curl's own exit status
# is what `set -e` actually sees. Found by infra/scripts/drills/test-ci-fails-closed.sh.
curl -sf "$STAGING_URL/healthz" > /dev/null
echo "OK: Gateway /healthz"
if [ -n "${OWNER_EMAIL:-}" ] && [ -n "${OWNER_PASSWORD:-}" ]; then
echo "==> Full acceptance run (platform/test/e2e-phase3b.js against staging)"
+14
View File
@@ -36,3 +36,17 @@ sed -e 's/{{ academy_domain }}/academy.example.com/g' -e 's/{{ management_cidr }
chmod a+r "$TMPDIR/Caddyfile"
docker run --rm -v "$TMPDIR:/etc/caddy:ro" caddy:2-alpine caddy validate --config /etc/caddy/Caddyfile 2>&1 | grep -q "Valid configuration" \
&& echo "OK: Caddyfile is valid" || { echo "FAIL: Caddyfile did not validate"; exit 1; }
# Phase 4.1: rn-lab-host-01's compose file (egress-proxy, lab-router, observability agents)
# was never covered here before — added when the lab-router service landed.
echo
echo "--- lab-host production compose ---"
LABHOST_TMPDIR="$(mktemp -d)"
sed \
-e 's/{{ registry }}/registry.example.com/g' \
-e 's/{{ image_tag }}/test/g' \
-e "s/{{ lab_router_port | default('8443') }}/8443/g" \
../lab-host/docker-compose.production.yml > "$LABHOST_TMPDIR/docker-compose.yml"
docker compose -f "$LABHOST_TMPDIR/docker-compose.yml" config --quiet
echo "OK: lab-host production compose file is well-formed YAML and passes docker compose's own schema validation"
rm -rf "$LABHOST_TMPDIR"
+15 -4
View File
@@ -23,6 +23,10 @@ entrypoint/Dockerfile unmodified; frontend bundle gained Phase 3B learning UX, s
`docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md` (routing), `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md`
(persistent sessions, CSRF), `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` §3 (hint gating).
- `egress-proxy/` — shared allowlisting proxy every Lab's outbound traffic passes through.
- `lab-router/` — Phase 4.1 addition. Runs on the Lab Host VM (or `dev-local`'s stand-in in
local dev, though `direct` mode there never exercises it); resolves an opaque routing token
to a local Lab container purely from Docker state, over mTLS, so the Gateway never needs a
Docker-internal address. Design: `docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1-3.
- `docker-compose.dev.yml` — the full stack (Postgres, egress proxy, Provisioner, Platform API,
Gateway) on one machine. This is the Phase 1/2A/2B **local dev exception** — production is
the two-VM Proxmox split described in `docs/LAB_ARCHITECTURE.md` §1.
@@ -77,13 +81,20 @@ those need a browser-equivalent client and aren't automated here.
See `provisioner/README.md` (Phase 1), `docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md` §9 (Phase
2A), `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md` (Phase 3A identity limitations), and
`docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` §20 (Phase 3B learning/assessment limitations) for the
full, honest lists.
full, honest lists. Phase 3B's "no automatic abandoned-attempt detection" and "no per-cohort
needs-attention threshold configuration" limitations were both closed in Phase 4.1 (§13/§17 of
`docs/PHASE4_1_PREPRODUCTION_HARDENING.md`); its raw-JSON pool_spec limitation was replaced with
a structured editor (§18) with an owner-only raw-JSON fallback still available. Phase 4.1's own
known limitation: real remote/self-paced delivery needs student email/password authentication,
not built — see `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`.
## Running this in production
`../infra/` has the full production infrastructure package (Ansible, Terraform, firewall,
monitoring, backup, CI/CD) for deploying the Platform API/Gateway/Provisioner/Postgres onto a
dedicated `rn-platform-01` VM, with student Labs on a separate `rn-lab-host-01` VM. See
dedicated `rn-platform-01` VM, with student Labs — and a new `lab-router` service
(`../platform/lab-router/`) — on a separate `rn-lab-host-01` VM. See
`../docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` for the overview and
`../docs/PRODUCTION_NETWORK_SECURITY.md` for a known gap in Gateway-to-Lab reachability across
two real VMs that must be closed before production go-live.
`../docs/PHASE4_1_PREPRODUCTION_HARDENING.md` for how Gateway-to-Lab reachability across two
real VMs was closed (a Lab Host Router + mTLS data plane, simulation-tested in
`../infra/scripts/simulate-two-host/`) and for the rest of Phase 4.1's pre-production hardening.
@@ -0,0 +1,20 @@
-- Abandoned/interrupted attempt handling — Phase 4.1, docs/PHASE4_1_PREPRODUCTION_HARDENING.md
-- §13. 'abandoned' was already a valid scenario_attempts.status value (migration 012's CHECK
-- constraint) but nothing ever set it — an attempt left open when a Lab was reaped, an
-- environment destroyed, a session ended, or a student moved on to a different scenario just
-- stayed 'in_progress' forever, an honest but incomplete signal. This is NOT an academic
-- failure marker — an abandoned attempt has no `result`, doesn't count against the student,
-- and the row is never deleted (see src/attempts.js abandonAttempt for the enforcement of
-- that). It exists so instructor views and needs-attention logic can tell "still working" from
-- "stopped, for a reason we know."
ALTER TABLE scenario_attempts ADD COLUMN IF NOT EXISTS abandoned_at TIMESTAMPTZ;
ALTER TABLE scenario_attempts ADD COLUMN IF NOT EXISTS abandoned_reason TEXT;
DO $$ BEGIN
ALTER TABLE scenario_attempts ADD CONSTRAINT scenario_attempts_abandoned_reason_check
CHECK (abandoned_reason IS NULL OR abandoned_reason IN
('environment_destroyed', 'session_ended', 'scenario_switched'));
EXCEPTION
WHEN duplicate_object THEN NULL;
END $$;
@@ -0,0 +1,15 @@
-- Postgres-backed login rate limiting — Phase 4.1 §15. Replaces the in-memory sliding-window
-- Map in security.js, which reset on every Platform API restart (an attacker could always just
-- wait for a deploy) and would silently stop being shared the moment there's more than one
-- Platform API instance. One small table, one cheap query per login attempt (this is a login
-- form, not a hot path) — no Redis, no new infrastructure dependency.
CREATE TABLE IF NOT EXISTS login_attempts (
id BIGSERIAL PRIMARY KEY,
rate_key TEXT NOT NULL, -- "ip:email", same composite key security.js always used
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- The hot query is "count attempts for this key within the last N minutes" — this index makes
-- that a range scan on one key instead of a table scan.
CREATE INDEX IF NOT EXISTS idx_login_attempts_key_time ON login_attempts(rate_key, created_at);
+44 -10
View File
@@ -7,9 +7,13 @@
// can do, per the Phase 3B RBAC matrix.
const { computeCurriculumState } = require('./curriculum-rules');
const { parsePagination, paginatedResponse } = require('./pagination'); // Phase 4.1 §16
const NEEDS_ATTENTION_FAIL_THRESHOLD = 3; // repeated validation failures on one scenario
const NEEDS_ATTENTION_STALE_DAYS = 7; // no progress in this many days
// Phase 4.1 §17 — configurable via env (environment-backed policy, matching the pattern
// sessions.js already uses for TTLs), not a database-backed settings table — that's more
// machinery than two numbers need. Sensible defaults if unset.
const NEEDS_ATTENTION_FAIL_THRESHOLD = Number(process.env.NEEDS_ATTENTION_FAIL_THRESHOLD || 3); // repeated validation failures on one scenario
const NEEDS_ATTENTION_STALE_DAYS = Number(process.env.NEEDS_ATTENTION_STALE_DAYS || 7); // no progress in this many days
module.exports = function registerAdminLearningRoutes(app, deps) {
const { instructorSession, rbac, curriculum, getPool, loadProgress, recordAudit } = deps;
@@ -70,9 +74,13 @@ module.exports = function registerAdminLearningRoutes(app, deps) {
FROM scenario_attempts WHERE student_id = $1 ORDER BY started_at DESC NULLS LAST LIMIT 100`,
[req.params.id]
);
// Phase 4.1 §16 — this is a per-student summary embedded in a wider dashboard view, not a
// standalone history list (that's /api/admin/assessments/:id/sessions, which got real
// pagination above) — a bounded LIMIT is the appropriate fix here, not a second paginator
// inside an already-composite response.
const { rows: assessmentSessions } = await pool.query(
`SELECT assessment_session_id, assessment_definition_id, type, title, status, started_at, submitted_at, result
FROM assessment_sessions WHERE student_id = $1 ORDER BY created_at DESC`,
FROM assessment_sessions WHERE student_id = $1 ORDER BY created_at DESC LIMIT 50`,
[req.params.id]
);
res.json({
@@ -152,6 +160,21 @@ module.exports = function registerAdminLearningRoutes(app, deps) {
res.json(signals);
});
// Phase 4.1 §18 — the scenario catalog (stage -> scenario id/title, no student-specific
// status) the structured assessment editor picks from, instead of an instructor hand-typing
// scenario ids into raw JSON. Same source data as the per-student progress view
// (admin-routes.js's /api/admin/students/:id/progress), just without the progress overlay.
admin.get('/api/admin/scenario-catalog', requireAnyInstructor, async (req, res) => {
const path = curriculum.loadCurriculumPath();
const scenarios = curriculum.loadScenarios();
if (!path) return res.json({ stages: [] });
const stages = path.stages.map((stage) => ({
id: stage.id, name: stage.name,
scenarios: stage.scenario_ids.map((sid) => ({ id: sid, title: scenarios.find((s) => s.id === sid)?.title || sid })),
}));
res.json({ stages });
});
// ── Assessment definitions ───────────────────────────────────────────────────────────────
admin.get('/api/admin/assessments', requireAnyInstructor, async (req, res) => {
const { rows } = await getPool().query(`SELECT * FROM assessment_definitions ORDER BY type, title`);
@@ -231,21 +254,32 @@ module.exports = function registerAdminLearningRoutes(app, deps) {
res.json({ ok: true });
});
// Phase 4.1 §16 — was fully unbounded (no LIMIT at all — the one endpoint here that could
// genuinely grow into thousands of rows over a semester of exam sessions). Now real
// pagination, same shape as audit-events/student-attempts.
admin.get('/api/admin/assessments/:id/sessions', requireAnyInstructor, async (req, res) => {
const pool = getPool();
const pg = parsePagination(req);
let scopeFilter = '';
const params = [req.params.id];
if (req.instructor.role === 'instructor') {
scopeFilter = `AND s.cohort_id IN (SELECT cohort_id FROM cohort_instructors WHERE instructor_id = $2)`;
params.push(req.instructor.instructor_id);
}
const { rows } = await pool.query(
`SELECT s.assessment_session_id, s.student_id, st.display_name, s.status, s.started_at, s.submitted_at, s.expires_at, s.result
FROM assessment_sessions s JOIN students st ON st.student_id = s.student_id
WHERE s.assessment_definition_id = $1 ${scopeFilter} ORDER BY s.created_at DESC`,
params
);
res.json(rows);
const pageParams = [...params, pg.pageSize, pg.offset];
const [{ rows }, { rows: countRows }] = await Promise.all([
pool.query(
`SELECT s.assessment_session_id, s.student_id, st.display_name, s.status, s.started_at, s.submitted_at, s.expires_at, s.result
FROM assessment_sessions s JOIN students st ON st.student_id = s.student_id
WHERE s.assessment_definition_id = $1 ${scopeFilter} ORDER BY s.created_at DESC LIMIT $${params.length + 1} OFFSET $${params.length + 2}`,
pageParams
),
pool.query(
`SELECT count(*)::int AS n FROM assessment_sessions s WHERE s.assessment_definition_id = $1 ${scopeFilter}`,
params
),
]);
res.json(paginatedResponse(rows, countRows[0].n, pg));
});
admin.post('/api/admin/assessment-sessions/:sessionId/cancel', requireAnyInstructor, async (req, res) => {
+42 -10
View File
@@ -14,10 +14,15 @@
// destroy, events) — the Admin UI never talks to the Provisioner directly.
const { listInstructors, createInstructor, assertNotLastOwner } = require('./instructors');
const attemptsLib = require('./attempts'); // Phase 4.1 §13 — abandonment on instructor-driven reset/destroy
const { parsePagination, paginatedResponse } = require('./pagination'); // Phase 4.1 §16
module.exports = function registerAdminRoutes(app, deps) {
const { instructorSession, rbac, provisioner, curriculum, getPool, loadProgress, recordAudit } = deps;
const { requireOwner, requireAdminOrOwner, requireAnyInstructor, requireCohortAccess, requireStudentCohortAccess } = rbac;
const {
requireOwner, requireAdminOrOwner, requireAnyInstructor, requireCohortAccess, requireStudentCohortAccess,
requireCohortWriteAccess, requireStudentCohortWriteAccess,
} = rbac;
// Tiny helper: every admin.<verb> call goes through instructorSession first, then whatever
// role/scope middleware is passed, then the handler (last function in the list — auto-wrapped
@@ -71,6 +76,9 @@ module.exports = function registerAdminRoutes(app, deps) {
`SELECT student_id, lab_host_id, status, error_reason, started_at
FROM environments WHERE status = 'error' ORDER BY started_at DESC NULLS LAST LIMIT 10`
);
// Phase 4.1 §26 — best-effort: a Provisioner hiccup here shouldn't break the whole
// dashboard, capacity is just omitted (null) rather than the request failing.
const capacity = await provisioner.capacity().then((r) => r.status === 200 ? r.body : null).catch(() => null);
res.json({
active_cohorts: cohorts.rows[0].n,
total_students: students.rows[0].n,
@@ -80,6 +88,7 @@ module.exports = function registerAdminRoutes(app, deps) {
error: envCounts.error || 0,
idle_or_destroyed: envCounts.destroyed || 0,
},
capacity, // { current, max } — max is null when admission control is unconfigured (see resource-policy.yaml)
recent_completions: recentCompletions.rows,
recent_failures: scoped ? [] : recentFailures.rows, // env-level failures aren't cohort-scoped data; keep them admin/owner-only
});
@@ -143,11 +152,15 @@ module.exports = function registerAdminRoutes(app, deps) {
res.json({ stages });
});
// Phase 4.1 §16 — was a flat LIMIT 100; now real pagination (see pagination.js).
admin.get('/api/admin/students/:id/attempts', requireAnyInstructor, requireStudentCohortAccess('id'), async (req, res) => {
const { rows } = await getPool().query(
`SELECT * FROM scenario_attempts WHERE student_id = $1 ORDER BY created_at DESC LIMIT 100`, [req.params.id]
);
res.json(rows);
const pool = getPool();
const pg = parsePagination(req);
const [{ rows }, { rows: countRows }] = await Promise.all([
pool.query(`SELECT * FROM scenario_attempts WHERE student_id = $1 ORDER BY created_at DESC LIMIT $2 OFFSET $3`, [req.params.id, pg.pageSize, pg.offset]),
pool.query(`SELECT count(*)::int AS n FROM scenario_attempts WHERE student_id = $1`, [req.params.id]),
]);
res.json(paginatedResponse(rows, countRows[0].n, pg));
});
admin.get('/api/admin/students/:id/environment', requireAnyInstructor, requireStudentCohortAccess('id'), async (req, res) => {
@@ -156,7 +169,7 @@ module.exports = function registerAdminRoutes(app, deps) {
res.json(body);
});
admin.post('/api/admin/students/:id/environment/provision', requireAnyInstructor, requireStudentCohortAccess('id'), async (req, res) => {
admin.post('/api/admin/students/:id/environment/provision', requireAnyInstructor, requireStudentCohortWriteAccess('id'), async (req, res) => {
const { status, body } = await provisioner.provision(req.params.id);
res.status(status === 409 ? 200 : status).json(body);
});
@@ -236,6 +249,12 @@ module.exports = function registerAdminRoutes(app, deps) {
admin.post('/api/admin/cohorts/:id/instructors', requireAdminOrOwner, async (req, res) => {
const { instructor_id, role_in_cohort } = req.body || {};
if (!instructor_id) return res.status(400).json({ error: 'instructor_id required' });
// Phase 4.1 §19 — role_in_cohort is now actually enforced (rbac.js's
// requireCohortWriteAccess/requireStudentCohortWriteAccess), so it's validated here rather
// than accepting any string that would silently never match either check.
if (role_in_cohort && !['teacher', 'viewer'].includes(role_in_cohort)) {
return res.status(400).json({ error: 'role_in_cohort must be teacher or viewer' });
}
await getPool().query(
`INSERT INTO cohort_instructors (cohort_id, instructor_id, role_in_cohort) VALUES ($1, $2, $3)
ON CONFLICT (cohort_id, instructor_id) DO UPDATE SET role_in_cohort = EXCLUDED.role_in_cohort`,
@@ -286,7 +305,7 @@ module.exports = function registerAdminRoutes(app, deps) {
});
// ── Classroom prep: Prepare Labs / Destroy Cohort Labs ──────────────────────────────────
admin.post('/api/admin/cohorts/:id/prepare-labs', requireAnyInstructor, requireCohortAccess('id'), async (req, res) => {
admin.post('/api/admin/cohorts/:id/prepare-labs', requireAnyInstructor, requireCohortWriteAccess('id'), async (req, res) => {
const { rows: students } = await getPool().query(
`SELECT s.student_id, s.display_name FROM cohort_enrollments e JOIN students s ON s.student_id = e.student_id
WHERE e.cohort_id = $1 AND e.enrollment_status = 'enrolled' AND s.status = 'active'`,
@@ -306,7 +325,7 @@ module.exports = function registerAdminRoutes(app, deps) {
});
});
admin.post('/api/admin/cohorts/:id/destroy-labs', requireAnyInstructor, requireCohortAccess('id'), async (req, res) => {
admin.post('/api/admin/cohorts/:id/destroy-labs', requireAnyInstructor, requireCohortWriteAccess('id'), async (req, res) => {
const { rows: students } = await getPool().query(
`SELECT student_id FROM cohort_enrollments WHERE cohort_id = $1 AND enrollment_status = 'enrolled'`,
[req.params.id]
@@ -315,6 +334,9 @@ module.exports = function registerAdminRoutes(app, deps) {
const { body } = await provisioner.listForStudent(s.student_id);
const active = body?.find((e) => e.status !== 'destroyed');
if (!active) return { student_id: s.student_id, status: 'none' };
// Phase 4.1 §13 — before the destroy, so any still-open attempt is honestly abandoned
// rather than left dangling once its environment no longer exists.
await attemptsLib.abandonOpenAttemptsForEnvironment(getPool(), active.environment_id);
await provisioner.destroy(active.environment_id);
return { student_id: s.student_id, status: 'destroyed' };
}));
@@ -328,12 +350,14 @@ module.exports = function registerAdminRoutes(app, deps) {
// resolving a bare environment id back to a cohort would need an extra Provisioner round
// trip on every call — a documented simplification, see docs/PHASE3A...) ───────────────────
admin.post('/api/admin/environments/:id/reset', requireAdminOrOwner, async (req, res) => {
await attemptsLib.abandonOpenAttemptsForEnvironment(getPool(), req.params.id); // Phase 4.1 §13
const { status, body } = await provisioner.reset(req.params.id);
await recordAudit(getPool(), { actorType: 'instructor', actorId: req.instructor.instructor_id, action: 'lab_reset', resourceType: 'environment', resourceId: req.params.id, result: status === 200 ? 'success' : 'failure', sourceIp: req.ip });
res.status(status).json(body);
});
admin.delete('/api/admin/environments/:id', requireAdminOrOwner, async (req, res) => {
await attemptsLib.abandonOpenAttemptsForEnvironment(getPool(), req.params.id); // Phase 4.1 §13
const { status, body } = await provisioner.destroy(req.params.id);
await recordAudit(getPool(), { actorType: 'instructor', actorId: req.instructor.instructor_id, action: 'lab_destroy', resourceType: 'environment', resourceId: req.params.id, result: status === 200 ? 'success' : 'failure', sourceIp: req.ip });
res.status(status).json(body);
@@ -423,8 +447,16 @@ module.exports = function registerAdminRoutes(app, deps) {
});
// ── Audit log — admin/owner ──────────────────────────────────────────────────────────────
// Phase 4.1 §16 — was a flat LIMIT 200 with no way to see anything older; now real
// server-side pagination (see pagination.js). Default page size (50) intentionally smaller
// than the old flat cap since this is now genuinely browsable instead of "the last 200 ever."
admin.get('/api/admin/audit-events', requireAdminOrOwner, async (req, res) => {
const { rows } = await getPool().query(`SELECT * FROM audit_events ORDER BY created_at DESC LIMIT 200`);
res.json(rows);
const pool = getPool();
const pg = parsePagination(req);
const [{ rows }, { rows: countRows }] = await Promise.all([
pool.query(`SELECT * FROM audit_events ORDER BY created_at DESC LIMIT $1 OFFSET $2`, [pg.pageSize, pg.offset]),
pool.query(`SELECT count(*)::int AS n FROM audit_events`),
]);
res.json(paginatedResponse(rows, countRows[0].n, pg));
});
};
+64 -6
View File
@@ -2,11 +2,55 @@
// session on a scenario (not per validation click, see migrations/012). Every mutation here
// updates that one row in place; nothing is ever inserted twice for the same open attempt.
//
// Deliberately does NOT try to detect "abandoned" attempts automatically (e.g. on Lab destroy)
// — that would need a reaper-style background job wired to Provisioner environment_events,
// which is more machinery than the instructional-visibility goal here needs. An attempt just
// stays 'in_progress' if the student never returns to finish it; instructors can already see
// "in_progress, N failed validations" as a real signal. Documented as a known limitation.
// Phase 4.1 (docs/PHASE4_1_PREPRODUCTION_HARDENING.md §13) added real abandonment handling —
// see abandonAttempt/abandonOpenAttemptsFor* below and their call sites (environment
// destroy/reset in server.js and admin-routes.js, logout in server.js, the session sweeper in
// sessions.js, and the "different scenario" transition inline in getOrCreateOpenAttempt just
// below). Still deliberately NOT a new standalone background job — every trigger either rides
// an existing request handler or the session sweeper that already runs periodically.
// An attempt is never silently deleted or marked as a failure by abandonment — result stays
// NULL, the row stays queryable, instructors see "abandoned (reason)" as a real, honest status
// rather than a punitive one. No-op (and does not overwrite an existing abandoned_reason) if
// the attempt is already terminal.
async function abandonAttempt(pool, attemptId, reason) {
const { rows } = await pool.query(
`UPDATE scenario_attempts SET status = 'abandoned', abandoned_at = now(), abandoned_reason = $2
WHERE attempt_id = $1 AND status = 'in_progress' RETURNING *`,
[attemptId, reason]
);
return rows[0] || null;
}
// Trigger: environment destroyed (idle reaper, student reset, admin reset/destroy — see
// server.js's student reset route and admin-routes.js's reset/destroy routes, all of which
// call this right after telling the Provisioner to tear the environment down). A reset creates
// a fresh environment_id in some flows and reuses the same one in others (see
// provisioner lifecycle.js's reset — same environment_id, fresh container) — either way, any
// attempt still open against the OLD container state is honestly "not going to be resumed as
// the same working session," so it's abandoned rather than left open indefinitely.
async function abandonOpenAttemptsForEnvironment(pool, environmentId) {
const { rows } = await pool.query(
`UPDATE scenario_attempts SET status = 'abandoned', abandoned_at = now(), abandoned_reason = 'environment_destroyed'
WHERE environment_id = $1 AND status = 'in_progress' RETURNING attempt_id`,
[environmentId]
);
return rows.length;
}
// Trigger: logout (server.js POST /api/logout) and the session sweeper's periodic pass for
// students whose sessions have all expired (sessions.js). Deliberately student-scoped, not
// environment-scoped — a student can log out while their Lab keeps running (it isn't
// destroyed until reaped or explicitly ended), but the WORKING SESSION on whatever scenario
// they had open is over the moment they're no longer authenticated to record more progress on it.
async function abandonOpenAttemptsForStudent(pool, studentId) {
const { rows } = await pool.query(
`UPDATE scenario_attempts SET status = 'abandoned', abandoned_at = now(), abandoned_reason = 'session_ended'
WHERE student_id = $1 AND status = 'in_progress' RETURNING attempt_id`,
[studentId]
);
return rows.length;
}
async function getOrCreateOpenAttempt(pool, { studentId, scenarioId, environmentId, cohortId, curriculumStage, assessmentSessionId }) {
const { rows: existing } = await pool.query(
@@ -17,6 +61,17 @@ async function getOrCreateOpenAttempt(pool, { studentId, scenarioId, environment
);
if (existing[0]) return existing[0];
// Trigger: student starts a different scenario while a previous one is still open (started
// via /start or implicitly via /validate, never finished). Only one scenario is "the one
// currently being worked on" per student+environment at a time from the app's own UX, so an
// open attempt on ANY OTHER scenario in this same environment is abandoned before the new
// one is created — never silently left as a phantom "still in progress" row.
await pool.query(
`UPDATE scenario_attempts SET status = 'abandoned', abandoned_at = now(), abandoned_reason = 'scenario_switched'
WHERE student_id = $1 AND environment_id = $2 AND scenario_id != $3 AND status = 'in_progress'`,
[studentId, environmentId, scenarioId]
);
const { rows } = await pool.query(
`INSERT INTO scenario_attempts
(student_id, scenario_id, environment_id, cohort_id, curriculum_stage, started_at, status, assessment_session_id)
@@ -80,4 +135,7 @@ async function getAttemptSummary(pool, { studentId, scenarioId, environmentId })
return rows[0] || null;
}
module.exports = { getOrCreateOpenAttempt, recordValidation, recordHintReveal, getAttemptSummary };
module.exports = {
getOrCreateOpenAttempt, recordValidation, recordHintReveal, getAttemptSummary,
abandonAttempt, abandonOpenAttemptsForEnvironment, abandonOpenAttemptsForStudent,
};
+1 -1
View File
@@ -18,7 +18,7 @@ module.exports = function registerInstructorAuthRoutes(app, { pool, sessions, in
app.post('/api/auth/instructor/login', async (req, res) => {
const { email, password } = req.body || {};
const rateKey = `${req.ip}:${String(email || '').toLowerCase()}`;
if (tooManyAttempts(rateKey, LOGIN_MAX_ATTEMPTS, LOGIN_WINDOW_MS)) {
if (await tooManyAttempts(pool, rateKey, LOGIN_MAX_ATTEMPTS, LOGIN_WINDOW_MS)) {
return res.status(429).json({ error: 'too_many_attempts', message: 'Too many login attempts. Try again in a few minutes.' });
}
if (!email || !password) return res.status(400).json({ error: 'email and password are required' });
+22
View File
@@ -0,0 +1,22 @@
// Stable server-side pagination — Phase 4.1 §16. Shared by every list endpoint that could
// otherwise grow unbounded (audit log, attempt histories, assessment session history, scenario
// analytics). Simple offset/limit, not cursor-based — this admin/instructor scale (hundreds to
// low thousands of rows, not a high-frequency infinite-scroll feed) doesn't need more than
// that, and offset pagination is what every other query in this codebase already reads like.
const DEFAULT_PAGE_SIZE = 50;
const MAX_PAGE_SIZE = 200;
// Reads ?page=&page_size= from a request, clamped to sane bounds — never trusts the client for
// an unbounded page_size (that would defeat the entire point).
function parsePagination(req) {
const page = Math.max(1, parseInt(req.query.page, 10) || 1);
const pageSize = Math.min(MAX_PAGE_SIZE, Math.max(1, parseInt(req.query.page_size, 10) || DEFAULT_PAGE_SIZE));
return { page, pageSize, offset: (page - 1) * pageSize };
}
function paginatedResponse(rows, total, { page, pageSize }) {
return { items: rows, page, page_size: pageSize, total, total_pages: Math.max(1, Math.ceil(total / pageSize)) };
}
module.exports = { parsePagination, paginatedResponse, DEFAULT_PAGE_SIZE, MAX_PAGE_SIZE };
+1
View File
@@ -33,6 +33,7 @@ const provisioner = {
reset: (id) => call('POST', `/environments/${encodeURIComponent(id)}/reset`),
destroy: (id) => call('DELETE', `/environments/${encodeURIComponent(id)}`),
events: (id) => call('GET', `/environments/${encodeURIComponent(id)}/events`),
capacity: () => call('GET', '/capacity'), // Phase 4.1 §26 — instructor dashboard capacity state
};
module.exports = { provisioner };
+57 -1
View File
@@ -78,5 +78,61 @@ module.exports = function createRbac(pool) {
};
}
return { requireRole, requireOwner, requireAdminOrOwner, requireAnyInstructor, requireCohortAccess, requireStudentCohortAccess };
// Phase 4.1 §19 — cohort_instructors.role_in_cohort existed since Phase 2B but was never
// read anywhere (a misleading field: it looked authoritative but did nothing). Minimal
// useful model: 'teacher' (default, existing full instructor capability within the cohort)
// vs 'viewer' (read-only — sees dashboards/progress/analytics via requireCohortAccess above,
// but cannot Prepare/Destroy Labs or provision a student's environment). owner/admin still
// bypass entirely, same as every other cohort-scoped check.
function requireCohortWriteAccess(cohortIdParam = 'id') {
return async (req, res, next) => {
const instructor = req.instructor;
if (['owner', 'admin'].includes(instructor.role)) return next();
const cohortId = req.params[cohortIdParam];
const { rows } = await pool.query(
`SELECT role_in_cohort FROM cohort_instructors WHERE cohort_id = $1 AND instructor_id = $2`,
[cohortId, instructor.instructor_id]
);
if (!rows.length || rows[0].role_in_cohort === 'viewer') {
await recordAudit(pool, {
actorType: 'instructor', actorId: instructor.instructor_id, action: 'authz_denied',
resourceType: 'cohort', resourceId: cohortId, result: 'failure', sourceIp: req.ip,
metadata: rows.length ? { reason: 'viewer_role' } : undefined,
});
return res.status(403).json({ error: 'forbidden' });
}
next();
};
}
// Same, keyed by student_id — resolves through cohort_enrollments the same way
// requireStudentCohortAccess does, then applies the same viewer check.
function requireStudentCohortWriteAccess(studentIdParam = 'id') {
return async (req, res, next) => {
const instructor = req.instructor;
if (['owner', 'admin'].includes(instructor.role)) return next();
const studentId = req.params[studentIdParam];
const { rows } = await pool.query(
`SELECT ci.role_in_cohort FROM cohort_enrollments e
JOIN cohort_instructors ci ON ci.cohort_id = e.cohort_id
WHERE e.student_id = $1 AND e.enrollment_status = 'enrolled' AND ci.instructor_id = $2`,
[studentId, instructor.instructor_id]
);
if (!rows.length || rows.every((r) => r.role_in_cohort === 'viewer')) {
await recordAudit(pool, {
actorType: 'instructor', actorId: instructor.instructor_id, action: 'authz_denied',
resourceType: 'student', resourceId: studentId, result: 'failure', sourceIp: req.ip,
metadata: rows.length ? { reason: 'viewer_role' } : undefined,
});
return res.status(403).json({ error: 'forbidden' });
}
next();
};
}
return {
requireRole, requireOwner, requireAdminOrOwner, requireAnyInstructor,
requireCohortAccess, requireStudentCohortAccess,
requireCohortWriteAccess, requireStudentCohortWriteAccess,
};
};
+28 -19
View File
@@ -33,25 +33,34 @@ function checkCsrf(req, cookieName) {
}
// ── Login rate limiting ─────────────────────────────────────────────────────────────────
// In-memory sliding window, keyed by "ip:identifier" (e.g. ip:email). Fine for a single-process
// Platform API — a shared store (Postgres/Redis) would be needed for multi-instance, not a
// concern at this scale (see docs/PHASE3A... known limitations).
const attempts = new Map();
function tooManyAttempts(key, max, windowMs) {
const now = Date.now();
const arr = (attempts.get(key) || []).filter((t) => now - t < windowMs);
arr.push(now);
attempts.set(key, arr);
return arr.length > max;
// Phase 4.1 §15: Postgres-backed (migrations/016_login_attempts.sql), not in-memory — the
// original in-memory sliding-window Map reset on every Platform API restart (an attacker could
// just wait for a deploy) and would silently stop being shared the moment there's more than
// one Platform API instance. This table is the fix for both, at essentially no cost: login is
// not a hot path, and the query below is a single indexed range scan.
//
// Deliberately still just a table + two queries, not a new rate-limiting service or Redis —
// the volume here (failed login attempts) never justifies that, and a Postgres row is already
// exactly as "shared/persistent" as this needs to be.
async function tooManyAttempts(pool, key, max, windowMs) {
// Insert THEN count (including the just-inserted row), matching the original in-memory
// version's semantics exactly: this call itself counts as one of the attempts.
await pool.query(`INSERT INTO login_attempts (rate_key) VALUES ($1)`, [key]);
const { rows } = await pool.query(
`SELECT count(*)::int AS n FROM login_attempts WHERE rate_key = $1 AND created_at > now() - ($2 || ' milliseconds')::interval`,
[key, windowMs]
);
return rows[0].n > max;
}
setInterval(() => {
const cutoff = Date.now() - 60 * 60 * 1000;
for (const [key, arr] of attempts) {
const kept = arr.filter((t) => t > cutoff);
if (kept.length) attempts.set(key, kept); else attempts.delete(key);
}
}, 15 * 60 * 1000).unref();
// Periodic cleanup so the table doesn't grow forever — cosmetic/storage only, mirrors
// sessions.js's own sweeper pattern rather than inventing a new one.
function startLoginAttemptSweeper(pool) {
setInterval(() => {
pool.query(`DELETE FROM login_attempts WHERE created_at < now() - interval '24 hours'`).catch((e) => {
console.error('[login attempt sweep] failed:', e.message);
});
}, 60 * 60 * 1000).unref();
}
module.exports = { makeCsrfToken, parseCookies, checkCsrf, tooManyAttempts };
module.exports = { makeCsrfToken, parseCookies, checkCsrf, tooManyAttempts, startLoginAttemptSweeper };
+38 -1
View File
@@ -16,7 +16,7 @@ const { provisioner } = require('./provisioner-client');
const curriculum = require('./curriculum');
const { loadProgress, saveOneProgress, touchStudent } = require('./progress');
const { recordAudit } = require('./audit');
const { parseCookies, checkCsrf, makeCsrfToken } = require('./security');
const { parseCookies, checkCsrf, makeCsrfToken, startLoginAttemptSweeper } = require('./security');
const { buildCookie, clearCookie } = require('./cookies');
const createRbac = require('./rbac');
const { INSTRUCTOR_COOKIE, CSRF_COOKIE: INSTRUCTOR_CSRF_COOKIE } = require('./instructor-auth-routes');
@@ -88,6 +88,26 @@ app.get('/healthz', async (req, res) => {
}
});
// GET /api/version — Phase 4.1 §25. Unauthenticated like /healthz, and deliberately just as
// minimal: an app version, the deployed Git SHA (or 'dev' locally — see
// infra/docker/platform/docker-compose.production.yml's GIT_SHA env var, set to the same
// value used for the image tag), and the most recently applied migration as a stand-in for
// schema version. No infrastructure detail, no secrets — this is "what build is this," not a
// diagnostics endpoint.
const PACKAGE_VERSION = require('../package.json').version;
app.get('/api/version', async (req, res) => {
let schemaVersion = null;
try {
const { rows } = await getPool().query('SELECT version FROM schema_migrations ORDER BY applied_at DESC LIMIT 1');
schemaVersion = rows[0]?.version || null;
} catch (e) { /* best-effort — an unreachable DB shouldn't make /api/version itself fail */ }
res.json({
version: PACKAGE_VERSION,
git_sha: process.env.GIT_SHA || 'dev',
schema_version: schemaVersion,
});
});
// ═══════════════════════════════════════════════════════════════════════════════════════
// STUDENT-FACING: roster / login / session
// ═══════════════════════════════════════════════════════════════════════════════════════
@@ -136,7 +156,12 @@ app.post('/api/login', async (req, res) => {
app.post('/api/logout', async (req, res) => {
if (!checkCsrf(req, STUDENT_CSRF_COOKIE)) return res.status(403).json({ error: 'csrf' });
const token = parseCookies(req)[STUDENT_COOKIE];
// Resolved BEFORE destroying, purely to know who to abandon open attempts for (Phase 4.1
// §13) — deliberately not behind studentSession middleware so logout still succeeds
// gracefully against an already-invalid/expired token, same as before this phase.
const studentId = await sessions.resolveStudentSession(getPool(), token);
await sessions.destroyStudentSession(getPool(), token);
if (studentId) await attemptsLib.abandonOpenAttemptsForStudent(getPool(), studentId);
res.append('Set-Cookie', clearCookie(STUDENT_COOKIE));
res.append('Set-Cookie', clearCookie(STUDENT_CSRF_COOKIE));
res.json({ ok: true });
@@ -268,6 +293,12 @@ app.post('/api/environment/launch', wrap(studentSession), async (req, res) => {
const retry = await provisioner.listForStudent(req.studentId);
return res.json(await publicEnvView(retry.body?.[0]));
}
// Phase 4.1 §26 — relayed as its own clean status/message, not folded into the generic
// provision_failed 502 below. The student sees "temporarily full, try again shortly," never
// a raw 500/502 — this is expected operational backpressure, not an application error.
if (status === 503 && body?.code === 'CAPACITY_EXCEEDED') {
return res.status(503).json({ error: 'capacity_exceeded', message: 'Lab capacity is temporarily full. Try again shortly.' });
}
if (status !== 201) return res.status(502).json({ error: 'provision_failed', message: body?.error });
res.json(await publicEnvView({ environment_id: body.environment_id, status: body.status, routing_token: body.routing_token }));
});
@@ -276,6 +307,11 @@ app.post('/api/environment/reset', wrap(studentSession), async (req, res) => {
const existing = await provisioner.listForStudent(req.studentId);
const env = existing.body?.find((e) => e.status !== 'destroyed');
if (!env) return res.status(404).json({ error: 'no_environment' });
// Before tearing the old container down — Phase 4.1 §13, see attempts.js's
// abandonOpenAttemptsForEnvironment. The reset reuses the same environment_id (fresh
// container, same DB row — provisioner lifecycle.js), so this must run BEFORE reset, not
// after, or the abandonment would incorrectly apply to attempts against the NEW container.
await attemptsLib.abandonOpenAttemptsForEnvironment(getPool(), env.environment_id);
const { status, body } = await provisioner.reset(env.environment_id);
if (status !== 200) return res.status(502).json({ error: 'reset_failed', message: body?.error });
res.json(await publicEnvView(body));
@@ -322,6 +358,7 @@ async function main() {
const applied = await runMigrations();
if (applied.length) console.log('Applied migrations:', applied.join(', '));
sessions.startSessionSweeper(getPool());
startLoginAttemptSweeper(getPool()); // Phase 4.1 §15
app.listen(PORT, () => console.log(`Platform API listening on :${PORT}`));
}
+17
View File
@@ -104,11 +104,28 @@ async function listInstructorSessions(pool, instructorId) {
// Periodic cleanup of long-dead rows so the table doesn't grow forever — cosmetic/storage only,
// not a security control (expired/revoked rows already fail resolveSession's WHERE clause).
//
// Phase 4.1 §13 also piggybacks the "session expires" abandonment trigger onto this same
// existing interval, rather than adding a new background job: any scenario_attempts row still
// 'in_progress' for a student with NO live session left (every session expired, idle-timed-out,
// or revoked, not just deleted by the sweep above) gets abandoned. Deliberately does the
// attempts update in the SAME pool.query call — one round trip, no separate job to keep in sync.
function startSessionSweeper(pool) {
setInterval(() => {
pool.query(`DELETE FROM sessions WHERE expires_at < now() - interval '7 days'`).catch((e) => {
console.error('[session sweep] failed:', e.message);
});
pool.query(`
UPDATE scenario_attempts SET status = 'abandoned', abandoned_at = now(), abandoned_reason = 'session_ended'
WHERE status = 'in_progress'
AND NOT EXISTS (
SELECT 1 FROM sessions
WHERE sessions.student_id = scenario_attempts.student_id
AND sessions.revoked_at IS NULL AND sessions.expires_at > now() AND sessions.idle_expires_at > now()
)
`).catch((e) => {
console.error('[session sweep] attempt abandonment failed:', e.message);
});
}, 60 * 60 * 1000).unref();
}
+128
View File
@@ -0,0 +1,128 @@
// Phase 4.1 — the ONE place the Gateway decides how to reach a student's Lab. Everything else
// in server.js just calls labTarget(env, token) and gets back a plain { url, headers, agent }
// it can hand to http-proxy or fetch, with zero knowledge of Docker, VMs, or lab_hosts. See
// docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1 and docs/PRODUCTION_NETWORK_SECURITY.md.
//
// env.lab_host_routing_mode / env.lab_host_router_endpoint come from the Provisioner's
// by-token response (a join against lab_hosts — see provisioner lifecycle.js
// getEnvironmentByTokenWithRouting). Two modes:
//
// 'direct' (default, dev/single-host) — unchanged from every prior phase: the Gateway
// reaches env.internal_ip:internal_port itself, because Gateway and the Lab share a Docker
// daemon/network namespace.
//
// 'router' (production, two real VMs) — the Gateway never touches a Docker-internal
// address. It reaches that lab_host's Lab Host Router (platform/lab-router) instead, over
// mTLS, carrying the environment's routing_token in a header so the Router — which lives on
// the SAME host as the Lab and can resolve the token from local `docker ps` alone — knows
// which local container to proxy to.
const fs = require('fs');
const http = require('http');
const https = require('https');
const { URL } = require('url');
const ROUTE_TOKEN_HEADER = 'x-rootnode-route-token';
// mTLS client identity the Gateway presents to a Lab Host Router. One shared cert/key/ca is
// enough for Phase 4.1's single lab_host; if a second lab_host ever needs a distinct client
// identity, this becomes a map keyed by lab_host_id — not needed yet (see docs
// "multi-host-ready" note, no scheduler built here).
const TLS_CA = process.env.LAB_ROUTER_TLS_CA;
const TLS_CERT = process.env.LAB_ROUTER_TLS_CERT;
const TLS_KEY = process.env.LAB_ROUTER_TLS_KEY;
let _agent = null;
function routerAgent() {
if (_agent) return _agent;
if (!TLS_CA || !TLS_CERT || !TLS_KEY) {
throw new Error(
'A lab_host is configured with routing_mode=router but LAB_ROUTER_TLS_CA/CERT/KEY are not set — refusing to connect to the Lab Host Router unauthenticated.'
);
}
_agent = new https.Agent({
ca: fs.readFileSync(TLS_CA),
cert: fs.readFileSync(TLS_CERT),
key: fs.readFileSync(TLS_KEY),
// The Router's server cert is issued by the same private CA (infra/scripts/docker-tls/
// generate-docker-tls.sh) — never fall back to the public CA trust store for this.
});
return _agent;
}
// env: the (possibly-cached) environment row from the Provisioner's by-token response.
// token: the routing token the browser's URL already carried (Gateway never needs the
// Provisioner to hand it back — see provisioner server.js redactRow's comment on why
// routing_token is redacted everywhere except the two routes that legitimately need it,
// neither of which is this one).
function labTarget(env, token) {
if (env.lab_host_routing_mode === 'router') {
if (!env.lab_host_router_endpoint) {
throw new Error(`lab_host '${env.lab_host_id}' has routing_mode=router but no router_endpoint`);
}
return {
url: env.lab_host_router_endpoint,
headers: { [ROUTE_TOKEN_HEADER]: token },
agent: routerAgent(),
mode: 'router',
// The browser's original Host header (e.g. "localhost:4210" or the real public origin)
// has nothing to do with the Router's own TLS identity — without this, http-proxy's
// outgoing HTTPS connection inherits that unrelated Host header for SNI/hostname
// verification and the Router's cert (correctly issued for ITS OWN name) fails to
// match. changeOrigin makes http-proxy rewrite the outgoing Host header (and therefore
// the TLS servername) to match the actual target — found by testing against the real
// Phase 4.1 simulation harness, not theoretical.
changeOrigin: true,
};
}
// direct — unchanged behavior from every phase before this one.
return { url: `http://${env.internal_ip}:${env.internal_port}`, headers: {}, agent: undefined, mode: 'direct', changeOrigin: false };
}
// Used by the two Gateway call sites that need to make a one-shot request to the Lab and read
// its JSON body (progress/answer/validate relay, hint-count stripping) — everywhere else
// (the actual proxy paths) uses http-proxy directly with the same target/agent/headers.
//
// Deliberately built on node:http/node:https rather than global fetch: fetch's undici
// dispatcher isn't a stable, dependency-free way to attach a custom mTLS agent across Node
// versions, and http.request/https.request support client certs natively with zero new
// dependencies (stdlib covers it — see the ladder in CLAUDE.md's coding-style guidance).
// Mimics just enough of fetch's response shape ({ status, json() }) that call sites barely
// change.
function labRequest(env, token, path, { method = 'GET', headers = {}, body } = {}) {
const target = labTarget(env, token);
const url = new URL(path, target.url);
const transport = url.protocol === 'https:' ? https : http;
const payload = body === undefined ? undefined : (typeof body === 'string' ? body : JSON.stringify(body));
return new Promise((resolve, reject) => {
const req = transport.request(
url,
{
method,
agent: target.agent,
headers: {
...headers,
...target.headers,
...(payload !== undefined ? { 'Content-Length': Buffer.byteLength(payload) } : {}),
},
},
(res) => {
const chunks = [];
res.on('data', (c) => chunks.push(c));
res.on('end', () => {
const text = Buffer.concat(chunks).toString('utf8');
resolve({
status: res.statusCode,
json: async () => JSON.parse(text),
});
});
}
);
req.on('error', reject);
if (payload !== undefined) req.write(payload);
req.end();
});
}
module.exports = { labTarget, labRequest, ROUTE_TOKEN_HEADER };
+34 -15
View File
@@ -19,6 +19,11 @@ const express = require('express');
const httpProxy = require('http-proxy');
const { provisioner } = require('./provisioner-client'); // still used ONLY for the token->environment lookup a Lab request needs; all mutating calls (provision/reset/destroy) now live in the Platform API.
const envCache = require('./env-cache');
// Phase 4.1 — the ONE place that knows how to reach a Lab (direct Docker-internal address, or
// via that lab_host's Lab Host Router). Every proxy.web/proxy.ws/labRequest call below uses
// labTarget's output instead of building `http://${env.internal_ip}:${env.internal_port}`
// itself. See lab-endpoint.js's header comment and docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1.
const { labTarget, labRequest } = require('./lab-endpoint');
const PORT = Number(process.env.GATEWAY_PORT) || 4200;
const PLATFORM_API_URL = process.env.PLATFORM_API_URL || 'http://localhost:4300';
@@ -90,6 +95,9 @@ async function resolveByToken(token) {
return body;
}
// internal_ip is set by the Provisioner as soon as a container exists, regardless of routing
// mode (it's a Docker-internal fact, independent of how the Gateway will reach it) — so this
// check stays valid unchanged for both 'direct' and 'router' lab_hosts.
function isReachable(env) {
return env && env.status === 'running' && env.internal_ip;
}
@@ -116,9 +124,9 @@ async function checkOwnership(req, env) {
// ── Progress relay (unchanged mechanism from Phase 2A, now passes environment_id +
// validation_count through so Platform API can log a real scenario_attempts row) ──────────
async function relayScenarioOutcome({ env, req, res, scenarioPath, extractOutcome }) {
async function relayScenarioOutcome({ env, token, req, res, scenarioPath, extractOutcome }) {
if (!isReachable(env)) return sendLabError(res, 503, 'lab_unavailable', 'Your lab is not currently running.');
const labRes = await fetch(`http://${env.internal_ip}:${env.internal_port}${scenarioPath}`, {
const labRes = await labRequest(env, token, scenarioPath, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(req.body || {}),
@@ -157,7 +165,7 @@ app.post('/lab/:token/api/scenarios/:scenarioId/validate', express.json(), async
const own = await checkOwnership(req, env);
if (!own.ok) return sendLabError(res, own.reason === 'invalid_token' ? 404 : 403, own.reason, 'This lab is not available to you.');
await relayScenarioOutcome({
env, req, res, scenarioPath: `/api/scenarios/${req.params.scenarioId}/validate`,
env, token: req.params.token, req, res, scenarioPath: `/api/scenarios/${req.params.scenarioId}/validate`,
extractOutcome: (body) => (typeof body.passed === 'boolean' ? body.passed : null),
});
});
@@ -167,7 +175,7 @@ app.post('/lab/:token/api/scenarios/:scenarioId/answer', express.json(), async (
const own = await checkOwnership(req, env);
if (!own.ok) return sendLabError(res, own.reason === 'invalid_token' ? 404 : 403, own.reason, 'This lab is not available to you.');
await relayScenarioOutcome({
env, req, res, scenarioPath: `/api/scenarios/${req.params.scenarioId}/answer`,
env, token: req.params.token, req, res, scenarioPath: `/api/scenarios/${req.params.scenarioId}/answer`,
extractOutcome: (body) => (typeof body.correct === 'boolean' ? body.correct : null),
});
});
@@ -223,7 +231,7 @@ app.get('/lab/:token/api/scenarios/:scenarioId', async (req, res) => {
if (!own.ok) return sendLabError(res, own.reason === 'invalid_token' ? 404 : 403, own.reason, 'This lab is not available to you.');
if (!isReachable(env)) return sendLabError(res, 503, 'lab_unavailable', 'Your lab is not currently running.');
try {
const labRes = await fetch(`http://${env.internal_ip}:${env.internal_port}/api/scenarios/${req.params.scenarioId}`);
const labRes = await labRequest(env, req.params.token, `/api/scenarios/${req.params.scenarioId}`);
const body = await labRes.json().catch(() => ({}));
const { hints, ...withoutHints } = body;
res.status(labRes.status).json({ ...withoutHints, hint_count: Array.isArray(hints) ? hints.length : 0 });
@@ -259,7 +267,8 @@ app.use('/lab/:token', async (req, res, next) => {
// for this browser.
res.append('Set-Cookie', `${LAB_ACTIVE_COOKIE}=1; Path=/; HttpOnly; SameSite=Lax; Max-Age=28800`);
req.url = req.url.replace(/^\/lab\/[^/]+/, '') || '/';
proxy.web(req, res, { target: `http://${env.internal_ip}:${env.internal_port}` });
const target = labTarget(env, token);
proxy.web(req, res, { target: target.url, agent: target.agent, headers: target.headers, changeOrigin: target.changeOrigin });
});
// Resolves "the environment this browser's session currently owns and can reach," for the
@@ -268,20 +277,23 @@ app.use('/lab/:token', async (req, res, next) => {
// the WS upgrade handler so there's exactly one place this logic lives. Gated on
// LAB_ACTIVE_COOKIE first — see its definition above for why "has a running env" alone isn't
// a safe signal (it's true on the dashboard too).
// Returns { env, token } (not just env) — Phase 4.1's labTarget needs the routing token
// alongside the environment row for router-mode lab_hosts, and this is the only place that
// derives the token from launch_path rather than the URL, so it has to hand both back.
async function resolveOwnedRunningEnvironment(req) {
const cookies = parseCookies(req);
if (!cookies[LAB_ACTIVE_COOKIE]) return null;
const token = cookies[STUDENT_COOKIE];
if (!token) return null;
const sessionToken = cookies[STUDENT_COOKIE];
if (!sessionToken) return null;
const studentId = await resolveSessionStudentId(req);
if (!studentId) return null;
try {
const envRes = await fetch(`${PLATFORM_API_URL}/api/environment/status`, { headers: { Cookie: `${STUDENT_COOKIE}=${token}` } });
const envRes = await fetch(`${PLATFORM_API_URL}/api/environment/status`, { headers: { Cookie: `${STUDENT_COOKIE}=${sessionToken}` } });
const envInfo = await envRes.json().catch(() => null);
if (envInfo?.status !== 'running' || !envInfo.launch_path) return null;
const lookupToken = envInfo.launch_path.split('/')[2];
const env = await resolveByToken(lookupToken);
return isReachable(env) && env.student_id === studentId ? env : null;
return isReachable(env) && env.student_id === studentId ? { env, token: lookupToken } : null;
} catch {
return null;
}
@@ -303,8 +315,11 @@ function looksLikeLabPath(pathname) {
// or the Platform API (student dashboard, admin portal, roster/curriculum/progress) ────────
app.use(async (req, res) => {
if (looksLikeLabPath(req.path)) {
const env = await resolveOwnedRunningEnvironment(req);
if (env) return proxy.web(req, res, { target: `http://${env.internal_ip}:${env.internal_port}` });
const owned = await resolveOwnedRunningEnvironment(req);
if (owned) {
const target = labTarget(owned.env, owned.token);
return proxy.web(req, res, { target: target.url, agent: target.agent, headers: target.headers, changeOrigin: target.changeOrigin });
}
}
proxy.web(req, res, { target: PLATFORM_API_URL });
});
@@ -319,12 +334,16 @@ server.on('upgrade', async (req, socket, head) => {
const own = await checkOwnership(req, env);
if (!own.ok || !isReachable(env)) return socket.destroy();
req.url = rest || '/';
return proxy.ws(req, socket, head, { target: `http://${env.internal_ip}:${env.internal_port}` });
const target = labTarget(env, token);
return proxy.ws(req, socket, head, { target: target.url, agent: target.agent, headers: target.headers, changeOrigin: target.changeOrigin });
}
// Unprefixed WS (the existing Lab UI's /shell-ws, hit while "inside" a lab session).
const env = await resolveOwnedRunningEnvironment(req);
if (env) return proxy.ws(req, socket, head, { target: `http://${env.internal_ip}:${env.internal_port}` });
const owned = await resolveOwnedRunningEnvironment(req);
if (owned) {
const target = labTarget(owned.env, owned.token);
return proxy.ws(req, socket, head, { target: target.url, agent: target.agent, headers: target.headers, changeOrigin: target.changeOrigin });
}
socket.destroy();
});
+7
View File
@@ -0,0 +1,7 @@
FROM node:20-slim
WORKDIR /app
COPY package.json ./
RUN npm install --production
COPY src/ ./src/
EXPOSE 8443
CMD ["node", "src/server.js"]
+706
View File
@@ -0,0 +1,706 @@
{
"name": "rootnode-lab-router",
"version": "0.1.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "rootnode-lab-router",
"version": "0.1.0",
"dependencies": {
"dockerode": "^4.0.2",
"http-proxy": "^1.18.1"
}
},
"node_modules/@balena/dockerignore": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/@balena/dockerignore/-/dockerignore-1.0.2.tgz",
"integrity": "sha512-wMue2Sy4GAVTk6Ic4tJVcnfdau+gx2EnG7S+uAEe+TWJFqE4YoWN4/H8MSLj4eYJKxGg26lZwboEniNiNwZQ6Q=="
},
"node_modules/@grpc/grpc-js": {
"version": "1.14.4",
"resolved": "https://registry.npmjs.org/@grpc/grpc-js/-/grpc-js-1.14.4.tgz",
"integrity": "sha512-k9Dj3DV/itK9D06Y8f190Qgop7/Ui+D0njFV3LHMPwPT75DpXLQohE9Wmz0QElrJnzsjB7KPWiKJbOl7IPDArQ==",
"dependencies": {
"@grpc/proto-loader": "^0.8.0",
"@js-sdsl/ordered-map": "^4.4.2"
},
"engines": {
"node": ">=12.10.0"
}
},
"node_modules/@grpc/grpc-js/node_modules/@grpc/proto-loader": {
"version": "0.8.1",
"resolved": "https://registry.npmjs.org/@grpc/proto-loader/-/proto-loader-0.8.1.tgz",
"integrity": "sha512-wtF6h+DY6M3YaDBPAmvuuA6jV8Sif9MjtOI5euKFWRgCDl5PeDpPsHR9u2l6St5ceY8AZgoNDww5+HvEsXFsGg==",
"dependencies": {
"lodash.camelcase": "^4.3.0",
"long": "^5.0.0",
"protobufjs": "^7.5.5",
"yargs": "^17.7.2"
},
"bin": {
"proto-loader-gen-types": "build/bin/proto-loader-gen-types.js"
},
"engines": {
"node": ">=6"
}
},
"node_modules/@grpc/proto-loader": {
"version": "0.7.15",
"resolved": "https://registry.npmjs.org/@grpc/proto-loader/-/proto-loader-0.7.15.tgz",
"integrity": "sha512-tMXdRCfYVixjuFK+Hk0Q1s38gV9zDiDJfWL3h1rv4Qc39oILCu1TRTDt7+fGUI8K4G1Fj125Hx/ru3azECWTyQ==",
"dependencies": {
"lodash.camelcase": "^4.3.0",
"long": "^5.0.0",
"protobufjs": "^7.2.5",
"yargs": "^17.7.2"
},
"bin": {
"proto-loader-gen-types": "build/bin/proto-loader-gen-types.js"
},
"engines": {
"node": ">=6"
}
},
"node_modules/@js-sdsl/ordered-map": {
"version": "4.4.2",
"resolved": "https://registry.npmjs.org/@js-sdsl/ordered-map/-/ordered-map-4.4.2.tgz",
"integrity": "sha512-iUKgm52T8HOE/makSxjqoWhe95ZJA1/G1sYsGev2JDKUSS14KAgg1LHb+Ba+IPow0xflbnSkOsZcO08C7w1gYw==",
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/js-sdsl"
}
},
"node_modules/@protobufjs/aspromise": {
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz",
"integrity": "sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ=="
},
"node_modules/@protobufjs/base64": {
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/@protobufjs/base64/-/base64-1.1.2.tgz",
"integrity": "sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg=="
},
"node_modules/@protobufjs/codegen": {
"version": "2.0.5",
"resolved": "https://registry.npmjs.org/@protobufjs/codegen/-/codegen-2.0.5.tgz",
"integrity": "sha512-zgXFLzW3Ap33e6d0Wlj4MGIm6Ce8O89n/apUaGNB/jx+hw+ruWEp7EwGUshdLKVRCxZW12fp9r40E1mQrf/34g=="
},
"node_modules/@protobufjs/eventemitter": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/@protobufjs/eventemitter/-/eventemitter-1.1.1.tgz",
"integrity": "sha512-vW1GmwMZNnL+gMRaovlh9yZX74kc+TTU3FObkkurpMaRtBfLP3ldjS9KQWlwZgraRE0+dheEEoAxdzcJQ8eXZg=="
},
"node_modules/@protobufjs/fetch": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/@protobufjs/fetch/-/fetch-1.1.1.tgz",
"integrity": "sha512-GpptLrs57adMSuHi3VNj0mAF8dwh36LMaYF6XyJ6JMWlVsc+t42tm1HSEDmOs3A8fC9yyeisgLhsTVQokOZ0zw==",
"dependencies": {
"@protobufjs/aspromise": "^1.1.1"
}
},
"node_modules/@protobufjs/float": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/@protobufjs/float/-/float-1.0.2.tgz",
"integrity": "sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ=="
},
"node_modules/@protobufjs/path": {
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/@protobufjs/path/-/path-1.1.2.tgz",
"integrity": "sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA=="
},
"node_modules/@protobufjs/pool": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/@protobufjs/pool/-/pool-1.1.0.tgz",
"integrity": "sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw=="
},
"node_modules/@protobufjs/utf8": {
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/@protobufjs/utf8/-/utf8-1.1.2.tgz",
"integrity": "sha512-b1UQwcEZ4yCnMCD8DAL1VlbvBJE9/IX4FTIp7BG1xYpf29SLazLSrqUkj4w7Y5y7cCVP6E5tcqqcI0xemPkHug=="
},
"node_modules/@types/node": {
"version": "26.2.0",
"resolved": "https://registry.npmjs.org/@types/node/-/node-26.2.0.tgz",
"integrity": "sha512-5IviulTZeRNp2vAJ514cc/HUlY5nZ9fCbq9DMyC52BrhFZACo3nI0R7qBxhQmo/d27NFe96ur/b7Wwxklda+kg==",
"dependencies": {
"undici-types": "~8.3.0"
}
},
"node_modules/ansi-regex": {
"version": "5.0.1",
"resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz",
"integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==",
"engines": {
"node": ">=8"
}
},
"node_modules/ansi-styles": {
"version": "4.3.0",
"resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-4.3.0.tgz",
"integrity": "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg==",
"dependencies": {
"color-convert": "^2.0.1"
},
"engines": {
"node": ">=8"
},
"funding": {
"url": "https://github.com/chalk/ansi-styles?sponsor=1"
}
},
"node_modules/asn1": {
"version": "0.2.6",
"resolved": "https://registry.npmjs.org/asn1/-/asn1-0.2.6.tgz",
"integrity": "sha512-ix/FxPn0MDjeyJ7i/yoHGFt/EX6LyNbxSEhPPXODPL+KB0VPk86UYfL0lMdy+KCnv+fmvIzySwaK5COwqVbWTQ==",
"dependencies": {
"safer-buffer": "~2.1.0"
}
},
"node_modules/base64-js": {
"version": "1.5.1",
"resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz",
"integrity": "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/feross"
},
{
"type": "patreon",
"url": "https://www.patreon.com/feross"
},
{
"type": "consulting",
"url": "https://feross.org/support"
}
]
},
"node_modules/bcrypt-pbkdf": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/bcrypt-pbkdf/-/bcrypt-pbkdf-1.0.2.tgz",
"integrity": "sha512-qeFIXtP4MSoi6NLqO12WfqARWWuCKi2Rn/9hJLEmtB5yTNr9DqFWkJRCf2qShWzPeAMRnOgCrq0sg/KLv5ES9w==",
"dependencies": {
"tweetnacl": "^0.14.3"
}
},
"node_modules/bl": {
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz",
"integrity": "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==",
"dependencies": {
"buffer": "^5.5.0",
"inherits": "^2.0.4",
"readable-stream": "^3.4.0"
}
},
"node_modules/buffer": {
"version": "5.7.1",
"resolved": "https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz",
"integrity": "sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/feross"
},
{
"type": "patreon",
"url": "https://www.patreon.com/feross"
},
{
"type": "consulting",
"url": "https://feross.org/support"
}
],
"dependencies": {
"base64-js": "^1.3.1",
"ieee754": "^1.1.13"
}
},
"node_modules/buildcheck": {
"version": "0.0.7",
"resolved": "https://registry.npmjs.org/buildcheck/-/buildcheck-0.0.7.tgz",
"integrity": "sha512-lHblz4ahamxpTmnsk+MNTRWsjYKv965MwOrSJyeD588rR3Jcu7swE+0wN5F+PbL5cjgu/9ObkhfzEPuofEMwLA==",
"optional": true,
"engines": {
"node": ">=10.0.0"
}
},
"node_modules/chownr": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz",
"integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg=="
},
"node_modules/cliui": {
"version": "8.0.1",
"resolved": "https://registry.npmjs.org/cliui/-/cliui-8.0.1.tgz",
"integrity": "sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ==",
"dependencies": {
"string-width": "^4.2.0",
"strip-ansi": "^6.0.1",
"wrap-ansi": "^7.0.0"
},
"engines": {
"node": ">=12"
}
},
"node_modules/color-convert": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz",
"integrity": "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==",
"dependencies": {
"color-name": "~1.1.4"
},
"engines": {
"node": ">=7.0.0"
}
},
"node_modules/color-name": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz",
"integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA=="
},
"node_modules/cpu-features": {
"version": "0.0.10",
"resolved": "https://registry.npmjs.org/cpu-features/-/cpu-features-0.0.10.tgz",
"integrity": "sha512-9IkYqtX3YHPCzoVg1Py+o9057a3i0fp7S530UWokCSaFVTc7CwXPRiOjRjBQQ18ZCNafx78YfnG+HALxtVmOGA==",
"hasInstallScript": true,
"optional": true,
"dependencies": {
"buildcheck": "~0.0.6",
"nan": "^2.19.0"
},
"engines": {
"node": ">=10.0.0"
}
},
"node_modules/debug": {
"version": "4.4.3",
"resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",
"integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==",
"dependencies": {
"ms": "^2.1.3"
},
"engines": {
"node": ">=6.0"
},
"peerDependenciesMeta": {
"supports-color": {
"optional": true
}
}
},
"node_modules/docker-modem": {
"version": "5.0.7",
"resolved": "https://registry.npmjs.org/docker-modem/-/docker-modem-5.0.7.tgz",
"integrity": "sha512-XJgGhoR/CLpqshm4d3L7rzH6t8NgDFUIIpztYlLHIApeJjMZKYJMz2zxPsYxnejq5h3ELYSw/RBsi3t5h7gNTA==",
"dependencies": {
"debug": "^4.1.1",
"readable-stream": "^3.5.0",
"split-ca": "^1.0.1",
"ssh2": "^1.15.0"
},
"engines": {
"node": ">= 8.0"
}
},
"node_modules/dockerode": {
"version": "4.0.12",
"resolved": "https://registry.npmjs.org/dockerode/-/dockerode-4.0.12.tgz",
"integrity": "sha512-/bCZd6KlGcjZO8Buqmi/vXuqEGVEZ0PNjx/biBNqJD3MhK9DmdiAuKxqfNhflgDESDIiBz3qF+0e55+CpnrUcw==",
"dependencies": {
"@balena/dockerignore": "^1.0.2",
"@grpc/grpc-js": "^1.11.1",
"@grpc/proto-loader": "^0.7.13",
"docker-modem": "^5.0.7",
"protobufjs": "^7.3.2",
"tar-fs": "^2.1.4",
"uuid": "^10.0.0"
},
"engines": {
"node": ">= 8.0"
}
},
"node_modules/emoji-regex": {
"version": "8.0.0",
"resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz",
"integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A=="
},
"node_modules/end-of-stream": {
"version": "1.4.5",
"resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz",
"integrity": "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==",
"dependencies": {
"once": "^1.4.0"
}
},
"node_modules/escalade": {
"version": "3.2.0",
"resolved": "https://registry.npmjs.org/escalade/-/escalade-3.2.0.tgz",
"integrity": "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA==",
"engines": {
"node": ">=6"
}
},
"node_modules/eventemitter3": {
"version": "4.0.7",
"resolved": "https://registry.npmjs.org/eventemitter3/-/eventemitter3-4.0.7.tgz",
"integrity": "sha512-8guHBZCwKnFhYdHr2ysuRWErTwhoN2X8XELRlrRwpmfeY2jjuUN4taQMsULKUVo1K4DvZl+0pgfyoysHxvmvEw=="
},
"node_modules/follow-redirects": {
"version": "1.16.0",
"resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.16.0.tgz",
"integrity": "sha512-y5rN/uOsadFT/JfYwhxRS5R7Qce+g3zG97+JrtFZlC9klX/W5hD7iiLzScI4nZqUS7DNUdhPgw4xI8W2LuXlUw==",
"funding": [
{
"type": "individual",
"url": "https://github.com/sponsors/RubenVerborgh"
}
],
"engines": {
"node": ">=4.0"
},
"peerDependenciesMeta": {
"debug": {
"optional": true
}
}
},
"node_modules/fs-constants": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz",
"integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow=="
},
"node_modules/get-caller-file": {
"version": "2.0.5",
"resolved": "https://registry.npmjs.org/get-caller-file/-/get-caller-file-2.0.5.tgz",
"integrity": "sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg==",
"engines": {
"node": "6.* || 8.* || >= 10.*"
}
},
"node_modules/http-proxy": {
"version": "1.18.1",
"resolved": "https://registry.npmjs.org/http-proxy/-/http-proxy-1.18.1.tgz",
"integrity": "sha512-7mz/721AbnJwIVbnaSv1Cz3Am0ZLT/UBwkC92VlxhXv/k/BBQfM2fXElQNC27BVGr0uwUpplYPQM9LnaBMR5NQ==",
"dependencies": {
"eventemitter3": "^4.0.0",
"follow-redirects": "^1.0.0",
"requires-port": "^1.0.0"
},
"engines": {
"node": ">=8.0.0"
}
},
"node_modules/ieee754": {
"version": "1.2.1",
"resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz",
"integrity": "sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/feross"
},
{
"type": "patreon",
"url": "https://www.patreon.com/feross"
},
{
"type": "consulting",
"url": "https://feross.org/support"
}
]
},
"node_modules/inherits": {
"version": "2.0.4",
"resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
"integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="
},
"node_modules/is-fullwidth-code-point": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz",
"integrity": "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==",
"engines": {
"node": ">=8"
}
},
"node_modules/lodash.camelcase": {
"version": "4.3.0",
"resolved": "https://registry.npmjs.org/lodash.camelcase/-/lodash.camelcase-4.3.0.tgz",
"integrity": "sha512-TwuEnCnxbc3rAvhf/LbG7tJUDzhqXyFnv3dtzLOPgCG/hODL7WFnsbwktkD7yUV0RrreP/l1PALq/YSg6VvjlA=="
},
"node_modules/long": {
"version": "5.3.2",
"resolved": "https://registry.npmjs.org/long/-/long-5.3.2.tgz",
"integrity": "sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA=="
},
"node_modules/mkdirp-classic": {
"version": "0.5.3",
"resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz",
"integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A=="
},
"node_modules/ms": {
"version": "2.1.3",
"resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
"integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="
},
"node_modules/nan": {
"version": "2.28.0",
"resolved": "https://registry.npmjs.org/nan/-/nan-2.28.0.tgz",
"integrity": "sha512-fTsDz99OTq2sVePhGdp4qQhggZFtKr64ZNVyVajRKtMOkJxYekplBh577PiJB12v/D3s2E5cGtOI45LWp6rnLQ==",
"optional": true
},
"node_modules/once": {
"version": "1.4.0",
"resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
"integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==",
"dependencies": {
"wrappy": "1"
}
},
"node_modules/protobufjs": {
"version": "7.6.5",
"resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-7.6.5.tgz",
"integrity": "sha512-/FPD0nUc9jH6rfFjji9IBqOz4pcSE3CsT1m7Ep6Mdb0LxSUMj8hgl6GomOvZzpNpAqqGaXA0P3VSrZLFzIhQrw==",
"hasInstallScript": true,
"dependencies": {
"@protobufjs/aspromise": "^1.1.2",
"@protobufjs/base64": "^1.1.2",
"@protobufjs/codegen": "^2.0.5",
"@protobufjs/eventemitter": "^1.1.1",
"@protobufjs/fetch": "^1.1.1",
"@protobufjs/float": "^1.0.2",
"@protobufjs/path": "^1.1.2",
"@protobufjs/pool": "^1.1.0",
"@protobufjs/utf8": "^1.1.1",
"@types/node": ">=13.7.0",
"long": "^5.3.2"
},
"engines": {
"node": ">=12.0.0"
}
},
"node_modules/pump": {
"version": "3.0.4",
"resolved": "https://registry.npmjs.org/pump/-/pump-3.0.4.tgz",
"integrity": "sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA==",
"dependencies": {
"end-of-stream": "^1.1.0",
"once": "^1.3.1"
}
},
"node_modules/readable-stream": {
"version": "3.6.2",
"resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz",
"integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==",
"dependencies": {
"inherits": "^2.0.3",
"string_decoder": "^1.1.1",
"util-deprecate": "^1.0.1"
},
"engines": {
"node": ">= 6"
}
},
"node_modules/require-directory": {
"version": "2.1.1",
"resolved": "https://registry.npmjs.org/require-directory/-/require-directory-2.1.1.tgz",
"integrity": "sha512-fGxEI7+wsG9xrvdjsrlmL22OMTTiHRwAMroiEeMgq8gzoLC/PQr7RsRDSTLUg/bZAZtF+TVIkHc6/4RIKrui+Q==",
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/requires-port": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/requires-port/-/requires-port-1.0.0.tgz",
"integrity": "sha512-KigOCHcocU3XODJxsu8i/j8T9tzT4adHiecwORRQ0ZZFcp7ahwXuRU1m+yuO90C5ZUyGeGfocHDI14M3L3yDAQ=="
},
"node_modules/safe-buffer": {
"version": "5.2.1",
"resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz",
"integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/feross"
},
{
"type": "patreon",
"url": "https://www.patreon.com/feross"
},
{
"type": "consulting",
"url": "https://feross.org/support"
}
]
},
"node_modules/safer-buffer": {
"version": "2.1.2",
"resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz",
"integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="
},
"node_modules/split-ca": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/split-ca/-/split-ca-1.0.1.tgz",
"integrity": "sha512-Q5thBSxp5t8WPTTJQS59LrGqOZqOsrhDGDVm8azCqIBjSBd7nd9o2PM+mDulQQkh8h//4U6hFZnc/mul8t5pWQ=="
},
"node_modules/ssh2": {
"version": "1.17.0",
"resolved": "https://registry.npmjs.org/ssh2/-/ssh2-1.17.0.tgz",
"integrity": "sha512-wPldCk3asibAjQ/kziWQQt1Wh3PgDFpC0XpwclzKcdT1vql6KeYxf5LIt4nlFkUeR8WuphYMKqUA56X4rjbfgQ==",
"hasInstallScript": true,
"dependencies": {
"asn1": "^0.2.6",
"bcrypt-pbkdf": "^1.0.2"
},
"engines": {
"node": ">=10.16.0"
},
"optionalDependencies": {
"cpu-features": "~0.0.10",
"nan": "^2.23.0"
}
},
"node_modules/string_decoder": {
"version": "1.3.0",
"resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz",
"integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==",
"dependencies": {
"safe-buffer": "~5.2.0"
}
},
"node_modules/string-width": {
"version": "4.2.3",
"resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz",
"integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==",
"dependencies": {
"emoji-regex": "^8.0.0",
"is-fullwidth-code-point": "^3.0.0",
"strip-ansi": "^6.0.1"
},
"engines": {
"node": ">=8"
}
},
"node_modules/strip-ansi": {
"version": "6.0.1",
"resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz",
"integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==",
"dependencies": {
"ansi-regex": "^5.0.1"
},
"engines": {
"node": ">=8"
}
},
"node_modules/tar-fs": {
"version": "2.1.5",
"resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.5.tgz",
"integrity": "sha512-OboTd8mmMhZDNPV+UjQcK9yKAatXu2aJ+r1w4im1Otd4M4fl2hwvdoXUxIYHFTHWK/3y3FarBP70v3vwmGlOxw==",
"dependencies": {
"chownr": "^1.1.1",
"mkdirp-classic": "^0.5.2",
"pump": "^3.0.0",
"tar-stream": "^2.1.4"
}
},
"node_modules/tar-stream": {
"version": "2.2.0",
"resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz",
"integrity": "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==",
"dependencies": {
"bl": "^4.0.3",
"end-of-stream": "^1.4.1",
"fs-constants": "^1.0.0",
"inherits": "^2.0.3",
"readable-stream": "^3.1.1"
},
"engines": {
"node": ">=6"
}
},
"node_modules/tweetnacl": {
"version": "0.14.5",
"resolved": "https://registry.npmjs.org/tweetnacl/-/tweetnacl-0.14.5.tgz",
"integrity": "sha512-KXXFFdAbFXY4geFIwoyNK+f5Z1b7swfXABfL7HXCmoIWMKU3dmS26672A4EeQtDzLKy7SXmfBu51JolvEKwtGA=="
},
"node_modules/undici-types": {
"version": "8.3.0",
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-8.3.0.tgz",
"integrity": "sha512-j375ScV60dom+YkPFIfTLcOiPxkN/buHz5GobjLhixFuANaNs3C9l4GmrWqejgXWJ7BbJcFYpTEUkS1Ge8bpZQ=="
},
"node_modules/util-deprecate": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",
"integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw=="
},
"node_modules/uuid": {
"version": "10.0.0",
"resolved": "https://registry.npmjs.org/uuid/-/uuid-10.0.0.tgz",
"integrity": "sha512-8XkAphELsDnEGrDxUOHB3RGvXz6TeuYSGEZBOjtTtPm2lwhGBjLgOzLHB63IUWfBpNucQjND6d3AOudO+H3RWQ==",
"deprecated": "uuid@10 and below is no longer supported. For ESM codebases, update to uuid@latest. For CommonJS codebases, use uuid@11 (but be aware this version will likely be deprecated in 2028).",
"funding": [
"https://github.com/sponsors/broofa",
"https://github.com/sponsors/ctavan"
],
"bin": {
"uuid": "dist/bin/uuid"
}
},
"node_modules/wrap-ansi": {
"version": "7.0.0",
"resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-7.0.0.tgz",
"integrity": "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==",
"dependencies": {
"ansi-styles": "^4.0.0",
"string-width": "^4.1.0",
"strip-ansi": "^6.0.0"
},
"engines": {
"node": ">=10"
},
"funding": {
"url": "https://github.com/chalk/wrap-ansi?sponsor=1"
}
},
"node_modules/wrappy": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
"integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ=="
},
"node_modules/y18n": {
"version": "5.0.8",
"resolved": "https://registry.npmjs.org/y18n/-/y18n-5.0.8.tgz",
"integrity": "sha512-0pfFzegeDWJHJIAmTLRP2DwHjdF5s7jo9tuztdQxAhINCdvS+3nGINqPd00AphqJR/0LhANUS6/+7SCb98YOfA==",
"engines": {
"node": ">=10"
}
},
"node_modules/yargs": {
"version": "17.7.3",
"resolved": "https://registry.npmjs.org/yargs/-/yargs-17.7.3.tgz",
"integrity": "sha512-GZtjxm/J/4TSxuL3FNYjCmLktBTnIw/rVmKSIyKeYAZpmJB2ig9VauCC5xsa82GNKVKDAqpOn3KVzNt0zmrU0g==",
"dependencies": {
"cliui": "^8.0.1",
"escalade": "^3.1.1",
"get-caller-file": "^2.0.5",
"require-directory": "^2.1.1",
"string-width": "^4.2.3",
"y18n": "^5.0.5",
"yargs-parser": "^21.1.1"
},
"engines": {
"node": ">=12"
}
},
"node_modules/yargs-parser": {
"version": "21.1.1",
"resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-21.1.1.tgz",
"integrity": "sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw==",
"engines": {
"node": ">=12"
}
}
}
}
+15
View File
@@ -0,0 +1,15 @@
{
"name": "rootnode-lab-router",
"version": "0.1.0",
"description": "RootNode Academy Phase 4.1 — Lab Host Router. Runs on the same Docker host as student Labs; resolves an opaque routing token to a local container and reverse-proxies the Platform Gateway's data-plane traffic to it. See docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3.",
"private": true,
"main": "src/server.js",
"scripts": {
"start": "node src/server.js",
"test": "node test/run.js"
},
"dependencies": {
"dockerode": "^4.0.2",
"http-proxy": "^1.18.1"
}
}
+29
View File
@@ -0,0 +1,29 @@
// Token -> local container resolution, split out from server.js so it can be unit-tested
// against a real (or fake) Docker client without needing TLS material or a bound port.
// See server.js's header comment for the full design rationale.
const ROUTING_TOKEN_LABEL = 'rootnode.routing-token';
function tokenFingerprint(token) {
return token ? `${token.slice(0, 6)}…(${token.length})` : '(none)';
}
// Resolves a routing token to a live local container's reachable base URL, purely from local
// Docker state — no Postgres, no Provisioner call. Returns null for "no such token" AND "token
// belonged to a destroyed/replaced container" — both cases the caller treats identically (404).
async function resolveTarget(docker, token) {
if (!token || typeof token !== 'string' || token.length < 16) return null; // reject garbage before even hitting Docker
const containers = await docker.listContainers({
filters: JSON.stringify({ label: [`${ROUTING_TOKEN_LABEL}=${token}`], status: ['running'] }),
});
if (containers.length !== 1) return null; // 0 = unknown/destroyed; >1 should never happen (label is per-env-unique) but fail closed either way
const info = containers[0];
const networks = info.NetworkSettings && info.NetworkSettings.Networks;
if (!networks) return null;
const netName = Object.keys(networks)[0];
const ip = netName && networks[netName].IPAddress;
if (!ip) return null;
return `http://${ip}:80`; // every Lab container listens on 80 internally — see docs/LAB_ARCHITECTURE.md
}
module.exports = { resolveTarget, tokenFingerprint, ROUTING_TOKEN_LABEL };
+103
View File
@@ -0,0 +1,103 @@
// RootNode Academy — Lab Host Router (Phase 4.1).
//
// Runs on the SAME Docker host as student Labs (rn-lab-host-01 in production). Its only job:
// take a request from the Platform Gateway carrying an opaque routing token, find the ONE
// local container labeled with that token, and reverse-proxy the request to it. See
// docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3 and docs/PRODUCTION_NETWORK_SECURITY.md.
//
// Deliberately narrow:
// - Never talks to Postgres, the Provisioner, or anything off this host — it resolves purely
// from local `docker ps` state (containers are labeled by the Provisioner at creation time,
// see provisioner lifecycle.js), so a destroyed/reset environment's old token simply has no
// matching container anymore. No separate "is this token still valid" check is needed.
// - Only one route exists: proxy-by-token. No container listing, no Docker-API passthrough,
// no endpoint that echoes back a container id, IP, or network name to the caller.
// - Requires mTLS — the Node https server itself rejects any connection that doesn't present
// a certificate signed by the configured CA (requestCert + rejectUnauthorized), so an
// unauthenticated caller never reaches route-resolution logic at all. This is the ONLY
// authentication; there is no bearer-token/API-key fallback (mTLS or nothing).
// - Binds to an internal address only — this process has no idea what "public" means, that's
// enforced by the firewall (infra/firewall/rn-lab-host-01.rules) and by never being given a
// public-facing port mapping in the production Compose file. Defense in depth: even if the
// firewall were misconfigured, mTLS still blocks anyone without the Gateway's client cert.
const fs = require('fs');
const https = require('https');
const httpProxy = require('http-proxy');
const Docker = require('dockerode');
const { resolveTarget, tokenFingerprint } = require('./resolve');
const PORT = Number(process.env.LAB_ROUTER_PORT) || 8443;
const TLS_CA = process.env.LAB_ROUTER_TLS_CA;
const TLS_CERT = process.env.LAB_ROUTER_TLS_CERT;
const TLS_KEY = process.env.LAB_ROUTER_TLS_KEY;
const ROUTE_TOKEN_HEADER = 'x-rootnode-route-token';
if (!TLS_CA || !TLS_CERT || !TLS_KEY) {
console.error('FATAL: LAB_ROUTER_TLS_CA/CERT/KEY are not set. Refusing to start an unauthenticated Lab Host Router.');
process.exit(1);
}
// Always the local socket — the Router only ever inspects containers on ITS OWN host, never a
// remote Docker daemon (that's the Provisioner's job, over a completely separate mTLS
// connection — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md §10 "control plane vs data plane").
const docker = new Docker({ socketPath: process.env.DOCKER_SOCKET || '/var/run/docker.sock' });
const proxy = httpProxy.createProxyServer({ ws: true });
proxy.on('error', (err, req, res) => {
console.error('[lab-router] proxy error:', err.message);
if (res && res.writeHead && !res.headersSent) {
res.writeHead(502, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({ error: 'lab_unreachable' }));
} else if (res && res.destroy) {
res.destroy();
}
});
async function handleRequest(req, res) {
const token = req.headers[ROUTE_TOKEN_HEADER];
let target;
try {
target = await resolveTarget(docker, token);
} catch (e) {
console.error('[lab-router] docker lookup failed:', e.message);
res.writeHead(502, { 'Content-Type': 'application/json' });
return res.end(JSON.stringify({ error: 'router_internal_error' }));
}
if (!target) {
console.log(`[lab-router] unknown/stale token ${tokenFingerprint(token)} -> 404`);
res.writeHead(404, { 'Content-Type': 'application/json' });
return res.end(JSON.stringify({ error: 'unknown_environment' }));
}
proxy.web(req, res, { target });
}
async function handleUpgrade(req, socket, head) {
const token = req.headers[ROUTE_TOKEN_HEADER];
let target;
try {
target = await resolveTarget(docker, token);
} catch (e) {
console.error('[lab-router] docker lookup failed (ws):', e.message);
return socket.destroy();
}
if (!target) {
console.log(`[lab-router] unknown/stale token ${tokenFingerprint(token)} -> ws rejected`);
return socket.destroy();
}
proxy.ws(req, socket, head, { target });
}
const tlsOptions = {
ca: fs.readFileSync(TLS_CA),
cert: fs.readFileSync(TLS_CERT),
key: fs.readFileSync(TLS_KEY),
requestCert: true,
rejectUnauthorized: true, // the entire authentication mechanism — see file header
};
const server = https.createServer(tlsOptions, handleRequest);
server.on('upgrade', handleUpgrade);
server.on('tlsClientError', (err) => console.warn('[lab-router] rejected unauthenticated/invalid TLS client:', err.message));
server.listen(PORT, () => console.log(`Lab Host Router listening on :${PORT} (mTLS required)`));
+73
View File
@@ -0,0 +1,73 @@
#!/usr/bin/env node
// Fast, isolated unit test for resolve.js's token->container logic — no live Docker daemon, no
// TLS material, no network. Uses a fake dockerode-shaped client so this runs in milliseconds
// as part of every regression pass. The REAL cross-host proxying behavior (mTLS handshake,
// actual container resolution, HTTP+WS traffic) is covered by the two-host simulation harness
// instead — see infra/scripts/simulate-two-host/README.md and
// docs/PHASE4_1_PREPRODUCTION_HARDENING.md §6, not duplicated here.
const assert = require('assert');
const { resolveTarget, tokenFingerprint } = require('../src/resolve');
function fakeDocker(containers) {
return {
listContainers: async ({ filters }) => {
const parsed = JSON.parse(filters);
const wantLabel = parsed.label[0];
const wantStatus = parsed.status[0];
return containers.filter((c) => c.Labels && `${Object.keys(c.Labels)[0]}=${Object.values(c.Labels)[0]}` === wantLabel && c.State === wantStatus);
},
};
}
async function main() {
const runningContainer = {
Labels: { 'rootnode.routing-token': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' },
State: 'running',
NetworkSettings: { Networks: { 'rootnode-lab-net-abc': { IPAddress: '172.20.0.5' } } },
};
{
const docker = fakeDocker([runningContainer]);
const target = await resolveTarget(docker, 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa');
assert.strictEqual(target, 'http://172.20.0.5:80', 'known running token resolves to its container');
console.log('ok — known token resolves to the correct container address');
}
{
const docker = fakeDocker([runningContainer]);
const target = await resolveTarget(docker, 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb');
assert.strictEqual(target, null, 'unknown token resolves to nothing');
console.log('ok — unknown token resolves to null (404 at the HTTP layer)');
}
{
// Simulates a destroyed/reset environment: the container that used to match this label is
// gone (not present in the list at all — a real reset/destroy removes the old container).
const docker = fakeDocker([]);
const target = await resolveTarget(docker, 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa');
assert.strictEqual(target, null, 'stale token (container destroyed) resolves to nothing');
console.log('ok — stale/destroyed-environment token resolves to null, same as unknown');
}
{
const docker = fakeDocker([runningContainer]);
for (const garbage of [undefined, null, '', 'short', 123, {}]) {
const target = await resolveTarget(docker, garbage);
assert.strictEqual(target, null, `garbage input ${JSON.stringify(garbage)} must not resolve`);
}
console.log('ok — garbage/malformed tokens are rejected before ever touching Docker');
}
{
assert.strictEqual(tokenFingerprint('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'), 'aaaaaa…(32)');
assert.strictEqual(tokenFingerprint(undefined), '(none)');
const fp = tokenFingerprint('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa');
assert.ok(!fp.includes('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'), 'fingerprint never contains the full token — never logged in plaintext');
console.log('ok — token fingerprint for logs never reveals the full token');
}
console.log('\nAll lab-router unit tests passed.');
}
main().catch((e) => { console.error(e); process.exit(1); });
@@ -1,19 +1,31 @@
# Phase 1 lab_hosts seed — applied to the DB at Provisioner startup (config.js
# applyLabHostsSeed), not managed via a runtime API (no host-management endpoints in Phase 1,
# matching "don't overbuild scheduling yet" — see docs/PHASE1_PLATFORM_FOUNDATION.md §4).
# lab_hosts seed — applied to the DB at Provisioner startup (config.js applyLabHostsSeed), not
# managed via a runtime API (no host-management endpoints yet — see
# docs/PHASE1_PLATFORM_FOUNDATION.md §4, "don't overbuild scheduling yet").
#
# Exactly one enabled row for Phase 1. Swapping this for a remote tcp://<lab-host-vm>:2376
# entry with TLS material is the entire code change needed to point at a real, separate Lab
# Host VM in production — see docs/LAB_ARCHITECTURE.md §1 and correction D.
# `endpoint`/`tls_*_path` — the Provisioner's CONTROL-plane connection to this host's Docker
# daemon (create/start/stop/remove containers). Unchanged since Phase 1.
#
# `routing_mode`/`router_endpoint` — Phase 4.1 addition, the Gateway's DATA-plane path to a
# running Lab on this host. See docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1 and
# docs/PRODUCTION_NETWORK_SECURITY.md.
# direct — Gateway reaches env.internal_ip:internal_port itself. Only correct when Gateway
# and this lab_host's Docker daemon are on the same network namespace (dev,
# single-host). This is the default and requires no router_endpoint.
# router — Gateway reaches this lab_host's Lab Host Router (platform/lab-router) instead,
# over mTLS, and never touches a Docker-internal address. Required once this
# lab_host is a genuinely separate VM from the Gateway.
lab_hosts:
- lab_host_id: dev-local
display_name: "Local development Docker socket"
endpoint: "unix:///var/run/docker.sock"
enabled: true
routing_mode: direct
# Production example (not active — shown for reference; uncomment and fill in when a real
# Lab Host VM exists):
# Production example (not active — shown for reference; uncomment and fill in once a real
# Lab Host VM exists). CONTROL plane: Provisioner -> dockerd, mTLS, port 2376. DATA plane:
# Gateway -> Lab Host Router, mTLS, router's own port (infra default 8443) — see
# infra/docker/lab-host/docker-compose.production.yml and infra/ansible/roles/lab-router.
# - lab_host_id: labhost-01
# display_name: "RootNode Lab Host VM 01"
# endpoint: "tcp://10.20.30.5:2376"
@@ -21,3 +33,5 @@ lab_hosts:
# tls_cert_path: /etc/rootnode/tls/labhost-01/cert.pem
# tls_key_path: /etc/rootnode/tls/labhost-01/key.pem
# enabled: true
# routing_mode: router
# router_endpoint: "https://10.20.30.5:8443"
@@ -20,3 +20,12 @@ health_check:
path: /api/health
timeout_ms: 240000 # k3s cold start in a nested container can be slow — matches the
# ~60-attempt wait already in scripts/entrypoint.sh
# Phase 4.1 §26 — admission control. Disabled (no cap) by default, on purpose: don't fabricate
# a concurrency number before it's measured against the real host (docs/CAPACITY_PLANNING.md).
# Set max_concurrent_labs from infra/scripts/capacity-model.js's real hard-admission-control
# output once the production host exists, so a full host rejects new Labs cleanly (503, "Lab
# capacity is temporarily full") instead of failing mid-provision once Docker itself runs out
# of room.
# admission_control:
# max_concurrent_labs: null
@@ -0,0 +1,32 @@
-- lab_hosts routing mode — Phase 4.1, docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1.
--
-- Removes the Gateway's dependency on env.internal_ip being reachable directly, which only
-- holds when Gateway and the Lab container share a Docker daemon (true in dev, false once
-- Labs run on a genuinely separate rn-lab-host-01 VM). A lab_host is either:
-- 'direct' — Gateway reaches env.internal_ip:internal_port itself. Correct when Gateway and
-- this lab_host's Docker daemon share a network namespace (dev, single-host).
-- 'router' — Gateway reaches this lab_host's Lab Host Router (router_endpoint) over mTLS
-- instead, carrying the environment's routing_token in a header; the Router
-- resolves that token to a local container on ITS OWN host and reverse-proxies.
-- Correct once this lab_host is a genuinely separate VM.
--
-- Existing 'dev-local' row keeps routing_mode='direct' (the default) — no behavior change for
-- current dev/single-host setups. All existing lab_hosts rows and every environments row are
-- untouched; this only adds columns.
ALTER TABLE lab_hosts ADD COLUMN IF NOT EXISTS routing_mode TEXT NOT NULL DEFAULT 'direct';
ALTER TABLE lab_hosts ADD COLUMN IF NOT EXISTS router_endpoint TEXT;
DO $$ BEGIN
ALTER TABLE lab_hosts ADD CONSTRAINT lab_hosts_routing_mode_check
CHECK (routing_mode IN ('direct', 'router'));
EXCEPTION
WHEN duplicate_object THEN NULL;
END $$;
DO $$ BEGIN
ALTER TABLE lab_hosts ADD CONSTRAINT lab_hosts_router_endpoint_required_check
CHECK (routing_mode = 'direct' OR router_endpoint IS NOT NULL);
EXCEPTION
WHEN duplicate_object THEN NULL;
END $$;
+9 -4
View File
@@ -24,14 +24,19 @@ function loadLabHostsSeed() {
async function applyLabHostsSeed(pool) {
const hosts = loadLabHostsSeed();
for (const h of hosts) {
const routingMode = h.routing_mode || 'direct'; // Phase 4.1 — see migration 004's comment
if (routingMode === 'router' && !h.router_endpoint) {
throw new Error(`lab_host '${h.lab_host_id}' has routing_mode: router but no router_endpoint configured`);
}
await pool.query(
`INSERT INTO lab_hosts (lab_host_id, display_name, endpoint, tls_ca_path, tls_cert_path, tls_key_path, enabled)
VALUES ($1, $2, $3, $4, $5, $6, $7)
`INSERT INTO lab_hosts (lab_host_id, display_name, endpoint, tls_ca_path, tls_cert_path, tls_key_path, enabled, routing_mode, router_endpoint)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
ON CONFLICT (lab_host_id) DO UPDATE SET
display_name = excluded.display_name, endpoint = excluded.endpoint,
tls_ca_path = excluded.tls_ca_path, tls_cert_path = excluded.tls_cert_path,
tls_key_path = excluded.tls_key_path, enabled = excluded.enabled`,
[h.lab_host_id, h.display_name, h.endpoint, h.tls_ca_path || null, h.tls_cert_path || null, h.tls_key_path || null, h.enabled !== false]
tls_key_path = excluded.tls_key_path, enabled = excluded.enabled,
routing_mode = excluded.routing_mode, router_endpoint = excluded.router_endpoint`,
[h.lab_host_id, h.display_name, h.endpoint, h.tls_ca_path || null, h.tls_cert_path || null, h.tls_key_path || null, h.enabled !== false, routingMode, h.router_endpoint || null]
);
}
return hosts.length;
+49 -1
View File
@@ -40,6 +40,21 @@ async function getEnvironment(pool, environmentId) {
return rows[0] || null;
}
// Phase 4.1 — the Gateway's read path (GET /environments/by-token/:token) needs to know HOW
// to reach the Lab, not just its Docker-internal address: is this lab_host 'direct' (Gateway
// reaches env.internal_ip itself) or 'router' (Gateway must go through that lab_host's Lab
// Host Router instead)? Joining lab_hosts here keeps that decision entirely data-driven —
// the Gateway never hardcodes or guesses a host's routing mode.
async function getEnvironmentByTokenWithRouting(pool, token) {
const { rows } = await pool.query(
`SELECT e.*, h.routing_mode AS lab_host_routing_mode, h.router_endpoint AS lab_host_router_endpoint
FROM environments e JOIN lab_hosts h ON h.lab_host_id = e.lab_host_id
WHERE e.routing_token = $1`,
[token]
);
return rows[0] || null;
}
async function listEnvironments(pool, studentId) {
const { rows } = await pool.query(
'SELECT * FROM environments WHERE student_id = $1 ORDER BY created_at DESC',
@@ -90,6 +105,23 @@ async function provisionEnvironment({ pool, studentId, resourcePolicy, imageRef,
throw Object.assign(new Error(`student '${studentId}' already has a non-terminal environment (${existing.environment_id})`), { code: 'ALREADY_EXISTS', existing });
}
// Admission control (Phase 4.1 §26) — checked BEFORE touching Docker at all, so a full host
// fails cleanly and fast (503, no partial network/container left behind to clean up) instead
// of failing mid-provision with a generic 500 once Docker itself runs out of room. No-op
// when maxConcurrentLabs isn't configured (see resourcePolicyFor's comment on why that's the
// default).
if (resourcePolicy.maxConcurrentLabs) {
const { rows } = await pool.query(
`SELECT count(*)::int AS n FROM environments WHERE status NOT IN ('destroyed', 'error')`
);
if (rows[0].n >= resourcePolicy.maxConcurrentLabs) {
throw Object.assign(
new Error('Lab capacity is temporarily full. Try again shortly.'),
{ code: 'CAPACITY_EXCEEDED', current: rows[0].n, max: resourcePolicy.maxConcurrentLabs }
);
}
}
const labHost = await selectLabHost(pool);
const docker = clientFor(labHost);
const routingToken = generateRoutingToken();
@@ -145,6 +177,15 @@ async function provisionEnvironment({ pool, studentId, resourcePolicy, imageRef,
const container = await docker.createContainer({
name: containerName,
Image: imageRef,
// Phase 4.1: labeled by routing token (not environment_id alone) so the Lab Host Router
// — which runs on this same Docker host and has no Postgres access — can resolve
// "which local container does this opaque token address" purely from local `docker ps`,
// with no cross-VM call back to the Provisioner needed at request time. See
// docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1, platform/lab-router/src/router.js.
Labels: {
'rootnode.routing-token': routingToken,
'rootnode.environment-id': env.environment_id,
},
HostConfig: {
Privileged: true, // required for k3s — see docs/LAB_ARCHITECTURE.md §7, not avoidable
NetworkMode: networkName,
@@ -321,6 +362,13 @@ async function resetEnvironment({ pool, environmentId, resourcePolicy, egressCon
const container = await docker.createContainer({
name: containerName,
Image: env.image_ref,
// Phase 4.1 — same routing-token label as provisionEnvironment; a reset keeps the same
// environment_id/routing_token (same DB row) so this must relabel the RECREATED container
// the same way, or the Lab Host Router would resolve stale/no container for this token.
Labels: {
'rootnode.routing-token': env.routing_token,
'rootnode.environment-id': environmentId,
},
HostConfig: {
Privileged: true,
NetworkMode: networkName,
@@ -453,7 +501,7 @@ async function reconcileOnStartup(pool) {
}
module.exports = {
generateRoutingToken, emitEvent, getEnvironment, listEnvironments, getActiveEnvironmentForStudent,
generateRoutingToken, emitEvent, getEnvironment, getEnvironmentByTokenWithRouting, listEnvironments, getActiveEnvironmentForStudent,
provisionEnvironment, destroyEnvironment, resetEnvironment,
suspendEnvironment, resumeEnvironment, pollHealth, shortId, reconcileOnStartup,
};
+25 -4
View File
@@ -40,11 +40,18 @@ function egressConfig() {
function resourcePolicyFor() {
const policy = getResourcePolicy().default;
// Phase 4.1 §26 — optional; null/unset (the default) means no cap is enforced, matching the
// Phase 4 rule against fabricating a concurrency number before it's measured. Set this from
// infra/scripts/capacity-model.js's real output once the host exists (see
// infra/environments/production/capacity.yml.example's hard-admission-control field) — see
// docs/PHASE4_1_PREPRODUCTION_HARDENING.md §26.
const admission = getResourcePolicy().admission_control || {};
return {
cpuLimit: policy.cpu_limit,
memLimitMb: policy.mem_limit_mb,
pidsLimit: policy.pids_limit,
storageLimitMb: policy.storage_limit_mb,
maxConcurrentLabs: admission.max_concurrent_labs || null,
};
}
@@ -81,7 +88,10 @@ function isValidUUID(v) {
function asyncRoute(fn) {
return (req, res) => fn(req, res).catch((e) => {
console.error(`[${req.requestId}]`, e);
const status = e.code === 'NOT_FOUND' ? 404 : e.code === 'ALREADY_EXISTS' ? 409 : 500;
// CAPACITY_EXCEEDED (Phase 4.1 §26) is 503, not 500 — the host being full is an expected,
// recoverable operational state, not an application error. See lifecycle.js's admission
// check, which runs BEFORE any Docker/network resources are touched.
const status = e.code === 'NOT_FOUND' ? 404 : e.code === 'ALREADY_EXISTS' ? 409 : e.code === 'CAPACITY_EXCEEDED' ? 503 : 500;
res.status(status).json({ error: e.message, code: e.code || 'INTERNAL', request_id: req.requestId });
});
}
@@ -98,6 +108,14 @@ app.get('/health', asyncRoute(async (req, res) => {
res.status(ok ? 200 : 503).json({ ok, db: dbOk, lab_hosts: hostChecks });
}));
// GET /capacity — Phase 4.1 §26. Internal-only, same shared-secret auth as everything else
// here — lets the Platform API's instructor dashboard show real capacity state instead of
// instructors only discovering a full host when a student's launch fails.
app.get('/capacity', asyncRoute(async (req, res) => {
const { rows } = await getPool().query(`SELECT count(*)::int AS n FROM environments WHERE status NOT IN ('destroyed', 'error')`);
res.json({ current: rows[0].n, max: resourcePolicyFor().maxConcurrentLabs });
}));
// POST /environments — { student_id }
app.post('/environments', asyncRoute(async (req, res) => {
const { student_id } = req.body || {};
@@ -130,9 +148,12 @@ app.get('/environments', asyncRoute(async (req, res) => {
// owning student_id) before proxying. Internal-only, same shared-secret auth as every other
// route here — the Gateway is the only caller, browsers never reach this directly.
app.get('/environments/by-token/:token', asyncRoute(async (req, res) => {
const { rows } = await getPool().query('SELECT * FROM environments WHERE routing_token = $1', [req.params.token]);
if (!rows.length) return res.status(404).json({ error: 'not found', request_id: req.requestId });
res.json(redactRow(rows[0]));
// Phase 4.1: also joins lab_host_routing_mode/lab_host_router_endpoint so the Gateway can
// decide direct-vs-router without knowing anything about Docker topology itself — see
// lifecycle.getEnvironmentByTokenWithRouting and docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1.
const env = await lifecycle.getEnvironmentByTokenWithRouting(getPool(), req.params.token);
if (!env) return res.status(404).json({ error: 'not found', request_id: req.requestId });
res.json(redactRow(env));
}));
// GET /environments/:id/token — Phase 2A addition. Every other read path deliberately redacts
+6 -2
View File
@@ -160,12 +160,16 @@ async function main() {
});
await test('audit log: security events were actually recorded', async () => {
// Phase 4.1 §16 — /api/admin/audit-events is now paginated (see platform/api/src/pagination.js);
// the events land on the most recent page, but this test has produced fewer than one
// page's worth by this point, so page 1's items are still every event so far.
const { body } = await owner.request('GET', '/api/admin/audit-events');
const actions = body.map((e) => e.action);
const items = body.items;
const actions = items.map((e) => e.action);
assert.ok(actions.includes('login'));
assert.ok(actions.includes('authz_denied'));
assert.ok(actions.includes('instructor_update'));
assert.ok(body.every((e) => !JSON.stringify(e.metadata || {}).match(/password/i)), 'audit metadata must never contain password material');
assert.ok(items.every((e) => !JSON.stringify(e.metadata || {}).match(/password/i)), 'audit metadata must never contain password material');
});
// ── Student session hardening ──────────────────────────────────────────────────────────