No more shared instructor password env var — the api service now needs only
PROVISIONER_SHARED_SECRET. First-run setup is npm run bootstrap-owner instead.
Non-HttpOnly rn_csrf_s/rn_csrf_i cookies, set on login, echoed back as X-CSRF-Token on every
state-changing request — defense in depth on top of (not instead of) SameSite=Lax. In-memory
sliding-window rate limiting on instructor login (8 attempts / 5 min per ip:email, no new
dependency).
Real bug found while wiring this up: the Gateway's own progress-relay call (triggered when a
student solves a scenario) is a server-to-server POST to /api/progress/record relaying the
browser's session — it needs to forward the CSRF token too, and needs to forward BOTH the
rn_csrf_s cookie and a matching X-CSRF-Token header (forwarding only the header still 403'd,
since the double-submit check needs both on the request Platform API actually receives).
Also tightens the Gateway's session-resolution cache from 5000ms to 2000ms, so a
revoked/expired session stops working at the Gateway within a tighter, tested bound.
Frontend (admin + student API clients) now reads the relevant rn_csrf_* cookie via
document.cookie and attaches it on every POST/PATCH/DELETE.
Three roles (owner > admin > instructor), enforced server-side on every /api/admin/* route
via reusable middleware (requireOwner/requireAdminOrOwner/requireAnyInstructor) — the
frontend hides controls a role can't use, but that's UX, never the actual boundary. A plain
'instructor' is further scoped to cohorts assigned via the new cohort_instructors table
(requireCohortAccess/requireStudentCohortAccess); owner/admin bypass that check entirely.
Owner-only: instructor account lifecycle, audit log. Admin/owner: cohort/student CRUD,
curriculum assignment. Any instructor role (cohort-scoped for 'instructor'): dashboard,
cohort/student views, Prepare/Destroy Labs, provisioning. Deliberately NOT cohort-scoped:
the raw /api/admin/environments/:id/* routes (no cohort context in a bare environment id
without an extra Provisioner round trip on every call) — a plain instructor still has full
Lab control via the cohort/student-scoped routes instead, which already carry the check.
server.js wires the new persistent-session middleware (studentSession/instructorSession —
re-fetches the instructor row from Postgres every request, so a role change or
deactivation takes effect on the very next request with no separate revoke needed) and adds
a last-resort error handler so a DB error in any async route returns 500 instead of
crashing the process or leaking a stack trace.
New audit_events table, distinct from the Provisioner's environment_events (Lab-lifecycle
only). recordAudit() is called from the auth/RBAC/admin layers added in this phase — login
success/failure, logout, password change, instructor CRUD, role changes, session
revocation, cohort/roster changes, curriculum assignment, Prepare/Destroy Labs, and every
403 authorization denial (recorded centrally by the RBAC middleware, not per-route). Never
records plaintext passwords, raw session tokens, Provisioner secrets, or raw routing tokens.
Replaces the single shared ADMIN_PASSWORD (Phase 2B) with real accounts: instructors table
(email/display_name/role/status), scrypt password hashing with a fresh random salt PER
INSTRUCTOR (Phase 2B used one fixed salt because there was exactly one shared password —
that assumption is gone), auth_identities table so a future OIDC login can map to the same
instructor_id without a schema change.
New POST /api/auth/instructor/login, POST /api/auth/logout, GET /api/auth/session,
POST /api/auth/change-password (old password required, revokes other sessions). No account
enumeration: identical response whether the email doesn't exist or the password is wrong.
bootstrap-owner.js is the only supported way to create the first account — no manual
database editing to stand up a fresh deployment. Refuses to run if an active owner already
exists unless the target email is an existing instructor (zero-owners recovery path).
Replaces the in-memory studentSessions/adminSessions Maps (Phase 2B — gone on every
restart) with a sessions table. Browser holds an opaque CSPRNG bearer token; only its
SHA-256 hash is ever stored server-side, same principle as password hashing. Each session
has an absolute expiry (fixed at creation) and a sliding idle expiry (pushed forward on
every authenticated request, capped at the absolute one) — one atomic UPDATE...RETURNING
per resolve, no read-then-write race. Student and instructor sessions stay fully separate
token namespaces, matching Phase 2B's cookie-name separation.
cookies.js centralizes Set-Cookie construction so the Secure attribute (COOKIE_SECURE=1,
for real HTTPS deployments) is applied consistently instead of hand-built per call site.
New docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md covers the Platform API
split, Postgres schema and migration guarantees/rollback, the session
cookie architecture (including the whoami-routing-bug fix), the admin
portal, the API/security boundary, known limitations, and Phase 3+
future work.
Updates ENGINEERING_SOP.md §8, LAB_ARCHITECTURE.md, README.md, and
platform/README.md to reflect Platform API as the sole Provisioner
caller and Postgres as authoritative for identity/progress as of
Phase 2B.
Full acceptance test: admin auth (+ student-can't-call-admin check),
cohort create, 3-student add, curriculum assign, concurrent Prepare Labs
(3/3 ready, instant-reuse timing check), two students completing
different scenarios, dashboard reflecting completions, admin reset with
progress-intact assertion, admin destroy-labs with progress-intact
assertion for everyone, and a regression check that no Lab container
publishes a host port. 12/12 passing.
Removes the old identity service/volume, adds the api service (host
networking, Postgres URL, Provisioner shared secret, ADMIN_PASSWORD,
both required via :?). gateway now points PLATFORM_API_URL at it instead
of the old IDENTITY_BACKEND_URL.
Replaces the Phase 2A pilot ?sid= query param with an HttpOnly,
SameSite=Lax session cookie. Gateway now derives student identity from
the session (via Platform API's /api/session/whoami) instead of trusting
a client-suppliable id, and checks ownership on every proxied HTTP and
WebSocket Lab request, not just the initial navigation.
Fixes a routing bug found during testing: the fallback route was using
'does this session own a running environment' to decide whether an
unprefixed /api/* call belonged to the Lab UI or the dashboard — true in
both places. Replaced with an explicit LAB_OWNED_PATHS allowlist of the
exact unprefixed paths the existing Lab UI actually calls.
e2e-phase2a.js rewritten to use a cookie jar instead of headers/query
params, with explicit impersonation-rejection tests added. 10/10 passing.
New platform/api/ service owns students, cohorts, curriculum, progress,
scenario attempts, and student/admin sessions. No k3s, no --privileged,
no terminal runtime — starts instantly, unlike the old identity service
which booted an unused k3s cluster just to serve roster/progress APIs.
Provisioner calls (student and admin) are consolidated here too, in
provisioner-client.js, so it's the only service that talks to the
Provisioner directly.
platform/test/e2e-phase2a.js — 15 assert-based tests against a live
platform/docker-compose.dev.yml stack, exercising the full student flow
through the Gateway exactly as a browser would: roster -> login ->
curriculum -> launch (real k3s boot) -> Gateway HTTP routing -> invalid-token
rejection -> cross-student ownership rejection (?sid= mismatch) ->
same-student access -> progress relay (solve in one student's lab, confirm
it shows in their curriculum and NOT the other student's) -> reset (stable
launch_path, progress survives).
All 15 pass. This is slow (each real environment launch is a genuine k3s
cold start) and deliberately not meant for every commit — see
platform/README.md for when to run it.
Does not cover what needs a browser/WebSocket client — real Lab image
terminal (node-pty, kubectl execution, scenario validation) and idle-reaper
timing were verified manually; see
docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md §8 for exactly what and how.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
New internal component (platform/gateway/) — the only student-facing path to
disposable Lab environments. Browsers never reach a Lab container, the
Provisioner, or Postgres directly.
Routes:
- /api/environment/{status,launch,reset} — student-facing glue (X-Student-Id),
calls the Provisioner internally (shared secret never reaches the browser).
Reuses an existing non-terminal environment on launch rather than always
provisioning a new one; reset destroys+recreates the same environment row.
- /lab/<token>/* — HTTP+WS proxy into that one student's Lab container,
resolved by opaque routing token via a new Provisioner endpoint
(GET /environments/by-token/:token). No raw container IP/port ever reaches
the browser. Rejects invalid tokens (404), unready environments (202/503),
and cross-student access via an optional ?sid= ownership check (403) — see
docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md "Security" for why token-alone,
student-id-alone, and env-id-alone are each individually insufficient and
this is explicitly pilot-grade, not production auth.
- everything else — proxied to the persistent identity/curriculum backend
(roster, login, curriculum, progress, platform-shell frontend), or to the
student's own Lab if a valid lab-session cookie is present.
Routing design note: the existing Lab UI makes root-relative calls
(fetch('/api/scenarios'), the /shell-ws WebSocket) with no idea it might be
served under a /lab/<token>/ prefix — preserving that UI unmodified was a
hard requirement. Solved with an HttpOnly, SameSite=Lax cookie set on the
first /lab/<token>/ response: every later request from that browser carries
it, so the unmodified Lab UI's own unprefixed calls resolve correctly once a
student has navigated into a lab session, with zero changes to its code.
/api/environment/* is explicitly exempted from cookie-based routing so it
never gets hijacked by an active lab session.
Also added platform/docker-compose.dev.yml (the full local stack: Postgres +
egress-proxy + Provisioner + a persistent "identity" instance + Gateway) and
platform/README.md tying the components together. The "identity" instance
required an explicit `network_mode: bridge` — under Compose's own
auto-created bridge network, k3s failed at startup with "no default routes
found in /proc/net/route"; documented as a known limitation, not fully
root-caused.
Found and fixed during testing: express.json() applied globally broke
http-proxy's re-streaming of proxied POST bodies (e.g. POST /api/roster hung
forever, since the body had already been consumed by the parser before
proxy.web() tried to forward it). Removed — none of the Gateway's own routes
actually need a parsed body.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
platform/provisioner/test/run.js — 12 assert-based integration tests against a
live docker-compose.dev.yml stack (no framework, matching the project's
existing convention). Covers: health/auth, provision with unique
network/container/token, duplicate-provision rejection, distinct
subnets/IPs for concurrent students, the egress firewall ordering fix
specifically (proxy ACCEPT before the RFC1918 DROP for the same range),
student-to-student isolation, RFC1918 blocking, reset (same identity, fresh
container), destroy (deterministic cleanup), and the audit trail (all
expected events present, routing token never logged).
All 12 pass against the default lightweight test image. Several properties
were validated manually during implementation rather than automated here —
idle reaper timing (needs a real threshold wait), remote TCP+TLS lab-host
(needs a Docker-in-Docker sidecar, too heavy for routine runs),
reconciliation-on-crash (needs an out-of-band container kill + process
restart), and real-Lab-image k3s behavior (environment-specific, see the
Phase 1 final report and platform/provisioner/README.md's known limitations)
— each is noted at the bottom of the file with why it's not automated here.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Background loop (src/reaper.js), config-driven idle/abandoned thresholds and
poll interval (no hard-coded numbers) — without this, "ephemeral by default"
is just a description: a closed browser tab does not stop a running k3s
cluster on its own.
Past the idle threshold: logs a warning event only. Past the abandoned
threshold: destroys via the same path DELETE /environments/:id uses, so
cleanup is identical either way, and logs event_type='reaped_idle'.
Failure-safe — one environment's teardown throwing never stops the reaper
from checking the rest, and a Postgres/Docker hiccup on one tick just means
"try again next tick," never a crashed process.
Uses COALESCE(last_active_at, created_at) rather than bare last_active_at.
Found during testing: an environment that fails during provisioning (before
ever reaching 'running') has a NULL last_active_at, and NULL < anything is
never true in SQL — without the fallback, a failed-and-never-cleaned-up
environment would never have aged out here regardless of how long it sat
there. This is the second, independent safety net on top of
provisionEnvironment's own best-effort cleanup on failure (see the
platform-provisioner commit).
Verified end-to-end: backdated an environment's last_active_at past the
abandoned threshold, waited for a real reaper tick, confirmed full
deterministic cleanup (container, network, firewall chain all gone) with the
correct actor='reaper' attribution in environment_events.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Egress enforcement is a firewall-layer control, not an application-layer
convention (correction B). HTTP_PROXY/HTTPS_PROXY configure containerd for a
clean pull path; they enforce nothing on their own.
- src/egress.js — per-environment dedicated iptables chain
(ROOTNODE-ENV-<short-id>), built with -A (append, order-preserving) rather
than the -I (insert-at-head) pattern an earlier design draft used, which
would have installed rules in reverse order and let the default-deny shadow
the proxy/DNS accepts. Required infrastructure (egress proxy, DNS) is
explicitly permitted first, then RFC1918 ranges are dropped unconditionally,
then a final default-deny catches the public internet. One jump rule per
environment hooks its subnet into its chain from DOCKER-USER (the chain
Docker guarantees it won't silently rewrite). Destroy is deterministic:
unhook, flush, delete — no comment-grep matching.
The ordering fix is empirically verified, not just reasoned about: the
egress proxy's own address is itself an RFC1918 address (it's Platform-side
infrastructure), and testing confirms it's reachable while the broader
RFC1918 range is still blocked — exactly the scenario the original ordering
bug would have broken.
- platform/egress-proxy/ — shared Squid instance (one per deployment, not
per-student), destination-allowlisted (dstdomain ACL against a plain-text
allowlist file), starting set covers docker.io/ghcr.io/gcr.io/quay.io/k8s.io.
Proxy performs domain allowlisting; the firewall chain above is what makes
it unbypassable, not the other way around.
The egress proxy's address is resolved per-student-network (each network's
Docker gateway IP, since the proxy runs with host networking and is present
on every bridge gateway) rather than a fixed hostname — a static address
wouldn't be reachable from more than one isolated student network at a time.
Found while testing with two simultaneous environments.
Verified: student-to-student isolation (A cannot reach B's internal IP),
RFC1918 blocking (A cannot reach a simulated trusted-infra address), and the
proxy allowlist itself (CONNECT tunnel to an allowlisted host succeeds,
non-allowlisted gets a clean 403 from Squid).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Internal-only service (platform/provisioner/) implementing the environment
lifecycle from docs/PHASE1_PLATFORM_FOUNDATION.md: provision -> running ->
(reset -> running | destroy -> destroyed), plus experimental suspend/resume
(decision #3 — never on the guaranteed path, never relied on for correctness).
- src/docker.js — the lab-host abstraction (correction D). Resolves a
lab_hosts row into a connected dockerode client, unix:// socket or tcp://
with TLS client-cert auth. Refuses to connect to a tcp:// endpoint with no
TLS material configured rather than connecting unauthenticated. Validated
against both a local socket and a real remote Docker-in-Docker TLS endpoint
during testing — same code path either way, only the lab_hosts config
differs, which is the entire point.
- src/lifecycle.js — provision/destroy/reset/suspend/resume orchestration,
wired to docker.js + egress.js + the Postgres registry. Every Lab container
is privileged (required for k3s, not avoidable), gets its own Docker network,
no published host ports, deterministic naming (rootnode-lab-<short-id> /
rootnode-lab-net-<short-id>), and a CSPRNG routing token. Includes
reconcileOnStartup(), implementing the "don't trust stale state after a
crash/restart" requirement from docs/LAB_ARCHITECTURE.md §8 — verified by
killing a container out-of-band and confirming a Provisioner restart flags
the mismatch instead of trusting the stale 'running' row.
Found and fixed during testing: a provisioning failure (e.g. readiness
timeout) left an orphaned container/network/firewall chain behind with only
the DB row marked 'error'. provisionEnvironment's catch block now does
best-effort teardown of whatever was already created before marking the row
error — confirmed by deliberately failing a provision and checking no
container/network/chain survived it.
- src/server.js — the internal HTTP API (X-Provisioner-Key shared-secret auth
on every route, matching the instructor-passcode pattern already used for
the roster endpoint). Input validation on student_id/environment_id,
structured JSON errors with a request id, routing tokens redacted from every
response.
- docker-compose.dev.yml — Postgres + egress-proxy + provisioner on one
machine (the documented Phase 1 dev exception). provisioner and egress-proxy
run with network_mode: host + NET_ADMIN/NET_RAW, which is what lets
src/egress.js manipulate the host's DOCKER-USER chain without needing
interactive host sudo.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
New Phase 1 Platform database — PostgreSQL, separate from the Lab image's own
SQLite (which keeps its current students/progress data untouched, decision #2 /
correction C in docs/LAB_ARCHITECTURE.md).
Schema (platform/provisioner/migrations/):
- lab_hosts: the abstraction that keeps Docker connectivity out of code —
endpoint (unix:// or tcp://+TLS), enabled flag. Phase 1 ships one row.
- environments: environment_id (UUID), student_id (opaque external identifier,
no FK — students table isn't in this database yet), lab_host_id (real FK),
status, routing_token (CSPRNG via crypto.randomBytes, opaque), container/
network identity, resource limits, lifecycle timestamps.
- environment_events: append-only audit trail, one row per lifecycle
transition, never logs the routing token in plaintext (see
lifecycle.js's redact()).
File-based migration runner (db.js) matching the check-before-alter idempotency
pattern already used in the Lab image's backend/server.js, applied on every
Provisioner boot.
Resource limits and the lab_hosts seed are config files
(config/resource-policy.yaml, config/lab-hosts.seed.yaml), not database tables —
there's no admin UI to manage them via a database yet, so a DB-backed policy
table would just be a write path nothing uses.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
ced4568
