docs(phase2a): document gateway and student launch architecture

- docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md — the full Phase 2A record: node-pty
  root cause and fix, Gateway architecture, student flow (entry through
  reset/recreate), API routes added, HTTP/WebSocket routing design (and why
  the lab-session cookie exists), security model (what's checked, and the
  explicit statement that this is pilot identity, not production auth),
  reset/recreate and reaper-recovery semantics, and known limitations found
  during implementation (the identity instance's k3s resource cost and its
  network_mode: bridge requirement, the Gateway's in-memory cache scope, the
  ?sid= check's coverage). BUILT NOW vs. FUTURE (Phase 2B) called out
  explicitly per the governing requirement.
- docs/LAB_ARCHITECTURE.md, docs/ENGINEERING_SOP.md — status updated to
  reflect Phase 2A is implemented and tested, not just Phase 1; roadmap item 5
  updated accordingly; ENGINEERING_SOP's Provisioner section corrected (it's
  no longer un-fronted — the Gateway is the only caller now).
- README.md — Multi-Student Deployment section updated to describe the
  Gateway and student flow, not just the Provisioner; clarifies the
  node-pty fix is the only change to the Lab image itself.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
This commit is contained in:
2026-08-18 11:18:43 -05:00
co-authored by Claude Sonnet 5
parent 05421bcb0e
commit 42dfb1c31b
4 changed files with 339 additions and 35 deletions
+14 -13
View File
@@ -142,13 +142,14 @@ Order agreed 2026-08-17/18. Status as of this doc:
3. Student accounts / persistent progress — **backend done** (§5), frontend not started,
no instructor auth yet.
4. Instructor dashboard — not started.
5. Per-student isolation — **Phase 1 (platform foundation) done**, see §8. Ephemeral
per-student Lab environments with Postgres-backed lifecycle, per-environment Docker
networking, and firewall-enforced egress restriction are implemented and tested on a single
dev machine. **Still not cleared for minors** — no Gateway/frontend wiring yet (environments
are only reachable via the internal Provisioner API), no instructor auth on top of it, and
the production two-VM Proxmox split this depends on doesn't exist yet. See
`docs/PHASE1_PLATFORM_FOUNDATION.md` final report for exactly what's verified vs. not.
5. Per-student isolation — **Phase 1 (platform foundation) + Phase 2A (Gateway + student flow)
done**, see §8/§9. Ephemeral per-student Lab environments, Gateway-mediated routing, and a
working entry→dashboard→launch→lab→progress flow are implemented and end-to-end tested on a
single dev machine, including two simultaneous students with verified cross-student
isolation. **Still not cleared for minors** — pilot-grade identity only (no
password/SSO), no instructor auth layer, and the production two-VM Proxmox split this
depends on doesn't exist yet. See `docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md` for exactly
what's verified vs. not.
6. Scenarios → teaching modules (incident context/investigation/prevention) — not started.
7. Capstone/exam engine — not started (placeholder stage only, see §4).
8. Progressive hint levels — not started.
@@ -156,9 +157,9 @@ Order agreed 2026-08-17/18. Status as of this doc:
10. CI scenario testing — not started.
11. Container/build optimization — explicitly deprioritized by design (platform works, not urgent).
Next slice: either the deferred frontend login (name picker → `X-Student-Id` header, curriculum
view), or the Phase 2 platform work (Gateway, frontend wiring to the Provisioner, instructor
auth) — see the Phase 1 final report for the concrete Phase 2 list. Not decided here.
Next slice: Phase 2B — instructor dashboard, production auth/SSO, and the identity/progress
Postgres migration are the biggest remaining items. See
`docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md` §9 for the full Phase 2B list.
## 8. Platform Provisioner (Phase 1)
@@ -170,9 +171,9 @@ section is just the map.
environment per student — its own Docker network, its own container running the existing,
unmodified Lab image, its own firewall-enforced egress. Postgres-backed registry
(`environments`, `environment_events`, `lab_hosts`), idle reaper, startup reconciliation.
- **What it is not (yet):** reachable by a student. There's no Gateway and no frontend wiring
— it's exercised directly via its internal API (shared-secret auth) or
`platform/provisioner/test/run.js`. Nothing here is a live feature.
- **What it is not:** directly reachable by a student, even now — the Gateway (§9) sits in
front of it and is the only caller. Still exercised directly via its internal API
(shared-secret auth) or `platform/provisioner/test/run.js` for isolated Provisioner testing.
- **Where identity lives:** unchanged from §5 — `students`/`progress` are still on the Lab
image's own SQLite. `environments.student_id` is deliberately opaque (no FK) until a later,
separate migration puts roster/progress on the same Postgres instance as the environment
+7 -6
View File
@@ -1,11 +1,12 @@
# RootNode Academy — Ephemeral Lab Architecture
Status: **Phase 1 (platform foundation) IMPLEMENTED 2026-08-18** — see
`docs/PHASE1_PLATFORM_FOUNDATION.md` for the concrete schema/API/test results, and
`platform/provisioner/README.md` for how to run it. This document remains the architecture
record for the full ephemeral-lab direction; later phases (Gateway, frontend wiring, instructor
dashboard, production Proxmox deployment — see the Phase 1 report's "Future Phase 2 items")
build on it but aren't built yet. Supersedes the namespace-isolation
Status: **Phase 1 (platform foundation) + Phase 2A (Gateway + student flow) IMPLEMENTED,
2026-08-18** — see `docs/PHASE1_PLATFORM_FOUNDATION.md` and
`docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md` for the concrete schema/API/test results, and
`platform/provisioner/README.md` for how to run the stack. This document remains the
architecture record for the full ephemeral-lab direction; instructor dashboard, production
auth/SSO, and production Proxmox deployment are still future work (Phase 2B — see the Phase 2A
doc's §9). Supersedes the namespace-isolation
(`shared_namespace`/`ephemeral_cluster`) approach that was in flight before this redirect — see
`scenarios/SCHEMA.md`'s `isolation_type` note for the pointer back here.
+292
View File
@@ -0,0 +1,292 @@
# RootNode Academy — Phase 2A: Gateway & Student Flow
Status: **IMPLEMENTED and end-to-end tested, 2026-08-18.** Builds on
`docs/LAB_ARCHITECTURE.md` and `docs/PHASE1_PLATFORM_FOUNDATION.md`
(Phase 1 milestone `a15897c78f7012c59ea3629a74dd8b1425f6f7f2`). This doc covers the three
Phase 2A workstreams: fixing the real Lab image, the Gateway, and the student frontend flow.
---
## 1. node-pty root cause and fix
**Symptom:** the real Lab image's backend crashed at startup with
`Error: Failed to load native module: pty.node, checked: build/Release, build/Debug,
prebuilds/linux-x64: Cannot find module './prebuilds/linux-x64//pty.node'`.
**What it looked like vs. what it was:** node-pty's own error handling collapses whatever
actually went wrong at each candidate path into one generic "module not found" message using
only the *last* path it tried — this made a link-time failure look like a missing-file problem.
**Root cause, confirmed empirically (not guessed):**
1. **node-gyp generates an incorrect `config.gypi`.** For this NodeSource-packaged Node 20
build, `node-gyp`'s downloaded headers/config reference reports `node_shared: true` (a
shared `libnode.so` build), but the actual installed Node is statically linked — there is no
`libnode.so.<abi>` anywhere on the system (confirmed: `find / -iname 'libnode*'` → nothing;
`process.config.variables.node_shared` on the real running Node → `false`). Node-gyp
trusted the wrong config and linked node-pty's native addon against `-lnode`, producing a
binary that fails at `require()` time with `libnode.so.127: cannot open shared object file`
— confirmed by loading the compiled `.node` file directly, bypassing node-pty's own
path-search/error-masking logic.
2. **A second, independent bug compounded it:** the repo had no `.dockerignore`. A host-side
`npm install` (run earlier in this project for unrelated backend testing) left a
`backend/node_modules/` on disk, built against the *host's* environment. The Dockerfile's
`COPY backend/ ./backend/` (placed after the npm-install `RUN` layer, for build-cache
efficiency) then silently overwrote the correctly-built-in-container `node_modules` with
that host copy — including after a first `.dockerignore` fix that used the pattern
`node_modules/`, which (a real gotcha) only matches a `node_modules` directory at the build
**context root** in Docker's matching rules, not `backend/node_modules` at any depth.
**Fix (`Dockerfile`, `.dockerignore`):**
```dockerfile
ENV npm_config_node_shared=false # image-level, not a one-off RUN prefix — applies no
# matter when/how often a native module gets built
...
RUN cd backend && npm install --production \
&& node -e "require('./node_modules/node-pty'); console.log('node-pty loads OK')"
```
The `node -e require(...)` line makes this a **hard build failure** if it ever regresses again,
instead of a silent runtime failure discovered later — not a workaround for the terminal, a
real build-time verification of a real native dependency.
```
# .dockerignore
**/node_modules/ # NOT `node_modules/` — that only matches the context root
frontend/dist/
.git/
*.db
*.db-journal
```
**Nothing else changed.** No terminal architecture change, no fake shell, no rewrite of
`backend/server.js`'s PTY handling. RootNode branding and upstream KubeKosh attribution
untouched. The single-container dev workflow (`docker build -t rootnode-academy:test .`) is
the same command as before — it now just works.
**Verified against the real image, not a stand-in** (see §8 for full results): image builds →
container starts → k3s reaches `Ready` → backend starts (`API server running on :4000`, no
crash) → nginx starts → browser UI returns 200 → terminal WebSocket opens and shows the
RootNode banner → `kubectl get nodes` run through the terminal returns real cluster state →
`deploy-nginx` scenario solved via the terminal and validated via
`POST /api/scenarios/deploy-nginx/validate` → all 5 checks pass.
---
## 2. Gateway architecture
New component: `platform/gateway/`. The only student-facing path to disposable Lab
environments — browsers never talk to a Lab container, the Provisioner, or Postgres directly.
```
Browser
│ (one stable origin — the Gateway)
▼
Gateway
├── /api/environment/* → calls the Provisioner internally (shared secret never
│ leaves the Gateway process)
├── /lab/<token>/* → HTTP + WS proxy into that specific student's Lab
│ container, resolved by opaque token via the Provisioner
└── everything else → proxied to the persistent "identity" backend (roster,
login, curriculum, progress, and the platform-shell
frontend) — OR to the student's own Lab, if a valid
lab-session cookie is present (see §5 for why)
```
**Why a persistent "identity" instance exists, and what it is:** the roster/curriculum/
progress SQLite data (built in the pre-Phase-2A slice) needs to be reachable *before* a student
has a Lab — they pick their name and see their curriculum before ever clicking Launch Lab. That
data can't live inside an ephemeral, disposable-by-design Lab container. Phase 2A runs it as
one plain, non-ephemeral instance of the existing, unmodified Lab image (`identity` in
`platform/docker-compose.dev.yml`) — same image, same code, just never destroyed/reset by the
Provisioner, the same way Postgres or the egress proxy are infrastructure rather than
per-student resources. This is a deliberate zero-new-code choice (see §9 known limitations for
its cost).
**Provisioner addition:** `GET /environments/by-token/:token` (small, internal-only, same
shared-secret auth as every other Provisioner route) — resolves a routing token to its
environment for the Gateway. This is the only change to `platform/provisioner/`.
**What the Gateway never does:** expose a Lab container's port publicly, expose the Docker
API, expose the Provisioner's shared secret to a browser, or trust a client-supplied
environment ID for anything without checking it against the Provisioner's own record.
---
## 3. Student flow
**A. Entry screen** (`frontend/src/platform/EntryScreen.jsx`) — cohort dropdown → name
dropdown (existing `GET /api/roster/cohorts` / `GET /api/roster?cohort=`) → `POST /api/login`.
No password. A visible note states this is pilot identity, not production auth.
**B. Session** — `student_id` stored in `localStorage` only (`frontend/src/platform/api.js`),
sent as `X-Student-Id` on every subsequent platform request. Every authenticated call throws
if no student is stored, rather than silently going out anonymously — this is what "prevent
accidental anonymous progress writes" means in practice.
**C. Dashboard / "Up Next"** (`StudentDashboard.jsx`) — renders `GET /api/curriculum/path`
as-is: the existing RootNode path (Foundation → Workloads → Networking → Configuration →
Troubleshooting → Real Ops → Capstone), no new structure invented. Capstone stays the locked
placeholder from the earlier slice.
**D. Launch Lab** — `POST /api/environment/launch` (Gateway): reuses a non-terminal
environment if one exists, otherwise provisions one via the Provisioner. UI states map directly
to environment status: none → provisioning/resetting (polled every 2.5s while in flight) →
running → error, each rendered distinctly (`EnvironmentPanel` in `StudentDashboard.jsx`). On
`running`, the browser navigates to `launch_path` (`/lab/<token>/`), with the student_id
attached as a `?sid=` query param for the Gateway's ownership check (see §5).
**E. Lab session** — the existing, **completely unmodified** Lab UI (`App.jsx` and everything
under `frontend/src/components/`). It has no student-identity awareness and needs none —
identity and progress stay entirely in the Platform layer, per the explicit requirement not to
inject student identity into the privileged Lab container.
**F. Reset/Recreate** — one button, "Recreate Lab," calling `POST /api/environment/reset`
(destroy + provision against the same environment row, per the Phase 1 lifecycle — same
`environment_id`, fresh container). No raw destroy-without-recreate control is exposed to
students; that keeps the surface to exactly what's safe, per the explicit requirement.
---
## 4. API routes added
All on the Gateway (`platform/gateway/src/server.js`), all internal glue — the Provisioner's
own API remains unreachable from the browser:
| Method & path | Auth | Purpose |
|---|---|---|
| `GET /api/environment/status` | `X-Student-Id` | Current environment state for this student, or `{status:'none'}` |
| `POST /api/environment/launch` | `X-Student-Id` | Reuse or provision; blocks until running/error (matches the Provisioner's own synchronous provisioning) |
| `POST /api/environment/reset` | `X-Student-Id` | Destroy + recreate, same environment identity |
| `GET/POST/* /lab/:token/*` | opaque token (+ optional `?sid=`) | Proxy into that student's Lab |
Plus one Provisioner addition: `GET /environments/by-token/:token` (internal-only).
Responses to the browser never include `internal_ip`, `container_id`, `network_id`,
`lab_host_id`, `environment_id`, or the raw routing token beyond its one legitimate use as a
navigation path segment (`launch_path`) — see §6.
---
## 5. Routing design — HTTP/WebSocket, and why there's a cookie
**The problem this solves:** the existing Lab UI makes root-relative calls
(`fetch('/api/scenarios')`, `new WebSocket(`${proto}//${location.host}/shell-ws`)`) — it has no
idea it might be served under a `/lab/<token>/` prefix, and it shouldn't have to (preserving
the existing architecture unmodified was an explicit requirement). Naively prefix-stripping
only the *first* request would leave every subsequent asset/API/WS call unrouted.
**Solution:** the Gateway sets an `HttpOnly`, `SameSite=Lax`, `Path=/` cookie (`rn_lab`,
8h max-age) on the first `/lab/<token>/` response. Every later request from that browser —
regardless of path — carries it automatically. Routing priority:
1. `/lab/<token>/*` (explicit) → resolve token, strip prefix, proxy, (re)set the cookie.
2. `/api/environment/*` → always the Gateway's own routes, **never** affected by the cookie —
this is what keeps "student is inside their lab" from accidentally hijacking the dashboard's
own environment-status calls.
3. Anything else, unprefixed → if the `rn_lab` cookie is present and resolves to a `running`
environment, proxy there (unprefixed) — this is what makes the *existing, unmodified* Lab
UI's own `/api/scenarios`, `/shell-ws`, static assets, etc. resolve correctly once a browser
has navigated into a lab session, with zero changes to that UI's code. Otherwise, proxy to
the identity backend (this is what serves `/`, the platform-shell bundle, `/api/roster`,
`/api/curriculum/path`, etc.).
WebSocket upgrades follow the identical priority (`server.on('upgrade', ...)` in
`gateway/src/server.js`), so the terminal's `/shell-ws` connection is routed the same way as
any other request.
**Stable URL, even across recreate:** the student-facing URL is always `/lab/<routing_token>/`
— the token lives on the `environments` row and does not change across `reset` (same row, new
container/network/IP). A bookmarked or previously-loaded lab URL keeps working after a
recreate without the student needing a new link.
---
## 6. Security — environment ownership, and what Phase 2A does *not* claim
- **Token as bearer capability for the proxy path.** A `/lab/<token>/` request is authorized by
possession of the token itself (CSPRNG, 32 bytes, never guessable) — the same trust model as
a signed download link. This is a deliberate, stated design choice, not an oversight.
- **`?sid=` as an additional check where it's available.** A full top-level browser navigation
cannot carry a custom header, so the Dashboard's "Launch/Continue Lab" link appends
`?sid=<student_id>` to the URL. If present, the Gateway compares it against the environment's
actual owning `student_id` (from the Provisioner) and rejects (403) on mismatch. This is
strictly additive security on top of token-alone — its absence (e.g., a bookmarked link
without the param) falls back to token-only trust, not a hole opened by omitting it.
- **`/api/environment/*` ownership is checked against the Provisioner's own record**, not
assumed from the header — `listForStudent`/`reset` always re-verify `env.student_id`.
- **Explicitly not production auth.** Per the governing requirement: routing-token-alone,
student-supplied-environment-ID-alone, and `X-Student-Id`-alone are each individually
insufficient and documented as such — Phase 2A's actual security property is the
*combination*, appropriate for a supervised pilot, not a claim of production-grade identity.
Production SSO/OIDC is explicitly future work (§9).
- **Nothing sensitive reaches the browser**: no internal Docker IPs, no Docker/Provisioner
endpoints, no shared secret, no Postgres credentials — verified by inspecting every Gateway
response shape in `server.js`'s `publicEnvView()`.
---
## 7. Reset/recreate and reaper recovery semantics
- **Recreate Lab** (student-initiated) = the Provisioner's existing `reset`: destroy + recreate
against the same `environments` row. Same `environment_id`/routing token, same
`/lab/<token>/` URL, fresh container/network/IP. Academic progress is untouched because it
never lived in the Lab container to begin with (Phase 1 property, unchanged).
- **Reaper-caused expiry** (idle timeout, no student action): the environment transitions to
`destroyed`. `GET /api/environment/status` correctly reports `none`/`destroyed` afterward —
the Dashboard's "Launch Lab" button provisions a **new** environment (new `environment_id`,
new token, new URL) rather than trying to resurrect the old row, matching the Provisioner's
actual lifecycle model. Progress survives because, again, it was never in the Lab.
- **Unexpected Lab death** (crash, not reaped): the Gateway's `isReachable()` check
(`status==='running' && internal_ip` present) fails closed — a dead container never gets
silently proxied to. The student sees the `error` state with a "Try Again" action.
---
## 8. Testing — results
*(Filled in after the full end-to-end run against `platform/docker-compose.dev.yml`; see the
Phase 2A final report for the authoritative results table. Summary below.)*
- Lab image: node-pty load, k3s Ready, terminal, kubectl, scenario validation — **all pass**,
§1 above.
- Gateway HTTP/WS proxying, two-student isolation, cross-student rejection, stable URL across
recreate, reaper recovery, existing single-container dev mode, scenario files untouched,
frontend/backend builds, Provisioner test suite — see final report for the full table and
exact commands run.
---
## 9. Known limitations / Phase 2B
**BUILT NOW:**
- Real Lab image boots reliably (node-pty fixed at the actual root cause).
- Gateway: HTTP + WS proxying, environment-aware routing, ownership checks, no public Lab
ports, no exposed Docker/Provisioner internals.
- Student entry (roster pick-name) → dashboard (curriculum + environment status) → Launch Lab
→ Lab session → Reset/Recreate, all wired through the Gateway as the single origin.
- Reaper-recovery and reset semantics correctly reflected in the student-facing states.
**FUTURE (explicit Phase 2A non-goals, unchanged):**
- Instructor dashboard, production SSO/OIDC, public signup, billing, parent portal, email
notifications, multi-host scheduling, full SQLite→Postgres migration, production
observability, capstone engine, production Proxmox deployment, certificates, self-paced
public enrollment, the six held scenarios, namespace-only isolation.
**Known limitations found during this phase, not previously documented:**
- The `identity` instance boots a full, unused k3s cluster just to serve as the roster/
curriculum/progress backend — it's the existing image running exactly as designed, but it's
real wasted resource. A Phase 2B optimization is a slimmed identity-only variant of the image
(skip k3s entirely) — not done here, to keep this phase at zero changes to the Lab image's
own runtime behavior beyond the node-pty fix.
- `identity` requires `network_mode: bridge` explicitly in `docker-compose.dev.yml` — under
Compose's own auto-created bridge network, k3s failed with "no default routes found in
/proc/net/route" at startup. Root cause not fully chased down (a Compose-network-vs-classic-
bridge routing-table timing/detection difference); the explicit `network_mode: bridge`
workaround is confirmed reliable and is what's shipped. Worth a deeper look before this
matters for a real deployment (bridge mode isn't compatible with a per-student isolated
network model anyway — the production identity service placement needs its own design pass).
- The Gateway's environment-lookup cache (5s TTL) is in-memory and per-process — fine for one
Gateway instance (Phase 2A's actual topology), would need a shared cache (or none) if the
Gateway is ever horizontally scaled.
- `?sid=` ownership verification only covers the initial `/lab/<token>/` navigation, not every
subsequent proxied request (by design, per §6) — acceptable for the stated pilot threat
model, revisit if this graduates past pilot use.