diff --git a/README.md b/README.md index fcf7866..c822423 100644 --- a/README.md +++ b/README.md @@ -127,22 +127,30 @@ Everything runs inside one image managed by `scripts/entrypoint.sh`. ## Multi-Student Deployment (RootNode Academy addition) -The upstream project assumes one learner, one container. RootNode Academy runs it across three -delivery modes. **Never publish the raw privileged container** — front it with isolation. +The upstream project assumes one learner, one container — this image is still exactly that, +unmodified. RootNode Academy multiplies it into a real platform via a separate provisioning +layer rather than changing the image itself: see `docs/LAB_ARCHITECTURE.md` for the +architecture and `platform/provisioner/` for the Phase 1 implementation (Postgres-backed +environment registry, per-student Docker networking, firewall-enforced egress, idle reaper — +see `platform/provisioner/README.md` for how to run it). -| Mode | Provisioning | Isolation | -| ---- | ------------ | --------- | -| In-person camp | Pre-spun container per seat, torn down after | Lab VLAN, no public ingress | -| Live cohort | Container per enrolled student for the session window | Per-student auth at the proxy | -| Self-paced | On-demand spin-up per session, idle-timeout teardown | Per-user volume + auth + resource limits | +**Never publish the raw privileged container** — front it with isolation. The one-container +`docker run --privileged -p 7554:80 ...` command above stays supported for local development +and scenario authoring; it is not how this gets deployed to real students. -Hardening checklist for shared infra: +Key properties of the Phase 1 platform (not this image): -- One container **per student** — never share a cluster between learners. -- Apply CPU/memory limits so one runaway pod can't starve a host. -- Put authentication in front (proxy-level), never expose `:7554` to the internet. -- Persist each student's `/data` to a dedicated volume. -- Idle-timeout + auto-teardown for self-paced sessions to control cost. +- One container **per student**, its own Docker network, its own firewall chain — never a + shared cluster between learners. +- Progress/identity live outside the disposable container (currently this image's own SQLite, + moving to the Platform's PostgreSQL in a later phase) — destroying a student's environment + never loses their academic record. +- CPU/memory/PID/storage limits are enforced per container, config-driven. +- Egress is restricted to an allowlisted proxy at the firewall layer, not left open. +- Idle/abandoned environments are automatically cleaned up. + +Production topology and what's built vs. not yet: `docs/LAB_ARCHITECTURE.md` and +`docs/PHASE1_PLATFORM_FOUNDATION.md`. --- diff --git a/docs/ENGINEERING_SOP.md b/docs/ENGINEERING_SOP.md new file mode 100644 index 0000000..34a3d77 --- /dev/null +++ b/docs/ENGINEERING_SOP.md @@ -0,0 +1,188 @@ +# RootNode Academy — Engineering SOP + +Living doc. Covers the "how" of building this platform — the "why" and business/curriculum +process live in the Business Pack (`~/Documents/CDAA Technologies/07 RootNode academy/RootNode_Academy_Business_Pack/`), +this doc doesn't duplicate that. Update this file in the same commit as the change it describes. + +## 1. Where this actually lives + +- **Repo:** `~/Dev/Gitea/ced4568/rootnode-academy` — remote is **Gitea** + (`git.synthossystems.com/ced4568/rootnode-academy`), not GitHub. +- **Gotcha:** `~/Dev/GitHub/ced4568/kubekosh` is a *different* clone of the same original + upstream fork, still pointed at GitHub, and it's behind (missing the RootNode logo commit + and the Real Ops enforce-limits scenario as of this writing). It's not deployed anywhere. + Don't edit it by mistake — always confirm `git remote -v` shows `gitea:` before making changes. +- Known-good baseline as of this doc: commit `6ddfb1c` ("add probe-port-mismatch and + enforce-limits to rootnode-real-ops bundle"). Everything below builds on top of that, + clean working tree. + +## 2. Architecture at a glance + +``` +backend/server.js Express API + SQLite (better-sqlite3) + WebSocket PTY terminal — the + "Lab image," single-tenant, unmodified by the Platform work below +frontend/ Vite + React SPA, built to frontend/dist and served by nginx +scenarios/data/ One JSON file per scenario (task or MCQ) +scenarios/bundles/ Flat scenario_id lists grouped by cert track (basics/appdev/admin/security/real-ops) +scenarios/rootnode-path.json The RootNode guided learning path (staged, see §4) +scripts/entrypoint.sh Container boot: starts k3s, nginx, node backend, sets the shell banner/PS1 +Dockerfile ubuntu:22.04 + k3s + node20 + nginx, single image, one port (80) + +platform/provisioner/ Phase 1 Platform Provisioner — disposable per-student Lab lifecycle, + PostgreSQL registry, egress firewall. See §8 and + platform/provisioner/README.md. +platform/egress-proxy/ Shared allowlisting egress proxy every Lab environment's traffic must + pass through. +``` + +Single Express process, single SQLite file at `/data/progress.db` (or `$PROGRESS_DB`), +single embedded k3s cluster **per Lab environment**. The Lab image itself is still +single-tenant by design — see §5 for identity, and §8 for how Phase 1 multiplies it into a +real multi-student platform without modifying it. + +## 3. Local dev / safe smoke-testing + +Never test schema changes against the real `/data/progress.db` first. Point at a scratch file: + +```bash +cd backend && npm install # first time only +node --check server.js # syntax check, catches typos fast +PROGRESS_DB=/tmp/scratch/progress.db node server.js & +curl -s http://localhost:4000/api/health +# ...exercise the endpoints you changed... +fuser -k 4000/tcp # stop it +``` + +To test a migration against **real prior data** without touching prod: copy the actual +`/data/progress.db` out of the running container/volume, point `PROGRESS_DB` at the copy, +boot, and verify the data you expect is still there before ever pointing at the real file. + +kubectl/k3s-backed endpoints (`/setup`, `/validate`, `/context`, the WS terminal) won't work +outside the container — that's expected. API-and-DB-only endpoints (roster, login, progress, +curriculum, bundles, scenarios list) all smoke-test fine on a bare host. + +## 4. Curriculum content + +Two separate structures, don't confuse them: + +- **Bundles** (`scenarios/bundles/*.json`) — flat scenario_id lists, one per cert track. + Loaded by `loadBundles()`/`GET /api/bundles`. Adding a bundle file here = adding a new + cert-track library. Every scenario referenced must have a matching file in `scenarios/data/`. +- **RootNode path** (`scenarios/rootnode-path.json`) — the guided onboarding sequence, + separate from bundles on purpose (it's *staged*, bundles are *flat*, and the bundle loader + would choke on a staged file if it were dropped in `scenarios/bundles/`). Loaded by + `loadCurriculumPath()`/`GET /api/curriculum/path`. Stages are Foundation → Workloads → + Networking → Configuration → Troubleshooting → Real Ops → Capstone, currently built by + reusing existing `1-k8s-basics` and `rootnode-real-ops` scenarios grouped by their existing + `category` field — no new scenario content was written for this. **Capstone stage is a + locked placeholder** (`"locked": true`, empty `scenario_ids`) — the capstone engine itself + is a separate, not-yet-built roadmap item (see §7, item 6). Do not put real scenarios in it + until that engine exists, or you'll reproduce the "duplicate beginner scenarios as capstone" + anti-pattern this path was built to avoid. + +To add/reorder a stage: edit `rootnode-path.json` directly, referencing existing +`scenarios/data/*.json` ids. No backend code change needed — it's read fresh via +`POST /api/cache/reload` or on server restart. + +## 5. Student identity — what's built, what isn't + +**Built (backend only, as of this doc):** +- `students` table: `student_id` (`student-NNN`, sequential), `display_name`, `cohort`, + `created_at`, `last_active_at`. +- Login model: **instructor roster, pick-your-name, no password.** `POST /api/roster` adds + a student (instructor-only action, currently unauthenticated at the API level — see + "not built" below). `GET /api/roster?cohort=X` lists names for a dropdown. + `POST /api/login` just validates the id exists and returns it; the frontend is expected to + hold onto `student_id` and send it back as an `X-Student-Id` header on every request. +- `progress` table now keyed on `(student_id, scenario_id)` instead of `scenario_id` alone — + each student's completion/attempts/hints are isolated. Verified by smoke test: two students + answering the same scenario don't affect each other's state, and pre-migration data was + preserved under `student_id = 'unassigned'`. +- Exam `sessions` now carry `student_id` (set at session start), so exam history and the + timer-reset-on-submit logic are correctly scoped per student instead of leaking across + whoever's browser hits the API next. + +**Not built yet — do not assume these exist:** +- **No frontend UI.** No login screen, no name picker, no roster-add form. The API works; + nothing calls it yet. This is the next slice. +- **No auth on `/api/roster` POST.** Anyone who can reach the API can add a "student." + Fine for a dev instance, not fine once this is reachable by actual students — needs an + instructor-only gate before that happens (ties into the instructor dashboard, roadmap + item 3). +- **No concurrency / isolation.** Exam sessions are still global-single-active + (`WHERE status='active' LIMIT 1` — see `server.js`), and there is one shared k3s cluster + for the whole instance. Per-student identity ≠ per-student environment. **Multiple students + cannot yet use this platform at the same time without stepping on each other.** That's + roadmap item 4 (isolation) and it gates real classroom/minors use, same as the plan called + out — this SOP is not claiming that gate is cleared. + +## 6. Schema migration pattern + +SQLite can't `ALTER` a primary key. The codebase's existing convention (see `progress` table +history) is: + +1. `PRAGMA table_info()` → check if the new column/shape is already there (idempotent, + safe to run on every boot). +2. Simple new nullable column → `ALTER TABLE ... ADD COLUMN`. +3. Primary-key or structural change → build `
_new` with the new shape, `INSERT ... + SELECT` the old data across with sane defaults for new required columns, `DROP TABLE`, + `RENAME TO`. Never drop data — give it a legacy/default value (`'unassigned'`, etc.) + instead of discarding rows. +4. Test the migration against a **copy of real data** (§3) before it ever runs against + `/data/progress.db` in the container. + +## 7. Roadmap status + +Order agreed 2026-08-17/18. Status as of this doc: + +1. RootNode identity/branding pass — **partial.** Shell banner fixed + (`⎈ RootNode Academy Lab — node: k8s-lab`). Upstream KubeKosh attribution left untouched in + footer/README/LICENSE. Broader "make Real Ops first-class" UI work not started. +2. RootNode curriculum mode — **backend done** (§4), frontend not started. +3. Student accounts / persistent progress — **backend done** (§5), frontend not started, + no instructor auth yet. +4. Instructor dashboard — not started. +5. Per-student isolation — **Phase 1 (platform foundation) done**, see §8. Ephemeral + per-student Lab environments with Postgres-backed lifecycle, per-environment Docker + networking, and firewall-enforced egress restriction are implemented and tested on a single + dev machine. **Still not cleared for minors** — no Gateway/frontend wiring yet (environments + are only reachable via the internal Provisioner API), no instructor auth on top of it, and + the production two-VM Proxmox split this depends on doesn't exist yet. See + `docs/PHASE1_PLATFORM_FOUNDATION.md` final report for exactly what's verified vs. not. +6. Scenarios → teaching modules (incident context/investigation/prevention) — not started. +7. Capstone/exam engine — not started (placeholder stage only, see §4). +8. Progressive hint levels — not started. +9. Production observability — not started. +10. CI scenario testing — not started. +11. Container/build optimization — explicitly deprioritized by design (platform works, not urgent). + +Next slice: either the deferred frontend login (name picker → `X-Student-Id` header, curriculum +view), or the Phase 2 platform work (Gateway, frontend wiring to the Provisioner, instructor +auth) — see the Phase 1 final report for the concrete Phase 2 list. Not decided here. + +## 8. Platform Provisioner (Phase 1) + +Full design: `docs/LAB_ARCHITECTURE.md` + `docs/PHASE1_PLATFORM_FOUNDATION.md`. Operational +reference (config, local dev, known limitations): `platform/provisioner/README.md`. This +section is just the map. + +- **What it is:** an internal-only service that provisions a disposable, fully isolated Lab + environment per student — its own Docker network, its own container running the existing, + unmodified Lab image, its own firewall-enforced egress. Postgres-backed registry + (`environments`, `environment_events`, `lab_hosts`), idle reaper, startup reconciliation. +- **What it is not (yet):** reachable by a student. There's no Gateway and no frontend wiring + — it's exercised directly via its internal API (shared-secret auth) or + `platform/provisioner/test/run.js`. Nothing here is a live feature. +- **Where identity lives:** unchanged from §5 — `students`/`progress` are still on the Lab + image's own SQLite. `environments.student_id` is deliberately opaque (no FK) until a later, + separate migration puts roster/progress on the same Postgres instance as the environment + registry. +- **Local dev:** `platform/provisioner/docker-compose.dev.yml` runs Postgres + the egress + proxy + the Provisioner on one machine — the documented Phase 1 dev exception. Production + is a two-VM Proxmox split (Platform VM / Lab Host VM); the dev compose file is explicitly + not that. +- **Test procedure:** `platform/provisioner/test/run.js` — 12 automated integration tests + against a live dev stack (isolation, firewall ordering, reset/destroy cleanup, audit + events). Some properties (idle reaper timing, remote TLS lab-host, reconciliation-on-crash, + real-image k3s behavior) were validated manually during implementation rather than + automated — see the Phase 1 final report for what and why. diff --git a/docs/LAB_ARCHITECTURE.md b/docs/LAB_ARCHITECTURE.md new file mode 100644 index 0000000..6389d78 --- /dev/null +++ b/docs/LAB_ARCHITECTURE.md @@ -0,0 +1,482 @@ +# RootNode Academy — Ephemeral Lab Architecture + +Status: **Phase 1 (platform foundation) IMPLEMENTED 2026-08-18** — see +`docs/PHASE1_PLATFORM_FOUNDATION.md` for the concrete schema/API/test results, and +`platform/provisioner/README.md` for how to run it. This document remains the architecture +record for the full ephemeral-lab direction; later phases (Gateway, frontend wiring, instructor +dashboard, production Proxmox deployment — see the Phase 1 report's "Future Phase 2 items") +build on it but aren't built yet. Supersedes the namespace-isolation +(`shared_namespace`/`ephemeral_cluster`) approach that was in flight before this redirect — see +`scenarios/SCHEMA.md`'s `isolation_type` note for the pointer back here. + +Committed direction: **one fully isolated, disposable environment per student. No shared +Kubernetes control plane, ever.** This is the long-term architecture, not a stopgap. + +**Decisions locked 2026-08-18** (resolves the four open questions this doc originally ended +with — see §2–§5 below for where each lands, and `docs/PHASE1_PLATFORM_FOUNDATION.md` for the +Phase 1 build plan against these decisions): +1. **Egress:** restricted, explicit allowlist. No unrestricted internet egress, ever. No route + to trusted RootNode management/CDAA/home/business networks, under any allowlist entry. +2. **Platform DB:** PostgreSQL. The Platform control plane is not built around SQLite. The + existing single-container local dev mode keeps using its own local SQLite as today — + unaffected, unrelated database. +3. **Suspend/resume:** experimental only, not load-bearing. The guaranteed lifecycle is + `provision → run → destroy → recreate`. Persistent academic state lives outside the Lab + specifically so destroying one never loses student progress — suspend/resume is a possible + future optimization, not something correctness depends on. +4. **Resource sizing:** not hard-coded. Quotas are policy/config-driven. Target host class is + ~16 cores/32 threads, 128GB RAM, SSD/NVMe — actual per-lab limits and concurrency get set + by benchmarking against that class, not guessed up front. + +**Corrections locked 2026-08-18** (round 2 — architectural fixes required before Phase 1 +implementation begins; folded into §1, §4, §7, §9 below, not just listed here): + +- **A. VM boundary.** Privileged Lab containers must never share an OS/VM boundary with the + Platform (Postgres, control-plane) in production — separate VMs under Proxmox, connected by + a Provisioner-initiated control channel only. The Lab Host is a *configured target*, not an + assumption that Docker is local. Single-machine is still fine for Phase 1 development — see §1/§9. +- **B. Egress is a firewall control, not an app convention.** `HTTP_PROXY`/`HTTPS_PROXY` + configure well-behaved apps for convenience; they enforce nothing. The firewall is the + control, and rule *ordering* matters — required infrastructure (proxy, DNS) must be + explicitly permitted before the blanket RFC1918 drops, or those drops shadow the allow + rules and nothing works. See §4 — this also corrects a real ordering bug in the original + design (explained there). +- **C. `student_id` is opaque in Phase 1.** No real foreign key to a `students` table — + students/progress stay on SQLite until a later, separate Platform DB migration. Documented, + not silently assumed. See §3/§9. +- **D. Lab host is an abstraction from day one**, even with exactly one configured host in + Phase 1 — the Provisioner is never allowed to hard-code "Docker is local." See §4. + +--- + +## 0. What already exists (grounding, not invention) + +The current repo (`~/Dev/Gitea/ced4568/rootnode-academy`) already documents almost this exact +intent in its own README, just never built out: + +> "The upstream project assumes one learner, one container. RootNode Academy runs it across +> three delivery modes... One container **per student** — never share a cluster between +> learners... Put authentication in front (proxy-level)... Idle-timeout + auto-teardown." + +And the existing image already **is** a complete single-student lab: nginx + React frontend + +Node/Express backend + `node-pty` WebSocket terminal + a single-node k3s cluster, all in one +`--privileged` container (confirmed required — k3s needs kernel namespaces/cgroups, per +`README.md:52`). That container is the unit this design multiplies, not something it replaces. + +What changes: today that one container also owns identity/progress (SQLite at +`/data/progress.db`, one volume per student, mounted by hand). This design pulls that +ownership out into a persistent Platform, and makes the container itself fully disposable. + +--- + +## 1. Proposed architecture + +Two roles — and in production, on **two separate VM/OS boundaries**, not just two logical +components sharing a box. Correction A drives this: `--privileged` Docker containers are not +a security boundary from their own Docker host, so the boundary has to be a real VM boundary, +not a process boundary. + +**Production topology:** + +``` +Proxmox VE (or equivalent hypervisor) +│ +├── RootNode Platform VM (trusted — never runs student code) +│ │ +│ │ Platform API roster · login · progress · curriculum · exam records · +│ │ environment registry · instructor controls +│ │ Platform DB PostgreSQL — environments + audit log now; +│ │ students/progress join later (§3/§9, correction C) +│ │ Provisioner talks OUT to a *configured* Lab Host endpoint — +│ │ never assumes Docker is local (§4, correction D) +│ │ Gateway opaque-token routing to student lab containers; +│ │ only component with a leg in both worlds +│ │ +│ ▼ (Provisioner-initiated control channel only — TCP+TLS to the Lab +│ Host's Docker API. Nothing on the Lab Host VM ever initiates a +│ connection back. See §4/§7 for why this direction matters.) +│ +└── RootNode Lab Host VM (untrusted — this is where privileged student + │ containers actually run; holds nothing the + │ Platform needs protected) + │ + ├── Student A's LAB ── own private network, own nested k3s, no identity + ├── Student B's LAB ── own private network, own nested k3s, no identity + └── Student C's LAB ── own private network, own nested k3s, no identity +``` + +Each Lab is **today's `Dockerfile` and `entrypoint.sh`, functionally unchanged** — still runs +k3s + the terminal + scenario setup/validate/teardown exactly as now. The only change inside +it: it stops being the system of record for who's using it or what they've completed (§3, §9). + +A compromised student container's worst case is the Lab Host VM — never Postgres, never the +Platform API, because there is no network path from that VM back to the Platform VM at all +(§4, §7). That containment is the actual point of the two-VM split, not a deployment nicety. + +**Phase 1 development exception:** running the Platform and a Lab container on one machine is +fine for local testing (§9) — the Provisioner just gets configured with a single `lab_host` +pointing at the local Docker socket instead of a remote TLS endpoint. The *code* never +special-cases "local" vs "remote" — same abstraction either way (§4/§9 of +`docs/PHASE1_PLATFORM_FOUNDATION.md`). + +--- + +## 2. Environment lifecycle + +State machine per environment, tracked in the Platform DB (`environments` table, §3): + +**Guaranteed path** (what correctness depends on): `requested → provisioning → running → +resetting → running` (reset loop, same route) and `(provisioning|running|error) → destroying +→ destroyed`. `provisioning → error` and `running → error` cover failed/lost environments; +`error → destroying → destroyed` (or `→ resetting → running`, when recoverable) is the way out. + +``` +requested → provisioning → running ─┬─→ resetting → running (clean slate, same route) + └─→ destroying → destroyed +provisioning → error (failed to reach ready state) +running → error (health check fails mid-session) +error → destroying → destroyed (or → resetting → running, if recoverable) + +[EXPERIMENTAL, not relied on for correctness — see decision #3] +running ──suspend──→ suspended ──resume──→ running (or → error, if resume fails) +``` + +- **provision** — Provisioner picks a host with capacity (§5), creates a private network for + this student (§4), starts a container from the pinned Lab image, generates a random + high-entropy routing token, waits for the existing readiness signal (k3s API ready + backend + `/api/health` OK — same check the current entrypoint already does), writes the `environments` + row, hands the route back to the Platform API. +- **start** — used only for `provisioning → running` (starting a freshly created container). + There is no supported path that starts a previously-stopped container in the guaranteed + lifecycle — see suspend/resume below. +- **suspend/resume — EXPERIMENTAL, decision #3.** `docker stop`/`docker start` on a k3s-in-container + Lab is *not assumed to work* and is not built as part of the guaranteed lifecycle. It's + worth testing (see the Phase 1 test plan) because if it turns out reliable it's a cheap + latency win later — but nothing in the platform depends on it succeeding. The only supported + way to bring an environment back is `destroy` + `provision` (or `reset`, which is that same + pair against one `environments` row/routing token). This is also *why* academic state living + outside the Lab (§3) isn't optional — it's what makes "just destroy and recreate" a fully + safe default instead of a data-loss risk. +- **reset** — destroy and recreate the container from the pinned image, **same routing token + and same `environments` row**, so the student's "return to my lab" link doesn't change. + This is the instructor-facing "reset student environment" control. +- **destroy** — stop + remove container, remove its private network, release the routing + token, mark the `environments` row `destroyed` with a timestamp. Progress already lives in + the Platform DB (§3), so destroying an environment never loses academic data — only + in-terminal, not-yet-validated work. +- **Idle reaper** (new, required — not optional) — background job in the Provisioner checks + `last_active_at` per running environment (heartbeat from the Gateway or from Lab `/api/health` + polling) and auto-suspends past an idle threshold, auto-destroys past a longer abandoned + threshold. Without this, "ephemeral by default" is just a description, not a guarantee — + a closed browser tab does not stop a running k3s cluster. + +--- + +## 3. Data model changes + +Everything below lives in the **Platform DB**, not inside any Lab container. + +Already built (previous slice, unaffected by this redirect — still correct under this +architecture): +- `students(student_id, display_name, cohort, created_at, last_active_at)` +- `progress(student_id, scenario_id, status, attempts, ...)` — composite key, already + per-student. +- `sessions` / `exam_progress` — exam records, already student-attributed. + +New: +- `environments(environment_id UUID PK, student_id, status, routing_token, container_id, + lab_host_id FK, internal_ip, cpu_limit, mem_limit_mb, storage_limit_mb, image_ref, created_at, + started_at, last_active_at, destroy_at, error_reason)` — **`student_id` is an opaque external + identifier here (correction C), not a foreign key.** Students/progress stay on the existing + SQLite in Phase 1 (see §9); Postgres has no `students` table yet, so there is nothing to key + against. The Provisioner trusts whatever `student_id` its caller passes — it does not + validate it against a roster. A real FK gets added when students/progress actually migrate + onto this Postgres instance, not before. `lab_host_id`, by contrast, *is* a real FK — both + tables live in the same Provisioner-owned Postgres DB (correction D, full schema in + `docs/PHASE1_PLATFORM_FOUNDATION.md`). +- `lab_hosts(lab_host_id PK, endpoint, enabled, ...)` — the abstraction correction D requires: + which Docker endpoint a given environment's container actually lives on. Phase 1 ships with + exactly one row (pointing at either a local socket in dev or the Lab Host VM's TLS endpoint + in production) — the table exists so environment creation is never coupled to "Docker is + local" as an assumption, even though there's no scheduling logic yet. +- `environment_events(id, environment_id, event_type, actor, detail, created_at)` — audit + trail for provision/start/stop/reset/destroy/error, needed for both incident response and + ordinary "why did my lab disappear" support. + +**Removed from the Lab image's responsibility:** the `students` and roster-facing parts of the +schema I added to `backend/server.js` in the previous slice move to be Platform-only. The Lab +image's own SQLite either goes away entirely or is kept only as scratch/ephemeral state that's +allowed to die with the container — it must not be treated as authoritative for anything. + +**Decision #2: PostgreSQL, not SQLite.** The Platform control plane — the environment registry +specifically, and the target for progress/roster once they migrate over — is Postgres. The +existing single-container local dev mode keeps its own local SQLite exactly as today; that's a +different, unrelated database serving a different purpose (single-tenant local testing), not +something this decision touches. See `docs/PHASE1_PLATFORM_FOUNDATION.md` for the exact schema +being stood up first. + +--- + +## 4. Networking model + +- **One private Docker network per student environment**, not one shared network for all + labs. Docker's default per-network isolation means Student A's container has no L2/L3 path + to Student B's container at all — this is stronger and simpler than trying to firewall + students apart on a shared network. +- **No published ports on Lab containers.** Today's image publishes `-p 7554:80`; under this + design that stops — the only thing that can reach a Lab container is the Gateway, over that + student's private network. + +### Egress — correction B: the firewall is the control, not the proxy env vars + +`HTTP_PROXY`/`HTTPS_PROXY` are set on each Lab container so well-behaved processes (containerd +pulling scenario images) route through the allowlisting proxy cleanly. **That is configuration, +not enforcement** — a process that ignores those env vars and dials out directly must still be +blocked, by something that doesn't depend on the container's cooperation. That something is a +host firewall rule, evaluated on every packet regardless of what any process inside the +container does. + +**Ordering bug in the original draft, corrected here:** the previous version of this section +listed proxy/DNS `ACCEPT` rules before the RFC1918 `DROP` rules in prose, but specified +installing them with `iptables -I` (insert-at-head) in that same top-to-bottom order — which +actually reverses the result, since each `-I` without a rule number pushes the *new* rule to +position 1. Run in that order, the RFC1918 drops end up evaluated *before* the proxy/DNS +accepts, silently blocking the proxy itself if it happens to sit on an RFC1918 address (it +will — it's Platform-side infrastructure). Corrected mechanism (exact rules in +`docs/PHASE1_PLATFORM_FOUNDATION.md` §5): each student network gets its own dedicated iptables +chain, built with `-A` (append, so rules land in the order they're added — no head-insertion +trap), containing explicit `ACCEPT` rules for the egress proxy, DNS, and any specifically +required Platform endpoint *first*, then the RFC1918 `DROP`s, then a final default-`DROP`. That +chain is reached from `DOCKER-USER` (the chain Docker guarantees it won't silently rewrite) via +one jump rule matching that student's subnet. + +**What no student lab may ever reach, regardless of allowlist state:** +- RootNode management networks +- the Platform database (Postgres) +- Docker/Proxmox management interfaces (including the Lab Host VM's own Docker API endpoint — + a Lab container talking back to the thing that provisioned it is exactly the escalation path + this design exists to prevent) +- CDAA infrastructure +- home/trusted networks +- other students' lab networks (already unreachable via Docker's own per-network isolation + above; the RFC1918 drop is a deliberate second layer, not redundant — it also catches any + case where two student networks' address ranges could otherwise see each other at the host + routing level) + +**Decision #1, restated with the ordering fix:** restricted egress, explicit allowlist, no +unrestricted internet access ever, enforced at the firewall layer with required infrastructure +explicitly permitted *before* the blanket denies — not the reverse, and not left implicit. + +### VM-to-VM boundary — correction A + +The Lab Host VM has **no route back to the Platform VM at all**, in either direction, except +the one the Provisioner itself opens. The Provisioner (on the Platform VM) connects *out* to +the Lab Host's Docker API over TCP+TLS with a client certificate only the Provisioner holds; +nothing on the Lab Host VM — student containers or the VM's own OS — can open a connection back +toward the Platform VM. This is enforced at the hypervisor/network level (Proxmox VLAN/firewall +rules between the two VMs), the same category of control as the CDAA/home boundary below, not +something Docker networking alone provides. A compromised student container's blast radius +stops at the Lab Host VM. + +**No route to trusted CDAA/home/business networks, full stop.** This is a host/hypervisor +boundary, not something Docker networking alone can guarantee — the box(es) these containers +run on need to sit on their own segment with no bridge/route to those networks, consistent with +the existing `RootNode_Academy_Business_Pack/06_Technology/01_Technical_Platform_Plan.md` +("dedicated student network/DMZ separated from trusted networks"). This design assumes that +host-level segmentation is a prerequisite, not something the app layer can create on its own. + +### Lab host abstraction — correction D + +The Provisioner never talks to "the local Docker socket" as a hard-coded assumption — it talks +to whichever `lab_hosts` row it's configured to use (§3), resolved to a Docker endpoint that +may be `unix:///var/run/docker.sock` (Phase 1 dev, single machine) or a remote +`tcp://:2376` with TLS (production, per the topology in §1). Phase 1 ships exactly +one enabled `lab_host` and zero scheduling logic — the abstraction exists so adding a second +host later is a config row, not a rewrite. + +- **Gateway** is the only component with a leg in both the public-facing network and every + student's private network. It resolves `routing_token → container internal IP:port` by + querying the Platform API (or a shared registry cache) per request/connection, then proxies + both HTTPS (scenario API, static frontend) and the WSS terminal connection. + +--- + +## 5. Resource model + +- Per-environment caps applied at container-create time: `--cpus`, `--memory` (with + `--memory-swap` pinned equal to `--memory` so nothing swaps), `--pids-limit` (fork-bomb + guard). Storage: needs a size-capped volume — exact mechanism (`--storage-opt size=`, a + loopback-backed bind mount, or a tmpfs cap) depends on the host's actual Docker storage + driver, which I haven't inspected — needs to be confirmed against the real deploy target + before picking one. +- **Decision #4: policy-driven, not hard-coded.** Limits come from config, not constants in + code — a resource-policy file the Provisioner reads at startup (and can reload), not numbers + baked into the create-container call. No specific numbers are chosen yet. Target host class + for planning purposes: ~16 cores/32 threads, 128GB RAM, SSD/NVMe. Actual per-lab limits and + how many environments run concurrently on that class get set by benchmarking (Phase 1's test + plan includes the first real measurements), not by guessing backward from the spec sheet. +- **Admission control:** Provisioner refuses to `provision` past the point where already-running + environments' summed limits would exceed host capacity. Simple counter/gate to start; can + become real bin-packing later if there's ever more than one host. + +--- + +## 6. Routing/auth model + +1. Student logs in at the Platform (existing roster/pick-your-name flow, unchanged). +2. Frontend asks the Platform for "my lab." If none running, Platform calls the Provisioner to + `provision` (or `start`, if one's suspended). +3. Platform returns an **opaque, high-entropy routing token** scoped to + `(student_id, environment_id)`, short TTL, revoked on `destroy`. +4. Frontend connects to the Gateway using that token — `https://labs./e//...` + and `wss://labs./e//shell-ws`. The Lab container itself never sees a + student identity; it's purely an execution surface addressed by an opaque route. +5. Gateway validates the token maps to a currently `running` environment before proxying — + rejects stale tokens from destroyed/reset environments outright, so there's no window where + an old link reaches a different (possibly now-different-student's) container. +6. **Instructor auth** extends the shared-passcode model already used for the roster-add + endpoint: instructor-only calls (`list environments`, `reset`, `terminate`, `resource usage`) + go through the Platform API under that same gate, which then calls the Provisioner + internally. Students and the public internet never talk to the Provisioner or the Docker + socket directly — only the Platform backend does. + +--- + +## 7. Security boundaries (defense in depth) + +1. **Browser ↔ Gateway** — public-facing, TLS, opaque routing tokens only. Nothing + cluster-admin-shaped is reachable from here. +2. **Gateway ↔ Lab container** — private per-student Docker network only, no published ports, + Gateway is the sole entry point. +3. **Inside the Lab container** — `--privileged` is required for k3s (confirmed, not + avoidable — this isn't a place to try to "hold the line" without breaking k3s). The + mitigation is containment, not avoidance: this privileged container has no reachable + network path to anything except the Gateway, runs as a throwaway that gets destroyed/reset + routinely, and cluster-admin *inside a student's own disposable cluster* is expected and + fine — that's the entire point of full isolation, a student doing CKS-tier cluster-admin + work only ever affects their own throwaway cluster. +4. **Lab container ↔ its own Docker host (correction A — the boundary that actually matters + here)** — `--privileged` is explicitly **not treated as a sufficient isolation boundary from + its own Docker host.** In production the Lab Host VM holding these containers is a separate + VM from the Platform VM, with no route back to it at all (§4). If a student ever escapes + their container, the ceiling is the Lab Host VM — never Postgres, never the Platform API, + never another student's environment. +5. **Provisioner ↔ Lab host's Docker API** — only the Provisioner can invoke it, over a + connection *it* initiates (TCP+TLS, client-cert auth, per §4's lab-host abstraction). The + Docker endpoint is never student-reachable, never internet-reachable, and nothing on the Lab + Host VM can initiate a connection back toward the Provisioner or the Platform VM. +6. **Platform VM ↔ other networks** — dedicated segment, no bridge to home/business/CDAA + trusted infrastructure, matching the existing Technical Platform Plan doc. Prerequisite, not + something this design builds. Same standard applies to the Lab Host VM independently — it + isn't exempted just because it's "the untrusted one." +7. **Routing tokens** — CSPRNG-generated, short TTL, invalidated on any lifecycle transition + out of `running`. + +--- + +## 8. Failure/recovery behavior + +- **Provisioning fails** (host at capacity, image pull fails, k3s doesn't reach ready state in + time) — environment marked `error`, student gets an explicit "your lab failed to start, try + again" message (never a silent hang), one automatic retry, then surfaced to the instructor + dashboard (future) as a flagged failure. +- **Mid-session crash** — Gateway's health check detects the upstream is unreachable, marks + the environment `error`, offers the student a "restart my lab" action (→ `reset`, since a + crashed in-container k3s is unlikely to resume cleanly). Because progress/identity/exam + records live in the Platform DB, this never loses the student's academic record — worst case + is losing in-progress, not-yet-validated terminal work. +- **Host failure / reboot** — every environment on that host becomes unreachable. On + Provisioner restart, it must **reconcile its registry against actual running containers** + rather than trust stale DB state — some containers may auto-restart via Docker's restart + policy, most won't; anything the registry claims is `running` that isn't actually running + gets marked `error`, not silently left wrong. +- **Idle/abandoned environments** — the reaper (§2) bounds resource leakage from crashed tabs + or students who just walked away, independent of any other failure path. +- **Core resilience property of this whole design:** because the Platform DB — not the + disposable environment — is the system of record, every failure mode above is recoverable + without academic data loss. That's the actual payoff of the split in §1/§3, not just an + architectural nicety. + +--- + +## 9. Migration path from the current single shared container + +- **Phase 0 (already true today, keep it):** running the existing image standalone — + `docker run --privileged -p 7554:80 ... rootnode-academy:latest` — is exactly the + single-user local dev/test mode you asked to keep available. No change needed to support + this; it already works and stays supported as-is. +- **Phase 1:** Build Provisioner + `environments`/`environment_events`/`lab_hosts` registry + around the **existing, unmodified** Lab image, on Postgres, with the egress firewall/proxy + from §4 in place from the start (not bolted on later). Development exception (correction A): + running Platform + Provisioner + one Lab container all on a single machine is fine here — the + Provisioner is configured with one `lab_host` pointing at the local Docker socket instead of + a remote TLS endpoint, same code path either way. Prove two lab environments can run side by + side, fully isolated from each other, before any production traffic or a second VM enters + the picture. `students`/`progress` stay on the existing SQLite through this phase + (correction C) — `environments.student_id` is opaque, not a foreign key, until a later, + separate migration actually moves roster/progress onto Postgres. +- **Phase 2 (the one real code change inside the Lab image):** narrow `backend/server.js` + inside the Lab image down to a "dumb executor" — scenario setup/validate/teardown/answer and + the WS terminal stay, but it stops owning `students`/roster and stops being the source of + truth for progress. Either it calls back to the Platform API to report validation results, + or — cleaner — the Platform API becomes the only thing the frontend talks to for + progress/curriculum/identity, and it calls the Lab container (via the route the Provisioner + already tracks) purely to execute against that student's cluster and get a pass/fail back. +- **Phase 3:** Wire instructor controls (list/reset/terminate/resource-usage) against the new + registry — this is also where the "no auth on roster-add" gap from the previous slice gets + closed for real, since instructor auth is now a load-bearing security boundary, not just + roster hygiene. +- **Phase 4 (cutover):** point the real domain at the Gateway instead of a single container. + Keep the Phase 0 single-container path available under local/dev tooling, per your explicit + requirement. +- **Rollback safety:** at every phase through Phase 3, the current single-shared-container path + keeps working completely unmodified — nothing about this migration risks the working + baseline until the Phase 4 cutover, which is also the only step that's meaningfully + reversible-with-effort rather than trivially reversible. + +--- + +## 10. Exact files/components this touches (once approved — no code written yet) + +**New, none of it touching the existing Lab image's core logic:** +- `platform/provisioner/` — Docker lifecycle management against a *configured* `lab_host` + endpoint (correction D — never a hard-coded local socket), `environments`/`lab_hosts` CRUD, + per-environment firewall chain management (correction B), idle reaper, admission control. +- `platform/egress-proxy/` — the shared allowlisting proxy every Lab container's egress must + pass through (correction B). +- `platform/gateway/` — token→route resolution, HTTPS + WSS proxying (candidate: Traefik with + a dynamic provider backed by the Platform API, or a small custom Node proxy — not decided, + not needed for this design review). +- Platform DB migrations: `environments`, `environment_events`, `lab_hosts` tables — full + schema in `docs/PHASE1_PLATFORM_FOUNDATION.md`. + +**Modified, once implementation starts:** +- `backend/server.js` — split as described in §9 Phase 2. This is presently both "the lab + image's backend" and "the only backend that exists"; under this design it forks into a + narrower Lab-side executor and a separate, persistent Platform-side API. Exact file layout + for that split is an implementation decision, not part of this design doc. +- `README.md` — the "Multi-Student Deployment" section (`README.md:128-146`) currently + documents per-student **volumes** for persistence; that persistence model is superseded by + §3 above and the README needs updating once this ships, so it doesn't contradict reality. +- `docs/ENGINEERING_SOP.md` — new section for the Platform/Provisioner/Gateway components and + the reconciliation/reaper operational procedures from §8. + +**Unchanged:** +- `Dockerfile`, `scripts/entrypoint.sh`, `scripts/nginx.conf` for the Lab image itself — it's + still one container running k3s + backend + terminal, exactly as today, just multiplied and + stripped of identity ownership. +- All scenario content (`scenarios/data/`, `scenarios/bundles/`, `scenarios/rootnode-path.json`) + — unaffected either way; every scenario now runs with full admin of its own disposable + cluster, so the earlier `shared_namespace`/`ephemeral_cluster` split has no bearing here. + +**New Dockerfile needed (not yet written):** one for the Provisioner+Gateway services — these +are new components, not variants of the existing Lab image. + +--- + +## Status + +All four open questions this doc originally ended with are resolved — see the decisions block +at the top and §2–§5 above. Next: `docs/PHASE1_PLATFORM_FOUNDATION.md` — the concrete build +plan for the platform foundation (Postgres environment registry, provisioner skeleton, +networking, egress, resource policy, reaper) against these decisions. diff --git a/docs/PHASE1_PLATFORM_FOUNDATION.md b/docs/PHASE1_PLATFORM_FOUNDATION.md new file mode 100644 index 0000000..8c2e0fc --- /dev/null +++ b/docs/PHASE1_PLATFORM_FOUNDATION.md @@ -0,0 +1,483 @@ +# RootNode Academy — Phase 1: Platform Foundation + +Status: **IMPLEMENTED, tested on a single dev machine, 2026-08-18.** Code lives in +`platform/provisioner/` and `platform/egress-proxy/`. This doc still reflects the corrected +design (four architecture corrections: VM boundary, egress-as-firewall-not-app-config, opaque +`student_id`, lab-host abstraction) — the implementation matches it, with a handful of +deviations found and fixed during testing, listed in "Implementation notes" at the end of each +section below where relevant, and summarized in `platform/provisioner/README.md`'s "Known +limitations." Builds on `docs/LAB_ARCHITECTURE.md` (read that first). + +**Scope — this phase only:** +Postgres-backed environment registry · lifecycle state model · provisioner service skeleton · +per-student private Docker network · disposable Lab container creation from the existing +image · configurable CPU/RAM/PID/storage limits · restricted egress · unique opaque routing +token · environment event/audit log · destroy/reset · idle-reaper design · current +single-container dev mode kept working, untouched. + +**Explicitly not in this phase:** instructor dashboard, self-paced public enrollment, any +scenario content change, unrestricted egress, removing local single-user mode. Nothing below +proposes touching `backend/server.js`, `Dockerfile`, `scripts/entrypoint.sh`, or +`scenarios/**` — the Lab image is used exactly as it exists today. + +--- + +## 1. Service boundaries + +Two new components. Neither is the Lab image; neither is public-facing. + +- **Provisioner** (`platform/provisioner/`) — new Node service. Owns: the Postgres + `environments`/`environment_events`/`lab_hosts` tables, the Docker Engine API connection (via + `dockerode`, not shelling out to the `docker` CLI), network/container lifecycle, the idle + reaper, and egress-firewall management for each student network. Exposes an internal HTTP API + (§4) gated by a shared-secret header — same pattern already used for the instructor-passcode + gate on the roster endpoint, so this doesn't introduce a new auth idiom. + **Docker connection is always resolved through `lab_hosts` (§2), never hard-coded** — Phase 1 + configures exactly one row, pointing at `unix:///var/run/docker.sock` for single-machine dev + or a `tcp://:2376` TLS endpoint for the two-VM production topology in + `docs/LAB_ARCHITECTURE.md` §1. Same code path either way; only the config differs. +- **Egress proxy** (`platform/egress-proxy/`) — one shared instance (not per-student), a + minimal HTTP(S) forward proxy (Squid) with a static host allowlist. Every Lab container's + `HTTP_PROXY`/`HTTPS_PROXY` env vars point at it; the student network's firewall rule permits + egress *only* to this proxy and to DNS, so there's no way for a Lab container to reach the + internet except through something the Platform's allowlist controls. + +Nothing else changes. No new Platform API / roster-progress migration in this phase — that's +explicitly out of scope (see §2 note on `student_id`). + +--- + +## 2. Exact DB schema (PostgreSQL) + +New database, e.g. `rootnode_platform`. Only the environment registry ships this phase — +`students`/`progress`/`sessions` stay on the existing SQLite inside the Lab image for now; +migrating those onto this Postgres instance is a later phase, not bundled in here. + +```sql +-- ── lab_hosts — correction D: the abstraction that keeps Docker connectivity out of code ── +CREATE TABLE lab_hosts ( + lab_host_id TEXT PRIMARY KEY, -- e.g. 'dev-local', 'labhost-01' + display_name TEXT NOT NULL, + endpoint TEXT NOT NULL, -- 'unix:///var/run/docker.sock' (dev) or + -- 'tcp://10.x.x.x:2376' (prod, TLS-verified) + tls_ca_path TEXT, -- null for the local-socket dev case + tls_cert_path TEXT, -- Provisioner's client cert for this host + tls_key_path TEXT, + enabled BOOLEAN NOT NULL DEFAULT true, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() + -- Future (not this phase): capacity/status columns for real scheduling once there's + -- more than one host. Phase 1 selects "the one enabled row" — no scheduler. +); + +CREATE TYPE environment_status AS ENUM ( + 'requested', 'provisioning', 'running', 'resetting', + 'suspended', -- experimental path only, see decision #3 + 'destroying', 'destroyed', 'error' +); + +CREATE TABLE environments ( + environment_id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + student_id TEXT NOT NULL, -- CORRECTION C: opaque external identifier, not a + -- foreign key. students/progress stay on the + -- existing SQLite through Phase 1 (see + -- docs/LAB_ARCHITECTURE.md §3/§9) — there is no + -- students table in this database yet, so there + -- is nothing to reference. The Provisioner trusts + -- whatever value its caller passes; it does not + -- validate against a roster. A real FK gets added + -- only when roster/progress actually migrate onto + -- this Postgres instance, in a later, separate + -- migration — not assumed or faked here. + lab_host_id TEXT NOT NULL REFERENCES lab_hosts(lab_host_id), -- real FK: both tables + -- are Provisioner-owned Postgres, correction D. + status environment_status NOT NULL DEFAULT 'requested', + routing_token TEXT NOT NULL, -- CSPRNG, opaque, unique + container_id TEXT, -- null until provisioned + container_name TEXT, + network_id TEXT, + network_name TEXT, + network_subnet CIDR, -- needed to install/remove the per-env firewall chain + image_ref TEXT NOT NULL, -- pinned Lab image, e.g. rootnode-academy:1.4.0 + cpu_limit NUMERIC NOT NULL, + mem_limit_mb INTEGER NOT NULL, + pids_limit INTEGER NOT NULL, + storage_limit_mb INTEGER NOT NULL, + internal_ip TEXT, -- set once running + internal_port INTEGER NOT NULL DEFAULT 80, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + started_at TIMESTAMPTZ, + last_active_at TIMESTAMPTZ, + destroy_at TIMESTAMPTZ, -- idle-reaper's scheduled destroy time + error_reason TEXT +); + +CREATE UNIQUE INDEX idx_environments_routing_token ON environments(routing_token); +CREATE INDEX idx_environments_student ON environments(student_id); +CREATE INDEX idx_environments_status ON environments(status); +CREATE INDEX idx_environments_lab_host ON environments(lab_host_id); +-- Phase 1 invariant: at most one non-terminal environment per student. +-- Enforced in application logic (see §4 POST /environments), not a DB constraint yet — +-- a partial unique index is the natural upgrade once this needs to be bulletproof under +-- concurrent requests, not needed for a single-instance provisioner in this phase. + +CREATE TABLE environment_events ( + id BIGSERIAL PRIMARY KEY, + environment_id UUID NOT NULL REFERENCES environments(environment_id), + event_type TEXT NOT NULL, -- provision_requested | provisioned | started | reset + -- | destroyed | error | reaped_idle | suspend | resume + actor TEXT NOT NULL, -- 'system' | 'instructor:' | 'student:' | 'reaper' + detail JSONB, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE INDEX idx_env_events_environment ON environment_events(environment_id); +``` + +**Resource policy and the egress allowlist are config files, not tables.** There's no admin UI +to manage them via a database yet (that's dashboard work, explicitly deferred), so a +DB-backed policy table would just be a write path nothing uses. `config/resource-policy.yaml` +(cpu/mem/pids/storage defaults, reloadable) and `platform/egress-proxy/allowlist.conf` (Squid +ACL) are the actual source of truth — plain files an operator edits, matching how +`entrypoint.sh` already does config via env vars rather than a database. + +--- + +## 3. Lifecycle state machine + +Exactly what `docs/LAB_ARCHITECTURE.md` §2 now specifies, decision #3 applied: + +``` +requested → provisioning → running ─┬─→ resetting → running (same environment_id + routing_token) + └─→ destroying → destroyed +provisioning → error +running → error +error → destroying → destroyed +error → resetting → running (when recoverable) + +[EXPERIMENTAL — not required to pass for Phase 1 sign-off] +running → suspended → running (resume) +running → suspended → error (resume fails) +``` + +`reset` is implemented as `destroy` + `provision` against the same DB row (new container, +network, and internal_ip; same `environment_id`/`routing_token` so a student's saved link +doesn't change). + +--- + +## 4. API endpoints (Provisioner, internal-only) + +All routes require `X-Provisioner-Key: `. Nothing here is reachable by a +student or the public internet in this phase — there's no frontend/gateway wiring yet, this +is tested directly (curl / a test script), which is also what makes it independently +verifiable before any later phase builds on top of it. + +| Method & path | Purpose | +|---|---| +| `POST /environments` | Body `{ student_id }` — `student_id` is opaque (correction C, §2), not validated against a roster. `lab_host_id` is not a caller-supplied parameter in Phase 1: the Provisioner selects the single enabled `lab_hosts` row itself (`SELECT ... WHERE enabled LIMIT 1`) — no scheduling logic, no host-selection API, matching "don't overbuild scheduling yet." Rejects if the student already has a non-terminal environment (Phase 1 invariant, see §2). Synchronous: creates network → installs firewall chain (§5) → creates+starts container → polls Lab `/api/health` → marks `running`, or `error` on timeout. Returns `{ environment_id, status, routing_token }`. | +| `GET /environments/:id` | Full row: status, container_id, limits, timestamps. | +| `GET /environments?student_id=` | A student's environments (expected: zero or one active). | +| `POST /environments/:id/reset` | Destroy + recreate, same `environment_id`/`routing_token`. | +| `DELETE /environments/:id` | Destroy: stop, remove container, remove network + its firewall rules, mark `destroyed`. | +| `POST /environments/:id/suspend` *(experimental)* | `docker stop`, mark `suspended`. Documented as unreliable — see test plan §7. | +| `POST /environments/:id/resume` *(experimental)* | `docker start` + re-poll readiness. On failure, mark `error` rather than leaving a wrong `suspended` status. | +| `GET /environments/:id/events` | Audit trail for that environment. | +| `GET /health` | Provisioner's own health: Docker reachable, Postgres reachable. | + +Synchronous provisioning is a deliberate simplification for this phase — no job queue, no +webhook callback. Given expected Phase 1 usage (developer testing, not a live cohort), a +blocking HTTP call that takes a few seconds while k3s comes up is fine; converting to +async-with-polling is the obvious upgrade if provisioning latency becomes a real UX problem +once this is behind an actual frontend. + +--- + +## 5. Docker / network operations (via `dockerode`, shown as CLI-equivalent for readability) + +**Provision — network:** +```bash +docker network create --driver bridge rootnode-lab-net- +# inspect to get the allocated subnet, store as environments.network_subnet +``` + +**Provision — egress firewall, corrected (§4 of `docs/LAB_ARCHITECTURE.md`, correction B).** +The previous draft chained individual `iptables -I DOCKER-USER ...` inserts in +ACCEPT-then-DROP order, which is backwards: `-I` without a rule number pushes each new rule to +*position 1*, so the rules end up installed in the *reverse* of the order they were run — +the final default-DROP would land at the top of `DOCKER-USER` and shadow everything below it, +including the proxy/DNS accepts. Fixed by giving each environment its own dedicated chain, +built with `-A` (append — rules land in the order added, no head-insertion trap), and reaching +it from `DOCKER-USER` with exactly one jump rule per environment: + +```bash +NET_SUBNET=$(docker network inspect rootnode-lab-net- -f '{{(index .IPAM.Config 0).Subnet}}') +CHAIN="ROOTNODE-ENV-" + +iptables -N "$CHAIN" # fresh, empty chain for this environment + +# Required infrastructure — explicitly permitted FIRST, before any blanket deny: +iptables -A "$CHAIN" -d -p tcp --dport 3128 -j ACCEPT +iptables -A "$CHAIN" -d -p udp --dport 53 -j ACCEPT +iptables -A "$CHAIN" -d -p tcp --dport 53 -j ACCEPT +# (any explicitly-required Platform endpoint would be an additional ACCEPT line here — +# none needed in Phase 1; the Lab image doesn't call back to the Platform yet) + +# Everything that must never be reachable, regardless of the above — unconditional, no +# allowlist entry can ever bypass these, satisfying "no route to trusted networks" as a +# hard rule rather than a policy convention: +iptables -A "$CHAIN" -d 10.0.0.0/8 -j DROP +iptables -A "$CHAIN" -d 172.16.0.0/12 -j DROP +iptables -A "$CHAIN" -d 192.168.0.0/16 -j DROP +iptables -A "$CHAIN" -j DROP # default deny, catches everything else (public internet) + +# Hook the student's subnet into this chain. -I here is fine and intentional: we want each +# new environment's jump rule evaluated ahead of any broader, pre-existing DOCKER-USER rule, +# and since every environment matches a disjoint source subnet, the relative order of jump +# rules against each other never matters — only that each is ahead of a catch-all, if one +# ever exists at the DOCKER-USER level. +iptables -I DOCKER-USER -s "$NET_SUBNET" -j "$CHAIN" +``` + +This also directly satisfies the explicit "must never reach" list from +`docs/LAB_ARCHITECTURE.md` §4: RootNode management networks, the Platform database, Docker/ +Proxmox management interfaces (including the Lab Host VM's own Docker API — notably, since +that API is what the Provisioner itself talks to from the *Platform* VM, a Lab container has +no way to reach it either way, but the RFC1918 drop covers it as a second, independent layer), +CDAA infrastructure, home/trusted networks, and other students' lab networks all fall inside +one of the three RFC1918 ranges above and are dropped unconditionally by this same ruleset — +one mechanism, not five separate ones to keep in sync. + +**Provision — container** (limits from `resource-policy.yaml`, no image/entrypoint change): +```bash +docker create \ + --name rootnode-lab- \ + --network rootnode-lab-net- \ + --privileged \ + --cpus --memory m --memory-swap m \ + --pids-limit \ + --storage-opt size=m \ + -e HTTP_PROXY=http://:3128 -e HTTPS_PROXY=http://:3128 \ + -e NO_PROXY=localhost,127.0.0.1,10.42.0.0/16,10.43.0.0/16 \ + +docker start rootnode-lab- +``` +**Implementation note: `` is resolved per environment, not a fixed +hostname/IP.** The egress proxy runs with host networking (`docker-compose.dev.yml`), so it's +present on every per-student bridge network's *gateway* interface — that gateway IP (from +`docker network inspect`'s `IPAM.Config[0].Gateway`, right alongside the subnet already used +for the firewall chain) is what's actually reachable from that specific environment's private +network. A single static proxy hostname would only be reachable from whichever one network +happened to be able to resolve/route to it — this wasn't apparent until testing with two +simultaneous environments on two different subnets. Same resolution is used for both the +firewall's `ACCEPT` rule destination (§ above) and this container's proxy env vars, so they +always agree. + +`NO_PROXY` carves out k3s's own internal pod/service CIDRs so in-cluster traffic never routes +through the proxy — only the container's *outbound-to-the-internet* path does. containerd +(started by the existing, unmodified `entrypoint.sh`) honors `HTTP_PROXY`/`HTTPS_PROXY` from +its process environment automatically, which is why no Lab image or entrypoint change is +needed for well-behaved image pulls to go through the proxy. + +**Correction B, stated plainly: these env vars configure containerd for a clean pull path — +they are not what makes egress restricted.** A process inside the container that ignores them +and dials out directly is stopped by the `ROOTNODE-ENV-` firewall chain above regardless, +because that chain evaluates every packet leaving the container's network, independent of +what any process inside chose to do. If the env vars were somehow wrong or absent, the only +consequence is broken image pulls (a visible, debuggable failure) — never a silent hole in the +egress boundary, because the boundary was never the env vars in the first place. + +Readiness: poll the container's internal IP (`docker inspect` → `NetworkSettings`) for +`GET /api/health` — the exact endpoint `entrypoint.sh` already waits on internally, so the +Provisioner is just doing the same check from outside instead of inventing a new one. + +**Destroy:** +```bash +docker stop rootnode-lab- --time 10 && docker rm rootnode-lab- + +# Unhook and remove this environment's dedicated chain — three ops, no rule-scanning/matching +# needed since it's one self-contained chain per environment, not tagged rules mixed into a +# shared one: +iptables -D DOCKER-USER -s "$NET_SUBNET" -j "$CHAIN" +iptables -F "$CHAIN" +iptables -X "$CHAIN" + +docker network rm rootnode-lab-net- +``` + +**Reset** = destroy, then provision again against the same `environments` row (new +`container_id`/`network_id`/`internal_ip`, unchanged `environment_id`/`routing_token`). + +**Idle reaper** (background loop in the Provisioner, e.g. every 60s): +- Query `environments WHERE status='running' AND COALESCE(last_active_at, created_at) < now() - idle_threshold`. +- Past a shorter "idle" threshold: log a warning event (future: notify student/instructor — + not built yet). +- Past a longer "abandoned" threshold: call the same destroy path as `DELETE /environments/:id`, + log `event_type='reaped_idle'`. Config-driven thresholds, not hard-coded (same policy file + as resource limits). + +**Implementation notes (found during testing, not in the original design):** +- **`COALESCE(last_active_at, created_at)`, not bare `last_active_at`.** An environment that + fails during provisioning (before ever reaching `running`) has a NULL `last_active_at` — + `NULL < anything` is never true in SQL, so without the fallback such an environment would + never age out via the reaper at all, regardless of how long ago it failed. +- **Provisioning failure now does best-effort cleanup of whatever was already created** + (container/network/firewall chain) before marking the row `error`, rather than leaving that + infrastructure orphaned. Found by testing: a deliberate readiness-timeout failure left a real + running container and firewall chain in place with only the DB row marked `error` — exactly + the kind of leak the reaper exists to prevent, except this case predates the reaper ever + having something to catch. Cleanup on the failure path directly is more honest than relying + on the reaper for a gap it wasn't actually catching. See `src/lifecycle.js`'s + `provisionEnvironment` catch block and `src/reaper.js`. +- **Startup reconciliation (docs/LAB_ARCHITECTURE.md §8) is implemented** as + `reconcileOnStartup()` in `src/lifecycle.js`, called once at Provisioner boot before the + reaper starts. Tested by killing a container out-of-band and restarting the Provisioner — + confirmed it correctly flags the mismatch as `error` rather than trusting the stale + `running` status. Any environment caught mid-transition (`provisioning`/`resetting`/ + `destroying`) by a crash is also always marked `error` on restart, never assumed to have + finished. + +--- + +## 6. Security assumptions (stated explicitly, not implied) + +- **Correction A — the Lab Host is not the Platform's trust boundary, the VM is.** In + production, the host running privileged Lab containers is a separate VM from the one running + Postgres/Provisioner/Platform API, with no route back from the Lab Host VM to the Platform VM + at all — enforced at the hypervisor/network level, not by this phase's code. Phase 1's + single-machine dev mode is a deliberate, documented exception (`docs/LAB_ARCHITECTURE.md` §1) + — it does not redefine what "isolated" means for the real deployment. +- The Docker host(s) sit on a segment with no route to trusted RootNode/CDAA/home/business + networks — a prerequisite this phase depends on but does not create (matches the existing + Business Pack Technical Platform Plan). Everything below assumes that's already true. +- No other host firewall tooling (ufw, firewalld) is silently rewriting `DOCKER-USER` or + removing the per-environment `ROOTNODE-ENV-` chains — needs a one-time check on the real + host before trusting the egress rules; not something this design can verify in advance. +- `--privileged` is required for k3s and is not being avoided (confirmed in + `docs/LAB_ARCHITECTURE.md` §7) — the security boundary is the VM split plus network + containment, not reduced container privilege. +- **Correction B — `HTTP_PROXY`/`HTTPS_PROXY` are not a security control**, stated here again + because it's the assumption most likely to get silently reintroduced during implementation: + the firewall chain enforces the boundary; the env vars only make well-behaved pulls work + cleanly through it. +- The egress proxy's allowlist is operator-controlled (a file the Provisioner's operator + edits), never student- or Lab-container-writable. +- The Provisioner API's shared secret is the only thing standing between "internal service" + and "anyone who can create/destroy arbitrary privileged containers on the host" — treat that + secret with the same care as the instructor passcode, not as a minor implementation detail. + This matters more, not less, once the Provisioner is issuing commands to a remote Docker + endpoint (correction D) — the TLS client cert for that connection carries the same weight. +- `storage_limit_mb` enforcement via `--storage-opt size=` depends on the host's Docker storage + driver supporting it (overlay2 with specific backing filesystems does; not all do) — unverified + until tested against the real host (§8). +- Nothing in this phase is internet-facing. The Provisioner API assumes internal-network-only + reachability; it does not implement its own TLS. The Provisioner→Lab-Host Docker connection + (correction D), by contrast, is TLS by design once it's not a local socket — that's a + different boundary with different stakes and isn't optional the way the Provisioner API's + own TLS currently is. + +--- + +## 7. Files to be created + +All new. Nothing existing is modified beyond the two docs already updated. + +``` +platform/provisioner/ +├── package.json +├── src/ +│ ├── server.js # Express app, routes from §4 +│ ├── db.js # Postgres connection (pg), migration runner +│ ├── docker.js # dockerode client factory — resolves a lab_hosts row to a connected +│ │ client (local socket or remote TLS), never a hard-coded default +│ ├── lifecycle.js # state machine + orchestration (calls docker.js + db.js) +│ ├── egress.js # per-environment ROOTNODE-ENV- chain create/hook/unhook/remove +│ └── reaper.js # idle-reaper background loop +├── migrations/ +│ ├── 001_lab_hosts.sql +│ ├── 002_environments.sql +│ └── 003_environment_events.sql +├── config/ +│ ├── resource-policy.yaml # cpu/mem/pids/storage defaults + idle/abandoned thresholds +│ └── lab-hosts.seed.yaml # the one Phase 1 lab_hosts row (endpoint + TLS paths), +│ # applied by the migration runner — not a runtime API, +│ # per "don't overbuild scheduling yet" +├── Dockerfile +└── docker-compose.dev.yml # Postgres + egress-proxy + provisioner, for local Phase 1 testing + +platform/egress-proxy/ +├── Dockerfile # minimal Squid image +└── allowlist.conf # Squid ACL: registry-1.docker.io, ghcr.io, etc. — starting set + # to be confirmed against what scenarios actually pull +``` + +No changes to `backend/server.js`, `Dockerfile`, `scripts/entrypoint.sh`, +`scripts/nginx.conf`, or anything under `scenarios/`. + +--- + +## 8. Test plan + +1. **Schema migration idempotency** — run `001_environments.sql` against a scratch Postgres + twice, confirm the second run is a no-op (same pattern already used for the SQLite + migrations: check-before-alter). +2. **Provision → destroy round trip** — create an environment, confirm the container is + `running` and reachable on its internal IP, confirm `environment_events` has one row per + transition, destroy, confirm container/network are gone and the environment's + `ROOTNODE-ENV-` chain no longer exists (`iptables -L ROOTNODE-ENV-` errors "chain + does not exist") and its `DOCKER-USER` jump rule is gone (`iptables -S DOCKER-USER | grep + ` returns nothing). +3. **Reset** — provision, note `environment_id`/`routing_token`, reset, confirm both are + unchanged, confirm `container_id` changed (genuinely new container), confirm no state + carried over (fresh k3s). +4. **Isolation** — provision two environments, confirm A cannot reach B at all + (`docker exec ping ` times out) — this is the core security property + and must be an explicit test, not an assumption. +5. **Egress allowlist, including the ordering fix specifically** — from inside a provisioned + environment: an allowlisted registry pull succeeds via the proxy; a non-allowlisted host + fails; a direct request to an RFC1918 address (simulating a trusted-network address) is + dropped regardless of allowlist state. Critically: **the egress proxy's own IP is itself an + RFC1918 address** — this test only actually exercises the fix from correction B if the proxy + is reachable *and* the broader RFC1918 range is still blocked. If the proxy were unreachable + here, that's the exact symptom the original ordering bug would have caused (the DROP rules + shadowing the ACCEPT rules) — this test is what would have caught it before it shipped. +6. **Idle reaper** — provision, force `last_active_at` backward past the abandoned threshold, + run one reaper tick, confirm the environment is destroyed and logged as `reaped_idle`. +7. **Suspend/resume (experimental, decision #3 — informational, not a pass/fail gate)** — + `docker stop`/`docker start` a running Lab container, time how long it takes (if ever) to + reach `/api/health` ready again versus a fresh `provision`. This is the actual answer to + the question `docs/LAB_ARCHITECTURE.md` flagged as untested — record the result either way, + but Phase 1 sign-off does not depend on this succeeding. +8. **Reconciliation on Provisioner restart** — manually create a container outside the + Provisioner's knowledge, restart the Provisioner, confirm it doesn't silently trust stale + `environments` rows — either adopts or flags the mismatch (per + `docs/LAB_ARCHITECTURE.md` §8). +9. **Resource limits hold** — inside a provisioned environment, confirm `--cpus`/`--memory`/ + `--pids-limit` are actually enforced (e.g. a deliberate memory-hog process gets OOM-killed + at the configured limit, not the host's full RAM); confirm whether `--storage-opt size=` + is honored on the real host's storage driver — this is the resource-model unknown from + `docs/LAB_ARCHITECTURE.md` §5, and this test is the first real benchmarking data point + toward the 16-core/128GB target sizing. +10. **Local dev-mode regression check** — confirm `docker run --privileged -p 7554:80 ... + rootnode-academy:latest` (today's standalone command) is completely unaffected — + `git diff` scoped to `backend/`, `Dockerfile`, `scripts/` should show zero changes from + this phase. +11. **Lab host abstraction actually abstracts (correction D)** — point the Provisioner's + `lab_hosts` config at a Docker daemon exposed over TCP+TLS on the *same* dev machine + (`dockerd -H tcp://127.0.0.1:2376 --tlsverify ...`) instead of the local Unix socket, with + no code change — only the `lab_hosts` row differs. Provision/destroy round trip (test 2) + must pass identically against that endpoint. This is the practical proof that "Docker is + local" was actually removed as an assumption, not just renamed — it doesn't require a + second real VM to verify, only a second way of reaching the same daemon. +12. **VM-boundary reasoning check (correction A, documentation-level, not a running test)** — + walk the failure scenario explicitly: if a student escaped their container onto the Lab + Host VM's OS, what could they reach? Confirm the answer stays "nothing on the Platform VM" + given the topology in `docs/LAB_ARCHITECTURE.md` §1 — i.e. confirm the design doesn't rely + on anything *inside* the Lab Host VM (a firewall rule, a credential, an agent) as the only + thing standing between a compromised container and Postgres. This is a design-review check + now; it becomes a real network-level test once a second VM exists to test against. + +Tests 1–6, 8, 10, 11 are runnable in a plain Docker environment with no special hardware. +Tests 5 and 9 need the actual target host (or something close to it) to produce meaningful +numbers, test 7 needs the real Lab image, and test 12 needs a second VM to be more than a +design-review exercise — flagging that distinction so "passed the test plan" doesn't get read +as "validated on production-class, two-VM hardware" when parts of it were necessarily +validated on a single dev machine.