docs(platform): document ephemeral lab architecture
- docs/LAB_ARCHITECTURE.md — the full ephemeral-per-student design: two-VM production topology (Platform VM / Lab Host VM, correction A), lifecycle, data model, networking/egress model (correction B), resource model, routing/auth model, security boundaries, failure/recovery behavior, migration path from the current single container, and the lab-host abstraction (correction D). Status updated to reflect Phase 1 is now implemented, pointing at the two docs below for specifics. - docs/PHASE1_PLATFORM_FOUNDATION.md — the Phase 1 build plan: exact Postgres schema, service boundaries, API routes, lifecycle state machine, Docker/ firewall commands, security assumptions, file layout, and a 12-test plan. Updated with "Implementation notes" at the points where testing surfaced real deviations from the original design: the per-network (not static) egress-proxy address resolution, the COALESCE reaper fix, and the provisioning-failure cleanup fix. - docs/ENGINEERING_SOP.md — new §8 mapping the platform/provisioner/ and platform/egress-proxy/ components, what's built vs. not (no Gateway, no frontend wiring — internal API only), and where identity still lives (unchanged: Lab image's own SQLite). Roadmap item 5 (per-student isolation) updated from "not started" to "Phase 1 done, not yet cleared for minors." - README.md — Multi-Student Deployment section rewritten: the old per-student- volume persistence model is superseded by the Platform's registry (progress already lives outside the disposable container as of the prior curriculum/ roster work); points at docs/LAB_ARCHITECTURE.md and platform/provisioner/ for the real architecture instead of describing three delivery modes that were never actually built out. Co-Authored-By: Claude Sonnet 5 <[email protected]>
This commit is contained in:
Claude Sonnet 5
@@ -127,22 +127,30 @@ Everything runs inside one image managed by `scripts/entrypoint.sh`.
|
||||
|
||||
## Multi-Student Deployment (RootNode Academy addition)
|
||||
|
||||
The upstream project assumes one learner, one container. RootNode Academy runs it across three
|
||||
delivery modes. **Never publish the raw privileged container** — front it with isolation.
|
||||
The upstream project assumes one learner, one container — this image is still exactly that,
|
||||
unmodified. RootNode Academy multiplies it into a real platform via a separate provisioning
|
||||
layer rather than changing the image itself: see `docs/LAB_ARCHITECTURE.md` for the
|
||||
architecture and `platform/provisioner/` for the Phase 1 implementation (Postgres-backed
|
||||
environment registry, per-student Docker networking, firewall-enforced egress, idle reaper —
|
||||
see `platform/provisioner/README.md` for how to run it).
|
||||
|
||||
| Mode | Provisioning | Isolation |
|
||||
| ---- | ------------ | --------- |
|
||||
| In-person camp | Pre-spun container per seat, torn down after | Lab VLAN, no public ingress |
|
||||
| Live cohort | Container per enrolled student for the session window | Per-student auth at the proxy |
|
||||
| Self-paced | On-demand spin-up per session, idle-timeout teardown | Per-user volume + auth + resource limits |
|
||||
**Never publish the raw privileged container** — front it with isolation. The one-container
|
||||
`docker run --privileged -p 7554:80 ...` command above stays supported for local development
|
||||
and scenario authoring; it is not how this gets deployed to real students.
|
||||
|
||||
Hardening checklist for shared infra:
|
||||
Key properties of the Phase 1 platform (not this image):
|
||||
|
||||
- One container **per student** — never share a cluster between learners.
|
||||
- Apply CPU/memory limits so one runaway pod can't starve a host.
|
||||
- Put authentication in front (proxy-level), never expose `:7554` to the internet.
|
||||
- Persist each student's `/data` to a dedicated volume.
|
||||
- Idle-timeout + auto-teardown for self-paced sessions to control cost.
|
||||
- One container **per student**, its own Docker network, its own firewall chain — never a
|
||||
shared cluster between learners.
|
||||
- Progress/identity live outside the disposable container (currently this image's own SQLite,
|
||||
moving to the Platform's PostgreSQL in a later phase) — destroying a student's environment
|
||||
never loses their academic record.
|
||||
- CPU/memory/PID/storage limits are enforced per container, config-driven.
|
||||
- Egress is restricted to an allowlisted proxy at the firewall layer, not left open.
|
||||
- Idle/abandoned environments are automatically cleaned up.
|
||||
|
||||
Production topology and what's built vs. not yet: `docs/LAB_ARCHITECTURE.md` and
|
||||
`docs/PHASE1_PLATFORM_FOUNDATION.md`.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
# RootNode Academy — Engineering SOP
|
||||
|
||||
Living doc. Covers the "how" of building this platform — the "why" and business/curriculum
|
||||
process live in the Business Pack (`~/Documents/CDAA Technologies/07 RootNode academy/RootNode_Academy_Business_Pack/`),
|
||||
this doc doesn't duplicate that. Update this file in the same commit as the change it describes.
|
||||
|
||||
## 1. Where this actually lives
|
||||
|
||||
- **Repo:** `~/Dev/Gitea/ced4568/rootnode-academy` — remote is **Gitea**
|
||||
(`git.synthossystems.com/ced4568/rootnode-academy`), not GitHub.
|
||||
- **Gotcha:** `~/Dev/GitHub/ced4568/kubekosh` is a *different* clone of the same original
|
||||
upstream fork, still pointed at GitHub, and it's behind (missing the RootNode logo commit
|
||||
and the Real Ops enforce-limits scenario as of this writing). It's not deployed anywhere.
|
||||
Don't edit it by mistake — always confirm `git remote -v` shows `gitea:` before making changes.
|
||||
- Known-good baseline as of this doc: commit `6ddfb1c` ("add probe-port-mismatch and
|
||||
enforce-limits to rootnode-real-ops bundle"). Everything below builds on top of that,
|
||||
clean working tree.
|
||||
|
||||
## 2. Architecture at a glance
|
||||
|
||||
```
|
||||
backend/server.js Express API + SQLite (better-sqlite3) + WebSocket PTY terminal — the
|
||||
"Lab image," single-tenant, unmodified by the Platform work below
|
||||
frontend/ Vite + React SPA, built to frontend/dist and served by nginx
|
||||
scenarios/data/ One JSON file per scenario (task or MCQ)
|
||||
scenarios/bundles/ Flat scenario_id lists grouped by cert track (basics/appdev/admin/security/real-ops)
|
||||
scenarios/rootnode-path.json The RootNode guided learning path (staged, see §4)
|
||||
scripts/entrypoint.sh Container boot: starts k3s, nginx, node backend, sets the shell banner/PS1
|
||||
Dockerfile ubuntu:22.04 + k3s + node20 + nginx, single image, one port (80)
|
||||
|
||||
platform/provisioner/ Phase 1 Platform Provisioner — disposable per-student Lab lifecycle,
|
||||
PostgreSQL registry, egress firewall. See §8 and
|
||||
platform/provisioner/README.md.
|
||||
platform/egress-proxy/ Shared allowlisting egress proxy every Lab environment's traffic must
|
||||
pass through.
|
||||
```
|
||||
|
||||
Single Express process, single SQLite file at `/data/progress.db` (or `$PROGRESS_DB`),
|
||||
single embedded k3s cluster **per Lab environment**. The Lab image itself is still
|
||||
single-tenant by design — see §5 for identity, and §8 for how Phase 1 multiplies it into a
|
||||
real multi-student platform without modifying it.
|
||||
|
||||
## 3. Local dev / safe smoke-testing
|
||||
|
||||
Never test schema changes against the real `/data/progress.db` first. Point at a scratch file:
|
||||
|
||||
```bash
|
||||
cd backend && npm install # first time only
|
||||
node --check server.js # syntax check, catches typos fast
|
||||
PROGRESS_DB=/tmp/scratch/progress.db node server.js &
|
||||
curl -s http://localhost:4000/api/health
|
||||
# ...exercise the endpoints you changed...
|
||||
fuser -k 4000/tcp # stop it
|
||||
```
|
||||
|
||||
To test a migration against **real prior data** without touching prod: copy the actual
|
||||
`/data/progress.db` out of the running container/volume, point `PROGRESS_DB` at the copy,
|
||||
boot, and verify the data you expect is still there before ever pointing at the real file.
|
||||
|
||||
kubectl/k3s-backed endpoints (`/setup`, `/validate`, `/context`, the WS terminal) won't work
|
||||
outside the container — that's expected. API-and-DB-only endpoints (roster, login, progress,
|
||||
curriculum, bundles, scenarios list) all smoke-test fine on a bare host.
|
||||
|
||||
## 4. Curriculum content
|
||||
|
||||
Two separate structures, don't confuse them:
|
||||
|
||||
- **Bundles** (`scenarios/bundles/*.json`) — flat scenario_id lists, one per cert track.
|
||||
Loaded by `loadBundles()`/`GET /api/bundles`. Adding a bundle file here = adding a new
|
||||
cert-track library. Every scenario referenced must have a matching file in `scenarios/data/`.
|
||||
- **RootNode path** (`scenarios/rootnode-path.json`) — the guided onboarding sequence,
|
||||
separate from bundles on purpose (it's *staged*, bundles are *flat*, and the bundle loader
|
||||
would choke on a staged file if it were dropped in `scenarios/bundles/`). Loaded by
|
||||
`loadCurriculumPath()`/`GET /api/curriculum/path`. Stages are Foundation → Workloads →
|
||||
Networking → Configuration → Troubleshooting → Real Ops → Capstone, currently built by
|
||||
reusing existing `1-k8s-basics` and `rootnode-real-ops` scenarios grouped by their existing
|
||||
`category` field — no new scenario content was written for this. **Capstone stage is a
|
||||
locked placeholder** (`"locked": true`, empty `scenario_ids`) — the capstone engine itself
|
||||
is a separate, not-yet-built roadmap item (see §7, item 6). Do not put real scenarios in it
|
||||
until that engine exists, or you'll reproduce the "duplicate beginner scenarios as capstone"
|
||||
anti-pattern this path was built to avoid.
|
||||
|
||||
To add/reorder a stage: edit `rootnode-path.json` directly, referencing existing
|
||||
`scenarios/data/*.json` ids. No backend code change needed — it's read fresh via
|
||||
`POST /api/cache/reload` or on server restart.
|
||||
|
||||
## 5. Student identity — what's built, what isn't
|
||||
|
||||
**Built (backend only, as of this doc):**
|
||||
- `students` table: `student_id` (`student-NNN`, sequential), `display_name`, `cohort`,
|
||||
`created_at`, `last_active_at`.
|
||||
- Login model: **instructor roster, pick-your-name, no password.** `POST /api/roster` adds
|
||||
a student (instructor-only action, currently unauthenticated at the API level — see
|
||||
"not built" below). `GET /api/roster?cohort=X` lists names for a dropdown.
|
||||
`POST /api/login` just validates the id exists and returns it; the frontend is expected to
|
||||
hold onto `student_id` and send it back as an `X-Student-Id` header on every request.
|
||||
- `progress` table now keyed on `(student_id, scenario_id)` instead of `scenario_id` alone —
|
||||
each student's completion/attempts/hints are isolated. Verified by smoke test: two students
|
||||
answering the same scenario don't affect each other's state, and pre-migration data was
|
||||
preserved under `student_id = 'unassigned'`.
|
||||
- Exam `sessions` now carry `student_id` (set at session start), so exam history and the
|
||||
timer-reset-on-submit logic are correctly scoped per student instead of leaking across
|
||||
whoever's browser hits the API next.
|
||||
|
||||
**Not built yet — do not assume these exist:**
|
||||
- **No frontend UI.** No login screen, no name picker, no roster-add form. The API works;
|
||||
nothing calls it yet. This is the next slice.
|
||||
- **No auth on `/api/roster` POST.** Anyone who can reach the API can add a "student."
|
||||
Fine for a dev instance, not fine once this is reachable by actual students — needs an
|
||||
instructor-only gate before that happens (ties into the instructor dashboard, roadmap
|
||||
item 3).
|
||||
- **No concurrency / isolation.** Exam sessions are still global-single-active
|
||||
(`WHERE status='active' LIMIT 1` — see `server.js`), and there is one shared k3s cluster
|
||||
for the whole instance. Per-student identity ≠ per-student environment. **Multiple students
|
||||
cannot yet use this platform at the same time without stepping on each other.** That's
|
||||
roadmap item 4 (isolation) and it gates real classroom/minors use, same as the plan called
|
||||
out — this SOP is not claiming that gate is cleared.
|
||||
|
||||
## 6. Schema migration pattern
|
||||
|
||||
SQLite can't `ALTER` a primary key. The codebase's existing convention (see `progress` table
|
||||
history) is:
|
||||
|
||||
1. `PRAGMA table_info(<table>)` → check if the new column/shape is already there (idempotent,
|
||||
safe to run on every boot).
|
||||
2. Simple new nullable column → `ALTER TABLE ... ADD COLUMN`.
|
||||
3. Primary-key or structural change → build `<table>_new` with the new shape, `INSERT ...
|
||||
SELECT` the old data across with sane defaults for new required columns, `DROP TABLE`,
|
||||
`RENAME TO`. Never drop data — give it a legacy/default value (`'unassigned'`, etc.)
|
||||
instead of discarding rows.
|
||||
4. Test the migration against a **copy of real data** (§3) before it ever runs against
|
||||
`/data/progress.db` in the container.
|
||||
|
||||
## 7. Roadmap status
|
||||
|
||||
Order agreed 2026-08-17/18. Status as of this doc:
|
||||
|
||||
1. RootNode identity/branding pass — **partial.** Shell banner fixed
|
||||
(`⎈ RootNode Academy Lab — node: k8s-lab`). Upstream KubeKosh attribution left untouched in
|
||||
footer/README/LICENSE. Broader "make Real Ops first-class" UI work not started.
|
||||
2. RootNode curriculum mode — **backend done** (§4), frontend not started.
|
||||
3. Student accounts / persistent progress — **backend done** (§5), frontend not started,
|
||||
no instructor auth yet.
|
||||
4. Instructor dashboard — not started.
|
||||
5. Per-student isolation — **Phase 1 (platform foundation) done**, see §8. Ephemeral
|
||||
per-student Lab environments with Postgres-backed lifecycle, per-environment Docker
|
||||
networking, and firewall-enforced egress restriction are implemented and tested on a single
|
||||
dev machine. **Still not cleared for minors** — no Gateway/frontend wiring yet (environments
|
||||
are only reachable via the internal Provisioner API), no instructor auth on top of it, and
|
||||
the production two-VM Proxmox split this depends on doesn't exist yet. See
|
||||
`docs/PHASE1_PLATFORM_FOUNDATION.md` final report for exactly what's verified vs. not.
|
||||
6. Scenarios → teaching modules (incident context/investigation/prevention) — not started.
|
||||
7. Capstone/exam engine — not started (placeholder stage only, see §4).
|
||||
8. Progressive hint levels — not started.
|
||||
9. Production observability — not started.
|
||||
10. CI scenario testing — not started.
|
||||
11. Container/build optimization — explicitly deprioritized by design (platform works, not urgent).
|
||||
|
||||
Next slice: either the deferred frontend login (name picker → `X-Student-Id` header, curriculum
|
||||
view), or the Phase 2 platform work (Gateway, frontend wiring to the Provisioner, instructor
|
||||
auth) — see the Phase 1 final report for the concrete Phase 2 list. Not decided here.
|
||||
|
||||
## 8. Platform Provisioner (Phase 1)
|
||||
|
||||
Full design: `docs/LAB_ARCHITECTURE.md` + `docs/PHASE1_PLATFORM_FOUNDATION.md`. Operational
|
||||
reference (config, local dev, known limitations): `platform/provisioner/README.md`. This
|
||||
section is just the map.
|
||||
|
||||
- **What it is:** an internal-only service that provisions a disposable, fully isolated Lab
|
||||
environment per student — its own Docker network, its own container running the existing,
|
||||
unmodified Lab image, its own firewall-enforced egress. Postgres-backed registry
|
||||
(`environments`, `environment_events`, `lab_hosts`), idle reaper, startup reconciliation.
|
||||
- **What it is not (yet):** reachable by a student. There's no Gateway and no frontend wiring
|
||||
— it's exercised directly via its internal API (shared-secret auth) or
|
||||
`platform/provisioner/test/run.js`. Nothing here is a live feature.
|
||||
- **Where identity lives:** unchanged from §5 — `students`/`progress` are still on the Lab
|
||||
image's own SQLite. `environments.student_id` is deliberately opaque (no FK) until a later,
|
||||
separate migration puts roster/progress on the same Postgres instance as the environment
|
||||
registry.
|
||||
- **Local dev:** `platform/provisioner/docker-compose.dev.yml` runs Postgres + the egress
|
||||
proxy + the Provisioner on one machine — the documented Phase 1 dev exception. Production
|
||||
is a two-VM Proxmox split (Platform VM / Lab Host VM); the dev compose file is explicitly
|
||||
not that.
|
||||
- **Test procedure:** `platform/provisioner/test/run.js` — 12 automated integration tests
|
||||
against a live dev stack (isolation, firewall ordering, reset/destroy cleanup, audit
|
||||
events). Some properties (idle reaper timing, remote TLS lab-host, reconciliation-on-crash,
|
||||
real-image k3s behavior) were validated manually during implementation rather than
|
||||
automated — see the Phase 1 final report for what and why.
|
||||
@@ -0,0 +1,482 @@
|
||||
# RootNode Academy — Ephemeral Lab Architecture
|
||||
|
||||
Status: **Phase 1 (platform foundation) IMPLEMENTED 2026-08-18** — see
|
||||
`docs/PHASE1_PLATFORM_FOUNDATION.md` for the concrete schema/API/test results, and
|
||||
`platform/provisioner/README.md` for how to run it. This document remains the architecture
|
||||
record for the full ephemeral-lab direction; later phases (Gateway, frontend wiring, instructor
|
||||
dashboard, production Proxmox deployment — see the Phase 1 report's "Future Phase 2 items")
|
||||
build on it but aren't built yet. Supersedes the namespace-isolation
|
||||
(`shared_namespace`/`ephemeral_cluster`) approach that was in flight before this redirect — see
|
||||
`scenarios/SCHEMA.md`'s `isolation_type` note for the pointer back here.
|
||||
|
||||
Committed direction: **one fully isolated, disposable environment per student. No shared
|
||||
Kubernetes control plane, ever.** This is the long-term architecture, not a stopgap.
|
||||
|
||||
**Decisions locked 2026-08-18** (resolves the four open questions this doc originally ended
|
||||
with — see §2–§5 below for where each lands, and `docs/PHASE1_PLATFORM_FOUNDATION.md` for the
|
||||
Phase 1 build plan against these decisions):
|
||||
1. **Egress:** restricted, explicit allowlist. No unrestricted internet egress, ever. No route
|
||||
to trusted RootNode management/CDAA/home/business networks, under any allowlist entry.
|
||||
2. **Platform DB:** PostgreSQL. The Platform control plane is not built around SQLite. The
|
||||
existing single-container local dev mode keeps using its own local SQLite as today —
|
||||
unaffected, unrelated database.
|
||||
3. **Suspend/resume:** experimental only, not load-bearing. The guaranteed lifecycle is
|
||||
`provision → run → destroy → recreate`. Persistent academic state lives outside the Lab
|
||||
specifically so destroying one never loses student progress — suspend/resume is a possible
|
||||
future optimization, not something correctness depends on.
|
||||
4. **Resource sizing:** not hard-coded. Quotas are policy/config-driven. Target host class is
|
||||
~16 cores/32 threads, 128GB RAM, SSD/NVMe — actual per-lab limits and concurrency get set
|
||||
by benchmarking against that class, not guessed up front.
|
||||
|
||||
**Corrections locked 2026-08-18** (round 2 — architectural fixes required before Phase 1
|
||||
implementation begins; folded into §1, §4, §7, §9 below, not just listed here):
|
||||
|
||||
- **A. VM boundary.** Privileged Lab containers must never share an OS/VM boundary with the
|
||||
Platform (Postgres, control-plane) in production — separate VMs under Proxmox, connected by
|
||||
a Provisioner-initiated control channel only. The Lab Host is a *configured target*, not an
|
||||
assumption that Docker is local. Single-machine is still fine for Phase 1 development — see §1/§9.
|
||||
- **B. Egress is a firewall control, not an app convention.** `HTTP_PROXY`/`HTTPS_PROXY`
|
||||
configure well-behaved apps for convenience; they enforce nothing. The firewall is the
|
||||
control, and rule *ordering* matters — required infrastructure (proxy, DNS) must be
|
||||
explicitly permitted before the blanket RFC1918 drops, or those drops shadow the allow
|
||||
rules and nothing works. See §4 — this also corrects a real ordering bug in the original
|
||||
design (explained there).
|
||||
- **C. `student_id` is opaque in Phase 1.** No real foreign key to a `students` table —
|
||||
students/progress stay on SQLite until a later, separate Platform DB migration. Documented,
|
||||
not silently assumed. See §3/§9.
|
||||
- **D. Lab host is an abstraction from day one**, even with exactly one configured host in
|
||||
Phase 1 — the Provisioner is never allowed to hard-code "Docker is local." See §4.
|
||||
|
||||
---
|
||||
|
||||
## 0. What already exists (grounding, not invention)
|
||||
|
||||
The current repo (`~/Dev/Gitea/ced4568/rootnode-academy`) already documents almost this exact
|
||||
intent in its own README, just never built out:
|
||||
|
||||
> "The upstream project assumes one learner, one container. RootNode Academy runs it across
|
||||
> three delivery modes... One container **per student** — never share a cluster between
|
||||
> learners... Put authentication in front (proxy-level)... Idle-timeout + auto-teardown."
|
||||
|
||||
And the existing image already **is** a complete single-student lab: nginx + React frontend +
|
||||
Node/Express backend + `node-pty` WebSocket terminal + a single-node k3s cluster, all in one
|
||||
`--privileged` container (confirmed required — k3s needs kernel namespaces/cgroups, per
|
||||
`README.md:52`). That container is the unit this design multiplies, not something it replaces.
|
||||
|
||||
What changes: today that one container also owns identity/progress (SQLite at
|
||||
`/data/progress.db`, one volume per student, mounted by hand). This design pulls that
|
||||
ownership out into a persistent Platform, and makes the container itself fully disposable.
|
||||
|
||||
---
|
||||
|
||||
## 1. Proposed architecture
|
||||
|
||||
Two roles — and in production, on **two separate VM/OS boundaries**, not just two logical
|
||||
components sharing a box. Correction A drives this: `--privileged` Docker containers are not
|
||||
a security boundary from their own Docker host, so the boundary has to be a real VM boundary,
|
||||
not a process boundary.
|
||||
|
||||
**Production topology:**
|
||||
|
||||
```
|
||||
Proxmox VE (or equivalent hypervisor)
|
||||
│
|
||||
├── RootNode Platform VM (trusted — never runs student code)
|
||||
│ │
|
||||
│ │ Platform API roster · login · progress · curriculum · exam records ·
|
||||
│ │ environment registry · instructor controls
|
||||
│ │ Platform DB PostgreSQL — environments + audit log now;
|
||||
│ │ students/progress join later (§3/§9, correction C)
|
||||
│ │ Provisioner talks OUT to a *configured* Lab Host endpoint —
|
||||
│ │ never assumes Docker is local (§4, correction D)
|
||||
│ │ Gateway opaque-token routing to student lab containers;
|
||||
│ │ only component with a leg in both worlds
|
||||
│ │
|
||||
│ ▼ (Provisioner-initiated control channel only — TCP+TLS to the Lab
|
||||
│ Host's Docker API. Nothing on the Lab Host VM ever initiates a
|
||||
│ connection back. See §4/§7 for why this direction matters.)
|
||||
│
|
||||
└── RootNode Lab Host VM (untrusted — this is where privileged student
|
||||
│ containers actually run; holds nothing the
|
||||
│ Platform needs protected)
|
||||
│
|
||||
├── Student A's LAB ── own private network, own nested k3s, no identity
|
||||
├── Student B's LAB ── own private network, own nested k3s, no identity
|
||||
└── Student C's LAB ── own private network, own nested k3s, no identity
|
||||
```
|
||||
|
||||
Each Lab is **today's `Dockerfile` and `entrypoint.sh`, functionally unchanged** — still runs
|
||||
k3s + the terminal + scenario setup/validate/teardown exactly as now. The only change inside
|
||||
it: it stops being the system of record for who's using it or what they've completed (§3, §9).
|
||||
|
||||
A compromised student container's worst case is the Lab Host VM — never Postgres, never the
|
||||
Platform API, because there is no network path from that VM back to the Platform VM at all
|
||||
(§4, §7). That containment is the actual point of the two-VM split, not a deployment nicety.
|
||||
|
||||
**Phase 1 development exception:** running the Platform and a Lab container on one machine is
|
||||
fine for local testing (§9) — the Provisioner just gets configured with a single `lab_host`
|
||||
pointing at the local Docker socket instead of a remote TLS endpoint. The *code* never
|
||||
special-cases "local" vs "remote" — same abstraction either way (§4/§9 of
|
||||
`docs/PHASE1_PLATFORM_FOUNDATION.md`).
|
||||
|
||||
---
|
||||
|
||||
## 2. Environment lifecycle
|
||||
|
||||
State machine per environment, tracked in the Platform DB (`environments` table, §3):
|
||||
|
||||
**Guaranteed path** (what correctness depends on): `requested → provisioning → running →
|
||||
resetting → running` (reset loop, same route) and `(provisioning|running|error) → destroying
|
||||
→ destroyed`. `provisioning → error` and `running → error` cover failed/lost environments;
|
||||
`error → destroying → destroyed` (or `→ resetting → running`, when recoverable) is the way out.
|
||||
|
||||
```
|
||||
requested → provisioning → running ─┬─→ resetting → running (clean slate, same route)
|
||||
└─→ destroying → destroyed
|
||||
provisioning → error (failed to reach ready state)
|
||||
running → error (health check fails mid-session)
|
||||
error → destroying → destroyed (or → resetting → running, if recoverable)
|
||||
|
||||
[EXPERIMENTAL, not relied on for correctness — see decision #3]
|
||||
running ──suspend──→ suspended ──resume──→ running (or → error, if resume fails)
|
||||
```
|
||||
|
||||
- **provision** — Provisioner picks a host with capacity (§5), creates a private network for
|
||||
this student (§4), starts a container from the pinned Lab image, generates a random
|
||||
high-entropy routing token, waits for the existing readiness signal (k3s API ready + backend
|
||||
`/api/health` OK — same check the current entrypoint already does), writes the `environments`
|
||||
row, hands the route back to the Platform API.
|
||||
- **start** — used only for `provisioning → running` (starting a freshly created container).
|
||||
There is no supported path that starts a previously-stopped container in the guaranteed
|
||||
lifecycle — see suspend/resume below.
|
||||
- **suspend/resume — EXPERIMENTAL, decision #3.** `docker stop`/`docker start` on a k3s-in-container
|
||||
Lab is *not assumed to work* and is not built as part of the guaranteed lifecycle. It's
|
||||
worth testing (see the Phase 1 test plan) because if it turns out reliable it's a cheap
|
||||
latency win later — but nothing in the platform depends on it succeeding. The only supported
|
||||
way to bring an environment back is `destroy` + `provision` (or `reset`, which is that same
|
||||
pair against one `environments` row/routing token). This is also *why* academic state living
|
||||
outside the Lab (§3) isn't optional — it's what makes "just destroy and recreate" a fully
|
||||
safe default instead of a data-loss risk.
|
||||
- **reset** — destroy and recreate the container from the pinned image, **same routing token
|
||||
and same `environments` row**, so the student's "return to my lab" link doesn't change.
|
||||
This is the instructor-facing "reset student environment" control.
|
||||
- **destroy** — stop + remove container, remove its private network, release the routing
|
||||
token, mark the `environments` row `destroyed` with a timestamp. Progress already lives in
|
||||
the Platform DB (§3), so destroying an environment never loses academic data — only
|
||||
in-terminal, not-yet-validated work.
|
||||
- **Idle reaper** (new, required — not optional) — background job in the Provisioner checks
|
||||
`last_active_at` per running environment (heartbeat from the Gateway or from Lab `/api/health`
|
||||
polling) and auto-suspends past an idle threshold, auto-destroys past a longer abandoned
|
||||
threshold. Without this, "ephemeral by default" is just a description, not a guarantee —
|
||||
a closed browser tab does not stop a running k3s cluster.
|
||||
|
||||
---
|
||||
|
||||
## 3. Data model changes
|
||||
|
||||
Everything below lives in the **Platform DB**, not inside any Lab container.
|
||||
|
||||
Already built (previous slice, unaffected by this redirect — still correct under this
|
||||
architecture):
|
||||
- `students(student_id, display_name, cohort, created_at, last_active_at)`
|
||||
- `progress(student_id, scenario_id, status, attempts, ...)` — composite key, already
|
||||
per-student.
|
||||
- `sessions` / `exam_progress` — exam records, already student-attributed.
|
||||
|
||||
New:
|
||||
- `environments(environment_id UUID PK, student_id, status, routing_token, container_id,
|
||||
lab_host_id FK, internal_ip, cpu_limit, mem_limit_mb, storage_limit_mb, image_ref, created_at,
|
||||
started_at, last_active_at, destroy_at, error_reason)` — **`student_id` is an opaque external
|
||||
identifier here (correction C), not a foreign key.** Students/progress stay on the existing
|
||||
SQLite in Phase 1 (see §9); Postgres has no `students` table yet, so there is nothing to key
|
||||
against. The Provisioner trusts whatever `student_id` its caller passes — it does not
|
||||
validate it against a roster. A real FK gets added when students/progress actually migrate
|
||||
onto this Postgres instance, not before. `lab_host_id`, by contrast, *is* a real FK — both
|
||||
tables live in the same Provisioner-owned Postgres DB (correction D, full schema in
|
||||
`docs/PHASE1_PLATFORM_FOUNDATION.md`).
|
||||
- `lab_hosts(lab_host_id PK, endpoint, enabled, ...)` — the abstraction correction D requires:
|
||||
which Docker endpoint a given environment's container actually lives on. Phase 1 ships with
|
||||
exactly one row (pointing at either a local socket in dev or the Lab Host VM's TLS endpoint
|
||||
in production) — the table exists so environment creation is never coupled to "Docker is
|
||||
local" as an assumption, even though there's no scheduling logic yet.
|
||||
- `environment_events(id, environment_id, event_type, actor, detail, created_at)` — audit
|
||||
trail for provision/start/stop/reset/destroy/error, needed for both incident response and
|
||||
ordinary "why did my lab disappear" support.
|
||||
|
||||
**Removed from the Lab image's responsibility:** the `students` and roster-facing parts of the
|
||||
schema I added to `backend/server.js` in the previous slice move to be Platform-only. The Lab
|
||||
image's own SQLite either goes away entirely or is kept only as scratch/ephemeral state that's
|
||||
allowed to die with the container — it must not be treated as authoritative for anything.
|
||||
|
||||
**Decision #2: PostgreSQL, not SQLite.** The Platform control plane — the environment registry
|
||||
specifically, and the target for progress/roster once they migrate over — is Postgres. The
|
||||
existing single-container local dev mode keeps its own local SQLite exactly as today; that's a
|
||||
different, unrelated database serving a different purpose (single-tenant local testing), not
|
||||
something this decision touches. See `docs/PHASE1_PLATFORM_FOUNDATION.md` for the exact schema
|
||||
being stood up first.
|
||||
|
||||
---
|
||||
|
||||
## 4. Networking model
|
||||
|
||||
- **One private Docker network per student environment**, not one shared network for all
|
||||
labs. Docker's default per-network isolation means Student A's container has no L2/L3 path
|
||||
to Student B's container at all — this is stronger and simpler than trying to firewall
|
||||
students apart on a shared network.
|
||||
- **No published ports on Lab containers.** Today's image publishes `-p 7554:80`; under this
|
||||
design that stops — the only thing that can reach a Lab container is the Gateway, over that
|
||||
student's private network.
|
||||
|
||||
### Egress — correction B: the firewall is the control, not the proxy env vars
|
||||
|
||||
`HTTP_PROXY`/`HTTPS_PROXY` are set on each Lab container so well-behaved processes (containerd
|
||||
pulling scenario images) route through the allowlisting proxy cleanly. **That is configuration,
|
||||
not enforcement** — a process that ignores those env vars and dials out directly must still be
|
||||
blocked, by something that doesn't depend on the container's cooperation. That something is a
|
||||
host firewall rule, evaluated on every packet regardless of what any process inside the
|
||||
container does.
|
||||
|
||||
**Ordering bug in the original draft, corrected here:** the previous version of this section
|
||||
listed proxy/DNS `ACCEPT` rules before the RFC1918 `DROP` rules in prose, but specified
|
||||
installing them with `iptables -I` (insert-at-head) in that same top-to-bottom order — which
|
||||
actually reverses the result, since each `-I` without a rule number pushes the *new* rule to
|
||||
position 1. Run in that order, the RFC1918 drops end up evaluated *before* the proxy/DNS
|
||||
accepts, silently blocking the proxy itself if it happens to sit on an RFC1918 address (it
|
||||
will — it's Platform-side infrastructure). Corrected mechanism (exact rules in
|
||||
`docs/PHASE1_PLATFORM_FOUNDATION.md` §5): each student network gets its own dedicated iptables
|
||||
chain, built with `-A` (append, so rules land in the order they're added — no head-insertion
|
||||
trap), containing explicit `ACCEPT` rules for the egress proxy, DNS, and any specifically
|
||||
required Platform endpoint *first*, then the RFC1918 `DROP`s, then a final default-`DROP`. That
|
||||
chain is reached from `DOCKER-USER` (the chain Docker guarantees it won't silently rewrite) via
|
||||
one jump rule matching that student's subnet.
|
||||
|
||||
**What no student lab may ever reach, regardless of allowlist state:**
|
||||
- RootNode management networks
|
||||
- the Platform database (Postgres)
|
||||
- Docker/Proxmox management interfaces (including the Lab Host VM's own Docker API endpoint —
|
||||
a Lab container talking back to the thing that provisioned it is exactly the escalation path
|
||||
this design exists to prevent)
|
||||
- CDAA infrastructure
|
||||
- home/trusted networks
|
||||
- other students' lab networks (already unreachable via Docker's own per-network isolation
|
||||
above; the RFC1918 drop is a deliberate second layer, not redundant — it also catches any
|
||||
case where two student networks' address ranges could otherwise see each other at the host
|
||||
routing level)
|
||||
|
||||
**Decision #1, restated with the ordering fix:** restricted egress, explicit allowlist, no
|
||||
unrestricted internet access ever, enforced at the firewall layer with required infrastructure
|
||||
explicitly permitted *before* the blanket denies — not the reverse, and not left implicit.
|
||||
|
||||
### VM-to-VM boundary — correction A
|
||||
|
||||
The Lab Host VM has **no route back to the Platform VM at all**, in either direction, except
|
||||
the one the Provisioner itself opens. The Provisioner (on the Platform VM) connects *out* to
|
||||
the Lab Host's Docker API over TCP+TLS with a client certificate only the Provisioner holds;
|
||||
nothing on the Lab Host VM — student containers or the VM's own OS — can open a connection back
|
||||
toward the Platform VM. This is enforced at the hypervisor/network level (Proxmox VLAN/firewall
|
||||
rules between the two VMs), the same category of control as the CDAA/home boundary below, not
|
||||
something Docker networking alone provides. A compromised student container's blast radius
|
||||
stops at the Lab Host VM.
|
||||
|
||||
**No route to trusted CDAA/home/business networks, full stop.** This is a host/hypervisor
|
||||
boundary, not something Docker networking alone can guarantee — the box(es) these containers
|
||||
run on need to sit on their own segment with no bridge/route to those networks, consistent with
|
||||
the existing `RootNode_Academy_Business_Pack/06_Technology/01_Technical_Platform_Plan.md`
|
||||
("dedicated student network/DMZ separated from trusted networks"). This design assumes that
|
||||
host-level segmentation is a prerequisite, not something the app layer can create on its own.
|
||||
|
||||
### Lab host abstraction — correction D
|
||||
|
||||
The Provisioner never talks to "the local Docker socket" as a hard-coded assumption — it talks
|
||||
to whichever `lab_hosts` row it's configured to use (§3), resolved to a Docker endpoint that
|
||||
may be `unix:///var/run/docker.sock` (Phase 1 dev, single machine) or a remote
|
||||
`tcp://<lab-host-vm>:2376` with TLS (production, per the topology in §1). Phase 1 ships exactly
|
||||
one enabled `lab_host` and zero scheduling logic — the abstraction exists so adding a second
|
||||
host later is a config row, not a rewrite.
|
||||
|
||||
- **Gateway** is the only component with a leg in both the public-facing network and every
|
||||
student's private network. It resolves `routing_token → container internal IP:port` by
|
||||
querying the Platform API (or a shared registry cache) per request/connection, then proxies
|
||||
both HTTPS (scenario API, static frontend) and the WSS terminal connection.
|
||||
|
||||
---
|
||||
|
||||
## 5. Resource model
|
||||
|
||||
- Per-environment caps applied at container-create time: `--cpus`, `--memory` (with
|
||||
`--memory-swap` pinned equal to `--memory` so nothing swaps), `--pids-limit` (fork-bomb
|
||||
guard). Storage: needs a size-capped volume — exact mechanism (`--storage-opt size=`, a
|
||||
loopback-backed bind mount, or a tmpfs cap) depends on the host's actual Docker storage
|
||||
driver, which I haven't inspected — needs to be confirmed against the real deploy target
|
||||
before picking one.
|
||||
- **Decision #4: policy-driven, not hard-coded.** Limits come from config, not constants in
|
||||
code — a resource-policy file the Provisioner reads at startup (and can reload), not numbers
|
||||
baked into the create-container call. No specific numbers are chosen yet. Target host class
|
||||
for planning purposes: ~16 cores/32 threads, 128GB RAM, SSD/NVMe. Actual per-lab limits and
|
||||
how many environments run concurrently on that class get set by benchmarking (Phase 1's test
|
||||
plan includes the first real measurements), not by guessing backward from the spec sheet.
|
||||
- **Admission control:** Provisioner refuses to `provision` past the point where already-running
|
||||
environments' summed limits would exceed host capacity. Simple counter/gate to start; can
|
||||
become real bin-packing later if there's ever more than one host.
|
||||
|
||||
---
|
||||
|
||||
## 6. Routing/auth model
|
||||
|
||||
1. Student logs in at the Platform (existing roster/pick-your-name flow, unchanged).
|
||||
2. Frontend asks the Platform for "my lab." If none running, Platform calls the Provisioner to
|
||||
`provision` (or `start`, if one's suspended).
|
||||
3. Platform returns an **opaque, high-entropy routing token** scoped to
|
||||
`(student_id, environment_id)`, short TTL, revoked on `destroy`.
|
||||
4. Frontend connects to the Gateway using that token — `https://labs.<domain>/e/<token>/...`
|
||||
and `wss://labs.<domain>/e/<token>/shell-ws`. The Lab container itself never sees a
|
||||
student identity; it's purely an execution surface addressed by an opaque route.
|
||||
5. Gateway validates the token maps to a currently `running` environment before proxying —
|
||||
rejects stale tokens from destroyed/reset environments outright, so there's no window where
|
||||
an old link reaches a different (possibly now-different-student's) container.
|
||||
6. **Instructor auth** extends the shared-passcode model already used for the roster-add
|
||||
endpoint: instructor-only calls (`list environments`, `reset`, `terminate`, `resource usage`)
|
||||
go through the Platform API under that same gate, which then calls the Provisioner
|
||||
internally. Students and the public internet never talk to the Provisioner or the Docker
|
||||
socket directly — only the Platform backend does.
|
||||
|
||||
---
|
||||
|
||||
## 7. Security boundaries (defense in depth)
|
||||
|
||||
1. **Browser ↔ Gateway** — public-facing, TLS, opaque routing tokens only. Nothing
|
||||
cluster-admin-shaped is reachable from here.
|
||||
2. **Gateway ↔ Lab container** — private per-student Docker network only, no published ports,
|
||||
Gateway is the sole entry point.
|
||||
3. **Inside the Lab container** — `--privileged` is required for k3s (confirmed, not
|
||||
avoidable — this isn't a place to try to "hold the line" without breaking k3s). The
|
||||
mitigation is containment, not avoidance: this privileged container has no reachable
|
||||
network path to anything except the Gateway, runs as a throwaway that gets destroyed/reset
|
||||
routinely, and cluster-admin *inside a student's own disposable cluster* is expected and
|
||||
fine — that's the entire point of full isolation, a student doing CKS-tier cluster-admin
|
||||
work only ever affects their own throwaway cluster.
|
||||
4. **Lab container ↔ its own Docker host (correction A — the boundary that actually matters
|
||||
here)** — `--privileged` is explicitly **not treated as a sufficient isolation boundary from
|
||||
its own Docker host.** In production the Lab Host VM holding these containers is a separate
|
||||
VM from the Platform VM, with no route back to it at all (§4). If a student ever escapes
|
||||
their container, the ceiling is the Lab Host VM — never Postgres, never the Platform API,
|
||||
never another student's environment.
|
||||
5. **Provisioner ↔ Lab host's Docker API** — only the Provisioner can invoke it, over a
|
||||
connection *it* initiates (TCP+TLS, client-cert auth, per §4's lab-host abstraction). The
|
||||
Docker endpoint is never student-reachable, never internet-reachable, and nothing on the Lab
|
||||
Host VM can initiate a connection back toward the Provisioner or the Platform VM.
|
||||
6. **Platform VM ↔ other networks** — dedicated segment, no bridge to home/business/CDAA
|
||||
trusted infrastructure, matching the existing Technical Platform Plan doc. Prerequisite, not
|
||||
something this design builds. Same standard applies to the Lab Host VM independently — it
|
||||
isn't exempted just because it's "the untrusted one."
|
||||
7. **Routing tokens** — CSPRNG-generated, short TTL, invalidated on any lifecycle transition
|
||||
out of `running`.
|
||||
|
||||
---
|
||||
|
||||
## 8. Failure/recovery behavior
|
||||
|
||||
- **Provisioning fails** (host at capacity, image pull fails, k3s doesn't reach ready state in
|
||||
time) — environment marked `error`, student gets an explicit "your lab failed to start, try
|
||||
again" message (never a silent hang), one automatic retry, then surfaced to the instructor
|
||||
dashboard (future) as a flagged failure.
|
||||
- **Mid-session crash** — Gateway's health check detects the upstream is unreachable, marks
|
||||
the environment `error`, offers the student a "restart my lab" action (→ `reset`, since a
|
||||
crashed in-container k3s is unlikely to resume cleanly). Because progress/identity/exam
|
||||
records live in the Platform DB, this never loses the student's academic record — worst case
|
||||
is losing in-progress, not-yet-validated terminal work.
|
||||
- **Host failure / reboot** — every environment on that host becomes unreachable. On
|
||||
Provisioner restart, it must **reconcile its registry against actual running containers**
|
||||
rather than trust stale DB state — some containers may auto-restart via Docker's restart
|
||||
policy, most won't; anything the registry claims is `running` that isn't actually running
|
||||
gets marked `error`, not silently left wrong.
|
||||
- **Idle/abandoned environments** — the reaper (§2) bounds resource leakage from crashed tabs
|
||||
or students who just walked away, independent of any other failure path.
|
||||
- **Core resilience property of this whole design:** because the Platform DB — not the
|
||||
disposable environment — is the system of record, every failure mode above is recoverable
|
||||
without academic data loss. That's the actual payoff of the split in §1/§3, not just an
|
||||
architectural nicety.
|
||||
|
||||
---
|
||||
|
||||
## 9. Migration path from the current single shared container
|
||||
|
||||
- **Phase 0 (already true today, keep it):** running the existing image standalone —
|
||||
`docker run --privileged -p 7554:80 ... rootnode-academy:latest` — is exactly the
|
||||
single-user local dev/test mode you asked to keep available. No change needed to support
|
||||
this; it already works and stays supported as-is.
|
||||
- **Phase 1:** Build Provisioner + `environments`/`environment_events`/`lab_hosts` registry
|
||||
around the **existing, unmodified** Lab image, on Postgres, with the egress firewall/proxy
|
||||
from §4 in place from the start (not bolted on later). Development exception (correction A):
|
||||
running Platform + Provisioner + one Lab container all on a single machine is fine here — the
|
||||
Provisioner is configured with one `lab_host` pointing at the local Docker socket instead of
|
||||
a remote TLS endpoint, same code path either way. Prove two lab environments can run side by
|
||||
side, fully isolated from each other, before any production traffic or a second VM enters
|
||||
the picture. `students`/`progress` stay on the existing SQLite through this phase
|
||||
(correction C) — `environments.student_id` is opaque, not a foreign key, until a later,
|
||||
separate migration actually moves roster/progress onto Postgres.
|
||||
- **Phase 2 (the one real code change inside the Lab image):** narrow `backend/server.js`
|
||||
inside the Lab image down to a "dumb executor" — scenario setup/validate/teardown/answer and
|
||||
the WS terminal stay, but it stops owning `students`/roster and stops being the source of
|
||||
truth for progress. Either it calls back to the Platform API to report validation results,
|
||||
or — cleaner — the Platform API becomes the only thing the frontend talks to for
|
||||
progress/curriculum/identity, and it calls the Lab container (via the route the Provisioner
|
||||
already tracks) purely to execute against that student's cluster and get a pass/fail back.
|
||||
- **Phase 3:** Wire instructor controls (list/reset/terminate/resource-usage) against the new
|
||||
registry — this is also where the "no auth on roster-add" gap from the previous slice gets
|
||||
closed for real, since instructor auth is now a load-bearing security boundary, not just
|
||||
roster hygiene.
|
||||
- **Phase 4 (cutover):** point the real domain at the Gateway instead of a single container.
|
||||
Keep the Phase 0 single-container path available under local/dev tooling, per your explicit
|
||||
requirement.
|
||||
- **Rollback safety:** at every phase through Phase 3, the current single-shared-container path
|
||||
keeps working completely unmodified — nothing about this migration risks the working
|
||||
baseline until the Phase 4 cutover, which is also the only step that's meaningfully
|
||||
reversible-with-effort rather than trivially reversible.
|
||||
|
||||
---
|
||||
|
||||
## 10. Exact files/components this touches (once approved — no code written yet)
|
||||
|
||||
**New, none of it touching the existing Lab image's core logic:**
|
||||
- `platform/provisioner/` — Docker lifecycle management against a *configured* `lab_host`
|
||||
endpoint (correction D — never a hard-coded local socket), `environments`/`lab_hosts` CRUD,
|
||||
per-environment firewall chain management (correction B), idle reaper, admission control.
|
||||
- `platform/egress-proxy/` — the shared allowlisting proxy every Lab container's egress must
|
||||
pass through (correction B).
|
||||
- `platform/gateway/` — token→route resolution, HTTPS + WSS proxying (candidate: Traefik with
|
||||
a dynamic provider backed by the Platform API, or a small custom Node proxy — not decided,
|
||||
not needed for this design review).
|
||||
- Platform DB migrations: `environments`, `environment_events`, `lab_hosts` tables — full
|
||||
schema in `docs/PHASE1_PLATFORM_FOUNDATION.md`.
|
||||
|
||||
**Modified, once implementation starts:**
|
||||
- `backend/server.js` — split as described in §9 Phase 2. This is presently both "the lab
|
||||
image's backend" and "the only backend that exists"; under this design it forks into a
|
||||
narrower Lab-side executor and a separate, persistent Platform-side API. Exact file layout
|
||||
for that split is an implementation decision, not part of this design doc.
|
||||
- `README.md` — the "Multi-Student Deployment" section (`README.md:128-146`) currently
|
||||
documents per-student **volumes** for persistence; that persistence model is superseded by
|
||||
§3 above and the README needs updating once this ships, so it doesn't contradict reality.
|
||||
- `docs/ENGINEERING_SOP.md` — new section for the Platform/Provisioner/Gateway components and
|
||||
the reconciliation/reaper operational procedures from §8.
|
||||
|
||||
**Unchanged:**
|
||||
- `Dockerfile`, `scripts/entrypoint.sh`, `scripts/nginx.conf` for the Lab image itself — it's
|
||||
still one container running k3s + backend + terminal, exactly as today, just multiplied and
|
||||
stripped of identity ownership.
|
||||
- All scenario content (`scenarios/data/`, `scenarios/bundles/`, `scenarios/rootnode-path.json`)
|
||||
— unaffected either way; every scenario now runs with full admin of its own disposable
|
||||
cluster, so the earlier `shared_namespace`/`ephemeral_cluster` split has no bearing here.
|
||||
|
||||
**New Dockerfile needed (not yet written):** one for the Provisioner+Gateway services — these
|
||||
are new components, not variants of the existing Lab image.
|
||||
|
||||
---
|
||||
|
||||
## Status
|
||||
|
||||
All four open questions this doc originally ended with are resolved — see the decisions block
|
||||
at the top and §2–§5 above. Next: `docs/PHASE1_PLATFORM_FOUNDATION.md` — the concrete build
|
||||
plan for the platform foundation (Postgres environment registry, provisioner skeleton,
|
||||
networking, egress, resource policy, reaper) against these decisions.
|
||||
@@ -0,0 +1,483 @@
|
||||
# RootNode Academy — Phase 1: Platform Foundation
|
||||
|
||||
Status: **IMPLEMENTED, tested on a single dev machine, 2026-08-18.** Code lives in
|
||||
`platform/provisioner/` and `platform/egress-proxy/`. This doc still reflects the corrected
|
||||
design (four architecture corrections: VM boundary, egress-as-firewall-not-app-config, opaque
|
||||
`student_id`, lab-host abstraction) — the implementation matches it, with a handful of
|
||||
deviations found and fixed during testing, listed in "Implementation notes" at the end of each
|
||||
section below where relevant, and summarized in `platform/provisioner/README.md`'s "Known
|
||||
limitations." Builds on `docs/LAB_ARCHITECTURE.md` (read that first).
|
||||
|
||||
**Scope — this phase only:**
|
||||
Postgres-backed environment registry · lifecycle state model · provisioner service skeleton ·
|
||||
per-student private Docker network · disposable Lab container creation from the existing
|
||||
image · configurable CPU/RAM/PID/storage limits · restricted egress · unique opaque routing
|
||||
token · environment event/audit log · destroy/reset · idle-reaper design · current
|
||||
single-container dev mode kept working, untouched.
|
||||
|
||||
**Explicitly not in this phase:** instructor dashboard, self-paced public enrollment, any
|
||||
scenario content change, unrestricted egress, removing local single-user mode. Nothing below
|
||||
proposes touching `backend/server.js`, `Dockerfile`, `scripts/entrypoint.sh`, or
|
||||
`scenarios/**` — the Lab image is used exactly as it exists today.
|
||||
|
||||
---
|
||||
|
||||
## 1. Service boundaries
|
||||
|
||||
Two new components. Neither is the Lab image; neither is public-facing.
|
||||
|
||||
- **Provisioner** (`platform/provisioner/`) — new Node service. Owns: the Postgres
|
||||
`environments`/`environment_events`/`lab_hosts` tables, the Docker Engine API connection (via
|
||||
`dockerode`, not shelling out to the `docker` CLI), network/container lifecycle, the idle
|
||||
reaper, and egress-firewall management for each student network. Exposes an internal HTTP API
|
||||
(§4) gated by a shared-secret header — same pattern already used for the instructor-passcode
|
||||
gate on the roster endpoint, so this doesn't introduce a new auth idiom.
|
||||
**Docker connection is always resolved through `lab_hosts` (§2), never hard-coded** — Phase 1
|
||||
configures exactly one row, pointing at `unix:///var/run/docker.sock` for single-machine dev
|
||||
or a `tcp://<lab-host-vm>:2376` TLS endpoint for the two-VM production topology in
|
||||
`docs/LAB_ARCHITECTURE.md` §1. Same code path either way; only the config differs.
|
||||
- **Egress proxy** (`platform/egress-proxy/`) — one shared instance (not per-student), a
|
||||
minimal HTTP(S) forward proxy (Squid) with a static host allowlist. Every Lab container's
|
||||
`HTTP_PROXY`/`HTTPS_PROXY` env vars point at it; the student network's firewall rule permits
|
||||
egress *only* to this proxy and to DNS, so there's no way for a Lab container to reach the
|
||||
internet except through something the Platform's allowlist controls.
|
||||
|
||||
Nothing else changes. No new Platform API / roster-progress migration in this phase — that's
|
||||
explicitly out of scope (see §2 note on `student_id`).
|
||||
|
||||
---
|
||||
|
||||
## 2. Exact DB schema (PostgreSQL)
|
||||
|
||||
New database, e.g. `rootnode_platform`. Only the environment registry ships this phase —
|
||||
`students`/`progress`/`sessions` stay on the existing SQLite inside the Lab image for now;
|
||||
migrating those onto this Postgres instance is a later phase, not bundled in here.
|
||||
|
||||
```sql
|
||||
-- ── lab_hosts — correction D: the abstraction that keeps Docker connectivity out of code ──
|
||||
CREATE TABLE lab_hosts (
|
||||
lab_host_id TEXT PRIMARY KEY, -- e.g. 'dev-local', 'labhost-01'
|
||||
display_name TEXT NOT NULL,
|
||||
endpoint TEXT NOT NULL, -- 'unix:///var/run/docker.sock' (dev) or
|
||||
-- 'tcp://10.x.x.x:2376' (prod, TLS-verified)
|
||||
tls_ca_path TEXT, -- null for the local-socket dev case
|
||||
tls_cert_path TEXT, -- Provisioner's client cert for this host
|
||||
tls_key_path TEXT,
|
||||
enabled BOOLEAN NOT NULL DEFAULT true,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
-- Future (not this phase): capacity/status columns for real scheduling once there's
|
||||
-- more than one host. Phase 1 selects "the one enabled row" — no scheduler.
|
||||
);
|
||||
|
||||
CREATE TYPE environment_status AS ENUM (
|
||||
'requested', 'provisioning', 'running', 'resetting',
|
||||
'suspended', -- experimental path only, see decision #3
|
||||
'destroying', 'destroyed', 'error'
|
||||
);
|
||||
|
||||
CREATE TABLE environments (
|
||||
environment_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
student_id TEXT NOT NULL, -- CORRECTION C: opaque external identifier, not a
|
||||
-- foreign key. students/progress stay on the
|
||||
-- existing SQLite through Phase 1 (see
|
||||
-- docs/LAB_ARCHITECTURE.md §3/§9) — there is no
|
||||
-- students table in this database yet, so there
|
||||
-- is nothing to reference. The Provisioner trusts
|
||||
-- whatever value its caller passes; it does not
|
||||
-- validate against a roster. A real FK gets added
|
||||
-- only when roster/progress actually migrate onto
|
||||
-- this Postgres instance, in a later, separate
|
||||
-- migration — not assumed or faked here.
|
||||
lab_host_id TEXT NOT NULL REFERENCES lab_hosts(lab_host_id), -- real FK: both tables
|
||||
-- are Provisioner-owned Postgres, correction D.
|
||||
status environment_status NOT NULL DEFAULT 'requested',
|
||||
routing_token TEXT NOT NULL, -- CSPRNG, opaque, unique
|
||||
container_id TEXT, -- null until provisioned
|
||||
container_name TEXT,
|
||||
network_id TEXT,
|
||||
network_name TEXT,
|
||||
network_subnet CIDR, -- needed to install/remove the per-env firewall chain
|
||||
image_ref TEXT NOT NULL, -- pinned Lab image, e.g. rootnode-academy:1.4.0
|
||||
cpu_limit NUMERIC NOT NULL,
|
||||
mem_limit_mb INTEGER NOT NULL,
|
||||
pids_limit INTEGER NOT NULL,
|
||||
storage_limit_mb INTEGER NOT NULL,
|
||||
internal_ip TEXT, -- set once running
|
||||
internal_port INTEGER NOT NULL DEFAULT 80,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
started_at TIMESTAMPTZ,
|
||||
last_active_at TIMESTAMPTZ,
|
||||
destroy_at TIMESTAMPTZ, -- idle-reaper's scheduled destroy time
|
||||
error_reason TEXT
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX idx_environments_routing_token ON environments(routing_token);
|
||||
CREATE INDEX idx_environments_student ON environments(student_id);
|
||||
CREATE INDEX idx_environments_status ON environments(status);
|
||||
CREATE INDEX idx_environments_lab_host ON environments(lab_host_id);
|
||||
-- Phase 1 invariant: at most one non-terminal environment per student.
|
||||
-- Enforced in application logic (see §4 POST /environments), not a DB constraint yet —
|
||||
-- a partial unique index is the natural upgrade once this needs to be bulletproof under
|
||||
-- concurrent requests, not needed for a single-instance provisioner in this phase.
|
||||
|
||||
CREATE TABLE environment_events (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
environment_id UUID NOT NULL REFERENCES environments(environment_id),
|
||||
event_type TEXT NOT NULL, -- provision_requested | provisioned | started | reset
|
||||
-- | destroyed | error | reaped_idle | suspend | resume
|
||||
actor TEXT NOT NULL, -- 'system' | 'instructor:<id>' | 'student:<id>' | 'reaper'
|
||||
detail JSONB,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE INDEX idx_env_events_environment ON environment_events(environment_id);
|
||||
```
|
||||
|
||||
**Resource policy and the egress allowlist are config files, not tables.** There's no admin UI
|
||||
to manage them via a database yet (that's dashboard work, explicitly deferred), so a
|
||||
DB-backed policy table would just be a write path nothing uses. `config/resource-policy.yaml`
|
||||
(cpu/mem/pids/storage defaults, reloadable) and `platform/egress-proxy/allowlist.conf` (Squid
|
||||
ACL) are the actual source of truth — plain files an operator edits, matching how
|
||||
`entrypoint.sh` already does config via env vars rather than a database.
|
||||
|
||||
---
|
||||
|
||||
## 3. Lifecycle state machine
|
||||
|
||||
Exactly what `docs/LAB_ARCHITECTURE.md` §2 now specifies, decision #3 applied:
|
||||
|
||||
```
|
||||
requested → provisioning → running ─┬─→ resetting → running (same environment_id + routing_token)
|
||||
└─→ destroying → destroyed
|
||||
provisioning → error
|
||||
running → error
|
||||
error → destroying → destroyed
|
||||
error → resetting → running (when recoverable)
|
||||
|
||||
[EXPERIMENTAL — not required to pass for Phase 1 sign-off]
|
||||
running → suspended → running (resume)
|
||||
running → suspended → error (resume fails)
|
||||
```
|
||||
|
||||
`reset` is implemented as `destroy` + `provision` against the same DB row (new container,
|
||||
network, and internal_ip; same `environment_id`/`routing_token` so a student's saved link
|
||||
doesn't change).
|
||||
|
||||
---
|
||||
|
||||
## 4. API endpoints (Provisioner, internal-only)
|
||||
|
||||
All routes require `X-Provisioner-Key: <shared secret>`. Nothing here is reachable by a
|
||||
student or the public internet in this phase — there's no frontend/gateway wiring yet, this
|
||||
is tested directly (curl / a test script), which is also what makes it independently
|
||||
verifiable before any later phase builds on top of it.
|
||||
|
||||
| Method & path | Purpose |
|
||||
|---|---|
|
||||
| `POST /environments` | Body `{ student_id }` — `student_id` is opaque (correction C, §2), not validated against a roster. `lab_host_id` is not a caller-supplied parameter in Phase 1: the Provisioner selects the single enabled `lab_hosts` row itself (`SELECT ... WHERE enabled LIMIT 1`) — no scheduling logic, no host-selection API, matching "don't overbuild scheduling yet." Rejects if the student already has a non-terminal environment (Phase 1 invariant, see §2). Synchronous: creates network → installs firewall chain (§5) → creates+starts container → polls Lab `/api/health` → marks `running`, or `error` on timeout. Returns `{ environment_id, status, routing_token }`. |
|
||||
| `GET /environments/:id` | Full row: status, container_id, limits, timestamps. |
|
||||
| `GET /environments?student_id=` | A student's environments (expected: zero or one active). |
|
||||
| `POST /environments/:id/reset` | Destroy + recreate, same `environment_id`/`routing_token`. |
|
||||
| `DELETE /environments/:id` | Destroy: stop, remove container, remove network + its firewall rules, mark `destroyed`. |
|
||||
| `POST /environments/:id/suspend` *(experimental)* | `docker stop`, mark `suspended`. Documented as unreliable — see test plan §7. |
|
||||
| `POST /environments/:id/resume` *(experimental)* | `docker start` + re-poll readiness. On failure, mark `error` rather than leaving a wrong `suspended` status. |
|
||||
| `GET /environments/:id/events` | Audit trail for that environment. |
|
||||
| `GET /health` | Provisioner's own health: Docker reachable, Postgres reachable. |
|
||||
|
||||
Synchronous provisioning is a deliberate simplification for this phase — no job queue, no
|
||||
webhook callback. Given expected Phase 1 usage (developer testing, not a live cohort), a
|
||||
blocking HTTP call that takes a few seconds while k3s comes up is fine; converting to
|
||||
async-with-polling is the obvious upgrade if provisioning latency becomes a real UX problem
|
||||
once this is behind an actual frontend.
|
||||
|
||||
---
|
||||
|
||||
## 5. Docker / network operations (via `dockerode`, shown as CLI-equivalent for readability)
|
||||
|
||||
**Provision — network:**
|
||||
```bash
|
||||
docker network create --driver bridge rootnode-lab-net-<short-env-id>
|
||||
# inspect to get the allocated subnet, store as environments.network_subnet
|
||||
```
|
||||
|
||||
**Provision — egress firewall, corrected (§4 of `docs/LAB_ARCHITECTURE.md`, correction B).**
|
||||
The previous draft chained individual `iptables -I DOCKER-USER ...` inserts in
|
||||
ACCEPT-then-DROP order, which is backwards: `-I` without a rule number pushes each new rule to
|
||||
*position 1*, so the rules end up installed in the *reverse* of the order they were run —
|
||||
the final default-DROP would land at the top of `DOCKER-USER` and shadow everything below it,
|
||||
including the proxy/DNS accepts. Fixed by giving each environment its own dedicated chain,
|
||||
built with `-A` (append — rules land in the order added, no head-insertion trap), and reaching
|
||||
it from `DOCKER-USER` with exactly one jump rule per environment:
|
||||
|
||||
```bash
|
||||
NET_SUBNET=$(docker network inspect rootnode-lab-net-<id> -f '{{(index .IPAM.Config 0).Subnet}}')
|
||||
CHAIN="ROOTNODE-ENV-<short-env-id>"
|
||||
|
||||
iptables -N "$CHAIN" # fresh, empty chain for this environment
|
||||
|
||||
# Required infrastructure — explicitly permitted FIRST, before any blanket deny:
|
||||
iptables -A "$CHAIN" -d <egress-proxy-ip> -p tcp --dport 3128 -j ACCEPT
|
||||
iptables -A "$CHAIN" -d <dns-resolver-ip> -p udp --dport 53 -j ACCEPT
|
||||
iptables -A "$CHAIN" -d <dns-resolver-ip> -p tcp --dport 53 -j ACCEPT
|
||||
# (any explicitly-required Platform endpoint would be an additional ACCEPT line here —
|
||||
# none needed in Phase 1; the Lab image doesn't call back to the Platform yet)
|
||||
|
||||
# Everything that must never be reachable, regardless of the above — unconditional, no
|
||||
# allowlist entry can ever bypass these, satisfying "no route to trusted networks" as a
|
||||
# hard rule rather than a policy convention:
|
||||
iptables -A "$CHAIN" -d 10.0.0.0/8 -j DROP
|
||||
iptables -A "$CHAIN" -d 172.16.0.0/12 -j DROP
|
||||
iptables -A "$CHAIN" -d 192.168.0.0/16 -j DROP
|
||||
iptables -A "$CHAIN" -j DROP # default deny, catches everything else (public internet)
|
||||
|
||||
# Hook the student's subnet into this chain. -I here is fine and intentional: we want each
|
||||
# new environment's jump rule evaluated ahead of any broader, pre-existing DOCKER-USER rule,
|
||||
# and since every environment matches a disjoint source subnet, the relative order of jump
|
||||
# rules against each other never matters — only that each is ahead of a catch-all, if one
|
||||
# ever exists at the DOCKER-USER level.
|
||||
iptables -I DOCKER-USER -s "$NET_SUBNET" -j "$CHAIN"
|
||||
```
|
||||
|
||||
This also directly satisfies the explicit "must never reach" list from
|
||||
`docs/LAB_ARCHITECTURE.md` §4: RootNode management networks, the Platform database, Docker/
|
||||
Proxmox management interfaces (including the Lab Host VM's own Docker API — notably, since
|
||||
that API is what the Provisioner itself talks to from the *Platform* VM, a Lab container has
|
||||
no way to reach it either way, but the RFC1918 drop covers it as a second, independent layer),
|
||||
CDAA infrastructure, home/trusted networks, and other students' lab networks all fall inside
|
||||
one of the three RFC1918 ranges above and are dropped unconditionally by this same ruleset —
|
||||
one mechanism, not five separate ones to keep in sync.
|
||||
|
||||
**Provision — container** (limits from `resource-policy.yaml`, no image/entrypoint change):
|
||||
```bash
|
||||
docker create \
|
||||
--name rootnode-lab-<short-env-id> \
|
||||
--network rootnode-lab-net-<short-env-id> \
|
||||
--privileged \
|
||||
--cpus <cpu_limit> --memory <mem_limit_mb>m --memory-swap <mem_limit_mb>m \
|
||||
--pids-limit <pids_limit> \
|
||||
--storage-opt size=<storage_limit_mb>m \
|
||||
-e HTTP_PROXY=http://<egress-proxy-host>:3128 -e HTTPS_PROXY=http://<egress-proxy-host>:3128 \
|
||||
-e NO_PROXY=localhost,127.0.0.1,10.42.0.0/16,10.43.0.0/16 \
|
||||
<pinned-lab-image-ref>
|
||||
docker start rootnode-lab-<short-env-id>
|
||||
```
|
||||
**Implementation note: `<egress-proxy-host>` is resolved per environment, not a fixed
|
||||
hostname/IP.** The egress proxy runs with host networking (`docker-compose.dev.yml`), so it's
|
||||
present on every per-student bridge network's *gateway* interface — that gateway IP (from
|
||||
`docker network inspect`'s `IPAM.Config[0].Gateway`, right alongside the subnet already used
|
||||
for the firewall chain) is what's actually reachable from that specific environment's private
|
||||
network. A single static proxy hostname would only be reachable from whichever one network
|
||||
happened to be able to resolve/route to it — this wasn't apparent until testing with two
|
||||
simultaneous environments on two different subnets. Same resolution is used for both the
|
||||
firewall's `ACCEPT` rule destination (§ above) and this container's proxy env vars, so they
|
||||
always agree.
|
||||
|
||||
`NO_PROXY` carves out k3s's own internal pod/service CIDRs so in-cluster traffic never routes
|
||||
through the proxy — only the container's *outbound-to-the-internet* path does. containerd
|
||||
(started by the existing, unmodified `entrypoint.sh`) honors `HTTP_PROXY`/`HTTPS_PROXY` from
|
||||
its process environment automatically, which is why no Lab image or entrypoint change is
|
||||
needed for well-behaved image pulls to go through the proxy.
|
||||
|
||||
**Correction B, stated plainly: these env vars configure containerd for a clean pull path —
|
||||
they are not what makes egress restricted.** A process inside the container that ignores them
|
||||
and dials out directly is stopped by the `ROOTNODE-ENV-<id>` firewall chain above regardless,
|
||||
because that chain evaluates every packet leaving the container's network, independent of
|
||||
what any process inside chose to do. If the env vars were somehow wrong or absent, the only
|
||||
consequence is broken image pulls (a visible, debuggable failure) — never a silent hole in the
|
||||
egress boundary, because the boundary was never the env vars in the first place.
|
||||
|
||||
Readiness: poll the container's internal IP (`docker inspect` → `NetworkSettings`) for
|
||||
`GET /api/health` — the exact endpoint `entrypoint.sh` already waits on internally, so the
|
||||
Provisioner is just doing the same check from outside instead of inventing a new one.
|
||||
|
||||
**Destroy:**
|
||||
```bash
|
||||
docker stop rootnode-lab-<id> --time 10 && docker rm rootnode-lab-<id>
|
||||
|
||||
# Unhook and remove this environment's dedicated chain — three ops, no rule-scanning/matching
|
||||
# needed since it's one self-contained chain per environment, not tagged rules mixed into a
|
||||
# shared one:
|
||||
iptables -D DOCKER-USER -s "$NET_SUBNET" -j "$CHAIN"
|
||||
iptables -F "$CHAIN"
|
||||
iptables -X "$CHAIN"
|
||||
|
||||
docker network rm rootnode-lab-net-<id>
|
||||
```
|
||||
|
||||
**Reset** = destroy, then provision again against the same `environments` row (new
|
||||
`container_id`/`network_id`/`internal_ip`, unchanged `environment_id`/`routing_token`).
|
||||
|
||||
**Idle reaper** (background loop in the Provisioner, e.g. every 60s):
|
||||
- Query `environments WHERE status='running' AND COALESCE(last_active_at, created_at) < now() - idle_threshold`.
|
||||
- Past a shorter "idle" threshold: log a warning event (future: notify student/instructor —
|
||||
not built yet).
|
||||
- Past a longer "abandoned" threshold: call the same destroy path as `DELETE /environments/:id`,
|
||||
log `event_type='reaped_idle'`. Config-driven thresholds, not hard-coded (same policy file
|
||||
as resource limits).
|
||||
|
||||
**Implementation notes (found during testing, not in the original design):**
|
||||
- **`COALESCE(last_active_at, created_at)`, not bare `last_active_at`.** An environment that
|
||||
fails during provisioning (before ever reaching `running`) has a NULL `last_active_at` —
|
||||
`NULL < anything` is never true in SQL, so without the fallback such an environment would
|
||||
never age out via the reaper at all, regardless of how long ago it failed.
|
||||
- **Provisioning failure now does best-effort cleanup of whatever was already created**
|
||||
(container/network/firewall chain) before marking the row `error`, rather than leaving that
|
||||
infrastructure orphaned. Found by testing: a deliberate readiness-timeout failure left a real
|
||||
running container and firewall chain in place with only the DB row marked `error` — exactly
|
||||
the kind of leak the reaper exists to prevent, except this case predates the reaper ever
|
||||
having something to catch. Cleanup on the failure path directly is more honest than relying
|
||||
on the reaper for a gap it wasn't actually catching. See `src/lifecycle.js`'s
|
||||
`provisionEnvironment` catch block and `src/reaper.js`.
|
||||
- **Startup reconciliation (docs/LAB_ARCHITECTURE.md §8) is implemented** as
|
||||
`reconcileOnStartup()` in `src/lifecycle.js`, called once at Provisioner boot before the
|
||||
reaper starts. Tested by killing a container out-of-band and restarting the Provisioner —
|
||||
confirmed it correctly flags the mismatch as `error` rather than trusting the stale
|
||||
`running` status. Any environment caught mid-transition (`provisioning`/`resetting`/
|
||||
`destroying`) by a crash is also always marked `error` on restart, never assumed to have
|
||||
finished.
|
||||
|
||||
---
|
||||
|
||||
## 6. Security assumptions (stated explicitly, not implied)
|
||||
|
||||
- **Correction A — the Lab Host is not the Platform's trust boundary, the VM is.** In
|
||||
production, the host running privileged Lab containers is a separate VM from the one running
|
||||
Postgres/Provisioner/Platform API, with no route back from the Lab Host VM to the Platform VM
|
||||
at all — enforced at the hypervisor/network level, not by this phase's code. Phase 1's
|
||||
single-machine dev mode is a deliberate, documented exception (`docs/LAB_ARCHITECTURE.md` §1)
|
||||
— it does not redefine what "isolated" means for the real deployment.
|
||||
- The Docker host(s) sit on a segment with no route to trusted RootNode/CDAA/home/business
|
||||
networks — a prerequisite this phase depends on but does not create (matches the existing
|
||||
Business Pack Technical Platform Plan). Everything below assumes that's already true.
|
||||
- No other host firewall tooling (ufw, firewalld) is silently rewriting `DOCKER-USER` or
|
||||
removing the per-environment `ROOTNODE-ENV-<id>` chains — needs a one-time check on the real
|
||||
host before trusting the egress rules; not something this design can verify in advance.
|
||||
- `--privileged` is required for k3s and is not being avoided (confirmed in
|
||||
`docs/LAB_ARCHITECTURE.md` §7) — the security boundary is the VM split plus network
|
||||
containment, not reduced container privilege.
|
||||
- **Correction B — `HTTP_PROXY`/`HTTPS_PROXY` are not a security control**, stated here again
|
||||
because it's the assumption most likely to get silently reintroduced during implementation:
|
||||
the firewall chain enforces the boundary; the env vars only make well-behaved pulls work
|
||||
cleanly through it.
|
||||
- The egress proxy's allowlist is operator-controlled (a file the Provisioner's operator
|
||||
edits), never student- or Lab-container-writable.
|
||||
- The Provisioner API's shared secret is the only thing standing between "internal service"
|
||||
and "anyone who can create/destroy arbitrary privileged containers on the host" — treat that
|
||||
secret with the same care as the instructor passcode, not as a minor implementation detail.
|
||||
This matters more, not less, once the Provisioner is issuing commands to a remote Docker
|
||||
endpoint (correction D) — the TLS client cert for that connection carries the same weight.
|
||||
- `storage_limit_mb` enforcement via `--storage-opt size=` depends on the host's Docker storage
|
||||
driver supporting it (overlay2 with specific backing filesystems does; not all do) — unverified
|
||||
until tested against the real host (§8).
|
||||
- Nothing in this phase is internet-facing. The Provisioner API assumes internal-network-only
|
||||
reachability; it does not implement its own TLS. The Provisioner→Lab-Host Docker connection
|
||||
(correction D), by contrast, is TLS by design once it's not a local socket — that's a
|
||||
different boundary with different stakes and isn't optional the way the Provisioner API's
|
||||
own TLS currently is.
|
||||
|
||||
---
|
||||
|
||||
## 7. Files to be created
|
||||
|
||||
All new. Nothing existing is modified beyond the two docs already updated.
|
||||
|
||||
```
|
||||
platform/provisioner/
|
||||
├── package.json
|
||||
├── src/
|
||||
│ ├── server.js # Express app, routes from §4
|
||||
│ ├── db.js # Postgres connection (pg), migration runner
|
||||
│ ├── docker.js # dockerode client factory — resolves a lab_hosts row to a connected
|
||||
│ │ client (local socket or remote TLS), never a hard-coded default
|
||||
│ ├── lifecycle.js # state machine + orchestration (calls docker.js + db.js)
|
||||
│ ├── egress.js # per-environment ROOTNODE-ENV-<id> chain create/hook/unhook/remove
|
||||
│ └── reaper.js # idle-reaper background loop
|
||||
├── migrations/
|
||||
│ ├── 001_lab_hosts.sql
|
||||
│ ├── 002_environments.sql
|
||||
│ └── 003_environment_events.sql
|
||||
├── config/
|
||||
│ ├── resource-policy.yaml # cpu/mem/pids/storage defaults + idle/abandoned thresholds
|
||||
│ └── lab-hosts.seed.yaml # the one Phase 1 lab_hosts row (endpoint + TLS paths),
|
||||
│ # applied by the migration runner — not a runtime API,
|
||||
│ # per "don't overbuild scheduling yet"
|
||||
├── Dockerfile
|
||||
└── docker-compose.dev.yml # Postgres + egress-proxy + provisioner, for local Phase 1 testing
|
||||
|
||||
platform/egress-proxy/
|
||||
├── Dockerfile # minimal Squid image
|
||||
└── allowlist.conf # Squid ACL: registry-1.docker.io, ghcr.io, etc. — starting set
|
||||
# to be confirmed against what scenarios actually pull
|
||||
```
|
||||
|
||||
No changes to `backend/server.js`, `Dockerfile`, `scripts/entrypoint.sh`,
|
||||
`scripts/nginx.conf`, or anything under `scenarios/`.
|
||||
|
||||
---
|
||||
|
||||
## 8. Test plan
|
||||
|
||||
1. **Schema migration idempotency** — run `001_environments.sql` against a scratch Postgres
|
||||
twice, confirm the second run is a no-op (same pattern already used for the SQLite
|
||||
migrations: check-before-alter).
|
||||
2. **Provision → destroy round trip** — create an environment, confirm the container is
|
||||
`running` and reachable on its internal IP, confirm `environment_events` has one row per
|
||||
transition, destroy, confirm container/network are gone and the environment's
|
||||
`ROOTNODE-ENV-<id>` chain no longer exists (`iptables -L ROOTNODE-ENV-<id>` errors "chain
|
||||
does not exist") and its `DOCKER-USER` jump rule is gone (`iptables -S DOCKER-USER | grep
|
||||
<id>` returns nothing).
|
||||
3. **Reset** — provision, note `environment_id`/`routing_token`, reset, confirm both are
|
||||
unchanged, confirm `container_id` changed (genuinely new container), confirm no state
|
||||
carried over (fresh k3s).
|
||||
4. **Isolation** — provision two environments, confirm A cannot reach B at all
|
||||
(`docker exec <A> ping <B's internal_ip>` times out) — this is the core security property
|
||||
and must be an explicit test, not an assumption.
|
||||
5. **Egress allowlist, including the ordering fix specifically** — from inside a provisioned
|
||||
environment: an allowlisted registry pull succeeds via the proxy; a non-allowlisted host
|
||||
fails; a direct request to an RFC1918 address (simulating a trusted-network address) is
|
||||
dropped regardless of allowlist state. Critically: **the egress proxy's own IP is itself an
|
||||
RFC1918 address** — this test only actually exercises the fix from correction B if the proxy
|
||||
is reachable *and* the broader RFC1918 range is still blocked. If the proxy were unreachable
|
||||
here, that's the exact symptom the original ordering bug would have caused (the DROP rules
|
||||
shadowing the ACCEPT rules) — this test is what would have caught it before it shipped.
|
||||
6. **Idle reaper** — provision, force `last_active_at` backward past the abandoned threshold,
|
||||
run one reaper tick, confirm the environment is destroyed and logged as `reaped_idle`.
|
||||
7. **Suspend/resume (experimental, decision #3 — informational, not a pass/fail gate)** —
|
||||
`docker stop`/`docker start` a running Lab container, time how long it takes (if ever) to
|
||||
reach `/api/health` ready again versus a fresh `provision`. This is the actual answer to
|
||||
the question `docs/LAB_ARCHITECTURE.md` flagged as untested — record the result either way,
|
||||
but Phase 1 sign-off does not depend on this succeeding.
|
||||
8. **Reconciliation on Provisioner restart** — manually create a container outside the
|
||||
Provisioner's knowledge, restart the Provisioner, confirm it doesn't silently trust stale
|
||||
`environments` rows — either adopts or flags the mismatch (per
|
||||
`docs/LAB_ARCHITECTURE.md` §8).
|
||||
9. **Resource limits hold** — inside a provisioned environment, confirm `--cpus`/`--memory`/
|
||||
`--pids-limit` are actually enforced (e.g. a deliberate memory-hog process gets OOM-killed
|
||||
at the configured limit, not the host's full RAM); confirm whether `--storage-opt size=`
|
||||
is honored on the real host's storage driver — this is the resource-model unknown from
|
||||
`docs/LAB_ARCHITECTURE.md` §5, and this test is the first real benchmarking data point
|
||||
toward the 16-core/128GB target sizing.
|
||||
10. **Local dev-mode regression check** — confirm `docker run --privileged -p 7554:80 ...
|
||||
rootnode-academy:latest` (today's standalone command) is completely unaffected —
|
||||
`git diff` scoped to `backend/`, `Dockerfile`, `scripts/` should show zero changes from
|
||||
this phase.
|
||||
11. **Lab host abstraction actually abstracts (correction D)** — point the Provisioner's
|
||||
`lab_hosts` config at a Docker daemon exposed over TCP+TLS on the *same* dev machine
|
||||
(`dockerd -H tcp://127.0.0.1:2376 --tlsverify ...`) instead of the local Unix socket, with
|
||||
no code change — only the `lab_hosts` row differs. Provision/destroy round trip (test 2)
|
||||
must pass identically against that endpoint. This is the practical proof that "Docker is
|
||||
local" was actually removed as an assumption, not just renamed — it doesn't require a
|
||||
second real VM to verify, only a second way of reaching the same daemon.
|
||||
12. **VM-boundary reasoning check (correction A, documentation-level, not a running test)** —
|
||||
walk the failure scenario explicitly: if a student escaped their container onto the Lab
|
||||
Host VM's OS, what could they reach? Confirm the answer stays "nothing on the Platform VM"
|
||||
given the topology in `docs/LAB_ARCHITECTURE.md` §1 — i.e. confirm the design doesn't rely
|
||||
on anything *inside* the Lab Host VM (a firewall rule, a credential, an agent) as the only
|
||||
thing standing between a compromised container and Postgres. This is a design-review check
|
||||
now; it becomes a real network-level test once a second VM exists to test against.
|
||||
|
||||
Tests 1–6, 8, 10, 11 are runnable in a plain Docker environment with no special hardware.
|
||||
Tests 5 and 9 need the actual target host (or something close to it) to produce meaningful
|
||||
numbers, test 7 needs the real Lab image, and test 12 needs a second VM to be more than a
|
||||
design-review exercise — flagging that distinction so "passed the test plan" doesn't get read
|
||||
as "validated on production-class, two-VM hardware" when parts of it were necessarily
|
||||
validated on a single dev machine.
|
||||
Reference in New Issue
Block a user