Files
rootnode-academy/docs/LAB_ARCHITECTURE.md
T
ced4568 bf1e1b5c76 docs(phase4): production infrastructure documentation + runbooks
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs
(PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR,
OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST,
and 12 runbooks (first deployment, normal deploy, rollback, Postgres
restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner
recovery, stuck Lab cleanup, failed migration, backup failure, disk
failure, Lab Host resource pressure).

PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine
architectural gap this phase surfaced: the Gateway's existing
env.internal_ip mechanism for reaching a Lab container assumes Gateway and
Lab share a Docker daemon, which breaks once Labs run on a genuinely
separate rn-lab-host-01 VM. The network-policy half is built (a firewall
allowance for Gateway->Lab-Host on a published port range); the
application half (Provisioner/Gateway address handling) is an explicit,
reviewed, NOT-YET-BUILT follow-up flagged there rather than silently
decided.

Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and
platform/README.md with Phase 4 pointers, consistent with how every prior
phase updated these.

Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
2026-08-18 17:13:51 -05:00

32 KiB

RootNode Academy — Ephemeral Lab Architecture

Status: Phase 1 (platform foundation) + Phase 2A (Gateway + student flow) + Phase 2B (Platform API split, Postgres academic data, server sessions, admin portal) + Phase 3A (persistent sessions, per-instructor accounts, RBAC, audit log) + Phase 3B (learning/assessment engine: attempt telemetry, progressive hints, curriculum rules, exam + capstone engine, achievements) IMPLEMENTED, 2026-08-18 — see docs/PHASE1_PLATFORM_FOUNDATION.md, docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md, docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md, docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md, and docs/PHASE3B_LEARNING_AND_ASSESSMENT.md for concrete schema/API/test results, and platform/README.md for how to run the stack. This document remains the architecture record for the full ephemeral-lab direction; production OIDC/SSO (an adapter interface exists, not a working integration) and production Proxmox deployment are still future work (Phase 4 — see the Phase 3B doc §21). Supersedes the namespace-isolation (shared_namespace/ephemeral_cluster) approach that was in flight before this redirect — see scenarios/SCHEMA.md's isolation_type note for the pointer back here.

Committed direction: one fully isolated, disposable environment per student. No shared Kubernetes control plane, ever. This is the long-term architecture, not a stopgap.

Decisions locked 2026-08-18 (resolves the four open questions this doc originally ended with — see §2–§5 below for where each lands, and docs/PHASE1_PLATFORM_FOUNDATION.md for the Phase 1 build plan against these decisions):

  1. Egress: restricted, explicit allowlist. No unrestricted internet egress, ever. No route to trusted RootNode management/CDAA/home/business networks, under any allowlist entry.
  2. Platform DB: PostgreSQL. The Platform control plane is not built around SQLite. The existing single-container local dev mode keeps using its own local SQLite as today — unaffected, unrelated database.
  3. Suspend/resume: experimental only, not load-bearing. The guaranteed lifecycle is provision → run → destroy → recreate. Persistent academic state lives outside the Lab specifically so destroying one never loses student progress — suspend/resume is a possible future optimization, not something correctness depends on.
  4. Resource sizing: not hard-coded. Quotas are policy/config-driven. Target host class is ~16 cores/32 threads, 128GB RAM, SSD/NVMe — actual per-lab limits and concurrency get set by benchmarking against that class, not guessed up front.

Corrections locked 2026-08-18 (round 2 — architectural fixes required before Phase 1 implementation begins; folded into §1, §4, §7, §9 below, not just listed here):

  • A. VM boundary. Privileged Lab containers must never share an OS/VM boundary with the Platform (Postgres, control-plane) in production — separate VMs under Proxmox, connected by a Provisioner-initiated control channel only. The Lab Host is a configured target, not an assumption that Docker is local. Single-machine is still fine for Phase 1 development — see §1/§9.
  • B. Egress is a firewall control, not an app convention. HTTP_PROXY/HTTPS_PROXY configure well-behaved apps for convenience; they enforce nothing. The firewall is the control, and rule ordering matters — required infrastructure (proxy, DNS) must be explicitly permitted before the blanket RFC1918 drops, or those drops shadow the allow rules and nothing works. See §4 — this also corrects a real ordering bug in the original design (explained there).
  • C. student_id is opaque in Phase 1. No real foreign key to a students table — students/progress stay on SQLite until a later, separate Platform DB migration. Documented, not silently assumed. See §3/§9.
  • D. Lab host is an abstraction from day one, even with exactly one configured host in Phase 1 — the Provisioner is never allowed to hard-code "Docker is local." See §4.

0. What already exists (grounding, not invention)

The current repo (~/Dev/Gitea/ced4568/rootnode-academy) already documents almost this exact intent in its own README, just never built out:

"The upstream project assumes one learner, one container. RootNode Academy runs it across three delivery modes... One container per student — never share a cluster between learners... Put authentication in front (proxy-level)... Idle-timeout + auto-teardown."

And the existing image already is a complete single-student lab: nginx + React frontend + Node/Express backend + node-pty WebSocket terminal + a single-node k3s cluster, all in one --privileged container (confirmed required — k3s needs kernel namespaces/cgroups, per README.md:52). That container is the unit this design multiplies, not something it replaces.

What changes: today that one container also owns identity/progress (SQLite at /data/progress.db, one volume per student, mounted by hand). This design pulls that ownership out into a persistent Platform, and makes the container itself fully disposable.


1. Proposed architecture

Two roles — and in production, on two separate VM/OS boundaries, not just two logical components sharing a box. Correction A drives this: --privileged Docker containers are not a security boundary from their own Docker host, so the boundary has to be a real VM boundary, not a process boundary.

Production topology:

Proxmox VE (or equivalent hypervisor)
│
├── RootNode Platform VM   (trusted — never runs student code)
│   │
│   │  Platform API   roster · login · progress · curriculum · exam records ·
│   │                 environment registry · instructor controls
│   │  Platform DB    PostgreSQL — environments + audit log now;
│   │                 students/progress join later (§3/§9, correction C)
│   │  Provisioner    talks OUT to a *configured* Lab Host endpoint —
│   │                 never assumes Docker is local (§4, correction D)
│   │  Gateway        opaque-token routing to student lab containers;
│   │                 only component with a leg in both worlds
│   │
│   ▼ (Provisioner-initiated control channel only — TCP+TLS to the Lab
│      Host's Docker API. Nothing on the Lab Host VM ever initiates a
│      connection back. See §4/§7 for why this direction matters.)
│
└── RootNode Lab Host VM   (untrusted — this is where privileged student
    │                       containers actually run; holds nothing the
    │                       Platform needs protected)
    │
    ├── Student A's LAB ── own private network, own nested k3s, no identity
    ├── Student B's LAB ── own private network, own nested k3s, no identity
    └── Student C's LAB ── own private network, own nested k3s, no identity

Each Lab is today's Dockerfile and entrypoint.sh, functionally unchanged — still runs k3s + the terminal + scenario setup/validate/teardown exactly as now. The only change inside it: it stops being the system of record for who's using it or what they've completed (§3, §9).

A compromised student container's worst case is the Lab Host VM — never Postgres, never the Platform API, because there is no network path from that VM back to the Platform VM at all (§4, §7). That containment is the actual point of the two-VM split, not a deployment nicety.

Phase 1 development exception: running the Platform and a Lab container on one machine is fine for local testing (§9) — the Provisioner just gets configured with a single lab_host pointing at the local Docker socket instead of a remote TLS endpoint. The code never special-cases "local" vs "remote" — same abstraction either way (§4/§9 of docs/PHASE1_PLATFORM_FOUNDATION.md).


2. Environment lifecycle

State machine per environment, tracked in the Platform DB (environments table, §3):

Guaranteed path (what correctness depends on): requested → provisioning → running → resetting → running (reset loop, same route) and (provisioning|running|error) → destroying → destroyed. provisioning → error and running → error cover failed/lost environments; error → destroying → destroyed (or → resetting → running, when recoverable) is the way out.

requested → provisioning → running ─┬─→ resetting → running (clean slate, same route)
                                     └─→ destroying → destroyed
provisioning → error (failed to reach ready state)
running → error (health check fails mid-session)
error → destroying → destroyed  (or → resetting → running, if recoverable)

[EXPERIMENTAL, not relied on for correctness — see decision #3]
running ──suspend──→ suspended ──resume──→ running (or → error, if resume fails)
  • provision — Provisioner picks a host with capacity (§5), creates a private network for this student (§4), starts a container from the pinned Lab image, generates a random high-entropy routing token, waits for the existing readiness signal (k3s API ready + backend /api/health OK — same check the current entrypoint already does), writes the environments row, hands the route back to the Platform API.
  • start — used only for provisioning → running (starting a freshly created container). There is no supported path that starts a previously-stopped container in the guaranteed lifecycle — see suspend/resume below.
  • suspend/resume — EXPERIMENTAL, decision #3. docker stop/docker start on a k3s-in-container Lab is not assumed to work and is not built as part of the guaranteed lifecycle. It's worth testing (see the Phase 1 test plan) because if it turns out reliable it's a cheap latency win later — but nothing in the platform depends on it succeeding. The only supported way to bring an environment back is destroy + provision (or reset, which is that same pair against one environments row/routing token). This is also why academic state living outside the Lab (§3) isn't optional — it's what makes "just destroy and recreate" a fully safe default instead of a data-loss risk.
  • reset — destroy and recreate the container from the pinned image, same routing token and same environments row, so the student's "return to my lab" link doesn't change. This is the instructor-facing "reset student environment" control.
  • destroy — stop + remove container, remove its private network, release the routing token, mark the environments row destroyed with a timestamp. Progress already lives in the Platform DB (§3), so destroying an environment never loses academic data — only in-terminal, not-yet-validated work.
  • Idle reaper (new, required — not optional) — background job in the Provisioner checks last_active_at per running environment (heartbeat from the Gateway or from Lab /api/health polling) and auto-suspends past an idle threshold, auto-destroys past a longer abandoned threshold. Without this, "ephemeral by default" is just a description, not a guarantee — a closed browser tab does not stop a running k3s cluster.

3. Data model changes

Everything below lives in the Platform DB, not inside any Lab container.

Already built (previous slice, unaffected by this redirect — still correct under this architecture):

  • students(student_id, display_name, cohort, created_at, last_active_at)
  • progress(student_id, scenario_id, status, attempts, ...) — composite key, already per-student.
  • sessions / exam_progress — exam records, already student-attributed.

New:

  • environments(environment_id UUID PK, student_id, status, routing_token, container_id, lab_host_id FK, internal_ip, cpu_limit, mem_limit_mb, storage_limit_mb, image_ref, created_at, started_at, last_active_at, destroy_at, error_reason) — student_id is an opaque external identifier here (correction C), not a foreign key. Students/progress stay on the existing SQLite in Phase 1 (see §9); Postgres has no students table yet, so there is nothing to key against. The Provisioner trusts whatever student_id its caller passes — it does not validate it against a roster. A real FK gets added when students/progress actually migrate onto this Postgres instance, not before. lab_host_id, by contrast, is a real FK — both tables live in the same Provisioner-owned Postgres DB (correction D, full schema in docs/PHASE1_PLATFORM_FOUNDATION.md).
  • lab_hosts(lab_host_id PK, endpoint, enabled, ...) — the abstraction correction D requires: which Docker endpoint a given environment's container actually lives on. Phase 1 ships with exactly one row (pointing at either a local socket in dev or the Lab Host VM's TLS endpoint in production) — the table exists so environment creation is never coupled to "Docker is local" as an assumption, even though there's no scheduling logic yet.
  • environment_events(id, environment_id, event_type, actor, detail, created_at) — audit trail for provision/start/stop/reset/destroy/error, needed for both incident response and ordinary "why did my lab disappear" support.

Removed from the Lab image's responsibility: the students and roster-facing parts of the schema I added to backend/server.js in the previous slice move to be Platform-only. The Lab image's own SQLite either goes away entirely or is kept only as scratch/ephemeral state that's allowed to die with the container — it must not be treated as authoritative for anything.

Decision #2: PostgreSQL, not SQLite. The Platform control plane — the environment registry specifically, and the target for progress/roster once they migrate over — is Postgres. The existing single-container local dev mode keeps its own local SQLite exactly as today; that's a different, unrelated database serving a different purpose (single-tenant local testing), not something this decision touches. See docs/PHASE1_PLATFORM_FOUNDATION.md for the exact schema being stood up first.


4. Networking model

  • One private Docker network per student environment, not one shared network for all labs. Docker's default per-network isolation means Student A's container has no L2/L3 path to Student B's container at all — this is stronger and simpler than trying to firewall students apart on a shared network.
  • No published ports on Lab containers. Today's image publishes -p 7554:80; under this design that stops — the only thing that can reach a Lab container is the Gateway, over that student's private network.

Egress — correction B: the firewall is the control, not the proxy env vars

HTTP_PROXY/HTTPS_PROXY are set on each Lab container so well-behaved processes (containerd pulling scenario images) route through the allowlisting proxy cleanly. That is configuration, not enforcement — a process that ignores those env vars and dials out directly must still be blocked, by something that doesn't depend on the container's cooperation. That something is a host firewall rule, evaluated on every packet regardless of what any process inside the container does.

Ordering bug in the original draft, corrected here: the previous version of this section listed proxy/DNS ACCEPT rules before the RFC1918 DROP rules in prose, but specified installing them with iptables -I (insert-at-head) in that same top-to-bottom order — which actually reverses the result, since each -I without a rule number pushes the new rule to position 1. Run in that order, the RFC1918 drops end up evaluated before the proxy/DNS accepts, silently blocking the proxy itself if it happens to sit on an RFC1918 address (it will — it's Platform-side infrastructure). Corrected mechanism (exact rules in docs/PHASE1_PLATFORM_FOUNDATION.md §5): each student network gets its own dedicated iptables chain, built with -A (append, so rules land in the order they're added — no head-insertion trap), containing explicit ACCEPT rules for the egress proxy, DNS, and any specifically required Platform endpoint first, then the RFC1918 DROPs, then a final default-DROP. That chain is reached from DOCKER-USER (the chain Docker guarantees it won't silently rewrite) via one jump rule matching that student's subnet.

What no student lab may ever reach, regardless of allowlist state:

  • RootNode management networks
  • the Platform database (Postgres)
  • Docker/Proxmox management interfaces (including the Lab Host VM's own Docker API endpoint — a Lab container talking back to the thing that provisioned it is exactly the escalation path this design exists to prevent)
  • CDAA infrastructure
  • home/trusted networks
  • other students' lab networks (already unreachable via Docker's own per-network isolation above; the RFC1918 drop is a deliberate second layer, not redundant — it also catches any case where two student networks' address ranges could otherwise see each other at the host routing level)

Decision #1, restated with the ordering fix: restricted egress, explicit allowlist, no unrestricted internet access ever, enforced at the firewall layer with required infrastructure explicitly permitted before the blanket denies — not the reverse, and not left implicit.

VM-to-VM boundary — correction A

The Lab Host VM has no route back to the Platform VM at all, in either direction, except the one the Provisioner itself opens. The Provisioner (on the Platform VM) connects out to the Lab Host's Docker API over TCP+TLS with a client certificate only the Provisioner holds; nothing on the Lab Host VM — student containers or the VM's own OS — can open a connection back toward the Platform VM. This is enforced at the hypervisor/network level (Proxmox VLAN/firewall rules between the two VMs), the same category of control as the CDAA/home boundary below, not something Docker networking alone provides. A compromised student container's blast radius stops at the Lab Host VM.

No route to trusted CDAA/home/business networks, full stop. This is a host/hypervisor boundary, not something Docker networking alone can guarantee — the box(es) these containers run on need to sit on their own segment with no bridge/route to those networks, consistent with the existing RootNode_Academy_Business_Pack/06_Technology/01_Technical_Platform_Plan.md ("dedicated student network/DMZ separated from trusted networks"). This design assumes that host-level segmentation is a prerequisite, not something the app layer can create on its own.

Lab host abstraction — correction D

The Provisioner never talks to "the local Docker socket" as a hard-coded assumption — it talks to whichever lab_hosts row it's configured to use (§3), resolved to a Docker endpoint that may be unix:///var/run/docker.sock (Phase 1 dev, single machine) or a remote tcp://<lab-host-vm>:2376 with TLS (production, per the topology in §1). Phase 1 ships exactly one enabled lab_host and zero scheduling logic — the abstraction exists so adding a second host later is a config row, not a rewrite.

  • Gateway is the only component with a leg in both the public-facing network and every student's private network. It resolves routing_token → container internal IP:port by querying the Platform API (or a shared registry cache) per request/connection, then proxies both HTTPS (scenario API, static frontend) and the WSS terminal connection.

5. Resource model

  • Per-environment caps applied at container-create time: --cpus, --memory (with --memory-swap pinned equal to --memory so nothing swaps), --pids-limit (fork-bomb guard). Storage: needs a size-capped volume — exact mechanism (--storage-opt size=, a loopback-backed bind mount, or a tmpfs cap) depends on the host's actual Docker storage driver, which I haven't inspected — needs to be confirmed against the real deploy target before picking one.
  • Decision #4: policy-driven, not hard-coded. Limits come from config, not constants in code — a resource-policy file the Provisioner reads at startup (and can reload), not numbers baked into the create-container call. No specific numbers are chosen yet. Target host class for planning purposes: ~16 cores/32 threads, 128GB RAM, SSD/NVMe. Actual per-lab limits and how many environments run concurrently on that class get set by benchmarking (Phase 1's test plan includes the first real measurements), not by guessing backward from the spec sheet.
  • Admission control: Provisioner refuses to provision past the point where already-running environments' summed limits would exceed host capacity. Simple counter/gate to start; can become real bin-packing later if there's ever more than one host.

6. Routing/auth model

  1. Student logs in at the Platform (existing roster/pick-your-name flow, unchanged).
  2. Frontend asks the Platform for "my lab." If none running, Platform calls the Provisioner to provision (or start, if one's suspended).
  3. Platform returns an opaque, high-entropy routing token scoped to (student_id, environment_id), short TTL, revoked on destroy.
  4. Frontend connects to the Gateway using that token — https://labs.<domain>/e/<token>/... and wss://labs.<domain>/e/<token>/shell-ws. The Lab container itself never sees a student identity; it's purely an execution surface addressed by an opaque route.
  5. Gateway validates the token maps to a currently running environment before proxying — rejects stale tokens from destroyed/reset environments outright, so there's no window where an old link reaches a different (possibly now-different-student's) container.
  6. Instructor auth extends the shared-passcode model already used for the roster-add endpoint: instructor-only calls (list environments, reset, terminate, resource usage) go through the Platform API under that same gate, which then calls the Provisioner internally. Students and the public internet never talk to the Provisioner or the Docker socket directly — only the Platform backend does.

7. Security boundaries (defense in depth)

  1. Browser ↔ Gateway — public-facing, TLS, opaque routing tokens only. Nothing cluster-admin-shaped is reachable from here.
  2. Gateway ↔ Lab container — private per-student Docker network only, no published ports, Gateway is the sole entry point.
  3. Inside the Lab container — --privileged is required for k3s (confirmed, not avoidable — this isn't a place to try to "hold the line" without breaking k3s). The mitigation is containment, not avoidance: this privileged container has no reachable network path to anything except the Gateway, runs as a throwaway that gets destroyed/reset routinely, and cluster-admin inside a student's own disposable cluster is expected and fine — that's the entire point of full isolation, a student doing CKS-tier cluster-admin work only ever affects their own throwaway cluster.
  4. Lab container ↔ its own Docker host (correction A — the boundary that actually matters here) — --privileged is explicitly not treated as a sufficient isolation boundary from its own Docker host. In production the Lab Host VM holding these containers is a separate VM from the Platform VM, with no route back to it at all (§4). If a student ever escapes their container, the ceiling is the Lab Host VM — never Postgres, never the Platform API, never another student's environment.
  5. Provisioner ↔ Lab host's Docker API — only the Provisioner can invoke it, over a connection it initiates (TCP+TLS, client-cert auth, per §4's lab-host abstraction). The Docker endpoint is never student-reachable, never internet-reachable, and nothing on the Lab Host VM can initiate a connection back toward the Provisioner or the Platform VM.
  6. Platform VM ↔ other networks — dedicated segment, no bridge to home/business/CDAA trusted infrastructure, matching the existing Technical Platform Plan doc. Prerequisite, not something this design builds. Same standard applies to the Lab Host VM independently — it isn't exempted just because it's "the untrusted one."
  7. Routing tokens — CSPRNG-generated, short TTL, invalidated on any lifecycle transition out of running.

8. Failure/recovery behavior

  • Provisioning fails (host at capacity, image pull fails, k3s doesn't reach ready state in time) — environment marked error, student gets an explicit "your lab failed to start, try again" message (never a silent hang), one automatic retry, then surfaced to the instructor dashboard (future) as a flagged failure.
  • Mid-session crash — Gateway's health check detects the upstream is unreachable, marks the environment error, offers the student a "restart my lab" action (→ reset, since a crashed in-container k3s is unlikely to resume cleanly). Because progress/identity/exam records live in the Platform DB, this never loses the student's academic record — worst case is losing in-progress, not-yet-validated terminal work.
  • Host failure / reboot — every environment on that host becomes unreachable. On Provisioner restart, it must reconcile its registry against actual running containers rather than trust stale DB state — some containers may auto-restart via Docker's restart policy, most won't; anything the registry claims is running that isn't actually running gets marked error, not silently left wrong.
  • Idle/abandoned environments — the reaper (§2) bounds resource leakage from crashed tabs or students who just walked away, independent of any other failure path.
  • Core resilience property of this whole design: because the Platform DB — not the disposable environment — is the system of record, every failure mode above is recoverable without academic data loss. That's the actual payoff of the split in §1/§3, not just an architectural nicety.

9. Migration path from the current single shared container

  • Phase 0 (already true today, keep it): running the existing image standalone — docker run --privileged -p 7554:80 ... rootnode-academy:latest — is exactly the single-user local dev/test mode you asked to keep available. No change needed to support this; it already works and stays supported as-is.
  • Phase 1: Build Provisioner + environments/environment_events/lab_hosts registry around the existing, unmodified Lab image, on Postgres, with the egress firewall/proxy from §4 in place from the start (not bolted on later). Development exception (correction A): running Platform + Provisioner + one Lab container all on a single machine is fine here — the Provisioner is configured with one lab_host pointing at the local Docker socket instead of a remote TLS endpoint, same code path either way. Prove two lab environments can run side by side, fully isolated from each other, before any production traffic or a second VM enters the picture. students/progress stay on the existing SQLite through this phase (correction C) — environments.student_id is opaque, not a foreign key, until a later, separate migration actually moves roster/progress onto Postgres.
  • Phase 2 (the one real code change inside the Lab image): narrow backend/server.js inside the Lab image down to a "dumb executor" — scenario setup/validate/teardown/answer and the WS terminal stay, but it stops owning students/roster and stops being the source of truth for progress. Either it calls back to the Platform API to report validation results, or — cleaner — the Platform API becomes the only thing the frontend talks to for progress/curriculum/identity, and it calls the Lab container (via the route the Provisioner already tracks) purely to execute against that student's cluster and get a pass/fail back.
  • Phase 3: Wire instructor controls (list/reset/terminate/resource-usage) against the new registry — this is also where the "no auth on roster-add" gap from the previous slice gets closed for real, since instructor auth is now a load-bearing security boundary, not just roster hygiene.
  • Phase 4 (cutover): point the real domain at the Gateway instead of a single container. Keep the Phase 0 single-container path available under local/dev tooling, per your explicit requirement.
  • Rollback safety: at every phase through Phase 3, the current single-shared-container path keeps working completely unmodified — nothing about this migration risks the working baseline until the Phase 4 cutover, which is also the only step that's meaningfully reversible-with-effort rather than trivially reversible.

10. Exact files/components this touches (once approved — no code written yet)

New, none of it touching the existing Lab image's core logic:

  • platform/provisioner/ — Docker lifecycle management against a configured lab_host endpoint (correction D — never a hard-coded local socket), environments/lab_hosts CRUD, per-environment firewall chain management (correction B), idle reaper, admission control.
  • platform/egress-proxy/ — the shared allowlisting proxy every Lab container's egress must pass through (correction B).
  • platform/gateway/ — token→route resolution, HTTPS + WSS proxying (candidate: Traefik with a dynamic provider backed by the Platform API, or a small custom Node proxy — not decided, not needed for this design review).
  • Platform DB migrations: environments, environment_events, lab_hosts tables — full schema in docs/PHASE1_PLATFORM_FOUNDATION.md.

Modified, once implementation starts:

  • backend/server.js — split as described in §9 Phase 2. This is presently both "the lab image's backend" and "the only backend that exists"; under this design it forks into a narrower Lab-side executor and a separate, persistent Platform-side API. Exact file layout for that split is an implementation decision, not part of this design doc.
  • README.md — the "Multi-Student Deployment" section (README.md:128-146) currently documents per-student volumes for persistence; that persistence model is superseded by §3 above and the README needs updating once this ships, so it doesn't contradict reality.
  • docs/ENGINEERING_SOP.md — new section for the Platform/Provisioner/Gateway components and the reconciliation/reaper operational procedures from §8.

Unchanged:

  • Dockerfile, scripts/entrypoint.sh, scripts/nginx.conf for the Lab image itself — it's still one container running k3s + backend + terminal, exactly as today, just multiplied and stripped of identity ownership.
  • All scenario content (scenarios/data/, scenarios/bundles/, scenarios/rootnode-path.json) — unaffected either way; every scenario now runs with full admin of its own disposable cluster, so the earlier shared_namespace/ephemeral_cluster split has no bearing here.

New Dockerfile needed (not yet written): one for the Provisioner+Gateway services — these are new components, not variants of the existing Lab image.


Status

All four open questions this doc originally ended with are resolved — see the decisions block at the top and §2–§5 above. Next: docs/PHASE1_PLATFORM_FOUNDATION.md — the concrete build plan for the platform foundation (Postgres environment registry, provisioner skeleton, networking, egress, resource policy, reaper) against these decisions.

Phase 4: this boundary in production

Phase 4 (docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md) codifies the Platform/Lab-Host split this document designed as two real, separately-firewalled Proxmox VMs (rn-platform-01, rn-lab-host-01) connected only by mutual-TLS Docker and a tightly scoped firewall allowance — see docs/PRODUCTION_NETWORK_SECURITY.md. That document also records a gap Phase 4 surfaced but did not resolve: the Gateway's env.internal_ip mechanism for reaching a Lab container assumes Gateway and Lab share a Docker daemon, which breaks once they're genuinely on separate VMs. Read that document's "Known gap" section before treating two-VM production as fully wired.