Root cause, confirmed empirically (not guessed):
1. node-gyp generates an incorrect build/config.gypi for this NodeSource-packaged
Node 20 build — it reports node_shared: true (expects a shared libnode.so),
but the actual installed Node is statically linked (no libnode.so.<abi>
exists anywhere on the system; process.config.variables.node_shared on the
real running Node is false). node-gyp trusted the wrong config and linked
node-pty's native addon against -lnode, producing a binary that fails at
require() time with "libnode.so.127: cannot open shared object file" —
confirmed by loading the compiled .node file directly, bypassing node-pty's
own path-search/error-masking logic (which collapses this into a generic
"Cannot find module .../pty.node" across every path it checks).
2. A second, independent bug compounded it: no .dockerignore existed. A
host-side `npm install` (run earlier for backend testing) left
backend/node_modules on disk, built against the host's own environment. The
Dockerfile's `COPY backend/ ./backend/` (placed after the npm-install RUN
layer, for build-cache efficiency) silently overwrote the correctly-built
node_modules with that host copy — including after a first .dockerignore
fix using the pattern `node_modules/`, which only matches a node_modules
directory at the build context ROOT in Docker's matching rules, not at any
depth (unlike .gitignore, where the same-looking pattern does match any
depth — a real, confirmed gotcha, not an assumption).
Fix:
- Dockerfile: `ENV npm_config_node_shared=false` (image-level, not a one-off
RUN prefix, so it can't be silently lost if a native module ever gets
rebuilt again for any reason) + a build-time `node -e "require(...)"` check
that fails the build loudly if this regresses, instead of shipping a silent
runtime failure.
- .dockerignore: `**/node_modules/` (recursive), not `node_modules/`.
No terminal architecture change, no fake shell, no changes to
backend/server.js's PTY handling. RootNode branding and upstream KubeKosh
attribution untouched. The single-container dev workflow
(`docker build -t rootnode-academy:test .`) is the same command as before —
it now just works.
Verified against the real image end-to-end, not a stand-in: build -> container
starts -> k3s reaches Ready -> backend starts, no crash -> nginx starts ->
browser UI 200s -> terminal WebSocket opens showing the RootNode banner ->
`kubectl get nodes` run through the terminal returns real cluster state ->
deploy-nginx scenario solved via the terminal -> all 5 validation checks pass.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
- docs/LAB_ARCHITECTURE.md — the full ephemeral-per-student design: two-VM
production topology (Platform VM / Lab Host VM, correction A), lifecycle,
data model, networking/egress model (correction B), resource model,
routing/auth model, security boundaries, failure/recovery behavior,
migration path from the current single container, and the lab-host
abstraction (correction D). Status updated to reflect Phase 1 is now
implemented, pointing at the two docs below for specifics.
- docs/PHASE1_PLATFORM_FOUNDATION.md — the Phase 1 build plan: exact Postgres
schema, service boundaries, API routes, lifecycle state machine, Docker/
firewall commands, security assumptions, file layout, and a 12-test plan.
Updated with "Implementation notes" at the points where testing surfaced
real deviations from the original design: the per-network (not static)
egress-proxy address resolution, the COALESCE reaper fix, and the
provisioning-failure cleanup fix.
- docs/ENGINEERING_SOP.md — new §8 mapping the platform/provisioner/ and
platform/egress-proxy/ components, what's built vs. not (no Gateway, no
frontend wiring — internal API only), and where identity still lives
(unchanged: Lab image's own SQLite). Roadmap item 5 (per-student isolation)
updated from "not started" to "Phase 1 done, not yet cleared for minors."
- README.md — Multi-Student Deployment section rewritten: the old per-student-
volume persistence model is superseded by the Platform's registry (progress
already lives outside the disposable container as of the prior curriculum/
roster work); points at docs/LAB_ARCHITECTURE.md and platform/provisioner/
for the real architecture instead of describing three delivery modes that
were never actually built out.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
platform/provisioner/test/run.js — 12 assert-based integration tests against a
live docker-compose.dev.yml stack (no framework, matching the project's
existing convention). Covers: health/auth, provision with unique
network/container/token, duplicate-provision rejection, distinct
subnets/IPs for concurrent students, the egress firewall ordering fix
specifically (proxy ACCEPT before the RFC1918 DROP for the same range),
student-to-student isolation, RFC1918 blocking, reset (same identity, fresh
container), destroy (deterministic cleanup), and the audit trail (all
expected events present, routing token never logged).
All 12 pass against the default lightweight test image. Several properties
were validated manually during implementation rather than automated here —
idle reaper timing (needs a real threshold wait), remote TCP+TLS lab-host
(needs a Docker-in-Docker sidecar, too heavy for routine runs),
reconciliation-on-crash (needs an out-of-band container kill + process
restart), and real-Lab-image k3s behavior (environment-specific, see the
Phase 1 final report and platform/provisioner/README.md's known limitations)
— each is noted at the bottom of the file with why it's not automated here.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Background loop (src/reaper.js), config-driven idle/abandoned thresholds and
poll interval (no hard-coded numbers) — without this, "ephemeral by default"
is just a description: a closed browser tab does not stop a running k3s
cluster on its own.
Past the idle threshold: logs a warning event only. Past the abandoned
threshold: destroys via the same path DELETE /environments/:id uses, so
cleanup is identical either way, and logs event_type='reaped_idle'.
Failure-safe — one environment's teardown throwing never stops the reaper
from checking the rest, and a Postgres/Docker hiccup on one tick just means
"try again next tick," never a crashed process.
Uses COALESCE(last_active_at, created_at) rather than bare last_active_at.
Found during testing: an environment that fails during provisioning (before
ever reaching 'running') has a NULL last_active_at, and NULL < anything is
never true in SQL — without the fallback, a failed-and-never-cleaned-up
environment would never have aged out here regardless of how long it sat
there. This is the second, independent safety net on top of
provisionEnvironment's own best-effort cleanup on failure (see the
platform-provisioner commit).
Verified end-to-end: backdated an environment's last_active_at past the
abandoned threshold, waited for a real reaper tick, confirmed full
deterministic cleanup (container, network, firewall chain all gone) with the
correct actor='reaper' attribution in environment_events.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Egress enforcement is a firewall-layer control, not an application-layer
convention (correction B). HTTP_PROXY/HTTPS_PROXY configure containerd for a
clean pull path; they enforce nothing on their own.
- src/egress.js — per-environment dedicated iptables chain
(ROOTNODE-ENV-<short-id>), built with -A (append, order-preserving) rather
than the -I (insert-at-head) pattern an earlier design draft used, which
would have installed rules in reverse order and let the default-deny shadow
the proxy/DNS accepts. Required infrastructure (egress proxy, DNS) is
explicitly permitted first, then RFC1918 ranges are dropped unconditionally,
then a final default-deny catches the public internet. One jump rule per
environment hooks its subnet into its chain from DOCKER-USER (the chain
Docker guarantees it won't silently rewrite). Destroy is deterministic:
unhook, flush, delete — no comment-grep matching.
The ordering fix is empirically verified, not just reasoned about: the
egress proxy's own address is itself an RFC1918 address (it's Platform-side
infrastructure), and testing confirms it's reachable while the broader
RFC1918 range is still blocked — exactly the scenario the original ordering
bug would have broken.
- platform/egress-proxy/ — shared Squid instance (one per deployment, not
per-student), destination-allowlisted (dstdomain ACL against a plain-text
allowlist file), starting set covers docker.io/ghcr.io/gcr.io/quay.io/k8s.io.
Proxy performs domain allowlisting; the firewall chain above is what makes
it unbypassable, not the other way around.
The egress proxy's address is resolved per-student-network (each network's
Docker gateway IP, since the proxy runs with host networking and is present
on every bridge gateway) rather than a fixed hostname — a static address
wouldn't be reachable from more than one isolated student network at a time.
Found while testing with two simultaneous environments.
Verified: student-to-student isolation (A cannot reach B's internal IP),
RFC1918 blocking (A cannot reach a simulated trusted-infra address), and the
proxy allowlist itself (CONNECT tunnel to an allowlisted host succeeds,
non-allowlisted gets a clean 403 from Squid).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Internal-only service (platform/provisioner/) implementing the environment
lifecycle from docs/PHASE1_PLATFORM_FOUNDATION.md: provision -> running ->
(reset -> running | destroy -> destroyed), plus experimental suspend/resume
(decision #3 — never on the guaranteed path, never relied on for correctness).
- src/docker.js — the lab-host abstraction (correction D). Resolves a
lab_hosts row into a connected dockerode client, unix:// socket or tcp://
with TLS client-cert auth. Refuses to connect to a tcp:// endpoint with no
TLS material configured rather than connecting unauthenticated. Validated
against both a local socket and a real remote Docker-in-Docker TLS endpoint
during testing — same code path either way, only the lab_hosts config
differs, which is the entire point.
- src/lifecycle.js — provision/destroy/reset/suspend/resume orchestration,
wired to docker.js + egress.js + the Postgres registry. Every Lab container
is privileged (required for k3s, not avoidable), gets its own Docker network,
no published host ports, deterministic naming (rootnode-lab-<short-id> /
rootnode-lab-net-<short-id>), and a CSPRNG routing token. Includes
reconcileOnStartup(), implementing the "don't trust stale state after a
crash/restart" requirement from docs/LAB_ARCHITECTURE.md §8 — verified by
killing a container out-of-band and confirming a Provisioner restart flags
the mismatch instead of trusting the stale 'running' row.
Found and fixed during testing: a provisioning failure (e.g. readiness
timeout) left an orphaned container/network/firewall chain behind with only
the DB row marked 'error'. provisionEnvironment's catch block now does
best-effort teardown of whatever was already created before marking the row
error — confirmed by deliberately failing a provision and checking no
container/network/chain survived it.
- src/server.js — the internal HTTP API (X-Provisioner-Key shared-secret auth
on every route, matching the instructor-passcode pattern already used for
the roster endpoint). Input validation on student_id/environment_id,
structured JSON errors with a request id, routing tokens redacted from every
response.
- docker-compose.dev.yml — Postgres + egress-proxy + provisioner on one
machine (the documented Phase 1 dev exception). provisioner and egress-proxy
run with network_mode: host + NET_ADMIN/NET_RAW, which is what lets
src/egress.js manipulate the host's DOCKER-USER chain without needing
interactive host sudo.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
New Phase 1 Platform database — PostgreSQL, separate from the Lab image's own
SQLite (which keeps its current students/progress data untouched, decision #2 /
correction C in docs/LAB_ARCHITECTURE.md).
Schema (platform/provisioner/migrations/):
- lab_hosts: the abstraction that keeps Docker connectivity out of code —
endpoint (unix:// or tcp://+TLS), enabled flag. Phase 1 ships one row.
- environments: environment_id (UUID), student_id (opaque external identifier,
no FK — students table isn't in this database yet), lab_host_id (real FK),
status, routing_token (CSPRNG via crypto.randomBytes, opaque), container/
network identity, resource limits, lifecycle timestamps.
- environment_events: append-only audit trail, one row per lifecycle
transition, never logs the routing token in plaintext (see
lifecycle.js's redact()).
File-based migration runner (db.js) matching the check-before-alter idempotency
pattern already used in the Lab image's backend/server.js, applied on every
Provisioner boot.
Resource limits and the lab_hosts seed are config files
(config/resource-policy.yaml, config/lab-hosts.seed.yaml), not database tables —
there's no admin UI to manage them via a database yet, so a DB-backed policy
table would just be a write path nothing uses.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Pre-Phase-1 platform work, approved and tested earlier but not yet committed:
- Terminal banner: "RootNode Academy Lab — node: k8s-lab", upstream KubeKosh
attribution left untouched in footer/README/LICENSE.
- RootNode curriculum mode: scenarios/rootnode-path.json (Foundation -> Workloads
-> Networking -> Configuration -> Troubleshooting -> Real Ops -> Capstone),
built from existing scenario content grouped by category, no new scenarios
invented. New GET /api/curriculum/path endpoint.
- Student roster + per-student progress: instructor-roster, pick-your-name login
(no password), students/environments-adjacent progress table migrated to a
(student_id, scenario_id) composite key so practice progress and exam sessions
are isolated per student instead of one global bucket. New /api/roster,
/api/login endpoints.
- scenarios/data/*.json: added isolation_type metadata (shared_namespace vs
ephemeral_cluster) from an audit of which scenarios execute cluster-scoped or
node-level commands. This was Tier-1-isolation groundwork that got superseded
same-day by the full-ephemeral-per-student architecture (see
docs/LAB_ARCHITECTURE.md) — the field is kept as descriptive metadata only,
per the note now in scenarios/SCHEMA.md. Not wasted: it's an accurate audit of
which scenarios touch cluster-scoped resources, useful independent of isolation
model.
- .gitignore added (node_modules/, frontend/dist/, *.db, /data/, .env) — none
existed before; without it, npm installs during testing would have landed in
a commit.
Known limitation still open: no frontend UI for any of this yet (login screen,
roster-add form, curriculum view) — API-only. No auth on the roster-add endpoint.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
- Split `scenarios/scenarios.json` into individual JSON files under `scenarios/data/` named `<scenario-id>.json`
- Split `scenarios/bundles.json` into individual JSON files under `scenarios/bundles/` named `<bundle-id>.json`
- Updated `backend/server.js` to dynamically load scenario and bundle files from their respective directories
- Updated documentation in `scenarios/SCHEMA.md` and `README.md` to reflect the new repository layout and contributor workflow
Signed-off-by: Abhinav Sinha <[email protected]>
ced4568andClaude Sonnet 5