Files
rootnode-academy/platform/provisioner
ced4568andClaude Sonnet 5 0116ee1522 feat(networking): enforce isolated lab egress policy
Egress enforcement is a firewall-layer control, not an application-layer
convention (correction B). HTTP_PROXY/HTTPS_PROXY configure containerd for a
clean pull path; they enforce nothing on their own.

- src/egress.js — per-environment dedicated iptables chain
  (ROOTNODE-ENV-<short-id>), built with -A (append, order-preserving) rather
  than the -I (insert-at-head) pattern an earlier design draft used, which
  would have installed rules in reverse order and let the default-deny shadow
  the proxy/DNS accepts. Required infrastructure (egress proxy, DNS) is
  explicitly permitted first, then RFC1918 ranges are dropped unconditionally,
  then a final default-deny catches the public internet. One jump rule per
  environment hooks its subnet into its chain from DOCKER-USER (the chain
  Docker guarantees it won't silently rewrite). Destroy is deterministic:
  unhook, flush, delete — no comment-grep matching.

  The ordering fix is empirically verified, not just reasoned about: the
  egress proxy's own address is itself an RFC1918 address (it's Platform-side
  infrastructure), and testing confirms it's reachable while the broader
  RFC1918 range is still blocked — exactly the scenario the original ordering
  bug would have broken.

- platform/egress-proxy/ — shared Squid instance (one per deployment, not
  per-student), destination-allowlisted (dstdomain ACL against a plain-text
  allowlist file), starting set covers docker.io/ghcr.io/gcr.io/quay.io/k8s.io.
  Proxy performs domain allowlisting; the firewall chain above is what makes
  it unbypassable, not the other way around.

  The egress proxy's address is resolved per-student-network (each network's
  Docker gateway IP, since the proxy runs with host networking and is present
  on every bridge gateway) rather than a fixed hostname — a static address
  wouldn't be reachable from more than one isolated student network at a time.
  Found while testing with two simultaneous environments.

Verified: student-to-student isolation (A cannot reach B's internal IP),
RFC1918 blocking (A cannot reach a simulated trusted-infra address), and the
proxy allowlist itself (CONNECT tunnel to an allowlisted host succeeds,
non-allowlisted gets a clean 403 from Squid).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-18 01:22:10 -05:00
..

RootNode Academy — Provisioner (Phase 1)

Internal-only service that provisions, resets, and destroys disposable per-student Lab environments. See docs/LAB_ARCHITECTURE.md and docs/PHASE1_PLATFORM_FOUNDATION.md at the repo root for the design this implements — read those first. This file is the operational reference: how to run it, configure it, and what its actual limitations are.

Quick start (local dev)

cd platform/provisioner
echo "PROVISIONER_SHARED_SECRET=$(openssl rand -hex 32)" > .env
docker compose -f docker-compose.dev.yml build
LAB_IMAGE_REF=rootnode-academy:test docker compose -f docker-compose.dev.yml up -d

That starts Postgres, the shared egress proxy, and the Provisioner, all on one machine — the documented Phase 1 dev exception (docs/LAB_ARCHITECTURE.md §1). It is not the production topology; nothing here represents the two-VM Proxmox split.

# Smoke test
curl -H "X-Provisioner-Key: $(grep PROVISIONER_SHARED_SECRET .env | cut -d= -f2)" http://localhost:4100/health

# Run the integration suite (needs the stack above already running)
source .env
PROVISIONER_SHARED_SECRET=$PROVISIONER_SHARED_SECRET LAB_IMAGE_REF=rootnode-academy:test node test/run.js

The test image rootnode-academy:test used by default is not the real Lab image — build that separately from the repo root (docker build -t rootnode-academy:test .) if you want the suite to exercise the real one instead of a placeholder. See "Known limitations" below for why the real image currently doesn't fully boot in every environment.

Configuration reference

All environment variables read by src/server.js:

Variable Required Default Meaning
PLATFORM_DB_URL no postgres://postgres:postgres@localhost:5433/rootnode_platform Postgres connection string
PROVISIONER_SHARED_SECRET yes — Internal API auth (X-Provisioner-Key header). Process refuses to start without it.
PROVISIONER_PORT no 4100 API listen port
LAB_IMAGE_REF no rootnode-academy:test Image every new environment is provisioned from
EGRESS_PROXY_PORT no 3128 Port the shared egress proxy listens on. The proxy's host is never a static config value — it's resolved per environment as that environment's own Docker network gateway IP (see src/lifecycle.js and docs/PHASE1_PLATFORM_FOUNDATION.md §5)
DNS_RESOLVER no 127.0.0.11 Docker's embedded per-network DNS — correct default for Docker-managed bridge networks
NO_PROXY_CIDRS no 10.42.0.0/16,10.43.0.0/16 k3s's internal pod/service CIDRs, carved out of the container's proxy config so in-cluster traffic never routes through the egress proxy

Two config files, not database tables (config.js), both operator-edited plain files, hot-reloadable via SIGHUP:

  • config/resource-policy.yaml — CPU/RAM/PID/storage limits, idle/abandoned thresholds, reaper poll interval, health-check path/timeout.
  • config/lab-hosts.seed.yaml — the lab_hosts row(s) applied at startup. Phase 1 ships exactly one, dev-local, pointing at the local Docker socket. Swapping this for a tcp://<lab-host-vm>:2376 entry with TLS material is the entire config change needed to point at a real, separate Lab Host VM — no code change (see src/docker.js and correction D in docs/LAB_ARCHITECTURE.md).

Both config files are baked into the image at build time (Dockerfile's COPY config/), not bind-mounted — changing a limit requires a rebuild in this Phase 1 setup, or a SIGHUP to the running process if you've bind-mounted config/ yourself for local iteration. This was a real friction point found during implementation testing (see "Known limitations").

Deployment prerequisites (production)

Per docs/LAB_ARCHITECTURE.md §1/§7 — none of this is built by Phase 1, it's what Phase 1 assumes already exists before it's deployed for real:

  • Two separate VMs under Proxmox (or equivalent): a Platform VM and a Lab Host VM, with no network route from the Lab Host VM back to the Platform VM in either direction except the Provisioner-initiated TLS connection to the Lab Host's Docker API.
  • Neither VM has a route to trusted RootNode/CDAA/home/business networks.
  • The Provisioner process needs NET_ADMIN/NET_RAW capability and visibility into the host network namespace it's managing egress firewall rules on (--network host + those two caps if containerized, as in docker-compose.dev.yml; native process privileges if run directly on the Lab Host VM instead).
  • TLS material (CA/cert/key) provisioned for the Provisioner→Lab-Host-VM Docker API connection, referenced by a lab_hosts row's tls_ca_path/tls_cert_path/tls_key_path.

Known limitations (found during Phase 1 implementation, not guessed)

  • The real Lab image's node-pty native module fails to load in this sandbox's specific environment, crashing the backend before it ever listens (Cannot find module './prebuilds/linux-x64//pty.node'). Confirmed via testing to be pre-existing and completely unrelated to the Provisioner — the identical crash happens running the unmodified image with a plain docker run --privileged, no Provisioner involved at all. Not fixed here (out of scope — "don't redesign the existing Lab image unless strictly necessary for Phase 1 operation," and it isn't; Phase 1's job is the provisioning mechanics, which are fully validated against a lightweight stand-in image). Needs investigation on the actual target deployment host before going live — may be an architecture/prebuild mismatch specific to this sandbox, or may reproduce elsewhere.
  • Config changes require an image rebuild (see above) — fine for a versioned production deploy, mildly annoying for local iteration. Bind-mounting config/ in docker-compose.dev.yml for local dev is a reasonable follow-up, not done here to keep the dev/prod image build path identical.
  • --storage-opt size= for per-environment storage limits is unverified — depends on the host's Docker storage driver (needs overlay2 with a backing filesystem that supports project quotas). Not confirmed working or failing on this dev machine; needs a direct check on the real deployment target.
  • Real per-lab CPU/RAM sizing numbers don't exist yet — config/resource-policy.yaml ships with placeholder defaults sized for a laptop, not the ~16-core/128GB target host. Benchmarking against real hardware is future work, not Phase 1.
  • Suspend/resume was only exercised against the lightweight test image, where a plain Node process restarts trivially. That result says nothing about whether k3s-in-container specifically survives docker stop/start — decision #3 already treats this as unproven, and this test run doesn't change that. See docs/PHASE1_PLATFORM_FOUNDATION.md §8 test 7.
  • The remote TCP+TLS lab_hosts code path was validated against a Docker-in-Docker sidecar, not a real second VM — proves the mechanism works, doesn't prove anything about the actual Proxmox Lab Host VM network path, which doesn't exist yet.