ced4568andClaude Sonnet 5
c8dd33522f
Internal-only service (platform/provisioner/) implementing the environment lifecycle from docs/PHASE1_PLATFORM_FOUNDATION.md: provision -> running -> (reset -> running | destroy -> destroyed), plus experimental suspend/resume (decision #3 — never on the guaranteed path, never relied on for correctness). - src/docker.js — the lab-host abstraction (correction D). Resolves a lab_hosts row into a connected dockerode client, unix:// socket or tcp:// with TLS client-cert auth. Refuses to connect to a tcp:// endpoint with no TLS material configured rather than connecting unauthenticated. Validated against both a local socket and a real remote Docker-in-Docker TLS endpoint during testing — same code path either way, only the lab_hosts config differs, which is the entire point. - src/lifecycle.js — provision/destroy/reset/suspend/resume orchestration, wired to docker.js + egress.js + the Postgres registry. Every Lab container is privileged (required for k3s, not avoidable), gets its own Docker network, no published host ports, deterministic naming (rootnode-lab-<short-id> / rootnode-lab-net-<short-id>), and a CSPRNG routing token. Includes reconcileOnStartup(), implementing the "don't trust stale state after a crash/restart" requirement from docs/LAB_ARCHITECTURE.md §8 — verified by killing a container out-of-band and confirming a Provisioner restart flags the mismatch instead of trusting the stale 'running' row. Found and fixed during testing: a provisioning failure (e.g. readiness timeout) left an orphaned container/network/firewall chain behind with only the DB row marked 'error'. provisionEnvironment's catch block now does best-effort teardown of whatever was already created before marking the row error — confirmed by deliberately failing a provision and checking no container/network/chain survived it. - src/server.js — the internal HTTP API (X-Provisioner-Key shared-secret auth on every route, matching the instructor-passcode pattern already used for the roster endpoint). Input validation on student_id/environment_id, structured JSON errors with a request id, routing tokens redacted from every response. - docker-compose.dev.yml — Postgres + egress-proxy + provisioner on one machine (the documented Phase 1 dev exception). provisioner and egress-proxy run with network_mode: host + NET_ADMIN/NET_RAW, which is what lets src/egress.js manipulate the host's DOCKER-USER chain without needing interactive host sudo. Co-Authored-By: Claude Sonnet 5 <[email protected]>
105 lines
7.0 KiB
Markdown
105 lines
7.0 KiB
Markdown
# RootNode Academy — Provisioner (Phase 1)
|
|
|
|
Internal-only service that provisions, resets, and destroys disposable per-student Lab
|
|
environments. See `docs/LAB_ARCHITECTURE.md` and `docs/PHASE1_PLATFORM_FOUNDATION.md` at the
|
|
repo root for the design this implements — read those first. This file is the operational
|
|
reference: how to run it, configure it, and what its actual limitations are.
|
|
|
|
## Quick start (local dev)
|
|
|
|
```bash
|
|
cd platform/provisioner
|
|
echo "PROVISIONER_SHARED_SECRET=$(openssl rand -hex 32)" > .env
|
|
docker compose -f docker-compose.dev.yml build
|
|
LAB_IMAGE_REF=rootnode-academy:test docker compose -f docker-compose.dev.yml up -d
|
|
```
|
|
|
|
That starts Postgres, the shared egress proxy, and the Provisioner, all on one machine — the
|
|
documented Phase 1 dev exception (`docs/LAB_ARCHITECTURE.md` §1). It is **not** the production
|
|
topology; nothing here represents the two-VM Proxmox split.
|
|
|
|
```bash
|
|
# Smoke test
|
|
curl -H "X-Provisioner-Key: $(grep PROVISIONER_SHARED_SECRET .env | cut -d= -f2)" http://localhost:4100/health
|
|
|
|
# Run the integration suite (needs the stack above already running)
|
|
source .env
|
|
PROVISIONER_SHARED_SECRET=$PROVISIONER_SHARED_SECRET LAB_IMAGE_REF=rootnode-academy:test node test/run.js
|
|
```
|
|
|
|
The test image `rootnode-academy:test` used by default is **not** the real Lab image — build
|
|
that separately from the repo root (`docker build -t rootnode-academy:test .`) if you want the
|
|
suite to exercise the real one instead of a placeholder. See "Known limitations" below for why
|
|
the real image currently doesn't fully boot in every environment.
|
|
|
|
## Configuration reference
|
|
|
|
All environment variables read by `src/server.js`:
|
|
|
|
| Variable | Required | Default | Meaning |
|
|
|---|---|---|---|
|
|
| `PLATFORM_DB_URL` | no | `postgres://postgres:postgres@localhost:5433/rootnode_platform` | Postgres connection string |
|
|
| `PROVISIONER_SHARED_SECRET` | **yes** | — | Internal API auth (`X-Provisioner-Key` header). Process refuses to start without it. |
|
|
| `PROVISIONER_PORT` | no | `4100` | API listen port |
|
|
| `LAB_IMAGE_REF` | no | `rootnode-academy:test` | Image every new environment is provisioned from |
|
|
| `EGRESS_PROXY_PORT` | no | `3128` | Port the shared egress proxy listens on. The proxy's *host* is never a static config value — it's resolved per environment as that environment's own Docker network gateway IP (see `src/lifecycle.js` and `docs/PHASE1_PLATFORM_FOUNDATION.md` §5) |
|
|
| `DNS_RESOLVER` | no | `127.0.0.11` | Docker's embedded per-network DNS — correct default for Docker-managed bridge networks |
|
|
| `NO_PROXY_CIDRS` | no | `10.42.0.0/16,10.43.0.0/16` | k3s's internal pod/service CIDRs, carved out of the container's proxy config so in-cluster traffic never routes through the egress proxy |
|
|
|
|
Two config files, not database tables (`config.js`), both operator-edited plain files, hot-reloadable via `SIGHUP`:
|
|
|
|
- `config/resource-policy.yaml` — CPU/RAM/PID/storage limits, idle/abandoned thresholds, reaper poll interval, health-check path/timeout.
|
|
- `config/lab-hosts.seed.yaml` — the `lab_hosts` row(s) applied at startup. Phase 1 ships exactly one, `dev-local`, pointing at the local Docker socket. Swapping this for a `tcp://<lab-host-vm>:2376` entry with TLS material is the entire config change needed to point at a real, separate Lab Host VM — no code change (see `src/docker.js` and correction D in `docs/LAB_ARCHITECTURE.md`).
|
|
|
|
**Both config files are baked into the image at build time** (`Dockerfile`'s `COPY config/`),
|
|
not bind-mounted — changing a limit requires a rebuild in this Phase 1 setup, or a `SIGHUP` to
|
|
the running process if you've bind-mounted `config/` yourself for local iteration. This was a
|
|
real friction point found during implementation testing (see "Known limitations").
|
|
|
|
## Deployment prerequisites (production)
|
|
|
|
Per `docs/LAB_ARCHITECTURE.md` §1/§7 — none of this is built by Phase 1, it's what Phase 1
|
|
assumes already exists before it's deployed for real:
|
|
|
|
- Two separate VMs under Proxmox (or equivalent): a Platform VM and a Lab Host VM, with no
|
|
network route from the Lab Host VM back to the Platform VM in either direction except the
|
|
Provisioner-initiated TLS connection to the Lab Host's Docker API.
|
|
- Neither VM has a route to trusted RootNode/CDAA/home/business networks.
|
|
- The Provisioner process needs `NET_ADMIN`/`NET_RAW` capability and visibility into the host
|
|
network namespace it's managing egress firewall rules on (`--network host` + those two caps
|
|
if containerized, as in `docker-compose.dev.yml`; native process privileges if run directly
|
|
on the Lab Host VM instead).
|
|
- TLS material (CA/cert/key) provisioned for the Provisioner→Lab-Host-VM Docker API connection,
|
|
referenced by a `lab_hosts` row's `tls_ca_path`/`tls_cert_path`/`tls_key_path`.
|
|
|
|
## Known limitations (found during Phase 1 implementation, not guessed)
|
|
|
|
- **The real Lab image's `node-pty` native module fails to load in this sandbox's specific
|
|
environment**, crashing the backend before it ever listens (`Cannot find module
|
|
'./prebuilds/linux-x64//pty.node'`). Confirmed via testing to be **pre-existing and
|
|
completely unrelated to the Provisioner** — the identical crash happens running the
|
|
unmodified image with a plain `docker run --privileged`, no Provisioner involved at all. Not
|
|
fixed here (out of scope — "don't redesign the existing Lab image unless strictly necessary
|
|
for Phase 1 operation," and it isn't; Phase 1's job is the provisioning mechanics, which are
|
|
fully validated against a lightweight stand-in image). Needs investigation on the actual
|
|
target deployment host before going live — may be an architecture/prebuild mismatch specific
|
|
to this sandbox, or may reproduce elsewhere.
|
|
- **Config changes require an image rebuild** (see above) — fine for a versioned production
|
|
deploy, mildly annoying for local iteration. Bind-mounting `config/` in
|
|
`docker-compose.dev.yml` for local dev is a reasonable follow-up, not done here to keep the
|
|
dev/prod image build path identical.
|
|
- **`--storage-opt size=` for per-environment storage limits is unverified** — depends on the
|
|
host's Docker storage driver (needs overlay2 with a backing filesystem that supports project
|
|
quotas). Not confirmed working or failing on this dev machine; needs a direct check on the
|
|
real deployment target.
|
|
- **Real per-lab CPU/RAM sizing numbers don't exist yet** — `config/resource-policy.yaml`
|
|
ships with placeholder defaults sized for a laptop, not the ~16-core/128GB target host.
|
|
Benchmarking against real hardware is future work, not Phase 1.
|
|
- **Suspend/resume was only exercised against the lightweight test image**, where a plain
|
|
Node process restarts trivially. That result says nothing about whether k3s-in-container
|
|
specifically survives `docker stop`/`start` — decision #3 already treats this as unproven,
|
|
and this test run doesn't change that. See `docs/PHASE1_PLATFORM_FOUNDATION.md` §8 test 7.
|
|
- **The remote TCP+TLS `lab_hosts` code path was validated against a Docker-in-Docker sidecar**,
|
|
not a real second VM — proves the mechanism works, doesn't prove anything about the actual
|
|
Proxmox Lab Host VM network path, which doesn't exist yet.
|