7 Commits
Author SHA1 Message Date
ced4568andClaude Sonnet 5 a15897c78f docs(platform): document ephemeral lab architecture
- docs/LAB_ARCHITECTURE.md — the full ephemeral-per-student design: two-VM
  production topology (Platform VM / Lab Host VM, correction A), lifecycle,
  data model, networking/egress model (correction B), resource model,
  routing/auth model, security boundaries, failure/recovery behavior,
  migration path from the current single container, and the lab-host
  abstraction (correction D). Status updated to reflect Phase 1 is now
  implemented, pointing at the two docs below for specifics.

- docs/PHASE1_PLATFORM_FOUNDATION.md — the Phase 1 build plan: exact Postgres
  schema, service boundaries, API routes, lifecycle state machine, Docker/
  firewall commands, security assumptions, file layout, and a 12-test plan.
  Updated with "Implementation notes" at the points where testing surfaced
  real deviations from the original design: the per-network (not static)
  egress-proxy address resolution, the COALESCE reaper fix, and the
  provisioning-failure cleanup fix.

- docs/ENGINEERING_SOP.md — new §8 mapping the platform/provisioner/ and
  platform/egress-proxy/ components, what's built vs. not (no Gateway, no
  frontend wiring — internal API only), and where identity still lives
  (unchanged: Lab image's own SQLite). Roadmap item 5 (per-student isolation)
  updated from "not started" to "Phase 1 done, not yet cleared for minors."

- README.md — Multi-Student Deployment section rewritten: the old per-student-
  volume persistence model is superseded by the Platform's registry (progress
  already lives outside the disposable container as of the prior curriculum/
  roster work); points at docs/LAB_ARCHITECTURE.md and platform/provisioner/
  for the real architecture instead of describing three delivery modes that
  were never actually built out.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-18 01:22:46 -05:00
ced4568andClaude Sonnet 5 17bd01045f test(platform): add Phase 1 integration coverage
platform/provisioner/test/run.js — 12 assert-based integration tests against a
live docker-compose.dev.yml stack (no framework, matching the project's
existing convention). Covers: health/auth, provision with unique
network/container/token, duplicate-provision rejection, distinct
subnets/IPs for concurrent students, the egress firewall ordering fix
specifically (proxy ACCEPT before the RFC1918 DROP for the same range),
student-to-student isolation, RFC1918 blocking, reset (same identity, fresh
container), destroy (deterministic cleanup), and the audit trail (all
expected events present, routing token never logged).

All 12 pass against the default lightweight test image. Several properties
were validated manually during implementation rather than automated here —
idle reaper timing (needs a real threshold wait), remote TCP+TLS lab-host
(needs a Docker-in-Docker sidecar, too heavy for routine runs),
reconciliation-on-crash (needs an out-of-band container kill + process
restart), and real-Lab-image k3s behavior (environment-specific, see the
Phase 1 final report and platform/provisioner/README.md's known limitations)
— each is noted at the bottom of the file with why it's not automated here.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-18 01:22:29 -05:00
ced4568andClaude Sonnet 5 9359abc9cb feat(reaper): add idle environment cleanup
Background loop (src/reaper.js), config-driven idle/abandoned thresholds and
poll interval (no hard-coded numbers) — without this, "ephemeral by default"
is just a description: a closed browser tab does not stop a running k3s
cluster on its own.

Past the idle threshold: logs a warning event only. Past the abandoned
threshold: destroys via the same path DELETE /environments/:id uses, so
cleanup is identical either way, and logs event_type='reaped_idle'.
Failure-safe — one environment's teardown throwing never stops the reaper
from checking the rest, and a Postgres/Docker hiccup on one tick just means
"try again next tick," never a crashed process.

Uses COALESCE(last_active_at, created_at) rather than bare last_active_at.
Found during testing: an environment that fails during provisioning (before
ever reaching 'running') has a NULL last_active_at, and NULL < anything is
never true in SQL — without the fallback, a failed-and-never-cleaned-up
environment would never have aged out here regardless of how long it sat
there. This is the second, independent safety net on top of
provisionEnvironment's own best-effort cleanup on failure (see the
platform-provisioner commit).

Verified end-to-end: backdated an environment's last_active_at past the
abandoned threshold, waited for a real reaper tick, confirmed full
deterministic cleanup (container, network, firewall chain all gone) with the
correct actor='reaper' attribution in environment_events.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-18 01:22:19 -05:00
ced4568andClaude Sonnet 5 0116ee1522 feat(networking): enforce isolated lab egress policy
Egress enforcement is a firewall-layer control, not an application-layer
convention (correction B). HTTP_PROXY/HTTPS_PROXY configure containerd for a
clean pull path; they enforce nothing on their own.

- src/egress.js — per-environment dedicated iptables chain
  (ROOTNODE-ENV-<short-id>), built with -A (append, order-preserving) rather
  than the -I (insert-at-head) pattern an earlier design draft used, which
  would have installed rules in reverse order and let the default-deny shadow
  the proxy/DNS accepts. Required infrastructure (egress proxy, DNS) is
  explicitly permitted first, then RFC1918 ranges are dropped unconditionally,
  then a final default-deny catches the public internet. One jump rule per
  environment hooks its subnet into its chain from DOCKER-USER (the chain
  Docker guarantees it won't silently rewrite). Destroy is deterministic:
  unhook, flush, delete — no comment-grep matching.

  The ordering fix is empirically verified, not just reasoned about: the
  egress proxy's own address is itself an RFC1918 address (it's Platform-side
  infrastructure), and testing confirms it's reachable while the broader
  RFC1918 range is still blocked — exactly the scenario the original ordering
  bug would have broken.

- platform/egress-proxy/ — shared Squid instance (one per deployment, not
  per-student), destination-allowlisted (dstdomain ACL against a plain-text
  allowlist file), starting set covers docker.io/ghcr.io/gcr.io/quay.io/k8s.io.
  Proxy performs domain allowlisting; the firewall chain above is what makes
  it unbypassable, not the other way around.

  The egress proxy's address is resolved per-student-network (each network's
  Docker gateway IP, since the proxy runs with host networking and is present
  on every bridge gateway) rather than a fixed hostname — a static address
  wouldn't be reachable from more than one isolated student network at a time.
  Found while testing with two simultaneous environments.

Verified: student-to-student isolation (A cannot reach B's internal IP),
RFC1918 blocking (A cannot reach a simulated trusted-infra address), and the
proxy allowlist itself (CONNECT tunnel to an allowlisted host succeeds,
non-allowlisted gets a clean 403 from Squid).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-18 01:22:10 -05:00
ced4568andClaude Sonnet 5 c8dd33522f feat(provisioner): add disposable lab lifecycle
Internal-only service (platform/provisioner/) implementing the environment
lifecycle from docs/PHASE1_PLATFORM_FOUNDATION.md: provision -> running ->
(reset -> running | destroy -> destroyed), plus experimental suspend/resume
(decision #3 — never on the guaranteed path, never relied on for correctness).

- src/docker.js — the lab-host abstraction (correction D). Resolves a
  lab_hosts row into a connected dockerode client, unix:// socket or tcp://
  with TLS client-cert auth. Refuses to connect to a tcp:// endpoint with no
  TLS material configured rather than connecting unauthenticated. Validated
  against both a local socket and a real remote Docker-in-Docker TLS endpoint
  during testing — same code path either way, only the lab_hosts config
  differs, which is the entire point.

- src/lifecycle.js — provision/destroy/reset/suspend/resume orchestration,
  wired to docker.js + egress.js + the Postgres registry. Every Lab container
  is privileged (required for k3s, not avoidable), gets its own Docker network,
  no published host ports, deterministic naming (rootnode-lab-<short-id> /
  rootnode-lab-net-<short-id>), and a CSPRNG routing token. Includes
  reconcileOnStartup(), implementing the "don't trust stale state after a
  crash/restart" requirement from docs/LAB_ARCHITECTURE.md §8 — verified by
  killing a container out-of-band and confirming a Provisioner restart flags
  the mismatch instead of trusting the stale 'running' row.

  Found and fixed during testing: a provisioning failure (e.g. readiness
  timeout) left an orphaned container/network/firewall chain behind with only
  the DB row marked 'error'. provisionEnvironment's catch block now does
  best-effort teardown of whatever was already created before marking the row
  error — confirmed by deliberately failing a provision and checking no
  container/network/chain survived it.

- src/server.js — the internal HTTP API (X-Provisioner-Key shared-secret auth
  on every route, matching the instructor-passcode pattern already used for
  the roster endpoint). Input validation on student_id/environment_id,
  structured JSON errors with a request id, routing tokens redacted from every
  response.

- docker-compose.dev.yml — Postgres + egress-proxy + provisioner on one
  machine (the documented Phase 1 dev exception). provisioner and egress-proxy
  run with network_mode: host + NET_ADMIN/NET_RAW, which is what lets
  src/egress.js manipulate the host's DOCKER-USER chain without needing
  interactive host sudo.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-18 01:21:52 -05:00
ced4568andClaude Sonnet 5 62241d26d2 feat(platform-db): add lab host and environment registry
New Phase 1 Platform database — PostgreSQL, separate from the Lab image's own
SQLite (which keeps its current students/progress data untouched, decision #2 /
correction C in docs/LAB_ARCHITECTURE.md).

Schema (platform/provisioner/migrations/):
- lab_hosts: the abstraction that keeps Docker connectivity out of code —
  endpoint (unix:// or tcp://+TLS), enabled flag. Phase 1 ships one row.
- environments: environment_id (UUID), student_id (opaque external identifier,
  no FK — students table isn't in this database yet), lab_host_id (real FK),
  status, routing_token (CSPRNG via crypto.randomBytes, opaque), container/
  network identity, resource limits, lifecycle timestamps.
- environment_events: append-only audit trail, one row per lifecycle
  transition, never logs the routing token in plaintext (see
  lifecycle.js's redact()).

File-based migration runner (db.js) matching the check-before-alter idempotency
pattern already used in the Lab image's backend/server.js, applied on every
Provisioner boot.

Resource limits and the lab_hosts seed are config files
(config/resource-policy.yaml, config/lab-hosts.seed.yaml), not database tables —
there's no admin UI to manage them via a database yet, so a DB-backed policy
table would just be a write path nothing uses.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-18 01:21:31 -05:00
ced4568andClaude Sonnet 5 60fbf0c385 feat(curriculum): add RootNode learning path, student roster, and scenario isolation metadata
Pre-Phase-1 platform work, approved and tested earlier but not yet committed:

- Terminal banner: "RootNode Academy Lab — node: k8s-lab", upstream KubeKosh
  attribution left untouched in footer/README/LICENSE.
- RootNode curriculum mode: scenarios/rootnode-path.json (Foundation -> Workloads
  -> Networking -> Configuration -> Troubleshooting -> Real Ops -> Capstone),
  built from existing scenario content grouped by category, no new scenarios
  invented. New GET /api/curriculum/path endpoint.
- Student roster + per-student progress: instructor-roster, pick-your-name login
  (no password), students/environments-adjacent progress table migrated to a
  (student_id, scenario_id) composite key so practice progress and exam sessions
  are isolated per student instead of one global bucket. New /api/roster,
  /api/login endpoints.
- scenarios/data/*.json: added isolation_type metadata (shared_namespace vs
  ephemeral_cluster) from an audit of which scenarios execute cluster-scoped or
  node-level commands. This was Tier-1-isolation groundwork that got superseded
  same-day by the full-ephemeral-per-student architecture (see
  docs/LAB_ARCHITECTURE.md) — the field is kept as descriptive metadata only,
  per the note now in scenarios/SCHEMA.md. Not wasted: it's an accurate audit of
  which scenarios touch cluster-scoped resources, useful independent of isolation
  model.
- .gitignore added (node_modules/, frontend/dist/, *.db, /data/, .env) — none
  existed before; without it, npm installs during testing would have landed in
  a commit.

Known limitation still open: no frontend UI for any of this yet (login screen,
roster-add form, curriculum view) — API-only. No auth on the roster-add endpoint.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-18 01:21:16 -05:00
130 changed files with 7416 additions and 68 deletions
+6
View File
@@ -0,0 +1,6 @@
node_modules/
frontend/dist/
*.db
*.db-journal
/data/
.env
+21 -13
View File
@@ -127,22 +127,30 @@ Everything runs inside one image managed by `scripts/entrypoint.sh`.
## Multi-Student Deployment (RootNode Academy addition)
The upstream project assumes one learner, one container. RootNode Academy runs it across three
delivery modes. **Never publish the raw privileged container** — front it with isolation.
The upstream project assumes one learner, one container — this image is still exactly that,
unmodified. RootNode Academy multiplies it into a real platform via a separate provisioning
layer rather than changing the image itself: see `docs/LAB_ARCHITECTURE.md` for the
architecture and `platform/provisioner/` for the Phase 1 implementation (Postgres-backed
environment registry, per-student Docker networking, firewall-enforced egress, idle reaper —
see `platform/provisioner/README.md` for how to run it).
| Mode | Provisioning | Isolation |
| ---- | ------------ | --------- |
| In-person camp | Pre-spun container per seat, torn down after | Lab VLAN, no public ingress |
| Live cohort | Container per enrolled student for the session window | Per-student auth at the proxy |
| Self-paced | On-demand spin-up per session, idle-timeout teardown | Per-user volume + auth + resource limits |
**Never publish the raw privileged container** — front it with isolation. The one-container
`docker run --privileged -p 7554:80 ...` command above stays supported for local development
and scenario authoring; it is not how this gets deployed to real students.
Hardening checklist for shared infra:
Key properties of the Phase 1 platform (not this image):
- One container **per student** — never share a cluster between learners.
- Apply CPU/memory limits so one runaway pod can't starve a host.
- Put authentication in front (proxy-level), never expose `:7554` to the internet.
- Persist each student's `/data` to a dedicated volume.
- Idle-timeout + auto-teardown for self-paced sessions to control cost.
- One container **per student**, its own Docker network, its own firewall chain — never a
shared cluster between learners.
- Progress/identity live outside the disposable container (currently this image's own SQLite,
moving to the Platform's PostgreSQL in a later phase) — destroying a student's environment
never loses their academic record.
- CPU/memory/PID/storage limits are enforced per container, config-driven.
- Egress is restricted to an allowlisted proxy at the firewall layer, not left open.
- Idle/abandoned environments are automatically cleaned up.
Production topology and what's built vs. not yet: `docs/LAB_ARCHITECTURE.md` and
`docs/PHASE1_PLATFORM_FOUNDATION.md`.
---
+1184
View File
File diff suppressed because it is too large Load Diff
+211 -43
View File
@@ -14,6 +14,7 @@ app.use(express.static(path.join(__dirname, '../frontend/dist')));
const SCENARIOS_DIR = path.join(__dirname, '../scenarios/data');
const BUNDLES_DIR = path.join(__dirname, '../scenarios/bundles');
const CURRICULUM_PATH_FILE = path.join(__dirname, '../scenarios/rootnode-path.json');
const DB_FILE = process.env.PROGRESS_DB || '/data/progress.db';
// ── SQLite progress store ─────────────────────────────────────────────────────
@@ -55,6 +56,51 @@ function getDb() {
console.error('Failed to run schema migrations on progress table:', e.message);
}
// ── Students / roster ────────────────────────────────────────────────────
// ponytail: pick-your-name login, no password — instructor adds students to the
// roster, student just selects their name. Trust model is "supervised classroom
// device", same as Clever/ClassLink instant login. Upgrade to per-student PIN
// if remote/unsupervised access is ever needed.
_db.exec(`
CREATE TABLE IF NOT EXISTS students (
student_id TEXT PRIMARY KEY,
display_name TEXT NOT NULL,
cohort TEXT NOT NULL,
created_at TEXT NOT NULL,
last_active_at TEXT
)
`);
// Migrate progress table to a per-student composite key (student_id, scenario_id).
// SQLite can't ALTER a primary key, so this rebuilds the table. Rows that predate
// student accounts are kept under student_id='unassigned' rather than dropped.
try {
const progressCols = _db.prepare("PRAGMA table_info(progress)").all().map(c => c.name);
if (!progressCols.includes('student_id')) {
console.log('Migrating progress table to per-student schema...');
_db.exec(`
CREATE TABLE progress_new (
student_id TEXT NOT NULL DEFAULT 'unassigned',
scenario_id TEXT NOT NULL,
status TEXT,
attempts INTEGER DEFAULT 0,
last_validated TEXT,
completed_at TEXT,
started_at TEXT,
notes TEXT,
time_spent_seconds INTEGER,
PRIMARY KEY (student_id, scenario_id)
);
INSERT INTO progress_new (student_id, scenario_id, status, attempts, last_validated, completed_at, started_at, notes, time_spent_seconds)
SELECT 'unassigned', scenario_id, status, attempts, last_validated, completed_at, started_at, notes, time_spent_seconds FROM progress;
DROP TABLE progress;
ALTER TABLE progress_new RENAME TO progress;
`);
}
} catch (e) {
console.error('Failed to migrate progress table to per-student schema:', e.message);
}
_db.exec(`
CREATE TABLE IF NOT EXISTS sessions (
id TEXT PRIMARY KEY,
@@ -75,6 +121,9 @@ function getDb() {
if (!cols.includes('scenario_ids')) {
_db.exec(`ALTER TABLE sessions ADD COLUMN scenario_ids TEXT`);
}
if (!cols.includes('student_id')) {
_db.exec(`ALTER TABLE sessions ADD COLUMN student_id TEXT`);
}
} catch (e) {
console.error('Failed to migrate sessions table:', e.message);
}
@@ -94,9 +143,26 @@ function getDb() {
return _db;
}
function loadProgress() {
// Resolves the acting student for a request. No session tokens yet (see students
// table comment) — frontend just sends back the student_id it got from /api/login.
// Falls back to 'unassigned' so the app still works before any student logs in
// (dev/test use, and pre-migration progress lives there too).
function getStudentId(req) {
return req.get('x-student-id') || req.query.studentId || 'unassigned';
}
function touchStudent(studentId) {
if (!studentId || studentId === 'unassigned') return;
try {
const rows = getDb().prepare('SELECT * FROM progress').all();
getDb().prepare(`UPDATE students SET last_active_at = datetime('now') WHERE student_id = ?`).run(studentId);
} catch (e) {
console.error('Failed to update student last_active_at:', e.message);
}
}
function loadProgress(studentId = 'unassigned') {
try {
const rows = getDb().prepare('SELECT * FROM progress WHERE student_id = ?').all(studentId);
return Object.fromEntries(rows.map(r => [r.scenario_id, {
status: r.status,
attempts: r.attempts,
@@ -109,13 +175,13 @@ function loadProgress() {
} catch { return {}; }
}
function saveProgress(progress) {
function saveProgress(studentId, progress) {
try {
const db = getDb();
const upsert = db.prepare(`
INSERT INTO progress (scenario_id, status, attempts, last_validated, completed_at, started_at, notes, time_spent_seconds)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(scenario_id) DO UPDATE SET
INSERT INTO progress (student_id, scenario_id, status, attempts, last_validated, completed_at, started_at, notes, time_spent_seconds)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(student_id, scenario_id) DO UPDATE SET
status = excluded.status,
attempts = excluded.attempts,
last_validated = excluded.last_validated,
@@ -127,6 +193,7 @@ function saveProgress(progress) {
const tx = db.transaction((entries) => {
for (const [id, p] of entries) {
upsert.run(
studentId,
id,
p.status || null,
p.attempts || 0,
@@ -153,11 +220,15 @@ function loadJsonDir(dir) {
let scenariosCache = [];
let bundlesCache = [];
let curriculumPathCache = null;
function reloadCache() {
try {
scenariosCache = loadJsonDir(SCENARIOS_DIR);
bundlesCache = loadJsonDir(BUNDLES_DIR);
curriculumPathCache = fs.existsSync(CURRICULUM_PATH_FILE)
? JSON.parse(fs.readFileSync(CURRICULUM_PATH_FILE, 'utf8'))
: null;
console.log(`Loaded ${scenariosCache.length} scenarios and ${bundlesCache.length} bundles into cache.`);
} catch (e) {
console.error('Failed to reload cache:', e.message);
@@ -175,6 +246,10 @@ function loadBundles() {
return bundlesCache;
}
function loadCurriculumPath() {
return curriculumPathCache;
}
async function runCommand(cmd, timeoutMs = 15000) {
try {
const { stdout } = await execAsync(cmd, {
@@ -223,31 +298,77 @@ function refreshPrompt(delayMs = 80) {
// ── Routes ────────────────────────────────────────────────────────────────────
// ── Roster / student login ──────────────────────────────────────────────────
// POST /api/roster — instructor adds a student to a cohort
app.post('/api/roster', (req, res) => {
const { display_name, cohort } = req.body;
if (!display_name || !cohort) {
return res.status(400).json({ error: 'display_name and cohort are required' });
}
try {
const db = getDb();
const count = db.prepare('SELECT COUNT(*) AS n FROM students').get().n;
const studentId = `student-${String(count + 1).padStart(3, '0')}`;
db.prepare(`INSERT INTO students (student_id, display_name, cohort, created_at) VALUES (?, ?, ?, datetime('now'))`)
.run(studentId, display_name.trim(), cohort.trim());
res.json({ student_id: studentId, display_name: display_name.trim(), cohort: cohort.trim() });
} catch (e) {
res.status(500).json({ error: e.message });
}
});
// GET /api/roster/cohorts — distinct cohort names, for the login cohort picker
app.get('/api/roster/cohorts', (req, res) => {
const rows = getDb().prepare('SELECT DISTINCT cohort FROM students ORDER BY cohort').all();
res.json(rows.map(r => r.cohort));
});
// GET /api/roster?cohort=root-fall-2026 — student names for the pick-your-name dropdown
app.get('/api/roster', (req, res) => {
const { cohort } = req.query;
const db = getDb();
const rows = cohort
? db.prepare('SELECT student_id, display_name, cohort, last_active_at FROM students WHERE cohort = ? ORDER BY display_name').all(cohort)
: db.prepare('SELECT student_id, display_name, cohort, last_active_at FROM students ORDER BY cohort, display_name').all();
res.json(rows);
});
// POST /api/login — student picks their name; no password (see students table comment)
app.post('/api/login', (req, res) => {
const { student_id } = req.body;
const student = getDb().prepare('SELECT student_id, display_name, cohort FROM students WHERE student_id = ?').get(student_id);
if (!student) return res.status(404).json({ error: 'Student not found' });
touchStudent(student.student_id);
res.json(student);
});
// POST /api/progress/reset — reset progress for scenario | category | bundle
app.post('/api/progress/reset', (req, res) => {
const { scope, scenarioId, category, bundleId } = req.body;
const studentId = getStudentId(req);
const db = getDb();
const resetProgressStmt = db.prepare(`
UPDATE progress
SET status = 'not_started',
attempts = 0,
completed_at = NULL,
last_validated = NULL
WHERE scenario_id = ?
UPDATE progress
SET status = 'not_started',
attempts = 0,
completed_at = NULL,
last_validated = NULL
WHERE scenario_id = ? AND student_id = ?
`);
try {
if (scope === 'scenario') {
resetProgressStmt.run(scenarioId);
resetProgressStmt.run(scenarioId, studentId);
} else if (scope === 'category') {
const scenarios = loadScenarios();
const ids = scenarios.filter(s => s.category === category).map(s => s.id);
const tx = db.transaction(ids => ids.forEach(id => resetProgressStmt.run(id)));
const tx = db.transaction(ids => ids.forEach(id => resetProgressStmt.run(id, studentId)));
tx(ids);
} else if (scope === 'bundle') {
const bundle = loadBundles().find(b => b.id === bundleId);
if (!bundle) return res.status(404).json({ error: 'Bundle not found' });
const tx = db.transaction(ids => ids.forEach(id => resetProgressStmt.run(id)));
const tx = db.transaction(ids => ids.forEach(id => resetProgressStmt.run(id, studentId)));
tx(bundle.scenario_ids);
} else {
return res.status(400).json({ error: 'Invalid scope' });
@@ -285,9 +406,10 @@ app.post('/api/sessions', (req, res) => {
db.prepare(`UPDATE sessions SET status='abandoned', submitted_at=datetime('now')
WHERE status='active'`).run();
const id = `sess_${Date.now()}_${Math.random().toString(36).slice(2, 8)}`;
db.prepare(`INSERT INTO sessions (id, bundle_id, started_at, status, exam_minutes, scenario_ids)
VALUES (?, ?, datetime('now'), 'active', ?, ?)`)
.run(id, bundleId, mins, JSON.stringify(sessionScenarioIds));
const studentId = getStudentId(req);
db.prepare(`INSERT INTO sessions (id, bundle_id, started_at, status, exam_minutes, scenario_ids, student_id)
VALUES (?, ?, datetime('now'), 'active', ?, ?, ?)`)
.run(id, bundleId, mins, JSON.stringify(sessionScenarioIds), studentId);
res.json({ id, bundleId, status: 'active', exam_minutes: mins, scenario_ids: sessionScenarioIds });
});
@@ -380,7 +502,8 @@ app.post('/api/sessions/:id/submit', (req, res) => {
for (const r of examProgressRows) examProgressMap[r.scenario_id] = r;
// Also pull time_spent_seconds from global progress (tracked per scenario during the exam)
const globalProgress = loadProgress();
// Use the session's own student, not whoever's request is submitting it.
const globalProgress = loadProgress(session.student_id || 'unassigned');
const snapshot = bundleScenarios.map(s => ({
id: s.id, title: s.title, weight: s.weight,
@@ -396,9 +519,10 @@ app.post('/api/sessions/:id/submit', (req, res) => {
duration_secs=?, snapshot=? WHERE id=?`)
.run(durationSecs, JSON.stringify(snapshot), req.params.id);
// Reset timers of all scenarios in this bundle
const clearTimer = db.prepare(`UPDATE progress SET started_at = NULL, time_spent_seconds = 0 WHERE scenario_id = ?`);
const tx = db.transaction(ids => ids.forEach(id => clearTimer.run(id)));
// Reset timers of all scenarios in this bundle, for this session's student only
const clearTimer = db.prepare(`UPDATE progress SET started_at = NULL, time_spent_seconds = 0 WHERE scenario_id = ? AND student_id = ?`);
const sid = session.student_id || 'unassigned';
const tx = db.transaction(ids => ids.forEach(id => clearTimer.run(id, sid)));
tx(bundle.scenario_ids);
res.json({ ok: true, snapshot, durationSecs });
@@ -416,8 +540,9 @@ app.post('/api/sessions/:id/abandon', (req, res) => {
WHERE id=? AND status='active'`).run(req.params.id);
if (bundle) {
const clearTimer = db.prepare(`UPDATE progress SET started_at = NULL, time_spent_seconds = 0 WHERE scenario_id = ?`);
const tx = db.transaction(ids => ids.forEach(id => clearTimer.run(id)));
const clearTimer = db.prepare(`UPDATE progress SET started_at = NULL, time_spent_seconds = 0 WHERE scenario_id = ? AND student_id = ?`);
const sid = session.student_id || 'unassigned';
const tx = db.transaction(ids => ids.forEach(id => clearTimer.run(id, sid)));
tx(bundle.scenario_ids);
}
@@ -475,7 +600,7 @@ app.post('/api/scenarios/:id/context', async (req, res) => {
app.get('/api/bundles', (req, res) => {
const bundles = loadBundles();
const scenarios = loadScenarios();
const progress = loadProgress();
const progress = loadProgress(getStudentId(req));
const result = bundles.map(b => {
const total = b.scenario_ids.length;
const completed = b.scenario_ids.filter(id => progress[id]?.status === 'completed').length;
@@ -484,10 +609,47 @@ app.get('/api/bundles', (req, res) => {
res.json(result);
});
// GET /api/curriculum/path — the guided RootNode learning path, staged with completion status
app.get('/api/curriculum/path', (req, res) => {
const path = loadCurriculumPath();
if (!path) return res.status(404).json({ error: 'Curriculum path not found' });
const scenarios = loadScenarios();
const progress = loadProgress(getStudentId(req));
let firstIncompleteStage = null;
const stages = path.stages.map(stage => {
const scenarioList = stage.scenario_ids.map(id => {
const s = scenarios.find(x => x.id === id);
return {
id,
title: s?.title || id,
difficulty: s?.difficulty || null,
status: progress[id]?.status || 'not_started'
};
});
const total = scenarioList.length;
const completed = scenarioList.filter(s => s.status === 'completed').length;
const isComplete = total > 0 && completed === total;
if (!isComplete && !stage.locked && firstIncompleteStage === null) {
firstIncompleteStage = stage.id;
}
return {
id: stage.id,
name: stage.name,
icon: stage.icon,
locked: !!stage.locked,
scenarios: scenarioList,
stats: { total, completed }
};
});
res.json({ id: path.id, name: path.name, tagline: path.tagline, stages, next_stage: firstIncompleteStage });
});
// GET /api/scenarios — list scenarios; optional ?bundle=<id> and ?session=<id> filter
app.get('/api/scenarios', (req, res) => {
const scenarios = loadScenarios()
const progress = loadProgress()
const progress = loadProgress(getStudentId(req))
const { bundle, session: sessionId } = req.query
// Session-scoped: return only session scenario_ids in their shuffled order
@@ -531,7 +693,7 @@ app.get('/api/scenarios/:id', (req, res) => {
const scenarios = loadScenarios();
const scenario = scenarios.find(s => s.id === req.params.id);
if (!scenario) return res.status(404).json({ error: 'Scenario not found' });
const progress = loadProgress();
const progress = loadProgress(getStudentId(req));
res.json({
...scenario,
progress: progress[scenario.id] || { status: 'not_started', attempts: 0 }
@@ -556,7 +718,8 @@ app.post('/api/scenarios/:id/setup', async (req, res) => {
}
// Mark scenario as in-progress
const progress = loadProgress();
const studentId = getStudentId(req);
const progress = loadProgress(studentId);
if (!progress[scenario.id] || progress[scenario.id].status === 'not_started') {
progress[scenario.id] = {
...progress[scenario.id],
@@ -564,7 +727,7 @@ app.post('/api/scenarios/:id/setup', async (req, res) => {
attempts: (progress[scenario.id]?.attempts || 0),
started_at: progress[scenario.id]?.started_at || new Date().toISOString()
};
saveProgress(progress);
saveProgress(studentId, progress);
}
res.json({ setup_results: results });
@@ -602,7 +765,8 @@ app.post('/api/scenarios/:id/validate', async (req, res) => {
}
// Update global practice progress
const progress = loadProgress();
const studentId = getStudentId(req);
const progress = loadProgress(studentId);
const prev = progress[scenario.id] || { attempts: 0 };
progress[scenario.id] = {
...prev,
@@ -611,7 +775,8 @@ app.post('/api/scenarios/:id/validate', async (req, res) => {
last_validated: new Date().toISOString(),
completed_at: allPassed ? new Date().toISOString() : prev.completed_at
};
saveProgress(progress);
saveProgress(studentId, progress);
touchStudent(studentId);
// Also update exam_progress if there's an active session
const db = getDb();
@@ -649,7 +814,8 @@ app.post('/api/scenarios/:id/answer', (req, res) => {
const correct = selected === scenario.correct_option;
// Update global practice progress
const progress = loadProgress();
const studentId = getStudentId(req);
const progress = loadProgress(studentId);
const prev = progress[scenario.id] || { attempts: 0 };
progress[scenario.id] = {
...prev,
@@ -659,7 +825,8 @@ app.post('/api/scenarios/:id/answer', (req, res) => {
last_validated: new Date().toISOString(),
completed_at: correct ? new Date().toISOString() : prev.completed_at
};
saveProgress(progress);
saveProgress(studentId, progress);
touchStudent(studentId);
// Also update exam_progress if there's an active session
const db = getDb();
@@ -702,7 +869,8 @@ app.post('/api/scenarios/:id/time', (req, res) => {
if (!activeSession) {
return res.json({ ok: true, message: 'No active session' });
}
const progress = loadProgress();
const studentId = getStudentId(req);
const progress = loadProgress(studentId);
if (!progress[req.params.id]) {
progress[req.params.id] = {
status: 'in_progress',
@@ -714,14 +882,14 @@ app.post('/api/scenarios/:id/time', (req, res) => {
progress[req.params.id].started_at = new Date().toISOString();
}
progress[req.params.id].time_spent_seconds = time_spent_seconds;
saveProgress(progress);
saveProgress(studentId, progress);
res.json({ ok: true });
});
// GET /api/progress — full progress summary
app.get('/api/progress', (req, res) => {
const scenarios = loadScenarios();
const progress = loadProgress();
const progress = loadProgress(getStudentId(req));
const total = scenarios.length;
const completed = Object.values(progress).filter(p => p.status === 'completed').length;
const totalWeight = scenarios.reduce((sum, s) => sum + (s.weight || 0), 0);
@@ -740,13 +908,13 @@ app.post('/api/progress/reset/:id', (req, res) => {
try {
const db = getDb();
db.prepare(`
UPDATE progress
SET status = 'not_started',
attempts = 0,
completed_at = NULL,
last_validated = NULL
WHERE scenario_id = ?
`).run(req.params.id);
UPDATE progress
SET status = 'not_started',
attempts = 0,
completed_at = NULL,
last_validated = NULL
WHERE scenario_id = ? AND student_id = ?
`).run(req.params.id, getStudentId(req));
res.json({ ok: true });
} catch (e) {
res.status(500).json({ error: e.message });
+188
View File
@@ -0,0 +1,188 @@
# RootNode Academy — Engineering SOP
Living doc. Covers the "how" of building this platform — the "why" and business/curriculum
process live in the Business Pack (`~/Documents/CDAA Technologies/07 RootNode academy/RootNode_Academy_Business_Pack/`),
this doc doesn't duplicate that. Update this file in the same commit as the change it describes.
## 1. Where this actually lives
- **Repo:** `~/Dev/Gitea/ced4568/rootnode-academy` — remote is **Gitea**
(`git.synthossystems.com/ced4568/rootnode-academy`), not GitHub.
- **Gotcha:** `~/Dev/GitHub/ced4568/kubekosh` is a *different* clone of the same original
upstream fork, still pointed at GitHub, and it's behind (missing the RootNode logo commit
and the Real Ops enforce-limits scenario as of this writing). It's not deployed anywhere.
Don't edit it by mistake — always confirm `git remote -v` shows `gitea:` before making changes.
- Known-good baseline as of this doc: commit `6ddfb1c` ("add probe-port-mismatch and
enforce-limits to rootnode-real-ops bundle"). Everything below builds on top of that,
clean working tree.
## 2. Architecture at a glance
```
backend/server.js Express API + SQLite (better-sqlite3) + WebSocket PTY terminal — the
"Lab image," single-tenant, unmodified by the Platform work below
frontend/ Vite + React SPA, built to frontend/dist and served by nginx
scenarios/data/ One JSON file per scenario (task or MCQ)
scenarios/bundles/ Flat scenario_id lists grouped by cert track (basics/appdev/admin/security/real-ops)
scenarios/rootnode-path.json The RootNode guided learning path (staged, see §4)
scripts/entrypoint.sh Container boot: starts k3s, nginx, node backend, sets the shell banner/PS1
Dockerfile ubuntu:22.04 + k3s + node20 + nginx, single image, one port (80)
platform/provisioner/ Phase 1 Platform Provisioner — disposable per-student Lab lifecycle,
PostgreSQL registry, egress firewall. See §8 and
platform/provisioner/README.md.
platform/egress-proxy/ Shared allowlisting egress proxy every Lab environment's traffic must
pass through.
```
Single Express process, single SQLite file at `/data/progress.db` (or `$PROGRESS_DB`),
single embedded k3s cluster **per Lab environment**. The Lab image itself is still
single-tenant by design — see §5 for identity, and §8 for how Phase 1 multiplies it into a
real multi-student platform without modifying it.
## 3. Local dev / safe smoke-testing
Never test schema changes against the real `/data/progress.db` first. Point at a scratch file:
```bash
cd backend && npm install # first time only
node --check server.js # syntax check, catches typos fast
PROGRESS_DB=/tmp/scratch/progress.db node server.js &
curl -s http://localhost:4000/api/health
# ...exercise the endpoints you changed...
fuser -k 4000/tcp # stop it
```
To test a migration against **real prior data** without touching prod: copy the actual
`/data/progress.db` out of the running container/volume, point `PROGRESS_DB` at the copy,
boot, and verify the data you expect is still there before ever pointing at the real file.
kubectl/k3s-backed endpoints (`/setup`, `/validate`, `/context`, the WS terminal) won't work
outside the container — that's expected. API-and-DB-only endpoints (roster, login, progress,
curriculum, bundles, scenarios list) all smoke-test fine on a bare host.
## 4. Curriculum content
Two separate structures, don't confuse them:
- **Bundles** (`scenarios/bundles/*.json`) — flat scenario_id lists, one per cert track.
Loaded by `loadBundles()`/`GET /api/bundles`. Adding a bundle file here = adding a new
cert-track library. Every scenario referenced must have a matching file in `scenarios/data/`.
- **RootNode path** (`scenarios/rootnode-path.json`) — the guided onboarding sequence,
separate from bundles on purpose (it's *staged*, bundles are *flat*, and the bundle loader
would choke on a staged file if it were dropped in `scenarios/bundles/`). Loaded by
`loadCurriculumPath()`/`GET /api/curriculum/path`. Stages are Foundation → Workloads →
Networking → Configuration → Troubleshooting → Real Ops → Capstone, currently built by
reusing existing `1-k8s-basics` and `rootnode-real-ops` scenarios grouped by their existing
`category` field — no new scenario content was written for this. **Capstone stage is a
locked placeholder** (`"locked": true`, empty `scenario_ids`) — the capstone engine itself
is a separate, not-yet-built roadmap item (see §7, item 6). Do not put real scenarios in it
until that engine exists, or you'll reproduce the "duplicate beginner scenarios as capstone"
anti-pattern this path was built to avoid.
To add/reorder a stage: edit `rootnode-path.json` directly, referencing existing
`scenarios/data/*.json` ids. No backend code change needed — it's read fresh via
`POST /api/cache/reload` or on server restart.
## 5. Student identity — what's built, what isn't
**Built (backend only, as of this doc):**
- `students` table: `student_id` (`student-NNN`, sequential), `display_name`, `cohort`,
`created_at`, `last_active_at`.
- Login model: **instructor roster, pick-your-name, no password.** `POST /api/roster` adds
a student (instructor-only action, currently unauthenticated at the API level — see
"not built" below). `GET /api/roster?cohort=X` lists names for a dropdown.
`POST /api/login` just validates the id exists and returns it; the frontend is expected to
hold onto `student_id` and send it back as an `X-Student-Id` header on every request.
- `progress` table now keyed on `(student_id, scenario_id)` instead of `scenario_id` alone —
each student's completion/attempts/hints are isolated. Verified by smoke test: two students
answering the same scenario don't affect each other's state, and pre-migration data was
preserved under `student_id = 'unassigned'`.
- Exam `sessions` now carry `student_id` (set at session start), so exam history and the
timer-reset-on-submit logic are correctly scoped per student instead of leaking across
whoever's browser hits the API next.
**Not built yet — do not assume these exist:**
- **No frontend UI.** No login screen, no name picker, no roster-add form. The API works;
nothing calls it yet. This is the next slice.
- **No auth on `/api/roster` POST.** Anyone who can reach the API can add a "student."
Fine for a dev instance, not fine once this is reachable by actual students — needs an
instructor-only gate before that happens (ties into the instructor dashboard, roadmap
item 3).
- **No concurrency / isolation.** Exam sessions are still global-single-active
(`WHERE status='active' LIMIT 1` — see `server.js`), and there is one shared k3s cluster
for the whole instance. Per-student identity ≠ per-student environment. **Multiple students
cannot yet use this platform at the same time without stepping on each other.** That's
roadmap item 4 (isolation) and it gates real classroom/minors use, same as the plan called
out — this SOP is not claiming that gate is cleared.
## 6. Schema migration pattern
SQLite can't `ALTER` a primary key. The codebase's existing convention (see `progress` table
history) is:
1. `PRAGMA table_info(<table>)` → check if the new column/shape is already there (idempotent,
safe to run on every boot).
2. Simple new nullable column → `ALTER TABLE ... ADD COLUMN`.
3. Primary-key or structural change → build `<table>_new` with the new shape, `INSERT ...
SELECT` the old data across with sane defaults for new required columns, `DROP TABLE`,
`RENAME TO`. Never drop data — give it a legacy/default value (`'unassigned'`, etc.)
instead of discarding rows.
4. Test the migration against a **copy of real data** (§3) before it ever runs against
`/data/progress.db` in the container.
## 7. Roadmap status
Order agreed 2026-08-17/18. Status as of this doc:
1. RootNode identity/branding pass — **partial.** Shell banner fixed
(`⎈ RootNode Academy Lab — node: k8s-lab`). Upstream KubeKosh attribution left untouched in
footer/README/LICENSE. Broader "make Real Ops first-class" UI work not started.
2. RootNode curriculum mode — **backend done** (§4), frontend not started.
3. Student accounts / persistent progress — **backend done** (§5), frontend not started,
no instructor auth yet.
4. Instructor dashboard — not started.
5. Per-student isolation — **Phase 1 (platform foundation) done**, see §8. Ephemeral
per-student Lab environments with Postgres-backed lifecycle, per-environment Docker
networking, and firewall-enforced egress restriction are implemented and tested on a single
dev machine. **Still not cleared for minors** — no Gateway/frontend wiring yet (environments
are only reachable via the internal Provisioner API), no instructor auth on top of it, and
the production two-VM Proxmox split this depends on doesn't exist yet. See
`docs/PHASE1_PLATFORM_FOUNDATION.md` final report for exactly what's verified vs. not.
6. Scenarios → teaching modules (incident context/investigation/prevention) — not started.
7. Capstone/exam engine — not started (placeholder stage only, see §4).
8. Progressive hint levels — not started.
9. Production observability — not started.
10. CI scenario testing — not started.
11. Container/build optimization — explicitly deprioritized by design (platform works, not urgent).
Next slice: either the deferred frontend login (name picker → `X-Student-Id` header, curriculum
view), or the Phase 2 platform work (Gateway, frontend wiring to the Provisioner, instructor
auth) — see the Phase 1 final report for the concrete Phase 2 list. Not decided here.
## 8. Platform Provisioner (Phase 1)
Full design: `docs/LAB_ARCHITECTURE.md` + `docs/PHASE1_PLATFORM_FOUNDATION.md`. Operational
reference (config, local dev, known limitations): `platform/provisioner/README.md`. This
section is just the map.
- **What it is:** an internal-only service that provisions a disposable, fully isolated Lab
environment per student — its own Docker network, its own container running the existing,
unmodified Lab image, its own firewall-enforced egress. Postgres-backed registry
(`environments`, `environment_events`, `lab_hosts`), idle reaper, startup reconciliation.
- **What it is not (yet):** reachable by a student. There's no Gateway and no frontend wiring
— it's exercised directly via its internal API (shared-secret auth) or
`platform/provisioner/test/run.js`. Nothing here is a live feature.
- **Where identity lives:** unchanged from §5 — `students`/`progress` are still on the Lab
image's own SQLite. `environments.student_id` is deliberately opaque (no FK) until a later,
separate migration puts roster/progress on the same Postgres instance as the environment
registry.
- **Local dev:** `platform/provisioner/docker-compose.dev.yml` runs Postgres + the egress
proxy + the Provisioner on one machine — the documented Phase 1 dev exception. Production
is a two-VM Proxmox split (Platform VM / Lab Host VM); the dev compose file is explicitly
not that.
- **Test procedure:** `platform/provisioner/test/run.js` — 12 automated integration tests
against a live dev stack (isolation, firewall ordering, reset/destroy cleanup, audit
events). Some properties (idle reaper timing, remote TLS lab-host, reconciliation-on-crash,
real-image k3s behavior) were validated manually during implementation rather than
automated — see the Phase 1 final report for what and why.
+482
View File
@@ -0,0 +1,482 @@
# RootNode Academy — Ephemeral Lab Architecture
Status: **Phase 1 (platform foundation) IMPLEMENTED 2026-08-18** — see
`docs/PHASE1_PLATFORM_FOUNDATION.md` for the concrete schema/API/test results, and
`platform/provisioner/README.md` for how to run it. This document remains the architecture
record for the full ephemeral-lab direction; later phases (Gateway, frontend wiring, instructor
dashboard, production Proxmox deployment — see the Phase 1 report's "Future Phase 2 items")
build on it but aren't built yet. Supersedes the namespace-isolation
(`shared_namespace`/`ephemeral_cluster`) approach that was in flight before this redirect — see
`scenarios/SCHEMA.md`'s `isolation_type` note for the pointer back here.
Committed direction: **one fully isolated, disposable environment per student. No shared
Kubernetes control plane, ever.** This is the long-term architecture, not a stopgap.
**Decisions locked 2026-08-18** (resolves the four open questions this doc originally ended
with — see §2–§5 below for where each lands, and `docs/PHASE1_PLATFORM_FOUNDATION.md` for the
Phase 1 build plan against these decisions):
1. **Egress:** restricted, explicit allowlist. No unrestricted internet egress, ever. No route
to trusted RootNode management/CDAA/home/business networks, under any allowlist entry.
2. **Platform DB:** PostgreSQL. The Platform control plane is not built around SQLite. The
existing single-container local dev mode keeps using its own local SQLite as today —
unaffected, unrelated database.
3. **Suspend/resume:** experimental only, not load-bearing. The guaranteed lifecycle is
`provision → run → destroy → recreate`. Persistent academic state lives outside the Lab
specifically so destroying one never loses student progress — suspend/resume is a possible
future optimization, not something correctness depends on.
4. **Resource sizing:** not hard-coded. Quotas are policy/config-driven. Target host class is
~16 cores/32 threads, 128GB RAM, SSD/NVMe — actual per-lab limits and concurrency get set
by benchmarking against that class, not guessed up front.
**Corrections locked 2026-08-18** (round 2 — architectural fixes required before Phase 1
implementation begins; folded into §1, §4, §7, §9 below, not just listed here):
- **A. VM boundary.** Privileged Lab containers must never share an OS/VM boundary with the
Platform (Postgres, control-plane) in production — separate VMs under Proxmox, connected by
a Provisioner-initiated control channel only. The Lab Host is a *configured target*, not an
assumption that Docker is local. Single-machine is still fine for Phase 1 development — see §1/§9.
- **B. Egress is a firewall control, not an app convention.** `HTTP_PROXY`/`HTTPS_PROXY`
configure well-behaved apps for convenience; they enforce nothing. The firewall is the
control, and rule *ordering* matters — required infrastructure (proxy, DNS) must be
explicitly permitted before the blanket RFC1918 drops, or those drops shadow the allow
rules and nothing works. See §4 — this also corrects a real ordering bug in the original
design (explained there).
- **C. `student_id` is opaque in Phase 1.** No real foreign key to a `students` table —
students/progress stay on SQLite until a later, separate Platform DB migration. Documented,
not silently assumed. See §3/§9.
- **D. Lab host is an abstraction from day one**, even with exactly one configured host in
Phase 1 — the Provisioner is never allowed to hard-code "Docker is local." See §4.
---
## 0. What already exists (grounding, not invention)
The current repo (`~/Dev/Gitea/ced4568/rootnode-academy`) already documents almost this exact
intent in its own README, just never built out:
> "The upstream project assumes one learner, one container. RootNode Academy runs it across
> three delivery modes... One container **per student** — never share a cluster between
> learners... Put authentication in front (proxy-level)... Idle-timeout + auto-teardown."
And the existing image already **is** a complete single-student lab: nginx + React frontend +
Node/Express backend + `node-pty` WebSocket terminal + a single-node k3s cluster, all in one
`--privileged` container (confirmed required — k3s needs kernel namespaces/cgroups, per
`README.md:52`). That container is the unit this design multiplies, not something it replaces.
What changes: today that one container also owns identity/progress (SQLite at
`/data/progress.db`, one volume per student, mounted by hand). This design pulls that
ownership out into a persistent Platform, and makes the container itself fully disposable.
---
## 1. Proposed architecture
Two roles — and in production, on **two separate VM/OS boundaries**, not just two logical
components sharing a box. Correction A drives this: `--privileged` Docker containers are not
a security boundary from their own Docker host, so the boundary has to be a real VM boundary,
not a process boundary.
**Production topology:**
```
Proxmox VE (or equivalent hypervisor)
│
├── RootNode Platform VM (trusted — never runs student code)
│ │
│ │ Platform API roster · login · progress · curriculum · exam records ·
│ │ environment registry · instructor controls
│ │ Platform DB PostgreSQL — environments + audit log now;
│ │ students/progress join later (§3/§9, correction C)
│ │ Provisioner talks OUT to a *configured* Lab Host endpoint —
│ │ never assumes Docker is local (§4, correction D)
│ │ Gateway opaque-token routing to student lab containers;
│ │ only component with a leg in both worlds
│ │
│ ▼ (Provisioner-initiated control channel only — TCP+TLS to the Lab
│ Host's Docker API. Nothing on the Lab Host VM ever initiates a
│ connection back. See §4/§7 for why this direction matters.)
│
└── RootNode Lab Host VM (untrusted — this is where privileged student
│ containers actually run; holds nothing the
│ Platform needs protected)
│
├── Student A's LAB ── own private network, own nested k3s, no identity
├── Student B's LAB ── own private network, own nested k3s, no identity
└── Student C's LAB ── own private network, own nested k3s, no identity
```
Each Lab is **today's `Dockerfile` and `entrypoint.sh`, functionally unchanged** — still runs
k3s + the terminal + scenario setup/validate/teardown exactly as now. The only change inside
it: it stops being the system of record for who's using it or what they've completed (§3, §9).
A compromised student container's worst case is the Lab Host VM — never Postgres, never the
Platform API, because there is no network path from that VM back to the Platform VM at all
(§4, §7). That containment is the actual point of the two-VM split, not a deployment nicety.
**Phase 1 development exception:** running the Platform and a Lab container on one machine is
fine for local testing (§9) — the Provisioner just gets configured with a single `lab_host`
pointing at the local Docker socket instead of a remote TLS endpoint. The *code* never
special-cases "local" vs "remote" — same abstraction either way (§4/§9 of
`docs/PHASE1_PLATFORM_FOUNDATION.md`).
---
## 2. Environment lifecycle
State machine per environment, tracked in the Platform DB (`environments` table, §3):
**Guaranteed path** (what correctness depends on): `requested → provisioning → running →
resetting → running` (reset loop, same route) and `(provisioning|running|error) → destroying
→ destroyed`. `provisioning → error` and `running → error` cover failed/lost environments;
`error → destroying → destroyed` (or `→ resetting → running`, when recoverable) is the way out.
```
requested → provisioning → running ─┬─→ resetting → running (clean slate, same route)
└─→ destroying → destroyed
provisioning → error (failed to reach ready state)
running → error (health check fails mid-session)
error → destroying → destroyed (or → resetting → running, if recoverable)
[EXPERIMENTAL, not relied on for correctness — see decision #3]
running ──suspend──→ suspended ──resume──→ running (or → error, if resume fails)
```
- **provision** — Provisioner picks a host with capacity (§5), creates a private network for
this student (§4), starts a container from the pinned Lab image, generates a random
high-entropy routing token, waits for the existing readiness signal (k3s API ready + backend
`/api/health` OK — same check the current entrypoint already does), writes the `environments`
row, hands the route back to the Platform API.
- **start** — used only for `provisioning → running` (starting a freshly created container).
There is no supported path that starts a previously-stopped container in the guaranteed
lifecycle — see suspend/resume below.
- **suspend/resume — EXPERIMENTAL, decision #3.** `docker stop`/`docker start` on a k3s-in-container
Lab is *not assumed to work* and is not built as part of the guaranteed lifecycle. It's
worth testing (see the Phase 1 test plan) because if it turns out reliable it's a cheap
latency win later — but nothing in the platform depends on it succeeding. The only supported
way to bring an environment back is `destroy` + `provision` (or `reset`, which is that same
pair against one `environments` row/routing token). This is also *why* academic state living
outside the Lab (§3) isn't optional — it's what makes "just destroy and recreate" a fully
safe default instead of a data-loss risk.
- **reset** — destroy and recreate the container from the pinned image, **same routing token
and same `environments` row**, so the student's "return to my lab" link doesn't change.
This is the instructor-facing "reset student environment" control.
- **destroy** — stop + remove container, remove its private network, release the routing
token, mark the `environments` row `destroyed` with a timestamp. Progress already lives in
the Platform DB (§3), so destroying an environment never loses academic data — only
in-terminal, not-yet-validated work.
- **Idle reaper** (new, required — not optional) — background job in the Provisioner checks
`last_active_at` per running environment (heartbeat from the Gateway or from Lab `/api/health`
polling) and auto-suspends past an idle threshold, auto-destroys past a longer abandoned
threshold. Without this, "ephemeral by default" is just a description, not a guarantee —
a closed browser tab does not stop a running k3s cluster.
---
## 3. Data model changes
Everything below lives in the **Platform DB**, not inside any Lab container.
Already built (previous slice, unaffected by this redirect — still correct under this
architecture):
- `students(student_id, display_name, cohort, created_at, last_active_at)`
- `progress(student_id, scenario_id, status, attempts, ...)` — composite key, already
per-student.
- `sessions` / `exam_progress` — exam records, already student-attributed.
New:
- `environments(environment_id UUID PK, student_id, status, routing_token, container_id,
lab_host_id FK, internal_ip, cpu_limit, mem_limit_mb, storage_limit_mb, image_ref, created_at,
started_at, last_active_at, destroy_at, error_reason)` — **`student_id` is an opaque external
identifier here (correction C), not a foreign key.** Students/progress stay on the existing
SQLite in Phase 1 (see §9); Postgres has no `students` table yet, so there is nothing to key
against. The Provisioner trusts whatever `student_id` its caller passes — it does not
validate it against a roster. A real FK gets added when students/progress actually migrate
onto this Postgres instance, not before. `lab_host_id`, by contrast, *is* a real FK — both
tables live in the same Provisioner-owned Postgres DB (correction D, full schema in
`docs/PHASE1_PLATFORM_FOUNDATION.md`).
- `lab_hosts(lab_host_id PK, endpoint, enabled, ...)` — the abstraction correction D requires:
which Docker endpoint a given environment's container actually lives on. Phase 1 ships with
exactly one row (pointing at either a local socket in dev or the Lab Host VM's TLS endpoint
in production) — the table exists so environment creation is never coupled to "Docker is
local" as an assumption, even though there's no scheduling logic yet.
- `environment_events(id, environment_id, event_type, actor, detail, created_at)` — audit
trail for provision/start/stop/reset/destroy/error, needed for both incident response and
ordinary "why did my lab disappear" support.
**Removed from the Lab image's responsibility:** the `students` and roster-facing parts of the
schema I added to `backend/server.js` in the previous slice move to be Platform-only. The Lab
image's own SQLite either goes away entirely or is kept only as scratch/ephemeral state that's
allowed to die with the container — it must not be treated as authoritative for anything.
**Decision #2: PostgreSQL, not SQLite.** The Platform control plane — the environment registry
specifically, and the target for progress/roster once they migrate over — is Postgres. The
existing single-container local dev mode keeps its own local SQLite exactly as today; that's a
different, unrelated database serving a different purpose (single-tenant local testing), not
something this decision touches. See `docs/PHASE1_PLATFORM_FOUNDATION.md` for the exact schema
being stood up first.
---
## 4. Networking model
- **One private Docker network per student environment**, not one shared network for all
labs. Docker's default per-network isolation means Student A's container has no L2/L3 path
to Student B's container at all — this is stronger and simpler than trying to firewall
students apart on a shared network.
- **No published ports on Lab containers.** Today's image publishes `-p 7554:80`; under this
design that stops — the only thing that can reach a Lab container is the Gateway, over that
student's private network.
### Egress — correction B: the firewall is the control, not the proxy env vars
`HTTP_PROXY`/`HTTPS_PROXY` are set on each Lab container so well-behaved processes (containerd
pulling scenario images) route through the allowlisting proxy cleanly. **That is configuration,
not enforcement** — a process that ignores those env vars and dials out directly must still be
blocked, by something that doesn't depend on the container's cooperation. That something is a
host firewall rule, evaluated on every packet regardless of what any process inside the
container does.
**Ordering bug in the original draft, corrected here:** the previous version of this section
listed proxy/DNS `ACCEPT` rules before the RFC1918 `DROP` rules in prose, but specified
installing them with `iptables -I` (insert-at-head) in that same top-to-bottom order — which
actually reverses the result, since each `-I` without a rule number pushes the *new* rule to
position 1. Run in that order, the RFC1918 drops end up evaluated *before* the proxy/DNS
accepts, silently blocking the proxy itself if it happens to sit on an RFC1918 address (it
will — it's Platform-side infrastructure). Corrected mechanism (exact rules in
`docs/PHASE1_PLATFORM_FOUNDATION.md` §5): each student network gets its own dedicated iptables
chain, built with `-A` (append, so rules land in the order they're added — no head-insertion
trap), containing explicit `ACCEPT` rules for the egress proxy, DNS, and any specifically
required Platform endpoint *first*, then the RFC1918 `DROP`s, then a final default-`DROP`. That
chain is reached from `DOCKER-USER` (the chain Docker guarantees it won't silently rewrite) via
one jump rule matching that student's subnet.
**What no student lab may ever reach, regardless of allowlist state:**
- RootNode management networks
- the Platform database (Postgres)
- Docker/Proxmox management interfaces (including the Lab Host VM's own Docker API endpoint —
a Lab container talking back to the thing that provisioned it is exactly the escalation path
this design exists to prevent)
- CDAA infrastructure
- home/trusted networks
- other students' lab networks (already unreachable via Docker's own per-network isolation
above; the RFC1918 drop is a deliberate second layer, not redundant — it also catches any
case where two student networks' address ranges could otherwise see each other at the host
routing level)
**Decision #1, restated with the ordering fix:** restricted egress, explicit allowlist, no
unrestricted internet access ever, enforced at the firewall layer with required infrastructure
explicitly permitted *before* the blanket denies — not the reverse, and not left implicit.
### VM-to-VM boundary — correction A
The Lab Host VM has **no route back to the Platform VM at all**, in either direction, except
the one the Provisioner itself opens. The Provisioner (on the Platform VM) connects *out* to
the Lab Host's Docker API over TCP+TLS with a client certificate only the Provisioner holds;
nothing on the Lab Host VM — student containers or the VM's own OS — can open a connection back
toward the Platform VM. This is enforced at the hypervisor/network level (Proxmox VLAN/firewall
rules between the two VMs), the same category of control as the CDAA/home boundary below, not
something Docker networking alone provides. A compromised student container's blast radius
stops at the Lab Host VM.
**No route to trusted CDAA/home/business networks, full stop.** This is a host/hypervisor
boundary, not something Docker networking alone can guarantee — the box(es) these containers
run on need to sit on their own segment with no bridge/route to those networks, consistent with
the existing `RootNode_Academy_Business_Pack/06_Technology/01_Technical_Platform_Plan.md`
("dedicated student network/DMZ separated from trusted networks"). This design assumes that
host-level segmentation is a prerequisite, not something the app layer can create on its own.
### Lab host abstraction — correction D
The Provisioner never talks to "the local Docker socket" as a hard-coded assumption — it talks
to whichever `lab_hosts` row it's configured to use (§3), resolved to a Docker endpoint that
may be `unix:///var/run/docker.sock` (Phase 1 dev, single machine) or a remote
`tcp://<lab-host-vm>:2376` with TLS (production, per the topology in §1). Phase 1 ships exactly
one enabled `lab_host` and zero scheduling logic — the abstraction exists so adding a second
host later is a config row, not a rewrite.
- **Gateway** is the only component with a leg in both the public-facing network and every
student's private network. It resolves `routing_token → container internal IP:port` by
querying the Platform API (or a shared registry cache) per request/connection, then proxies
both HTTPS (scenario API, static frontend) and the WSS terminal connection.
---
## 5. Resource model
- Per-environment caps applied at container-create time: `--cpus`, `--memory` (with
`--memory-swap` pinned equal to `--memory` so nothing swaps), `--pids-limit` (fork-bomb
guard). Storage: needs a size-capped volume — exact mechanism (`--storage-opt size=`, a
loopback-backed bind mount, or a tmpfs cap) depends on the host's actual Docker storage
driver, which I haven't inspected — needs to be confirmed against the real deploy target
before picking one.
- **Decision #4: policy-driven, not hard-coded.** Limits come from config, not constants in
code — a resource-policy file the Provisioner reads at startup (and can reload), not numbers
baked into the create-container call. No specific numbers are chosen yet. Target host class
for planning purposes: ~16 cores/32 threads, 128GB RAM, SSD/NVMe. Actual per-lab limits and
how many environments run concurrently on that class get set by benchmarking (Phase 1's test
plan includes the first real measurements), not by guessing backward from the spec sheet.
- **Admission control:** Provisioner refuses to `provision` past the point where already-running
environments' summed limits would exceed host capacity. Simple counter/gate to start; can
become real bin-packing later if there's ever more than one host.
---
## 6. Routing/auth model
1. Student logs in at the Platform (existing roster/pick-your-name flow, unchanged).
2. Frontend asks the Platform for "my lab." If none running, Platform calls the Provisioner to
`provision` (or `start`, if one's suspended).
3. Platform returns an **opaque, high-entropy routing token** scoped to
`(student_id, environment_id)`, short TTL, revoked on `destroy`.
4. Frontend connects to the Gateway using that token — `https://labs.<domain>/e/<token>/...`
and `wss://labs.<domain>/e/<token>/shell-ws`. The Lab container itself never sees a
student identity; it's purely an execution surface addressed by an opaque route.
5. Gateway validates the token maps to a currently `running` environment before proxying —
rejects stale tokens from destroyed/reset environments outright, so there's no window where
an old link reaches a different (possibly now-different-student's) container.
6. **Instructor auth** extends the shared-passcode model already used for the roster-add
endpoint: instructor-only calls (`list environments`, `reset`, `terminate`, `resource usage`)
go through the Platform API under that same gate, which then calls the Provisioner
internally. Students and the public internet never talk to the Provisioner or the Docker
socket directly — only the Platform backend does.
---
## 7. Security boundaries (defense in depth)
1. **Browser ↔ Gateway** — public-facing, TLS, opaque routing tokens only. Nothing
cluster-admin-shaped is reachable from here.
2. **Gateway ↔ Lab container** — private per-student Docker network only, no published ports,
Gateway is the sole entry point.
3. **Inside the Lab container** — `--privileged` is required for k3s (confirmed, not
avoidable — this isn't a place to try to "hold the line" without breaking k3s). The
mitigation is containment, not avoidance: this privileged container has no reachable
network path to anything except the Gateway, runs as a throwaway that gets destroyed/reset
routinely, and cluster-admin *inside a student's own disposable cluster* is expected and
fine — that's the entire point of full isolation, a student doing CKS-tier cluster-admin
work only ever affects their own throwaway cluster.
4. **Lab container ↔ its own Docker host (correction A — the boundary that actually matters
here)** — `--privileged` is explicitly **not treated as a sufficient isolation boundary from
its own Docker host.** In production the Lab Host VM holding these containers is a separate
VM from the Platform VM, with no route back to it at all (§4). If a student ever escapes
their container, the ceiling is the Lab Host VM — never Postgres, never the Platform API,
never another student's environment.
5. **Provisioner ↔ Lab host's Docker API** — only the Provisioner can invoke it, over a
connection *it* initiates (TCP+TLS, client-cert auth, per §4's lab-host abstraction). The
Docker endpoint is never student-reachable, never internet-reachable, and nothing on the Lab
Host VM can initiate a connection back toward the Provisioner or the Platform VM.
6. **Platform VM ↔ other networks** — dedicated segment, no bridge to home/business/CDAA
trusted infrastructure, matching the existing Technical Platform Plan doc. Prerequisite, not
something this design builds. Same standard applies to the Lab Host VM independently — it
isn't exempted just because it's "the untrusted one."
7. **Routing tokens** — CSPRNG-generated, short TTL, invalidated on any lifecycle transition
out of `running`.
---
## 8. Failure/recovery behavior
- **Provisioning fails** (host at capacity, image pull fails, k3s doesn't reach ready state in
time) — environment marked `error`, student gets an explicit "your lab failed to start, try
again" message (never a silent hang), one automatic retry, then surfaced to the instructor
dashboard (future) as a flagged failure.
- **Mid-session crash** — Gateway's health check detects the upstream is unreachable, marks
the environment `error`, offers the student a "restart my lab" action (→ `reset`, since a
crashed in-container k3s is unlikely to resume cleanly). Because progress/identity/exam
records live in the Platform DB, this never loses the student's academic record — worst case
is losing in-progress, not-yet-validated terminal work.
- **Host failure / reboot** — every environment on that host becomes unreachable. On
Provisioner restart, it must **reconcile its registry against actual running containers**
rather than trust stale DB state — some containers may auto-restart via Docker's restart
policy, most won't; anything the registry claims is `running` that isn't actually running
gets marked `error`, not silently left wrong.
- **Idle/abandoned environments** — the reaper (§2) bounds resource leakage from crashed tabs
or students who just walked away, independent of any other failure path.
- **Core resilience property of this whole design:** because the Platform DB — not the
disposable environment — is the system of record, every failure mode above is recoverable
without academic data loss. That's the actual payoff of the split in §1/§3, not just an
architectural nicety.
---
## 9. Migration path from the current single shared container
- **Phase 0 (already true today, keep it):** running the existing image standalone —
`docker run --privileged -p 7554:80 ... rootnode-academy:latest` — is exactly the
single-user local dev/test mode you asked to keep available. No change needed to support
this; it already works and stays supported as-is.
- **Phase 1:** Build Provisioner + `environments`/`environment_events`/`lab_hosts` registry
around the **existing, unmodified** Lab image, on Postgres, with the egress firewall/proxy
from §4 in place from the start (not bolted on later). Development exception (correction A):
running Platform + Provisioner + one Lab container all on a single machine is fine here — the
Provisioner is configured with one `lab_host` pointing at the local Docker socket instead of
a remote TLS endpoint, same code path either way. Prove two lab environments can run side by
side, fully isolated from each other, before any production traffic or a second VM enters
the picture. `students`/`progress` stay on the existing SQLite through this phase
(correction C) — `environments.student_id` is opaque, not a foreign key, until a later,
separate migration actually moves roster/progress onto Postgres.
- **Phase 2 (the one real code change inside the Lab image):** narrow `backend/server.js`
inside the Lab image down to a "dumb executor" — scenario setup/validate/teardown/answer and
the WS terminal stay, but it stops owning `students`/roster and stops being the source of
truth for progress. Either it calls back to the Platform API to report validation results,
or — cleaner — the Platform API becomes the only thing the frontend talks to for
progress/curriculum/identity, and it calls the Lab container (via the route the Provisioner
already tracks) purely to execute against that student's cluster and get a pass/fail back.
- **Phase 3:** Wire instructor controls (list/reset/terminate/resource-usage) against the new
registry — this is also where the "no auth on roster-add" gap from the previous slice gets
closed for real, since instructor auth is now a load-bearing security boundary, not just
roster hygiene.
- **Phase 4 (cutover):** point the real domain at the Gateway instead of a single container.
Keep the Phase 0 single-container path available under local/dev tooling, per your explicit
requirement.
- **Rollback safety:** at every phase through Phase 3, the current single-shared-container path
keeps working completely unmodified — nothing about this migration risks the working
baseline until the Phase 4 cutover, which is also the only step that's meaningfully
reversible-with-effort rather than trivially reversible.
---
## 10. Exact files/components this touches (once approved — no code written yet)
**New, none of it touching the existing Lab image's core logic:**
- `platform/provisioner/` — Docker lifecycle management against a *configured* `lab_host`
endpoint (correction D — never a hard-coded local socket), `environments`/`lab_hosts` CRUD,
per-environment firewall chain management (correction B), idle reaper, admission control.
- `platform/egress-proxy/` — the shared allowlisting proxy every Lab container's egress must
pass through (correction B).
- `platform/gateway/` — token→route resolution, HTTPS + WSS proxying (candidate: Traefik with
a dynamic provider backed by the Platform API, or a small custom Node proxy — not decided,
not needed for this design review).
- Platform DB migrations: `environments`, `environment_events`, `lab_hosts` tables — full
schema in `docs/PHASE1_PLATFORM_FOUNDATION.md`.
**Modified, once implementation starts:**
- `backend/server.js` — split as described in §9 Phase 2. This is presently both "the lab
image's backend" and "the only backend that exists"; under this design it forks into a
narrower Lab-side executor and a separate, persistent Platform-side API. Exact file layout
for that split is an implementation decision, not part of this design doc.
- `README.md` — the "Multi-Student Deployment" section (`README.md:128-146`) currently
documents per-student **volumes** for persistence; that persistence model is superseded by
§3 above and the README needs updating once this ships, so it doesn't contradict reality.
- `docs/ENGINEERING_SOP.md` — new section for the Platform/Provisioner/Gateway components and
the reconciliation/reaper operational procedures from §8.
**Unchanged:**
- `Dockerfile`, `scripts/entrypoint.sh`, `scripts/nginx.conf` for the Lab image itself — it's
still one container running k3s + backend + terminal, exactly as today, just multiplied and
stripped of identity ownership.
- All scenario content (`scenarios/data/`, `scenarios/bundles/`, `scenarios/rootnode-path.json`)
— unaffected either way; every scenario now runs with full admin of its own disposable
cluster, so the earlier `shared_namespace`/`ephemeral_cluster` split has no bearing here.
**New Dockerfile needed (not yet written):** one for the Provisioner+Gateway services — these
are new components, not variants of the existing Lab image.
---
## Status
All four open questions this doc originally ended with are resolved — see the decisions block
at the top and §2–§5 above. Next: `docs/PHASE1_PLATFORM_FOUNDATION.md` — the concrete build
plan for the platform foundation (Postgres environment registry, provisioner skeleton,
networking, egress, resource policy, reaper) against these decisions.
+483
View File
@@ -0,0 +1,483 @@
# RootNode Academy — Phase 1: Platform Foundation
Status: **IMPLEMENTED, tested on a single dev machine, 2026-08-18.** Code lives in
`platform/provisioner/` and `platform/egress-proxy/`. This doc still reflects the corrected
design (four architecture corrections: VM boundary, egress-as-firewall-not-app-config, opaque
`student_id`, lab-host abstraction) — the implementation matches it, with a handful of
deviations found and fixed during testing, listed in "Implementation notes" at the end of each
section below where relevant, and summarized in `platform/provisioner/README.md`'s "Known
limitations." Builds on `docs/LAB_ARCHITECTURE.md` (read that first).
**Scope — this phase only:**
Postgres-backed environment registry · lifecycle state model · provisioner service skeleton ·
per-student private Docker network · disposable Lab container creation from the existing
image · configurable CPU/RAM/PID/storage limits · restricted egress · unique opaque routing
token · environment event/audit log · destroy/reset · idle-reaper design · current
single-container dev mode kept working, untouched.
**Explicitly not in this phase:** instructor dashboard, self-paced public enrollment, any
scenario content change, unrestricted egress, removing local single-user mode. Nothing below
proposes touching `backend/server.js`, `Dockerfile`, `scripts/entrypoint.sh`, or
`scenarios/**` — the Lab image is used exactly as it exists today.
---
## 1. Service boundaries
Two new components. Neither is the Lab image; neither is public-facing.
- **Provisioner** (`platform/provisioner/`) — new Node service. Owns: the Postgres
`environments`/`environment_events`/`lab_hosts` tables, the Docker Engine API connection (via
`dockerode`, not shelling out to the `docker` CLI), network/container lifecycle, the idle
reaper, and egress-firewall management for each student network. Exposes an internal HTTP API
(§4) gated by a shared-secret header — same pattern already used for the instructor-passcode
gate on the roster endpoint, so this doesn't introduce a new auth idiom.
**Docker connection is always resolved through `lab_hosts` (§2), never hard-coded** — Phase 1
configures exactly one row, pointing at `unix:///var/run/docker.sock` for single-machine dev
or a `tcp://<lab-host-vm>:2376` TLS endpoint for the two-VM production topology in
`docs/LAB_ARCHITECTURE.md` §1. Same code path either way; only the config differs.
- **Egress proxy** (`platform/egress-proxy/`) — one shared instance (not per-student), a
minimal HTTP(S) forward proxy (Squid) with a static host allowlist. Every Lab container's
`HTTP_PROXY`/`HTTPS_PROXY` env vars point at it; the student network's firewall rule permits
egress *only* to this proxy and to DNS, so there's no way for a Lab container to reach the
internet except through something the Platform's allowlist controls.
Nothing else changes. No new Platform API / roster-progress migration in this phase — that's
explicitly out of scope (see §2 note on `student_id`).
---
## 2. Exact DB schema (PostgreSQL)
New database, e.g. `rootnode_platform`. Only the environment registry ships this phase —
`students`/`progress`/`sessions` stay on the existing SQLite inside the Lab image for now;
migrating those onto this Postgres instance is a later phase, not bundled in here.
```sql
-- ── lab_hosts — correction D: the abstraction that keeps Docker connectivity out of code ──
CREATE TABLE lab_hosts (
lab_host_id TEXT PRIMARY KEY, -- e.g. 'dev-local', 'labhost-01'
display_name TEXT NOT NULL,
endpoint TEXT NOT NULL, -- 'unix:///var/run/docker.sock' (dev) or
-- 'tcp://10.x.x.x:2376' (prod, TLS-verified)
tls_ca_path TEXT, -- null for the local-socket dev case
tls_cert_path TEXT, -- Provisioner's client cert for this host
tls_key_path TEXT,
enabled BOOLEAN NOT NULL DEFAULT true,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
-- Future (not this phase): capacity/status columns for real scheduling once there's
-- more than one host. Phase 1 selects "the one enabled row" — no scheduler.
);
CREATE TYPE environment_status AS ENUM (
'requested', 'provisioning', 'running', 'resetting',
'suspended', -- experimental path only, see decision #3
'destroying', 'destroyed', 'error'
);
CREATE TABLE environments (
environment_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
student_id TEXT NOT NULL, -- CORRECTION C: opaque external identifier, not a
-- foreign key. students/progress stay on the
-- existing SQLite through Phase 1 (see
-- docs/LAB_ARCHITECTURE.md §3/§9) — there is no
-- students table in this database yet, so there
-- is nothing to reference. The Provisioner trusts
-- whatever value its caller passes; it does not
-- validate against a roster. A real FK gets added
-- only when roster/progress actually migrate onto
-- this Postgres instance, in a later, separate
-- migration — not assumed or faked here.
lab_host_id TEXT NOT NULL REFERENCES lab_hosts(lab_host_id), -- real FK: both tables
-- are Provisioner-owned Postgres, correction D.
status environment_status NOT NULL DEFAULT 'requested',
routing_token TEXT NOT NULL, -- CSPRNG, opaque, unique
container_id TEXT, -- null until provisioned
container_name TEXT,
network_id TEXT,
network_name TEXT,
network_subnet CIDR, -- needed to install/remove the per-env firewall chain
image_ref TEXT NOT NULL, -- pinned Lab image, e.g. rootnode-academy:1.4.0
cpu_limit NUMERIC NOT NULL,
mem_limit_mb INTEGER NOT NULL,
pids_limit INTEGER NOT NULL,
storage_limit_mb INTEGER NOT NULL,
internal_ip TEXT, -- set once running
internal_port INTEGER NOT NULL DEFAULT 80,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
started_at TIMESTAMPTZ,
last_active_at TIMESTAMPTZ,
destroy_at TIMESTAMPTZ, -- idle-reaper's scheduled destroy time
error_reason TEXT
);
CREATE UNIQUE INDEX idx_environments_routing_token ON environments(routing_token);
CREATE INDEX idx_environments_student ON environments(student_id);
CREATE INDEX idx_environments_status ON environments(status);
CREATE INDEX idx_environments_lab_host ON environments(lab_host_id);
-- Phase 1 invariant: at most one non-terminal environment per student.
-- Enforced in application logic (see §4 POST /environments), not a DB constraint yet —
-- a partial unique index is the natural upgrade once this needs to be bulletproof under
-- concurrent requests, not needed for a single-instance provisioner in this phase.
CREATE TABLE environment_events (
id BIGSERIAL PRIMARY KEY,
environment_id UUID NOT NULL REFERENCES environments(environment_id),
event_type TEXT NOT NULL, -- provision_requested | provisioned | started | reset
-- | destroyed | error | reaped_idle | suspend | resume
actor TEXT NOT NULL, -- 'system' | 'instructor:<id>' | 'student:<id>' | 'reaper'
detail JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX idx_env_events_environment ON environment_events(environment_id);
```
**Resource policy and the egress allowlist are config files, not tables.** There's no admin UI
to manage them via a database yet (that's dashboard work, explicitly deferred), so a
DB-backed policy table would just be a write path nothing uses. `config/resource-policy.yaml`
(cpu/mem/pids/storage defaults, reloadable) and `platform/egress-proxy/allowlist.conf` (Squid
ACL) are the actual source of truth — plain files an operator edits, matching how
`entrypoint.sh` already does config via env vars rather than a database.
---
## 3. Lifecycle state machine
Exactly what `docs/LAB_ARCHITECTURE.md` §2 now specifies, decision #3 applied:
```
requested → provisioning → running ─┬─→ resetting → running (same environment_id + routing_token)
└─→ destroying → destroyed
provisioning → error
running → error
error → destroying → destroyed
error → resetting → running (when recoverable)
[EXPERIMENTAL — not required to pass for Phase 1 sign-off]
running → suspended → running (resume)
running → suspended → error (resume fails)
```
`reset` is implemented as `destroy` + `provision` against the same DB row (new container,
network, and internal_ip; same `environment_id`/`routing_token` so a student's saved link
doesn't change).
---
## 4. API endpoints (Provisioner, internal-only)
All routes require `X-Provisioner-Key: <shared secret>`. Nothing here is reachable by a
student or the public internet in this phase — there's no frontend/gateway wiring yet, this
is tested directly (curl / a test script), which is also what makes it independently
verifiable before any later phase builds on top of it.
| Method & path | Purpose |
|---|---|
| `POST /environments` | Body `{ student_id }` — `student_id` is opaque (correction C, §2), not validated against a roster. `lab_host_id` is not a caller-supplied parameter in Phase 1: the Provisioner selects the single enabled `lab_hosts` row itself (`SELECT ... WHERE enabled LIMIT 1`) — no scheduling logic, no host-selection API, matching "don't overbuild scheduling yet." Rejects if the student already has a non-terminal environment (Phase 1 invariant, see §2). Synchronous: creates network → installs firewall chain (§5) → creates+starts container → polls Lab `/api/health` → marks `running`, or `error` on timeout. Returns `{ environment_id, status, routing_token }`. |
| `GET /environments/:id` | Full row: status, container_id, limits, timestamps. |
| `GET /environments?student_id=` | A student's environments (expected: zero or one active). |
| `POST /environments/:id/reset` | Destroy + recreate, same `environment_id`/`routing_token`. |
| `DELETE /environments/:id` | Destroy: stop, remove container, remove network + its firewall rules, mark `destroyed`. |
| `POST /environments/:id/suspend` *(experimental)* | `docker stop`, mark `suspended`. Documented as unreliable — see test plan §7. |
| `POST /environments/:id/resume` *(experimental)* | `docker start` + re-poll readiness. On failure, mark `error` rather than leaving a wrong `suspended` status. |
| `GET /environments/:id/events` | Audit trail for that environment. |
| `GET /health` | Provisioner's own health: Docker reachable, Postgres reachable. |
Synchronous provisioning is a deliberate simplification for this phase — no job queue, no
webhook callback. Given expected Phase 1 usage (developer testing, not a live cohort), a
blocking HTTP call that takes a few seconds while k3s comes up is fine; converting to
async-with-polling is the obvious upgrade if provisioning latency becomes a real UX problem
once this is behind an actual frontend.
---
## 5. Docker / network operations (via `dockerode`, shown as CLI-equivalent for readability)
**Provision — network:**
```bash
docker network create --driver bridge rootnode-lab-net-<short-env-id>
# inspect to get the allocated subnet, store as environments.network_subnet
```
**Provision — egress firewall, corrected (§4 of `docs/LAB_ARCHITECTURE.md`, correction B).**
The previous draft chained individual `iptables -I DOCKER-USER ...` inserts in
ACCEPT-then-DROP order, which is backwards: `-I` without a rule number pushes each new rule to
*position 1*, so the rules end up installed in the *reverse* of the order they were run —
the final default-DROP would land at the top of `DOCKER-USER` and shadow everything below it,
including the proxy/DNS accepts. Fixed by giving each environment its own dedicated chain,
built with `-A` (append — rules land in the order added, no head-insertion trap), and reaching
it from `DOCKER-USER` with exactly one jump rule per environment:
```bash
NET_SUBNET=$(docker network inspect rootnode-lab-net-<id> -f '{{(index .IPAM.Config 0).Subnet}}')
CHAIN="ROOTNODE-ENV-<short-env-id>"
iptables -N "$CHAIN" # fresh, empty chain for this environment
# Required infrastructure — explicitly permitted FIRST, before any blanket deny:
iptables -A "$CHAIN" -d <egress-proxy-ip> -p tcp --dport 3128 -j ACCEPT
iptables -A "$CHAIN" -d <dns-resolver-ip> -p udp --dport 53 -j ACCEPT
iptables -A "$CHAIN" -d <dns-resolver-ip> -p tcp --dport 53 -j ACCEPT
# (any explicitly-required Platform endpoint would be an additional ACCEPT line here —
# none needed in Phase 1; the Lab image doesn't call back to the Platform yet)
# Everything that must never be reachable, regardless of the above — unconditional, no
# allowlist entry can ever bypass these, satisfying "no route to trusted networks" as a
# hard rule rather than a policy convention:
iptables -A "$CHAIN" -d 10.0.0.0/8 -j DROP
iptables -A "$CHAIN" -d 172.16.0.0/12 -j DROP
iptables -A "$CHAIN" -d 192.168.0.0/16 -j DROP
iptables -A "$CHAIN" -j DROP # default deny, catches everything else (public internet)
# Hook the student's subnet into this chain. -I here is fine and intentional: we want each
# new environment's jump rule evaluated ahead of any broader, pre-existing DOCKER-USER rule,
# and since every environment matches a disjoint source subnet, the relative order of jump
# rules against each other never matters — only that each is ahead of a catch-all, if one
# ever exists at the DOCKER-USER level.
iptables -I DOCKER-USER -s "$NET_SUBNET" -j "$CHAIN"
```
This also directly satisfies the explicit "must never reach" list from
`docs/LAB_ARCHITECTURE.md` §4: RootNode management networks, the Platform database, Docker/
Proxmox management interfaces (including the Lab Host VM's own Docker API — notably, since
that API is what the Provisioner itself talks to from the *Platform* VM, a Lab container has
no way to reach it either way, but the RFC1918 drop covers it as a second, independent layer),
CDAA infrastructure, home/trusted networks, and other students' lab networks all fall inside
one of the three RFC1918 ranges above and are dropped unconditionally by this same ruleset —
one mechanism, not five separate ones to keep in sync.
**Provision — container** (limits from `resource-policy.yaml`, no image/entrypoint change):
```bash
docker create \
--name rootnode-lab-<short-env-id> \
--network rootnode-lab-net-<short-env-id> \
--privileged \
--cpus <cpu_limit> --memory <mem_limit_mb>m --memory-swap <mem_limit_mb>m \
--pids-limit <pids_limit> \
--storage-opt size=<storage_limit_mb>m \
-e HTTP_PROXY=http://<egress-proxy-host>:3128 -e HTTPS_PROXY=http://<egress-proxy-host>:3128 \
-e NO_PROXY=localhost,127.0.0.1,10.42.0.0/16,10.43.0.0/16 \
<pinned-lab-image-ref>
docker start rootnode-lab-<short-env-id>
```
**Implementation note: `<egress-proxy-host>` is resolved per environment, not a fixed
hostname/IP.** The egress proxy runs with host networking (`docker-compose.dev.yml`), so it's
present on every per-student bridge network's *gateway* interface — that gateway IP (from
`docker network inspect`'s `IPAM.Config[0].Gateway`, right alongside the subnet already used
for the firewall chain) is what's actually reachable from that specific environment's private
network. A single static proxy hostname would only be reachable from whichever one network
happened to be able to resolve/route to it — this wasn't apparent until testing with two
simultaneous environments on two different subnets. Same resolution is used for both the
firewall's `ACCEPT` rule destination (§ above) and this container's proxy env vars, so they
always agree.
`NO_PROXY` carves out k3s's own internal pod/service CIDRs so in-cluster traffic never routes
through the proxy — only the container's *outbound-to-the-internet* path does. containerd
(started by the existing, unmodified `entrypoint.sh`) honors `HTTP_PROXY`/`HTTPS_PROXY` from
its process environment automatically, which is why no Lab image or entrypoint change is
needed for well-behaved image pulls to go through the proxy.
**Correction B, stated plainly: these env vars configure containerd for a clean pull path —
they are not what makes egress restricted.** A process inside the container that ignores them
and dials out directly is stopped by the `ROOTNODE-ENV-<id>` firewall chain above regardless,
because that chain evaluates every packet leaving the container's network, independent of
what any process inside chose to do. If the env vars were somehow wrong or absent, the only
consequence is broken image pulls (a visible, debuggable failure) — never a silent hole in the
egress boundary, because the boundary was never the env vars in the first place.
Readiness: poll the container's internal IP (`docker inspect` → `NetworkSettings`) for
`GET /api/health` — the exact endpoint `entrypoint.sh` already waits on internally, so the
Provisioner is just doing the same check from outside instead of inventing a new one.
**Destroy:**
```bash
docker stop rootnode-lab-<id> --time 10 && docker rm rootnode-lab-<id>
# Unhook and remove this environment's dedicated chain — three ops, no rule-scanning/matching
# needed since it's one self-contained chain per environment, not tagged rules mixed into a
# shared one:
iptables -D DOCKER-USER -s "$NET_SUBNET" -j "$CHAIN"
iptables -F "$CHAIN"
iptables -X "$CHAIN"
docker network rm rootnode-lab-net-<id>
```
**Reset** = destroy, then provision again against the same `environments` row (new
`container_id`/`network_id`/`internal_ip`, unchanged `environment_id`/`routing_token`).
**Idle reaper** (background loop in the Provisioner, e.g. every 60s):
- Query `environments WHERE status='running' AND COALESCE(last_active_at, created_at) < now() - idle_threshold`.
- Past a shorter "idle" threshold: log a warning event (future: notify student/instructor —
not built yet).
- Past a longer "abandoned" threshold: call the same destroy path as `DELETE /environments/:id`,
log `event_type='reaped_idle'`. Config-driven thresholds, not hard-coded (same policy file
as resource limits).
**Implementation notes (found during testing, not in the original design):**
- **`COALESCE(last_active_at, created_at)`, not bare `last_active_at`.** An environment that
fails during provisioning (before ever reaching `running`) has a NULL `last_active_at` —
`NULL < anything` is never true in SQL, so without the fallback such an environment would
never age out via the reaper at all, regardless of how long ago it failed.
- **Provisioning failure now does best-effort cleanup of whatever was already created**
(container/network/firewall chain) before marking the row `error`, rather than leaving that
infrastructure orphaned. Found by testing: a deliberate readiness-timeout failure left a real
running container and firewall chain in place with only the DB row marked `error` — exactly
the kind of leak the reaper exists to prevent, except this case predates the reaper ever
having something to catch. Cleanup on the failure path directly is more honest than relying
on the reaper for a gap it wasn't actually catching. See `src/lifecycle.js`'s
`provisionEnvironment` catch block and `src/reaper.js`.
- **Startup reconciliation (docs/LAB_ARCHITECTURE.md §8) is implemented** as
`reconcileOnStartup()` in `src/lifecycle.js`, called once at Provisioner boot before the
reaper starts. Tested by killing a container out-of-band and restarting the Provisioner —
confirmed it correctly flags the mismatch as `error` rather than trusting the stale
`running` status. Any environment caught mid-transition (`provisioning`/`resetting`/
`destroying`) by a crash is also always marked `error` on restart, never assumed to have
finished.
---
## 6. Security assumptions (stated explicitly, not implied)
- **Correction A — the Lab Host is not the Platform's trust boundary, the VM is.** In
production, the host running privileged Lab containers is a separate VM from the one running
Postgres/Provisioner/Platform API, with no route back from the Lab Host VM to the Platform VM
at all — enforced at the hypervisor/network level, not by this phase's code. Phase 1's
single-machine dev mode is a deliberate, documented exception (`docs/LAB_ARCHITECTURE.md` §1)
— it does not redefine what "isolated" means for the real deployment.
- The Docker host(s) sit on a segment with no route to trusted RootNode/CDAA/home/business
networks — a prerequisite this phase depends on but does not create (matches the existing
Business Pack Technical Platform Plan). Everything below assumes that's already true.
- No other host firewall tooling (ufw, firewalld) is silently rewriting `DOCKER-USER` or
removing the per-environment `ROOTNODE-ENV-<id>` chains — needs a one-time check on the real
host before trusting the egress rules; not something this design can verify in advance.
- `--privileged` is required for k3s and is not being avoided (confirmed in
`docs/LAB_ARCHITECTURE.md` §7) — the security boundary is the VM split plus network
containment, not reduced container privilege.
- **Correction B — `HTTP_PROXY`/`HTTPS_PROXY` are not a security control**, stated here again
because it's the assumption most likely to get silently reintroduced during implementation:
the firewall chain enforces the boundary; the env vars only make well-behaved pulls work
cleanly through it.
- The egress proxy's allowlist is operator-controlled (a file the Provisioner's operator
edits), never student- or Lab-container-writable.
- The Provisioner API's shared secret is the only thing standing between "internal service"
and "anyone who can create/destroy arbitrary privileged containers on the host" — treat that
secret with the same care as the instructor passcode, not as a minor implementation detail.
This matters more, not less, once the Provisioner is issuing commands to a remote Docker
endpoint (correction D) — the TLS client cert for that connection carries the same weight.
- `storage_limit_mb` enforcement via `--storage-opt size=` depends on the host's Docker storage
driver supporting it (overlay2 with specific backing filesystems does; not all do) — unverified
until tested against the real host (§8).
- Nothing in this phase is internet-facing. The Provisioner API assumes internal-network-only
reachability; it does not implement its own TLS. The Provisioner→Lab-Host Docker connection
(correction D), by contrast, is TLS by design once it's not a local socket — that's a
different boundary with different stakes and isn't optional the way the Provisioner API's
own TLS currently is.
---
## 7. Files to be created
All new. Nothing existing is modified beyond the two docs already updated.
```
platform/provisioner/
├── package.json
├── src/
│ ├── server.js # Express app, routes from §4
│ ├── db.js # Postgres connection (pg), migration runner
│ ├── docker.js # dockerode client factory — resolves a lab_hosts row to a connected
│ │ client (local socket or remote TLS), never a hard-coded default
│ ├── lifecycle.js # state machine + orchestration (calls docker.js + db.js)
│ ├── egress.js # per-environment ROOTNODE-ENV-<id> chain create/hook/unhook/remove
│ └── reaper.js # idle-reaper background loop
├── migrations/
│ ├── 001_lab_hosts.sql
│ ├── 002_environments.sql
│ └── 003_environment_events.sql
├── config/
│ ├── resource-policy.yaml # cpu/mem/pids/storage defaults + idle/abandoned thresholds
│ └── lab-hosts.seed.yaml # the one Phase 1 lab_hosts row (endpoint + TLS paths),
│ # applied by the migration runner — not a runtime API,
│ # per "don't overbuild scheduling yet"
├── Dockerfile
└── docker-compose.dev.yml # Postgres + egress-proxy + provisioner, for local Phase 1 testing
platform/egress-proxy/
├── Dockerfile # minimal Squid image
└── allowlist.conf # Squid ACL: registry-1.docker.io, ghcr.io, etc. — starting set
# to be confirmed against what scenarios actually pull
```
No changes to `backend/server.js`, `Dockerfile`, `scripts/entrypoint.sh`,
`scripts/nginx.conf`, or anything under `scenarios/`.
---
## 8. Test plan
1. **Schema migration idempotency** — run `001_environments.sql` against a scratch Postgres
twice, confirm the second run is a no-op (same pattern already used for the SQLite
migrations: check-before-alter).
2. **Provision → destroy round trip** — create an environment, confirm the container is
`running` and reachable on its internal IP, confirm `environment_events` has one row per
transition, destroy, confirm container/network are gone and the environment's
`ROOTNODE-ENV-<id>` chain no longer exists (`iptables -L ROOTNODE-ENV-<id>` errors "chain
does not exist") and its `DOCKER-USER` jump rule is gone (`iptables -S DOCKER-USER | grep
<id>` returns nothing).
3. **Reset** — provision, note `environment_id`/`routing_token`, reset, confirm both are
unchanged, confirm `container_id` changed (genuinely new container), confirm no state
carried over (fresh k3s).
4. **Isolation** — provision two environments, confirm A cannot reach B at all
(`docker exec <A> ping <B's internal_ip>` times out) — this is the core security property
and must be an explicit test, not an assumption.
5. **Egress allowlist, including the ordering fix specifically** — from inside a provisioned
environment: an allowlisted registry pull succeeds via the proxy; a non-allowlisted host
fails; a direct request to an RFC1918 address (simulating a trusted-network address) is
dropped regardless of allowlist state. Critically: **the egress proxy's own IP is itself an
RFC1918 address** — this test only actually exercises the fix from correction B if the proxy
is reachable *and* the broader RFC1918 range is still blocked. If the proxy were unreachable
here, that's the exact symptom the original ordering bug would have caused (the DROP rules
shadowing the ACCEPT rules) — this test is what would have caught it before it shipped.
6. **Idle reaper** — provision, force `last_active_at` backward past the abandoned threshold,
run one reaper tick, confirm the environment is destroyed and logged as `reaped_idle`.
7. **Suspend/resume (experimental, decision #3 — informational, not a pass/fail gate)** —
`docker stop`/`docker start` a running Lab container, time how long it takes (if ever) to
reach `/api/health` ready again versus a fresh `provision`. This is the actual answer to
the question `docs/LAB_ARCHITECTURE.md` flagged as untested — record the result either way,
but Phase 1 sign-off does not depend on this succeeding.
8. **Reconciliation on Provisioner restart** — manually create a container outside the
Provisioner's knowledge, restart the Provisioner, confirm it doesn't silently trust stale
`environments` rows — either adopts or flags the mismatch (per
`docs/LAB_ARCHITECTURE.md` §8).
9. **Resource limits hold** — inside a provisioned environment, confirm `--cpus`/`--memory`/
`--pids-limit` are actually enforced (e.g. a deliberate memory-hog process gets OOM-killed
at the configured limit, not the host's full RAM); confirm whether `--storage-opt size=`
is honored on the real host's storage driver — this is the resource-model unknown from
`docs/LAB_ARCHITECTURE.md` §5, and this test is the first real benchmarking data point
toward the 16-core/128GB target sizing.
10. **Local dev-mode regression check** — confirm `docker run --privileged -p 7554:80 ...
rootnode-academy:latest` (today's standalone command) is completely unaffected —
`git diff` scoped to `backend/`, `Dockerfile`, `scripts/` should show zero changes from
this phase.
11. **Lab host abstraction actually abstracts (correction D)** — point the Provisioner's
`lab_hosts` config at a Docker daemon exposed over TCP+TLS on the *same* dev machine
(`dockerd -H tcp://127.0.0.1:2376 --tlsverify ...`) instead of the local Unix socket, with
no code change — only the `lab_hosts` row differs. Provision/destroy round trip (test 2)
must pass identically against that endpoint. This is the practical proof that "Docker is
local" was actually removed as an assumption, not just renamed — it doesn't require a
second real VM to verify, only a second way of reaching the same daemon.
12. **VM-boundary reasoning check (correction A, documentation-level, not a running test)** —
walk the failure scenario explicitly: if a student escaped their container onto the Lab
Host VM's OS, what could they reach? Confirm the answer stays "nothing on the Platform VM"
given the topology in `docs/LAB_ARCHITECTURE.md` §1 — i.e. confirm the design doesn't rely
on anything *inside* the Lab Host VM (a firewall rule, a credential, an agent) as the only
thing standing between a compromised container and Postgres. This is a design-review check
now; it becomes a real network-level test once a second VM exists to test against.
Tests 1–6, 8, 10, 11 are runnable in a plain Docker environment with no special hardware.
Tests 5 and 9 need the actual target host (or something close to it) to produce meaningful
numbers, test 7 needs the real Lab image, and test 12 needs a second VM to be more than a
design-review exercise — flagging that distinction so "passed the test plan" doesn't get read
as "validated on production-class, two-VM hardware" when parts of it were necessarily
validated on a single dev machine.
+2996
View File
File diff suppressed because it is too large Load Diff
+10
View File
@@ -0,0 +1,10 @@
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y --no-install-recommends squid \
&& rm -rf /var/lib/apt/lists/*
COPY squid.conf /etc/squid/squid.conf
COPY allowlist.txt /etc/squid/allowlist.txt
EXPOSE 3128
CMD ["squid", "-N", "-d", "1"]
+17
View File
@@ -0,0 +1,17 @@
# RootNode Academy egress allowlist — one domain per line, matched with Squid's dstdomain ACL
# (a leading "." matches the domain and all subdomains). Configuration-driven, operator-edited,
# never student- or Lab-container-writable (docs/PHASE1_PLATFORM_FOUNDATION.md §6).
#
# Starting set covers what the existing scenario content actually pulls (nginx, busybox-style
# images used across scenarios/data/*.json setup_commands) — confirm/expand against real usage
# before relying on this for a live cohort; this is a first pass, not a final audit.
.docker.io
.docker.com
registry-1.docker.io
auth.docker.io
production.cloudflare.docker.com
.ghcr.io
.gcr.io
.k8s.io
.quay.io
+22
View File
@@ -0,0 +1,22 @@
# RootNode Academy shared egress proxy — one instance, not per-student.
# Destination allowlisting only; the firewall (egress.js) is the actual enforcement boundary
# that makes this proxy unbypassable, not the other way around
# (docs/PHASE1_PLATFORM_FOUNDATION.md §5/§6, correction B).
http_port 3128
acl allowed_dst dstdomain "/etc/squid/allowlist.txt"
acl SSL_ports port 443
acl Safe_ports port 80 443
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow allowed_dst
http_access deny all
# Audit/troubleshooting log — requirement #7. Squid's default access log format already
# includes timestamp, client IP, method, URL, and result code, which is what "why did this
# scenario's image pull fail" support needs.
access_log /var/log/squid/access.log squid
cache deny all
+6
View File
@@ -0,0 +1,6 @@
# Copy to .env and fill in a real secret before running docker-compose.dev.yml.
# Generate one with: openssl rand -hex 32
PROVISIONER_SHARED_SECRET=
# Optional overrides — see README.md's "Configuration reference" for the full list and defaults.
# LAB_IMAGE_REF=rootnode-academy:test
+19
View File
@@ -0,0 +1,19 @@
FROM node:20-slim
# iptables is required at runtime for egress.js — the Provisioner manipulates the host's
# DOCKER-USER chain directly (see docs/PHASE1_PLATFORM_FOUNDATION.md §5), which requires this
# container to run with --network host --cap-add=NET_ADMIN --cap-add=NET_RAW (see
# docker-compose.dev.yml). This is not a "nice to have" — without those run flags the
# Provisioner cannot enforce egress at all, only fail loudly, which it does.
RUN apt-get update && apt-get install -y --no-install-recommends iptables ca-certificates \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY package.json ./
RUN npm install --production
COPY src/ ./src/
COPY migrations/ ./migrations/
COPY config/ ./config/
EXPOSE 4100
CMD ["node", "src/server.js"]
+104
View File
@@ -0,0 +1,104 @@
# RootNode Academy — Provisioner (Phase 1)
Internal-only service that provisions, resets, and destroys disposable per-student Lab
environments. See `docs/LAB_ARCHITECTURE.md` and `docs/PHASE1_PLATFORM_FOUNDATION.md` at the
repo root for the design this implements — read those first. This file is the operational
reference: how to run it, configure it, and what its actual limitations are.
## Quick start (local dev)
```bash
cd platform/provisioner
echo "PROVISIONER_SHARED_SECRET=$(openssl rand -hex 32)" > .env
docker compose -f docker-compose.dev.yml build
LAB_IMAGE_REF=rootnode-academy:test docker compose -f docker-compose.dev.yml up -d
```
That starts Postgres, the shared egress proxy, and the Provisioner, all on one machine — the
documented Phase 1 dev exception (`docs/LAB_ARCHITECTURE.md` §1). It is **not** the production
topology; nothing here represents the two-VM Proxmox split.
```bash
# Smoke test
curl -H "X-Provisioner-Key: $(grep PROVISIONER_SHARED_SECRET .env | cut -d= -f2)" http://localhost:4100/health
# Run the integration suite (needs the stack above already running)
source .env
PROVISIONER_SHARED_SECRET=$PROVISIONER_SHARED_SECRET LAB_IMAGE_REF=rootnode-academy:test node test/run.js
```
The test image `rootnode-academy:test` used by default is **not** the real Lab image — build
that separately from the repo root (`docker build -t rootnode-academy:test .`) if you want the
suite to exercise the real one instead of a placeholder. See "Known limitations" below for why
the real image currently doesn't fully boot in every environment.
## Configuration reference
All environment variables read by `src/server.js`:
| Variable | Required | Default | Meaning |
|---|---|---|---|
| `PLATFORM_DB_URL` | no | `postgres://postgres:postgres@localhost:5433/rootnode_platform` | Postgres connection string |
| `PROVISIONER_SHARED_SECRET` | **yes** | — | Internal API auth (`X-Provisioner-Key` header). Process refuses to start without it. |
| `PROVISIONER_PORT` | no | `4100` | API listen port |
| `LAB_IMAGE_REF` | no | `rootnode-academy:test` | Image every new environment is provisioned from |
| `EGRESS_PROXY_PORT` | no | `3128` | Port the shared egress proxy listens on. The proxy's *host* is never a static config value — it's resolved per environment as that environment's own Docker network gateway IP (see `src/lifecycle.js` and `docs/PHASE1_PLATFORM_FOUNDATION.md` §5) |
| `DNS_RESOLVER` | no | `127.0.0.11` | Docker's embedded per-network DNS — correct default for Docker-managed bridge networks |
| `NO_PROXY_CIDRS` | no | `10.42.0.0/16,10.43.0.0/16` | k3s's internal pod/service CIDRs, carved out of the container's proxy config so in-cluster traffic never routes through the egress proxy |
Two config files, not database tables (`config.js`), both operator-edited plain files, hot-reloadable via `SIGHUP`:
- `config/resource-policy.yaml` — CPU/RAM/PID/storage limits, idle/abandoned thresholds, reaper poll interval, health-check path/timeout.
- `config/lab-hosts.seed.yaml` — the `lab_hosts` row(s) applied at startup. Phase 1 ships exactly one, `dev-local`, pointing at the local Docker socket. Swapping this for a `tcp://<lab-host-vm>:2376` entry with TLS material is the entire config change needed to point at a real, separate Lab Host VM — no code change (see `src/docker.js` and correction D in `docs/LAB_ARCHITECTURE.md`).
**Both config files are baked into the image at build time** (`Dockerfile`'s `COPY config/`),
not bind-mounted — changing a limit requires a rebuild in this Phase 1 setup, or a `SIGHUP` to
the running process if you've bind-mounted `config/` yourself for local iteration. This was a
real friction point found during implementation testing (see "Known limitations").
## Deployment prerequisites (production)
Per `docs/LAB_ARCHITECTURE.md` §1/§7 — none of this is built by Phase 1, it's what Phase 1
assumes already exists before it's deployed for real:
- Two separate VMs under Proxmox (or equivalent): a Platform VM and a Lab Host VM, with no
network route from the Lab Host VM back to the Platform VM in either direction except the
Provisioner-initiated TLS connection to the Lab Host's Docker API.
- Neither VM has a route to trusted RootNode/CDAA/home/business networks.
- The Provisioner process needs `NET_ADMIN`/`NET_RAW` capability and visibility into the host
network namespace it's managing egress firewall rules on (`--network host` + those two caps
if containerized, as in `docker-compose.dev.yml`; native process privileges if run directly
on the Lab Host VM instead).
- TLS material (CA/cert/key) provisioned for the Provisioner→Lab-Host-VM Docker API connection,
referenced by a `lab_hosts` row's `tls_ca_path`/`tls_cert_path`/`tls_key_path`.
## Known limitations (found during Phase 1 implementation, not guessed)
- **The real Lab image's `node-pty` native module fails to load in this sandbox's specific
environment**, crashing the backend before it ever listens (`Cannot find module
'./prebuilds/linux-x64//pty.node'`). Confirmed via testing to be **pre-existing and
completely unrelated to the Provisioner** — the identical crash happens running the
unmodified image with a plain `docker run --privileged`, no Provisioner involved at all. Not
fixed here (out of scope — "don't redesign the existing Lab image unless strictly necessary
for Phase 1 operation," and it isn't; Phase 1's job is the provisioning mechanics, which are
fully validated against a lightweight stand-in image). Needs investigation on the actual
target deployment host before going live — may be an architecture/prebuild mismatch specific
to this sandbox, or may reproduce elsewhere.
- **Config changes require an image rebuild** (see above) — fine for a versioned production
deploy, mildly annoying for local iteration. Bind-mounting `config/` in
`docker-compose.dev.yml` for local dev is a reasonable follow-up, not done here to keep the
dev/prod image build path identical.
- **`--storage-opt size=` for per-environment storage limits is unverified** — depends on the
host's Docker storage driver (needs overlay2 with a backing filesystem that supports project
quotas). Not confirmed working or failing on this dev machine; needs a direct check on the
real deployment target.
- **Real per-lab CPU/RAM sizing numbers don't exist yet** — `config/resource-policy.yaml`
ships with placeholder defaults sized for a laptop, not the ~16-core/128GB target host.
Benchmarking against real hardware is future work, not Phase 1.
- **Suspend/resume was only exercised against the lightweight test image**, where a plain
Node process restarts trivially. That result says nothing about whether k3s-in-container
specifically survives `docker stop`/`start` — decision #3 already treats this as unproven,
and this test run doesn't change that. See `docs/PHASE1_PLATFORM_FOUNDATION.md` §8 test 7.
- **The remote TCP+TLS `lab_hosts` code path was validated against a Docker-in-Docker sidecar**,
not a real second VM — proves the mechanism works, doesn't prove anything about the actual
Proxmox Lab Host VM network path, which doesn't exist yet.
@@ -0,0 +1,23 @@
# Phase 1 lab_hosts seed — applied to the DB at Provisioner startup (config.js
# applyLabHostsSeed), not managed via a runtime API (no host-management endpoints in Phase 1,
# matching "don't overbuild scheduling yet" — see docs/PHASE1_PLATFORM_FOUNDATION.md §4).
#
# Exactly one enabled row for Phase 1. Swapping this for a remote tcp://<lab-host-vm>:2376
# entry with TLS material is the entire code change needed to point at a real, separate Lab
# Host VM in production — see docs/LAB_ARCHITECTURE.md §1 and correction D.
lab_hosts:
- lab_host_id: dev-local
display_name: "Local development Docker socket"
endpoint: "unix:///var/run/docker.sock"
enabled: true
# Production example (not active — shown for reference; uncomment and fill in when a real
# Lab Host VM exists):
# - lab_host_id: labhost-01
# display_name: "RootNode Lab Host VM 01"
# endpoint: "tcp://10.20.30.5:2376"
# tls_ca_path: /etc/rootnode/tls/labhost-01/ca.pem
# tls_cert_path: /etc/rootnode/tls/labhost-01/cert.pem
# tls_key_path: /etc/rootnode/tls/labhost-01/key.pem
# enabled: true
@@ -0,0 +1,22 @@
# Phase 1 resource policy — config-driven, not hard-coded (decision #4).
# No specific production numbers chosen yet; target host class for planning is ~16 cores/32
# threads, 128GB RAM, SSD/NVMe. Actual per-lab limits get set by benchmarking (see
# docs/PHASE1_PLATFORM_FOUNDATION.md §8 test 9), not guessed backward from the spec sheet.
# These defaults are sized for local single-machine Phase 1 dev/testing only.
default:
cpu_limit: 2.0 # vCPUs
mem_limit_mb: 4096
pids_limit: 512
storage_limit_mb: 8192 # only enforced if the host's Docker storage driver supports
# --storage-opt size= (see known-limitations in the SOP)
idle_threshold_minutes: 30 # past this: log a warning event only
abandoned_threshold_minutes: 120 # past this: reaper destroys for real
reaper_poll_interval_seconds: 60
health_check:
path: /api/health
timeout_ms: 240000 # k3s cold start in a nested container can be slow — matches the
# ~60-attempt wait already in scripts/entrypoint.sh
@@ -0,0 +1,54 @@
# Phase 1 local development/testing stack — Postgres + egress-proxy + provisioner, all on one
# machine (docs/LAB_ARCHITECTURE.md §1's documented dev exception). Nothing here represents the
# production two-VM topology; it's what makes the provisioner testable on a laptop.
#
# provisioner and egress-proxy run with network_mode: host + NET_ADMIN/NET_RAW because:
# - egress.js manipulates the host's DOCKER-USER iptables chain directly (§5) — that's only
# possible if the process can see the host network namespace.
# - egress-proxy needs to be reachable from every per-student bridge network's gateway IP,
# which only works if it's bound across all host interfaces (host networking), not sandboxed
# inside its own compose-managed bridge network.
# Because provisioner uses host networking, it reaches Postgres via localhost:5433 (Postgres's
# published host port), not the compose service name.
services:
postgres:
image: postgres:16-alpine
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: rootnode_platform
ports:
- "5433:5432"
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 5s
timeout: 3s
retries: 10
egress-proxy:
build: ../egress-proxy
network_mode: host
# Squid listens on :3128 across every host interface, including each per-student bridge's
# gateway — that's the whole reason this needs host networking rather than a normal
# compose network.
provisioner:
build: .
network_mode: host
cap_add:
- NET_ADMIN
- NET_RAW
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
PLATFORM_DB_URL: postgres://postgres:postgres@localhost:5433/rootnode_platform
PROVISIONER_SHARED_SECRET: ${PROVISIONER_SHARED_SECRET:?set a real secret, do not use a default}
PROVISIONER_PORT: "4100"
LAB_IMAGE_REF: ${LAB_IMAGE_REF:-rootnode-academy:test}
EGRESS_PROXY_PORT: "3128"
DNS_RESOLVER: "127.0.0.11"
NO_PROXY_CIDRS: "10.42.0.0/16,10.43.0.0/16"
depends_on:
postgres:
condition: service_healthy
@@ -0,0 +1,16 @@
-- lab_hosts — correction D: the abstraction that keeps Docker connectivity out of code.
-- See docs/LAB_ARCHITECTURE.md §3/§4 and docs/PHASE1_PLATFORM_FOUNDATION.md §2.
CREATE TABLE IF NOT EXISTS lab_hosts (
lab_host_id TEXT PRIMARY KEY,
display_name TEXT NOT NULL,
endpoint TEXT NOT NULL, -- 'unix:///var/run/docker.sock' (dev) or
-- 'tcp://10.x.x.x:2376' (prod, TLS-verified)
tls_ca_path TEXT, -- null for the local-socket dev case
tls_cert_path TEXT,
tls_key_path TEXT,
enabled BOOLEAN NOT NULL DEFAULT true,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
-- Future (not Phase 1): capacity/status columns for real scheduling once there's more
-- than one host. Phase 1 selects "the one enabled row" — no scheduler.
);
@@ -0,0 +1,51 @@
-- environments — the disposable per-student lab registry.
-- See docs/LAB_ARCHITECTURE.md §3 and docs/PHASE1_PLATFORM_FOUNDATION.md §2.
DO $$ BEGIN
CREATE TYPE environment_status AS ENUM (
'requested', 'provisioning', 'running', 'resetting',
'suspended', -- experimental path only, see decision #3 / correction A
'destroying', 'destroyed', 'error'
);
EXCEPTION
WHEN duplicate_object THEN NULL;
END $$;
CREATE TABLE IF NOT EXISTS environments (
environment_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
student_id TEXT NOT NULL, -- opaque external identifier (correction C) —
-- no FK: students table isn't in this database
-- until a later, separate Platform DB migration.
-- The Provisioner trusts its caller on this value.
lab_host_id TEXT NOT NULL REFERENCES lab_hosts(lab_host_id), -- real FK: both
-- tables are Provisioner-owned Postgres.
status environment_status NOT NULL DEFAULT 'requested',
routing_token TEXT NOT NULL, -- CSPRNG, opaque, unique
container_id TEXT,
container_name TEXT,
network_id TEXT,
network_name TEXT,
network_subnet CIDR, -- needed to install/remove the per-env firewall chain
image_ref TEXT NOT NULL,
cpu_limit NUMERIC NOT NULL,
mem_limit_mb INTEGER NOT NULL,
pids_limit INTEGER NOT NULL,
storage_limit_mb INTEGER NOT NULL,
internal_ip TEXT,
internal_port INTEGER NOT NULL DEFAULT 80,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
started_at TIMESTAMPTZ,
last_active_at TIMESTAMPTZ,
destroy_at TIMESTAMPTZ, -- idle-reaper's scheduled destroy time
error_reason TEXT
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_environments_routing_token ON environments(routing_token);
CREATE INDEX IF NOT EXISTS idx_environments_student ON environments(student_id);
CREATE INDEX IF NOT EXISTS idx_environments_status ON environments(status);
CREATE INDEX IF NOT EXISTS idx_environments_lab_host ON environments(lab_host_id);
-- Phase 1 invariant "at most one non-terminal environment per student" is enforced in
-- application logic (lifecycle.js), not a DB constraint yet — a partial unique index is the
-- natural upgrade once this needs to be bulletproof under concurrent requests, not needed for
-- a single-instance provisioner in this phase.
@@ -0,0 +1,17 @@
-- environment_events — audit trail for every lifecycle transition.
-- See docs/LAB_ARCHITECTURE.md §3/§8 and docs/PHASE1_PLATFORM_FOUNDATION.md §2/§9.
-- Never log secrets or routing tokens in plaintext here (see lifecycle.js emitEvent()).
CREATE TABLE IF NOT EXISTS environment_events (
id BIGSERIAL PRIMARY KEY,
environment_id UUID NOT NULL REFERENCES environments(environment_id),
event_type TEXT NOT NULL, -- provision_requested | host_selected | network_created |
-- firewall_configured | container_created | ready |
-- reset | destroyed | error | reaped_idle | suspend | resume
actor TEXT NOT NULL, -- 'system' | 'instructor:<id>' | 'student:<id>' | 'reaper'
detail JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_env_events_environment ON environment_events(environment_id);
CREATE INDEX IF NOT EXISTS idx_env_events_type ON environment_events(event_type);
+17
View File
@@ -0,0 +1,17 @@
{
"name": "rootnode-provisioner",
"version": "0.1.0",
"description": "RootNode Academy Phase 1 platform provisioner — disposable per-student lab lifecycle.",
"private": true,
"main": "src/server.js",
"scripts": {
"start": "node src/server.js",
"test": "node test/run.js"
},
"dependencies": {
"dockerode": "^4.0.2",
"express": "^4.19.2",
"js-yaml": "^4.1.0",
"pg": "^8.11.5"
}
}
+50
View File
@@ -0,0 +1,50 @@
// Loads the two Phase 1 config files (resource-policy.yaml, lab-hosts.seed.yaml) — plain
// files an operator edits, not database tables, since there's no admin UI to manage them via
// a database yet (see docs/PHASE1_PLATFORM_FOUNDATION.md §2). Reloadable on SIGHUP so a limits
// change doesn't require a restart.
const fs = require('fs');
const path = require('path');
const yaml = require('js-yaml');
const CONFIG_DIR = process.env.PROVISIONER_CONFIG_DIR || path.join(__dirname, '../config');
function loadResourcePolicy() {
const raw = yaml.load(fs.readFileSync(path.join(CONFIG_DIR, 'resource-policy.yaml'), 'utf8'));
return raw;
}
function loadLabHostsSeed() {
const raw = yaml.load(fs.readFileSync(path.join(CONFIG_DIR, 'lab-hosts.seed.yaml'), 'utf8'));
return raw.lab_hosts || [];
}
// Upserts the seed rows into lab_hosts — applied at startup, not exposed as a runtime API
// (no lab-host management endpoints in Phase 1, matching "don't overbuild scheduling yet").
async function applyLabHostsSeed(pool) {
const hosts = loadLabHostsSeed();
for (const h of hosts) {
await pool.query(
`INSERT INTO lab_hosts (lab_host_id, display_name, endpoint, tls_ca_path, tls_cert_path, tls_key_path, enabled)
VALUES ($1, $2, $3, $4, $5, $6, $7)
ON CONFLICT (lab_host_id) DO UPDATE SET
display_name = excluded.display_name, endpoint = excluded.endpoint,
tls_ca_path = excluded.tls_ca_path, tls_cert_path = excluded.tls_cert_path,
tls_key_path = excluded.tls_key_path, enabled = excluded.enabled`,
[h.lab_host_id, h.display_name, h.endpoint, h.tls_ca_path || null, h.tls_cert_path || null, h.tls_key_path || null, h.enabled !== false]
);
}
return hosts.length;
}
let _resourcePolicy = null;
function getResourcePolicy() {
if (!_resourcePolicy) _resourcePolicy = loadResourcePolicy();
return _resourcePolicy;
}
function reloadResourcePolicy() {
_resourcePolicy = loadResourcePolicy();
return _resourcePolicy;
}
module.exports = { loadResourcePolicy, loadLabHostsSeed, applyLabHostsSeed, getResourcePolicy, reloadResourcePolicy };
+61
View File
@@ -0,0 +1,61 @@
// Postgres connection + migration runner for the Platform DB.
//
// Migration pattern matches the one already established in the Lab image's
// backend/server.js (check-before-alter, idempotent) — here it's file-based rather than
// inline, since this is a fresh schema with a real migrations/ directory instead of a single
// growing CREATE TABLE IF NOT EXISTS block.
const fs = require('fs');
const path = require('path');
const { Pool } = require('pg');
const MIGRATIONS_DIR = path.join(__dirname, '../migrations');
let _pool = null;
function getPool() {
if (_pool) return _pool;
_pool = new Pool({
connectionString: process.env.PLATFORM_DB_URL || 'postgres://postgres:postgres@localhost:5433/rootnode_platform',
});
return _pool;
}
// Applies every .sql file in migrations/ that hasn't run yet, in filename order.
// Tracked in schema_migrations so re-running is a no-op — same idempotency guarantee the
// existing SQLite migrations already rely on.
async function runMigrations() {
const pool = getPool();
await pool.query(`
CREATE TABLE IF NOT EXISTS schema_migrations (
version TEXT PRIMARY KEY,
applied_at TIMESTAMPTZ NOT NULL DEFAULT now()
)
`);
const applied = new Set(
(await pool.query('SELECT version FROM schema_migrations')).rows.map(r => r.version)
);
const files = fs.readdirSync(MIGRATIONS_DIR).filter(f => f.endsWith('.sql')).sort();
const ranNow = [];
for (const file of files) {
if (applied.has(file)) continue;
const sql = fs.readFileSync(path.join(MIGRATIONS_DIR, file), 'utf8');
const client = await pool.connect();
try {
await client.query('BEGIN');
await client.query(sql);
await client.query('INSERT INTO schema_migrations (version) VALUES ($1)', [file]);
await client.query('COMMIT');
ranNow.push(file);
} catch (e) {
await client.query('ROLLBACK');
throw new Error(`Migration ${file} failed: ${e.message}`);
} finally {
client.release();
}
}
return ranNow;
}
module.exports = { getPool, runMigrations };
+66
View File
@@ -0,0 +1,66 @@
// Lab-host abstraction (correction D) — resolves a `lab_hosts` row into a connected dockerode
// client. Nothing in this module or its callers may assume Docker is local; the endpoint
// always comes from a `lab_hosts` row, whether that row points at a Unix socket (Phase 1 dev)
// or a remote TCP+TLS endpoint (production Lab Host VM, per docs/LAB_ARCHITECTURE.md §1).
const fs = require('fs');
const Docker = require('dockerode');
// Caches one dockerode client per lab_host_id — cheap to keep around, avoids re-parsing TLS
// material on every call.
const clientCache = new Map();
function buildClient(labHost) {
if (labHost.endpoint.startsWith('unix://')) {
return new Docker({ socketPath: labHost.endpoint.replace('unix://', '') });
}
if (labHost.endpoint.startsWith('tcp://')) {
const url = new URL(labHost.endpoint.replace('tcp://', 'http://'));
const opts = { host: url.hostname, port: Number(url.port) || 2376, protocol: 'https' };
if (labHost.tls_ca_path && labHost.tls_cert_path && labHost.tls_key_path) {
opts.ca = fs.readFileSync(labHost.tls_ca_path);
opts.cert = fs.readFileSync(labHost.tls_cert_path);
opts.key = fs.readFileSync(labHost.tls_key_path);
} else {
// ponytail: no TLS material configured for a remote endpoint is a real misconfiguration,
// not a silently-degraded mode — fail loudly rather than connect unauthenticated to a
// remote Docker API (see docs/PHASE1_PLATFORM_FOUNDATION.md §6 on why that cert matters
// as much as the Provisioner API's own shared secret).
throw new Error(
`lab_host '${labHost.lab_host_id}' has a tcp:// endpoint but no TLS cert/key/ca configured — refusing to connect unauthenticated.`
);
}
return new Docker(opts);
}
throw new Error(`lab_host '${labHost.lab_host_id}' has an unrecognized endpoint scheme: ${labHost.endpoint}`);
}
function clientFor(labHost) {
let client = clientCache.get(labHost.lab_host_id);
if (!client) {
client = buildClient(labHost);
clientCache.set(labHost.lab_host_id, client);
}
return client;
}
// Phase 1: no scheduler, just "the one enabled row" (docs/PHASE1_PLATFORM_FOUNDATION.md §4).
async function selectLabHost(pool) {
const { rows } = await pool.query(
`SELECT * FROM lab_hosts WHERE enabled = true ORDER BY created_at ASC LIMIT 1`
);
if (!rows.length) throw new Error('No enabled lab_hosts row configured — cannot provision.');
return rows[0];
}
async function healthCheck(labHost) {
try {
const client = clientFor(labHost);
const info = await client.info();
return { ok: true, containers: info.Containers, serverVersion: info.ServerVersion };
} catch (e) {
return { ok: false, error: e.message };
}
}
module.exports = { clientFor, selectLabHost, healthCheck };
+102
View File
@@ -0,0 +1,102 @@
// Per-environment firewall chain management — the actual security enforcement for egress
// (correction B: HTTP_PROXY/HTTPS_PROXY are configuration, not the control; this is the
// control). See docs/PHASE1_PLATFORM_FOUNDATION.md §5 for the full design and the ordering
// bug this replaces.
//
// Each environment gets its own dedicated chain, built with -A (append — order-preserving,
// no head-insertion trap), reached from DOCKER-USER by exactly one jump rule matching that
// student's subnet. Requires the process to have NET_ADMIN/NET_RAW and see the host network
// namespace (in production: the Provisioner running on/against the Lab Host; in Phase 1 dev:
// a --network host --cap-add=NET_ADMIN --cap-add=NET_RAW container — see
// platform/provisioner/docker-compose.dev.yml).
const { execFile } = require('child_process');
const { promisify } = require('util');
const execFileAsync = promisify(execFile);
const RFC1918_RANGES = ['10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16'];
function chainName(environmentId) {
// iptables chain names top out around 28 chars on the legacy backend — keep this well
// under that. Deterministic from the environment id so setup/teardown always agree.
const shortId = environmentId.replace(/-/g, '').slice(0, 12);
return `ROOTNODE-ENV-${shortId}`;
}
async function iptables(args) {
try {
return await execFileAsync('iptables', args);
} catch (e) {
throw new Error(`iptables ${args.join(' ')} failed: ${e.stderr || e.message}`);
}
}
// Best-effort — used during teardown/cleanup where "already gone" is a success, not a failure.
async function iptablesIgnoreMissing(args) {
try {
await execFileAsync('iptables', args);
} catch (e) {
// "No chain/target/match by that name" (ENOENT-equivalent for iptables) is fine on cleanup.
if (!/No chain|does not exist|matching rule exist/i.test(e.stderr || '')) {
throw new Error(`iptables ${args.join(' ')} failed: ${e.stderr || e.message}`);
}
}
}
/**
* Installs the egress chain for one environment and hooks it into DOCKER-USER.
* @param {object} opts
* @param {string} opts.environmentId
* @param {string} opts.subnet - e.g. '172.19.0.0/24' (from docker network inspect)
* @param {string} opts.egressProxyHost
* @param {number} opts.egressProxyPort
* @param {string} opts.dnsResolver
* @param {string[]} [opts.extraAllowedHosts] - explicitly-required Platform endpoints, if any
*/
async function setupEgressChain({ environmentId, subnet, egressProxyHost, egressProxyPort, dnsResolver, extraAllowedHosts = [] }) {
const chain = chainName(environmentId);
await iptables(['-N', chain]);
// Required infrastructure — explicitly permitted FIRST, before any blanket deny.
await iptables(['-A', chain, '-d', egressProxyHost, '-p', 'tcp', '--dport', String(egressProxyPort), '-j', 'ACCEPT']);
await iptables(['-A', chain, '-d', dnsResolver, '-p', 'udp', '--dport', '53', '-j', 'ACCEPT']);
await iptables(['-A', chain, '-d', dnsResolver, '-p', 'tcp', '--dport', '53', '-j', 'ACCEPT']);
for (const host of extraAllowedHosts) {
await iptables(['-A', chain, '-d', host, '-j', 'ACCEPT']);
}
// Everything that must never be reachable, regardless of the above — unconditional.
for (const range of RFC1918_RANGES) {
await iptables(['-A', chain, '-d', range, '-j', 'DROP']);
}
await iptables(['-A', chain, '-j', 'DROP']); // default deny — catches the public internet too
// Hook this student's subnet into the chain. -I is intentional here: new environments'
// jump rules should be evaluated ahead of any broader, pre-existing DOCKER-USER rule.
// Since every environment matches a disjoint source subnet, the order of jump rules
// relative to each other never matters.
await iptables(['-I', 'DOCKER-USER', '-s', subnet, '-j', chain]);
return { chain };
}
async function teardownEgressChain({ environmentId, subnet }) {
const chain = chainName(environmentId);
await iptablesIgnoreMissing(['-D', 'DOCKER-USER', '-s', subnet, '-j', chain]);
await iptablesIgnoreMissing(['-F', chain]);
await iptablesIgnoreMissing(['-X', chain]);
return { chain };
}
async function chainExists(environmentId) {
const chain = chainName(environmentId);
try {
await execFileAsync('iptables', ['-L', chain, '-n']);
return true;
} catch {
return false;
}
}
module.exports = { chainName, setupEgressChain, teardownEgressChain, chainExists, RFC1918_RANGES };
+459
View File
@@ -0,0 +1,459 @@
// Environment lifecycle orchestration — the state machine from
// docs/PHASE1_PLATFORM_FOUNDATION.md §3, wired to docker.js (lab-host abstraction), egress.js
// (firewall), and db.js (registry + audit).
//
// Guaranteed path: requested -> provisioning -> running -> resetting -> running
// (provisioning|running|error) -> destroying -> destroyed
// Suspend/resume is experimental only (decision #3 / correction A) — see suspendEnvironment/
// resumeEnvironment below, both explicitly opt-in and never on the guaranteed path.
const crypto = require('crypto');
const http = require('http');
const { selectLabHost, clientFor } = require('./docker');
const { setupEgressChain, teardownEgressChain } = require('./egress');
function shortId(environmentId) {
return environmentId.replace(/-/g, '').slice(0, 12);
}
function generateRoutingToken() {
return crypto.randomBytes(32).toString('base64url'); // opaque, CSPRNG, URL-safe
}
// Never let a routing token reach the audit log in plaintext (requirement #9).
function redact(detail) {
if (!detail || typeof detail !== 'object') return detail;
const copy = { ...detail };
if (copy.routing_token) copy.routing_token = '[redacted]';
return copy;
}
async function emitEvent(pool, environmentId, eventType, actor, detail = null) {
await pool.query(
`INSERT INTO environment_events (environment_id, event_type, actor, detail) VALUES ($1, $2, $3, $4)`,
[environmentId, eventType, actor, detail ? JSON.stringify(redact(detail)) : null]
);
}
async function getEnvironment(pool, environmentId) {
const { rows } = await pool.query('SELECT * FROM environments WHERE environment_id = $1', [environmentId]);
return rows[0] || null;
}
async function listEnvironments(pool, studentId) {
const { rows } = await pool.query(
'SELECT * FROM environments WHERE student_id = $1 ORDER BY created_at DESC',
[studentId]
);
return rows;
}
async function getActiveEnvironmentForStudent(pool, studentId) {
const { rows } = await pool.query(
`SELECT * FROM environments WHERE student_id = $1
AND status NOT IN ('destroyed', 'error') ORDER BY created_at DESC LIMIT 1`,
[studentId]
);
return rows[0] || null;
}
async function pollHealth(ip, port, path, timeoutMs) {
const deadline = Date.now() + timeoutMs;
while (Date.now() < deadline) {
const ok = await new Promise((resolve) => {
const req = http.get({ host: ip, port, path, timeout: 2000 }, (res) => {
res.resume();
resolve(res.statusCode >= 200 && res.statusCode < 300);
});
req.on('error', () => resolve(false));
req.on('timeout', () => { req.destroy(); resolve(false); });
});
if (ok) return true;
await new Promise((r) => setTimeout(r, 2000));
}
return false;
}
/**
* provision — the full requested -> provisioning -> running (or error) path.
* @param {object} opts
* @param {import('pg').Pool} opts.pool
* @param {string} opts.studentId - opaque, not validated against a roster (correction C)
* @param {object} opts.resourcePolicy - { cpuLimit, memLimitMb, pidsLimit, storageLimitMb }
* @param {string} opts.imageRef
* @param {object} opts.egressConfig - { proxyHost, proxyPort, dnsResolver, extraAllowedHosts, noProxyCidrs }
* @param {object} [opts.health] - { path='/api/health', timeoutMs=120000 } — overridable for testing
*/
async function provisionEnvironment({ pool, studentId, resourcePolicy, imageRef, egressConfig, health = {} }) {
const existing = await getActiveEnvironmentForStudent(pool, studentId);
if (existing) {
throw Object.assign(new Error(`student '${studentId}' already has a non-terminal environment (${existing.environment_id})`), { code: 'ALREADY_EXISTS', existing });
}
const labHost = await selectLabHost(pool);
const docker = clientFor(labHost);
const routingToken = generateRoutingToken();
const insertResult = await pool.query(
`INSERT INTO environments
(student_id, lab_host_id, status, routing_token, image_ref, cpu_limit, mem_limit_mb, pids_limit, storage_limit_mb)
VALUES ($1, $2, 'requested', $3, $4, $5, $6, $7, $8)
RETURNING *`,
[studentId, labHost.lab_host_id, routingToken, imageRef,
resourcePolicy.cpuLimit, resourcePolicy.memLimitMb, resourcePolicy.pidsLimit, resourcePolicy.storageLimitMb]
);
const env = insertResult.rows[0];
await emitEvent(pool, env.environment_id, 'provision_requested', 'system', { student_id: studentId });
await emitEvent(pool, env.environment_id, 'host_selected', 'system', { lab_host_id: labHost.lab_host_id });
try {
await pool.query(`UPDATE environments SET status = 'provisioning' WHERE environment_id = $1`, [env.environment_id]);
const id = shortId(env.environment_id);
const networkName = `rootnode-lab-net-${id}`;
const containerName = `rootnode-lab-${id}`;
// Network
const network = await docker.createNetwork({ Name: networkName, Driver: 'bridge' });
const netInfo = await network.inspect();
const subnet = netInfo.IPAM.Config[0].Subnet;
// The egress proxy runs with host networking (see docker-compose.dev.yml), so it's present
// on every bridge network's gateway interface — that gateway IP, not a fixed hostname, is
// what's actually reachable from *this* environment's private network. Resolving it per
// network (rather than one static proxyHost) is what makes the proxy reachable at all once
// there's more than one isolated student network.
const proxyHost = netInfo.IPAM.Config[0].Gateway;
await pool.query(
`UPDATE environments SET network_id = $1, network_name = $2, network_subnet = $3 WHERE environment_id = $4`,
[network.id, networkName, subnet, env.environment_id]
);
await emitEvent(pool, env.environment_id, 'network_created', 'system', { network_name: networkName, subnet });
// Firewall — installed before the container starts, so there's never a window where the
// container has network access but no restriction in place.
await setupEgressChain({
environmentId: env.environment_id,
subnet,
egressProxyHost: proxyHost,
egressProxyPort: egressConfig.proxyPort,
dnsResolver: egressConfig.dnsResolver,
extraAllowedHosts: egressConfig.extraAllowedHosts || [],
});
await emitEvent(pool, env.environment_id, 'firewall_configured', 'system', { subnet });
// Container
const container = await docker.createContainer({
name: containerName,
Image: imageRef,
HostConfig: {
Privileged: true, // required for k3s — see docs/LAB_ARCHITECTURE.md §7, not avoidable
NetworkMode: networkName,
NanoCpus: Math.round(resourcePolicy.cpuLimit * 1e9),
Memory: resourcePolicy.memLimitMb * 1024 * 1024,
MemorySwap: resourcePolicy.memLimitMb * 1024 * 1024, // pinned equal to Memory — no swap
PidsLimit: resourcePolicy.pidsLimit,
PublishAllPorts: false, // no public host port mappings — requirement #5
StorageOpt: resourcePolicy.storageLimitMb ? { size: `${resourcePolicy.storageLimitMb}m` } : undefined,
},
Env: [
`HTTP_PROXY=http://${proxyHost}:${egressConfig.proxyPort}`,
`HTTPS_PROXY=http://${proxyHost}:${egressConfig.proxyPort}`,
`NO_PROXY=localhost,127.0.0.1,${(egressConfig.noProxyCidrs || []).join(',')}`,
],
});
await container.start();
const containerInfo = await container.inspect();
const internalIp = containerInfo.NetworkSettings.Networks[networkName].IPAddress;
await pool.query(
`UPDATE environments SET container_id = $1, container_name = $2, internal_ip = $3, started_at = now() WHERE environment_id = $4`,
[container.id, containerName, internalIp, env.environment_id]
);
await emitEvent(pool, env.environment_id, 'container_created', 'system', { container_name: containerName, internal_ip: internalIp });
// Readiness
const healthPath = health.path || '/api/health';
const timeoutMs = health.timeoutMs || 120000;
const ready = await pollHealth(internalIp, env.internal_port, healthPath, timeoutMs);
if (!ready) {
throw new Error(`environment ${env.environment_id} did not become ready within ${timeoutMs}ms`);
}
const { rows } = await pool.query(
`UPDATE environments SET status = 'running', last_active_at = now() WHERE environment_id = $1 RETURNING *`,
[env.environment_id]
);
await emitEvent(pool, env.environment_id, 'ready', 'system', {});
return rows[0];
} catch (e) {
// Best-effort teardown of whatever got created before the failure — a provisioning
// failure must not leave an orphaned container/network/firewall chain running
// indefinitely. Found by testing: a readiness timeout left a real container and firewall
// chain in place with only the DB row marked 'error', which is exactly the kind of
// resource leak requirement #10 (idle reaper) exists to prevent — except this environment
// never reaches 'running', so it would never have had a last_active_at to age out on
// (see the reaper fix in reaper.js). Cleaning up here directly is more honest than relying
// on the reaper to catch a case it wasn't actually able to catch.
try {
const fresh = await getEnvironment(pool, env.environment_id);
const labHostRes = await pool.query('SELECT * FROM lab_hosts WHERE lab_host_id = $1', [fresh.lab_host_id]);
const docker = clientFor(labHostRes.rows[0]);
if (fresh.container_id) {
await docker.getContainer(fresh.container_id).remove({ force: true }).catch(() => {});
}
if (fresh.network_subnet) {
await teardownEgressChain({ environmentId: fresh.environment_id, subnet: fresh.network_subnet }).catch(() => {});
}
if (fresh.network_id) {
await docker.getNetwork(fresh.network_id).remove().catch(() => {});
}
} catch (cleanupErr) {
// Cleanup itself failing is exactly what the reaper's COALESCE(last_active_at,
// created_at) fallback exists to catch eventually — log and move on, don't mask the
// original provisioning error with a cleanup error.
await emitEvent(pool, env.environment_id, 'error', 'system', { stage: 'provision_failure_cleanup', message: cleanupErr.message }).catch(() => {});
}
await pool.query(
`UPDATE environments SET status = 'error', error_reason = $1 WHERE environment_id = $2`,
[e.message, env.environment_id]
);
await emitEvent(pool, env.environment_id, 'error', 'system', { message: e.message });
throw e;
}
}
/**
* destroy — stop+remove container, remove network+firewall chain, mark destroyed.
* Never touches academic data — that lives entirely outside the Lab (Platform SQLite today,
* see docs/LAB_ARCHITECTURE.md §3) — destroying an environment is purely infrastructure teardown.
*/
async function destroyEnvironment({ pool, environmentId, actor = 'system' }) {
const env = await getEnvironment(pool, environmentId);
if (!env) throw Object.assign(new Error('environment not found'), { code: 'NOT_FOUND' });
if (env.status === 'destroyed') return env;
await pool.query(`UPDATE environments SET status = 'destroying' WHERE environment_id = $1`, [environmentId]);
const labHostRes = await pool.query('SELECT * FROM lab_hosts WHERE lab_host_id = $1', [env.lab_host_id]);
const docker = clientFor(labHostRes.rows[0]);
if (env.container_id) {
try {
const container = docker.getContainer(env.container_id);
await container.stop({ t: 10 }).catch(() => {}); // already stopped/gone is fine
await container.remove({ force: true }).catch(() => {});
} catch (e) {
// Container already gone — proceed with cleanup regardless (failure-safe, per requirement #10).
}
}
if (env.network_subnet) {
await teardownEgressChain({ environmentId: env.environment_id, subnet: env.network_subnet }).catch((e) => {
// Log via event but don't block destroy on firewall cleanup failure — an orphaned
// per-env chain for a destroyed environment is a cleanup task, not a live security hole
// (its subnet is no longer bound to any running container).
return emitEvent(pool, environmentId, 'error', actor, { stage: 'firewall_teardown', message: e.message });
});
}
if (env.network_id) {
try {
await docker.getNetwork(env.network_id).remove();
} catch (e) { /* already gone is fine */ }
}
const { rows } = await pool.query(
`UPDATE environments SET status = 'destroyed' WHERE environment_id = $1 RETURNING *`,
[environmentId]
);
await emitEvent(pool, environmentId, 'destroyed', actor, {});
return rows[0];
}
/**
* reset — destroy + provision again, same environment_id/routing_token (same DB row).
*/
async function resetEnvironment({ pool, environmentId, resourcePolicy, egressConfig, health, actor = 'system' }) {
const env = await getEnvironment(pool, environmentId);
if (!env) throw Object.assign(new Error('environment not found'), { code: 'NOT_FOUND' });
await pool.query(`UPDATE environments SET status = 'resetting' WHERE environment_id = $1`, [environmentId]);
await emitEvent(pool, environmentId, 'reset', actor, {});
const labHostRes = await pool.query('SELECT * FROM lab_hosts WHERE lab_host_id = $1', [env.lab_host_id]);
const docker = clientFor(labHostRes.rows[0]);
if (env.container_id) {
const container = docker.getContainer(env.container_id);
await container.stop({ t: 10 }).catch(() => {});
await container.remove({ force: true }).catch(() => {});
}
if (env.network_subnet) {
await teardownEgressChain({ environmentId: env.environment_id, subnet: env.network_subnet }).catch(() => {});
}
if (env.network_id) {
await docker.getNetwork(env.network_id).remove().catch(() => {});
}
// Recreate against the SAME row: same environment_id/routing_token, fresh everything else.
const id = shortId(env.environment_id);
const networkName = `rootnode-lab-net-${id}`;
const containerName = `rootnode-lab-${id}`;
const network = await docker.createNetwork({ Name: networkName, Driver: 'bridge' });
const netInfo = await network.inspect();
const subnet = netInfo.IPAM.Config[0].Subnet;
const proxyHost = netInfo.IPAM.Config[0].Gateway; // see provisionEnvironment for why this is per-network, not static
await pool.query(
`UPDATE environments SET network_id = $1, network_name = $2, network_subnet = $3, container_id = NULL, container_name = NULL, internal_ip = NULL, error_reason = NULL WHERE environment_id = $4`,
[network.id, networkName, subnet, environmentId]
);
await emitEvent(pool, environmentId, 'network_created', actor, { network_name: networkName, subnet });
await setupEgressChain({
environmentId, subnet,
egressProxyHost: proxyHost, egressProxyPort: egressConfig.proxyPort,
dnsResolver: egressConfig.dnsResolver, extraAllowedHosts: egressConfig.extraAllowedHosts || [],
});
await emitEvent(pool, environmentId, 'firewall_configured', actor, { subnet });
const container = await docker.createContainer({
name: containerName,
Image: env.image_ref,
HostConfig: {
Privileged: true,
NetworkMode: networkName,
NanoCpus: Math.round(Number(env.cpu_limit) * 1e9),
Memory: env.mem_limit_mb * 1024 * 1024,
MemorySwap: env.mem_limit_mb * 1024 * 1024,
PidsLimit: env.pids_limit,
PublishAllPorts: false,
StorageOpt: env.storage_limit_mb ? { size: `${env.storage_limit_mb}m` } : undefined,
},
Env: [
`HTTP_PROXY=http://${proxyHost}:${egressConfig.proxyPort}`,
`HTTPS_PROXY=http://${proxyHost}:${egressConfig.proxyPort}`,
`NO_PROXY=localhost,127.0.0.1,${(egressConfig.noProxyCidrs || []).join(',')}`,
],
});
await container.start();
const containerInfo = await container.inspect();
const internalIp = containerInfo.NetworkSettings.Networks[networkName].IPAddress;
await pool.query(
`UPDATE environments SET container_id = $1, container_name = $2, internal_ip = $3, started_at = now() WHERE environment_id = $4`,
[container.id, containerName, internalIp, environmentId]
);
await emitEvent(pool, environmentId, 'container_created', actor, { container_name: containerName, internal_ip: internalIp });
const healthPath = (health && health.path) || '/api/health';
const timeoutMs = (health && health.timeoutMs) || 120000;
const ready = await pollHealth(internalIp, env.internal_port, healthPath, timeoutMs);
if (!ready) {
await pool.query(`UPDATE environments SET status = 'error', error_reason = $1 WHERE environment_id = $2`,
[`reset: did not become ready within ${timeoutMs}ms`, environmentId]);
await emitEvent(pool, environmentId, 'error', actor, { stage: 'reset_readiness' });
throw new Error(`environment ${environmentId} did not become ready after reset`);
}
const { rows } = await pool.query(
`UPDATE environments SET status = 'running', last_active_at = now() WHERE environment_id = $1 RETURNING *`,
[environmentId]
);
await emitEvent(pool, environmentId, 'ready', actor, {});
return rows[0];
}
// ── Suspend/resume — EXPERIMENTAL ONLY (decision #3 / correction A) ────────────────────────
// Never called by the reaper or any guaranteed-path logic. Exists so the test plan
// (docs/PHASE1_PLATFORM_FOUNDATION.md §8 test 7) can measure whether it's viable at all.
async function suspendEnvironment({ pool, environmentId, actor = 'system' }) {
const env = await getEnvironment(pool, environmentId);
if (!env) throw Object.assign(new Error('environment not found'), { code: 'NOT_FOUND' });
const labHostRes = await pool.query('SELECT * FROM lab_hosts WHERE lab_host_id = $1', [env.lab_host_id]);
const docker = clientFor(labHostRes.rows[0]);
await docker.getContainer(env.container_id).stop({ t: 10 });
const { rows } = await pool.query(`UPDATE environments SET status = 'suspended' WHERE environment_id = $1 RETURNING *`, [environmentId]);
await emitEvent(pool, environmentId, 'suspend', actor, { experimental: true });
return rows[0];
}
async function resumeEnvironment({ pool, environmentId, health = {}, actor = 'system' }) {
const env = await getEnvironment(pool, environmentId);
if (!env) throw Object.assign(new Error('environment not found'), { code: 'NOT_FOUND' });
const labHostRes = await pool.query('SELECT * FROM lab_hosts WHERE lab_host_id = $1', [env.lab_host_id]);
const docker = clientFor(labHostRes.rows[0]);
await docker.getContainer(env.container_id).start();
const healthPath = health.path || '/api/health';
const timeoutMs = health.timeoutMs || 120000;
const ready = await pollHealth(env.internal_ip, env.internal_port, healthPath, timeoutMs);
if (!ready) {
await pool.query(`UPDATE environments SET status = 'error', error_reason = 'resume: did not become ready' WHERE environment_id = $1`, [environmentId]);
await emitEvent(pool, environmentId, 'error', actor, { stage: 'resume_readiness', experimental: true });
throw new Error(`environment ${environmentId} did not become ready after resume (suspend/resume is experimental — see decision #3)`);
}
const { rows } = await pool.query(`UPDATE environments SET status = 'running', last_active_at = now() WHERE environment_id = $1 RETURNING *`, [environmentId]);
await emitEvent(pool, environmentId, 'resume', actor, { experimental: true });
return rows[0];
}
// ── Reconciliation on startup — docs/LAB_ARCHITECTURE.md §8 ────────────────────────────────
// "On Provisioner restart, it must reconcile its registry against actual running containers
// rather than trust stale DB state." Anything the registry claims is running/in-flight that
// isn't actually running on its lab host gets marked error, not silently left wrong.
async function reconcileOnStartup(pool) {
const { rows } = await pool.query(
`SELECT * FROM environments WHERE status IN ('provisioning', 'running', 'resetting', 'destroying', 'suspended')`
);
const results = { checked: rows.length, flagged: 0, ok: 0 };
for (const env of rows) {
try {
const labHostRes = await pool.query('SELECT * FROM lab_hosts WHERE lab_host_id = $1', [env.lab_host_id]);
if (!labHostRes.rows.length) {
await pool.query(`UPDATE environments SET status = 'error', error_reason = $1 WHERE environment_id = $2`,
['reconcile: lab_host no longer configured', env.environment_id]);
await emitEvent(pool, env.environment_id, 'error', 'system', { stage: 'reconcile', reason: 'lab_host missing' });
results.flagged++;
continue;
}
const docker = clientFor(labHostRes.rows[0]);
let actuallyRunning = false;
if (env.container_id) {
try {
const info = await docker.getContainer(env.container_id).inspect();
actuallyRunning = info.State && info.State.Running;
} catch {
actuallyRunning = false; // container doesn't exist on this host at all
}
}
const expectedRunning = env.status === 'running' || env.status === 'suspended';
if (expectedRunning && !actuallyRunning) {
await pool.query(`UPDATE environments SET status = 'error', error_reason = $1 WHERE environment_id = $2`,
['reconcile: DB claimed running/suspended but container is not actually running', env.environment_id]);
await emitEvent(pool, env.environment_id, 'error', 'system', { stage: 'reconcile', reason: 'container not running' });
results.flagged++;
} else if (['provisioning', 'resetting', 'destroying'].includes(env.status)) {
// Any environment caught mid-transition by a Provisioner crash/restart is not
// trustworthy either way — never silently assume it finished.
await pool.query(`UPDATE environments SET status = 'error', error_reason = $1 WHERE environment_id = $2`,
[`reconcile: caught mid-transition (${env.status}) on restart`, env.environment_id]);
await emitEvent(pool, env.environment_id, 'error', 'system', { stage: 'reconcile', reason: `mid-transition:${env.status}` });
results.flagged++;
} else {
results.ok++;
}
} catch (e) {
await emitEvent(pool, env.environment_id, 'error', 'system', { stage: 'reconcile', message: e.message }).catch(() => {});
results.flagged++;
}
}
return results;
}
module.exports = {
generateRoutingToken, emitEvent, getEnvironment, listEnvironments, getActiveEnvironmentForStudent,
provisionEnvironment, destroyEnvironment, resetEnvironment,
suspendEnvironment, resumeEnvironment, pollHealth, shortId, reconcileOnStartup,
};
+71
View File
@@ -0,0 +1,71 @@
// Idle reaper — requirement #10. Without this, "ephemeral by default" is just a description:
// a closed browser tab does not stop a running k3s cluster on its own.
//
// Two thresholds, both config-driven (never hard-coded):
// - idleThresholdMinutes — past this, log a warning event only (future: notify student).
// - abandonedThresholdMinutes — past this, destroy for real via the same path DELETE
// /environments/:id uses, so cleanup is identical either way.
//
// Uses COALESCE(last_active_at, created_at): an environment that never reached 'running' has
// a NULL last_active_at, and `NULL < ...` is never true in SQL — without this fallback, an
// environment stuck in 'error' from a failed provision would never age out here at all. This
// is a second, independent safety net on top of provisionEnvironment's own best-effort cleanup
// on failure (lifecycle.js) — found by testing, not designed in from the start.
//
// Failure-safe: one environment's teardown throwing never stops the reaper from checking the
// rest, and a Postgres/Docker hiccup on one tick just means "try again next tick" rather than
// crashing the process.
const { destroyEnvironment, emitEvent } = require('./lifecycle');
function startReaper({ pool, idleThresholdMinutes, abandonedThresholdMinutes, pollIntervalSeconds }) {
let stopped = false;
let timer = null;
async function tick() {
if (stopped) return;
try {
const idleRows = await pool.query(
`SELECT environment_id FROM environments
WHERE status = 'running'
AND COALESCE(last_active_at, created_at) < now() - ($1 || ' minutes')::interval
AND COALESCE(last_active_at, created_at) >= now() - ($2 || ' minutes')::interval`,
[String(idleThresholdMinutes), String(abandonedThresholdMinutes)]
);
for (const row of idleRows.rows) {
await emitEvent(pool, row.environment_id, 'idle_warning', 'reaper', { idleThresholdMinutes }).catch(() => {});
}
const abandonedRows = await pool.query(
`SELECT environment_id FROM environments
WHERE status IN ('running', 'error')
AND COALESCE(last_active_at, created_at) < now() - ($1 || ' minutes')::interval`,
[String(abandonedThresholdMinutes)]
);
for (const row of abandonedRows.rows) {
try {
await destroyEnvironment({ pool, environmentId: row.environment_id, actor: 'reaper' });
await emitEvent(pool, row.environment_id, 'reaped_idle', 'reaper', { abandonedThresholdMinutes });
} catch (e) {
// One bad environment must not stop the reaper from handling the rest.
await emitEvent(pool, row.environment_id, 'error', 'reaper', { stage: 'reap', message: e.message }).catch(() => {});
}
}
} catch (e) {
// Postgres/Docker unreachable this tick — try again next tick, don't crash the process.
console.error('[reaper] tick failed:', e.message);
} finally {
if (!stopped) timer = setTimeout(tick, pollIntervalSeconds * 1000);
}
}
timer = setTimeout(tick, 0);
return {
stop() {
stopped = true;
if (timer) clearTimeout(timer);
},
};
}
module.exports = { startReaper };
+214
View File
@@ -0,0 +1,214 @@
// RootNode Academy — Phase 1 Provisioner API.
// Internal-only (requirement #3/#8) — every route below requires X-Provisioner-Key. Nothing
// here is reachable by a student or the public internet in this phase; there's no frontend/
// Gateway wiring yet, so this is exercised directly (curl / the test suite in test/run.js).
//
// See docs/PHASE1_PLATFORM_FOUNDATION.md §4 for the approved route list this implements.
const express = require('express');
const { randomUUID } = require('crypto');
const { getPool, runMigrations } = require('./db');
const { applyLabHostsSeed, getResourcePolicy, reloadResourcePolicy } = require('./config');
const { healthCheck: dockerHealthCheck } = require('./docker');
const lifecycle = require('./lifecycle');
const { startReaper } = require('./reaper');
const PORT = Number(process.env.PROVISIONER_PORT) || 4100;
const SHARED_SECRET = process.env.PROVISIONER_SHARED_SECRET;
const LAB_IMAGE_REF = process.env.LAB_IMAGE_REF || 'rootnode-academy:test';
// No fixed EGRESS_PROXY_HOST: the proxy runs with host networking and is reached via each
// environment's own network gateway IP, resolved per-environment in lifecycle.js — a single
// static hostname wouldn't be reachable from every isolated student network. Only the port
// is fixed/shared.
const EGRESS_PROXY_PORT = Number(process.env.EGRESS_PROXY_PORT) || 3128;
const DNS_RESOLVER = process.env.DNS_RESOLVER || '127.0.0.11'; // Docker's embedded DNS by default
const NO_PROXY_CIDRS = (process.env.NO_PROXY_CIDRS || '10.42.0.0/16,10.43.0.0/16').split(',');
if (!SHARED_SECRET) {
console.error('FATAL: PROVISIONER_SHARED_SECRET is not set. Refusing to start with an unauthenticated internal API.');
process.exit(1);
}
function egressConfig() {
return {
proxyPort: EGRESS_PROXY_PORT,
dnsResolver: DNS_RESOLVER,
extraAllowedHosts: [], // none required in Phase 1 — the Lab image doesn't call back to the Platform yet
noProxyCidrs: NO_PROXY_CIDRS,
};
}
function resourcePolicyFor() {
const policy = getResourcePolicy().default;
return {
cpuLimit: policy.cpu_limit,
memLimitMb: policy.mem_limit_mb,
pidsLimit: policy.pids_limit,
storageLimitMb: policy.storage_limit_mb,
};
}
function healthOpts() {
const hc = getResourcePolicy().health_check || {};
return { path: hc.path || '/api/health', timeoutMs: hc.timeout_ms || 120000 };
}
const app = express();
app.use(express.json());
// Every request gets a request id for structured error correlation, and every response is
// JSON — no HTML error pages, no leaking stack traces to a caller.
app.use((req, res, next) => {
req.requestId = randomUUID();
next();
});
function requireAuth(req, res, next) {
const key = req.get('X-Provisioner-Key');
if (!key || key !== SHARED_SECRET) {
return res.status(401).json({ error: 'unauthorized', request_id: req.requestId });
}
next();
}
function isValidStudentId(v) {
return typeof v === 'string' && v.length > 0 && v.length <= 128 && /^[a-zA-Z0-9_.-]+$/.test(v);
}
function isValidUUID(v) {
return typeof v === 'string' && /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(v);
}
function asyncRoute(fn) {
return (req, res) => fn(req, res).catch((e) => {
console.error(`[${req.requestId}]`, e);
const status = e.code === 'NOT_FOUND' ? 404 : e.code === 'ALREADY_EXISTS' ? 409 : 500;
res.status(status).json({ error: e.message, code: e.code || 'INTERNAL', request_id: req.requestId });
});
}
app.use(requireAuth);
// GET /health — Provisioner's own health: Docker reachable, Postgres reachable.
app.get('/health', asyncRoute(async (req, res) => {
const pool = getPool();
const dbOk = await pool.query('SELECT 1').then(() => true).catch(() => false);
const labHosts = await pool.query('SELECT * FROM lab_hosts WHERE enabled = true').catch(() => ({ rows: [] }));
const hostChecks = await Promise.all(labHosts.rows.map(async (h) => ({ lab_host_id: h.lab_host_id, ...(await dockerHealthCheck(h)) })));
const ok = dbOk && hostChecks.every((h) => h.ok);
res.status(ok ? 200 : 503).json({ ok, db: dbOk, lab_hosts: hostChecks });
}));
// POST /environments — { student_id }
app.post('/environments', asyncRoute(async (req, res) => {
const { student_id } = req.body || {};
if (!isValidStudentId(student_id)) {
return res.status(400).json({ error: 'student_id is required and must be a simple identifier', request_id: req.requestId });
}
const env = await lifecycle.provisionEnvironment({
pool: getPool(),
studentId: student_id,
resourcePolicy: resourcePolicyFor(),
imageRef: LAB_IMAGE_REF,
egressConfig: egressConfig(),
health: healthOpts(),
});
res.status(201).json({ environment_id: env.environment_id, status: env.status, routing_token: env.routing_token });
}));
// GET /environments?student_id=
app.get('/environments', asyncRoute(async (req, res) => {
const { student_id } = req.query;
if (!isValidStudentId(student_id)) {
return res.status(400).json({ error: 'student_id query param is required', request_id: req.requestId });
}
const envs = await lifecycle.listEnvironments(getPool(), student_id);
res.json(envs.map(redactRow));
}));
// GET /environments/:id
app.get('/environments/:id', asyncRoute(async (req, res) => {
if (!isValidUUID(req.params.id)) return res.status(400).json({ error: 'invalid environment id', request_id: req.requestId });
const env = await lifecycle.getEnvironment(getPool(), req.params.id);
if (!env) return res.status(404).json({ error: 'not found', request_id: req.requestId });
res.json(redactRow(env));
}));
// POST /environments/:id/reset
app.post('/environments/:id/reset', asyncRoute(async (req, res) => {
if (!isValidUUID(req.params.id)) return res.status(400).json({ error: 'invalid environment id', request_id: req.requestId });
const env = await lifecycle.resetEnvironment({
pool: getPool(), environmentId: req.params.id,
resourcePolicy: resourcePolicyFor(), egressConfig: egressConfig(), health: healthOpts(),
actor: req.get('X-Actor') || 'system',
});
res.json(redactRow(env));
}));
// DELETE /environments/:id
app.delete('/environments/:id', asyncRoute(async (req, res) => {
if (!isValidUUID(req.params.id)) return res.status(400).json({ error: 'invalid environment id', request_id: req.requestId });
const env = await lifecycle.destroyEnvironment({ pool: getPool(), environmentId: req.params.id, actor: req.get('X-Actor') || 'system' });
res.json(redactRow(env));
}));
// POST /environments/:id/suspend — EXPERIMENTAL, see decision #3 / correction A.
app.post('/environments/:id/suspend', asyncRoute(async (req, res) => {
if (!isValidUUID(req.params.id)) return res.status(400).json({ error: 'invalid environment id', request_id: req.requestId });
const env = await lifecycle.suspendEnvironment({ pool: getPool(), environmentId: req.params.id, actor: req.get('X-Actor') || 'system' });
res.json({ ...redactRow(env), experimental: true });
}));
// POST /environments/:id/resume — EXPERIMENTAL, see decision #3 / correction A.
app.post('/environments/:id/resume', asyncRoute(async (req, res) => {
if (!isValidUUID(req.params.id)) return res.status(400).json({ error: 'invalid environment id', request_id: req.requestId });
const env = await lifecycle.resumeEnvironment({ pool: getPool(), environmentId: req.params.id, health: healthOpts(), actor: req.get('X-Actor') || 'system' });
res.json({ ...redactRow(env), experimental: true });
}));
// GET /environments/:id/events
app.get('/environments/:id/events', asyncRoute(async (req, res) => {
if (!isValidUUID(req.params.id)) return res.status(400).json({ error: 'invalid environment id', request_id: req.requestId });
const { rows } = await getPool().query(
'SELECT id, event_type, actor, detail, created_at FROM environment_events WHERE environment_id = $1 ORDER BY created_at ASC',
[req.params.id]
);
res.json(rows);
}));
function redactRow(env) {
const { routing_token, ...rest } = env;
return { ...rest, routing_token: routing_token ? '[redacted]' : null };
}
process.on('SIGHUP', () => {
console.log('SIGHUP received, reloading resource-policy.yaml');
try { reloadResourcePolicy(); } catch (e) { console.error('reload failed:', e.message); }
});
async function main() {
const pool = getPool();
const applied = await runMigrations();
if (applied.length) console.log('Applied migrations:', applied.join(', '));
const seeded = await applyLabHostsSeed(pool);
console.log(`Applied lab_hosts seed (${seeded} row(s))`);
const reconcileResult = await lifecycle.reconcileOnStartup(pool);
console.log(`Startup reconciliation: checked=${reconcileResult.checked} ok=${reconcileResult.ok} flagged=${reconcileResult.flagged}`);
const policy = getResourcePolicy();
const reaper = startReaper({
pool,
idleThresholdMinutes: policy.idle_threshold_minutes,
abandonedThresholdMinutes: policy.abandoned_threshold_minutes,
pollIntervalSeconds: policy.reaper_poll_interval_seconds,
});
process.on('SIGTERM', () => { reaper.stop(); process.exit(0); });
app.listen(PORT, () => console.log(`Provisioner API listening on :${PORT}`));
}
if (require.main === module) {
main().catch((e) => { console.error('FATAL:', e); process.exit(1); });
}
module.exports = { app, main };
+198
View File
@@ -0,0 +1,198 @@
#!/usr/bin/env node
// Phase 1 integration test suite — runs against a live docker-compose.dev.yml stack.
// Mirrors the manual verification run during implementation (see docs/PHASE1_PLATFORM_FOUNDATION.md
// §8 and the final report in the Phase 1 PR/commit history). No framework — assert-based, per
// project convention (ponytail: no test frameworks unless the project already has one; this
// one doesn't).
//
// Usage:
// cd platform/provisioner
// docker compose -f docker-compose.dev.yml up -d
// PROVISIONER_SHARED_SECRET=<from .env> LAB_IMAGE_REF=rootnode-academy:test node test/run.js
//
// Defaults to the lightweight fake-lab test image (LAB_IMAGE_REF) for speed — the mechanics
// under test here (network/container/firewall/registry lifecycle) don't depend on what's
// running inside the container. The real Lab image (k3s) was validated separately by hand —
// see the "cannot be fully automated here" note at the bottom of this file.
const assert = require('assert');
const { execFile } = require('child_process');
const { promisify } = require('util');
const execFileAsync = promisify(execFile);
const BASE = process.env.PROVISIONER_URL || 'http://localhost:4100';
const KEY = process.env.PROVISIONER_SHARED_SECRET;
if (!KEY) { console.error('Set PROVISIONER_SHARED_SECRET'); process.exit(1); }
async function api(method, path, body) {
const res = await fetch(`${BASE}${path}`, {
method,
headers: { 'Content-Type': 'application/json', 'X-Provisioner-Key': KEY },
body: body ? JSON.stringify(body) : undefined,
});
const text = await res.text();
let json;
try { json = JSON.parse(text); } catch { json = text; }
return { status: res.status, body: json };
}
async function iptablesRules(chain) {
try {
const { stdout } = await execFileAsync('docker', [
'run', '--rm', '--network', 'host', '--cap-add=NET_ADMIN', '--cap-add=NET_RAW', 'alpine',
'sh', '-c', `apk add -q iptables >/dev/null 2>&1; iptables -S ${chain} 2>&1`,
]);
return stdout;
} catch (e) { return e.stdout || e.message; }
}
let passed = 0, failed = 0;
async function test(name, fn) {
try {
await fn();
console.log(` ok — ${name}`);
passed++;
} catch (e) {
console.error(` FAIL — ${name}\n ${e.message}`);
failed++;
}
}
async function main() {
console.log('Provisioner integration tests\n');
// 1. Health
await test('GET /health reports ok with db and lab_hosts reachable', async () => {
const { status, body } = await api('GET', '/health');
assert.equal(status, 200);
assert.equal(body.ok, true);
assert.equal(body.db, true);
assert.ok(body.lab_hosts.length >= 1);
});
// 2. Auth
await test('unauthenticated request is rejected', async () => {
const res = await fetch(`${BASE}/health`);
assert.equal(res.status, 401);
});
// 3. Provision -> running, unique network/container/token
let envA, envB;
await test('provision creates a running environment with a unique routing token', async () => {
const { status, body } = await api('POST', '/environments', { student_id: 'test-student-A' });
assert.equal(status, 201);
assert.equal(body.status, 'running');
assert.ok(body.routing_token && body.routing_token.length >= 32);
envA = body;
});
await test('a second student gets a distinct environment/network/token', async () => {
const { status, body } = await api('POST', '/environments', { student_id: 'test-student-B' });
assert.equal(status, 201);
assert.notEqual(body.environment_id, envA.environment_id);
assert.notEqual(body.routing_token, envA.routing_token);
envB = body;
});
await test('duplicate provision for the same student is rejected (409)', async () => {
const { status } = await api('POST', '/environments', { student_id: 'test-student-A' });
assert.equal(status, 409);
});
await test('environments have distinct internal IPs on distinct subnets', async () => {
const a = (await api('GET', `/environments/${envA.environment_id}`)).body;
const b = (await api('GET', `/environments/${envB.environment_id}`)).body;
assert.notEqual(a.internal_ip, b.internal_ip);
assert.notEqual(a.network_subnet, b.network_subnet);
});
// 4. Egress firewall ordering — the exact regression the original -I bug would have caused.
await test('firewall chain permits the (RFC1918) proxy address before the RFC1918 blanket drop', async () => {
const a = (await api('GET', `/environments/${envA.environment_id}`)).body;
const shortId = envA.environment_id.replace(/-/g, '').slice(0, 12);
const rules = await iptablesRules(`ROOTNODE-ENV-${shortId}`);
const lines = rules.trim().split('\n').filter((l) => l.startsWith('-A'));
const acceptIdx = lines.findIndex((l) => l.includes('ACCEPT') && l.includes('3128'));
const dropIdx = lines.findIndex((l) => l.includes('DROP') && l.includes('172.16.0.0/12'));
assert.ok(acceptIdx >= 0 && dropIdx >= 0, 'expected both an ACCEPT-to-proxy and a DROP-172.16 rule');
assert.ok(acceptIdx < dropIdx, 'ACCEPT must come before DROP — this is the bug the redesign fixed');
});
// 5. Isolation — the core security property.
await test('student A cannot reach student B (student-to-student isolation)', async () => {
const a = (await api('GET', `/environments/${envA.environment_id}`)).body;
const b = (await api('GET', `/environments/${envB.environment_id}`)).body;
const shortA = envA.environment_id.replace(/-/g, '').slice(0, 12);
try {
await execFileAsync('docker', ['exec', `rootnode-lab-${shortA}`, 'node', '-e',
`const h=require('http');const r=h.get({host:'${b.internal_ip}',port:80,path:'/api/health',timeout:2000},()=>process.exit(1));r.on('error',()=>process.exit(0));r.on('timeout',()=>{r.destroy();process.exit(0)});`,
]);
} catch (e) {
// execFile throws on non-zero exit; our script exits 0 when correctly blocked.
if (e.code !== 0) throw new Error(`expected blocked (exit 0), got exit ${e.code}`);
}
});
await test('student A cannot reach an arbitrary RFC1918 address (simulated trusted infra)', async () => {
const shortA = envA.environment_id.replace(/-/g, '').slice(0, 12);
try {
await execFileAsync('docker', ['exec', `rootnode-lab-${shortA}`, 'node', '-e',
`const h=require('http');const r=h.get({host:'10.1.2.3',port:80,path:'/',timeout:2000},()=>process.exit(1));r.on('error',()=>process.exit(0));r.on('timeout',()=>{r.destroy();process.exit(0)});`,
]);
} catch (e) {
if (e.code !== 0) throw new Error(`expected blocked (exit 0), got exit ${e.code}`);
}
});
// 6. Reset — same identity, fresh container.
await test('reset preserves environment_id/routing_token but recreates the container', async () => {
const before = (await api('GET', `/environments/${envA.environment_id}`)).body;
const { status, body } = await api('POST', `/environments/${envA.environment_id}/reset`);
assert.equal(status, 200);
assert.equal(body.environment_id, envA.environment_id);
assert.notEqual(body.container_id, before.container_id);
});
// 7. Destroy — deterministic cleanup.
await test('destroy removes container, network, and firewall chain', async () => {
const shortB = envB.environment_id.replace(/-/g, '').slice(0, 12);
const { status, body } = await api('DELETE', `/environments/${envB.environment_id}`);
assert.equal(status, 200);
assert.equal(body.status, 'destroyed');
const rules = await iptablesRules(`ROOTNODE-ENV-${shortB}`);
assert.ok(/No chain|does not exist/i.test(rules), 'firewall chain should be gone after destroy');
});
// 8. Events — full audit trail, no secrets.
await test('environment_events records the lifecycle and never contains the routing token', async () => {
const { body: events } = await api('GET', `/environments/${envA.environment_id}/events`);
const types = events.map((e) => e.event_type);
for (const expected of ['provision_requested', 'host_selected', 'network_created', 'firewall_configured', 'container_created', 'ready']) {
assert.ok(types.includes(expected), `missing event: ${expected}`);
}
const serialized = JSON.stringify(events);
assert.ok(!serialized.includes(envA.routing_token) || envA.routing_token === undefined, 'routing token must never appear in event details');
});
// Cleanup
await api('DELETE', `/environments/${envA.environment_id}`);
console.log(`\n${passed} passed, ${failed} failed`);
process.exit(failed ? 1 : 0);
}
main().catch((e) => { console.error('Test run crashed:', e); process.exit(1); });
// ── Not exercised by this script — see the Phase 1 final report for why ──────────────────
// - Idle reaper timing (validated manually with a backdated last_active_at + real 60s wait;
// automating that here would just make the suite slow for no extra confidence).
// - Suspend/resume against the real Lab image / k3s specifically (experimental, decision #3
// — validated manually; correctness never depends on this).
// - Remote TCP+TLS lab_host endpoint (validated manually against a Docker-in-Docker sidecar;
// not wired into this script to avoid a heavy, flaky dependency in routine test runs).
// - Reconciliation-on-restart (validated manually by killing a container out-of-band and
// restarting the Provisioner; requires an out-of-band kill + process restart that doesn't
// fit a single-process test script cleanly).
// - Anything requiring the real Lab Host VM / Proxmox topology — see
// docs/PHASE1_PLATFORM_FOUNDATION.md §8 tests 5, 7, 9, 12 for what's explicitly documented
// as untestable on a single dev machine rather than faked.
+15
View File
@@ -53,6 +53,7 @@ Scenarios are defined as JSON objects. A scenario can be either a hands-on conso
"id": "unique-kebab-case-id", // string — unique scenario identifier
"title": "Human-readable Title", // string — shown in sidebar list
"category": "Workloads", // string — groups scenarios in sidebar accordion
"isolation_type": "shared_namespace", // "shared_namespace" | "ephemeral_cluster" — see below
"difficulty": "Easy", // "Easy" | "Medium" | "Hard"
"type": "task", // "task" | "mcq"
"weight": 7, // number — points value (used for final grade scoring)
@@ -66,6 +67,18 @@ Scenarios are defined as JSON objects. A scenario can be either a hands-on conso
---
### `isolation_type` — informational, not a security boundary
Earlier draft of this schema used `isolation_type` (`shared_namespace` | `ephemeral_cluster`)
to decide which scenarios could safely run under shared-cluster RBAC. That namespace-isolation
approach was superseded before it shipped — see `docs/LAB_ARCHITECTURE.md`. Every student now
gets a fully separate disposable environment with its own nested cluster, so no scenario
content is unsafe for any other student's environment; the field no longer gates anything.
It's kept on scenario records as descriptive metadata only (which scenarios exercise
cluster-scoped/node-level operations), not read by any authorization logic.
---
### Hints Schema
Each hint is rendered as a collapsible card inside the Hints tab of the UI:
```jsonc
@@ -149,6 +162,7 @@ Renders a questionnaire block. No terminal is shown. The user answers by selecti
"id": "scale-deployment",
"title": "Scale a Deployment",
"category": "Workloads",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 4,
@@ -187,6 +201,7 @@ Renders a questionnaire block. No terminal is shown. The user answers by selecti
"id": "service-types-mcq",
"title": "Kubernetes Service Types",
"category": "Networking",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "mcq",
"weight": 3,
@@ -2,6 +2,7 @@
"id": "annotate-resource-basics",
"title": "Annotating Kubernetes Resources",
"category": "Core Concepts",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 3,
+1
View File
@@ -2,6 +2,7 @@
"id": "broken-deployment",
"title": "Debug a Failing Deployment",
"category": "Troubleshooting",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "task",
"weight": 8,
+1
View File
@@ -2,6 +2,7 @@
"id": "cks-apparmor-profile",
"title": "AppArmor Profiles",
"category": "Workload Security",
"isolation_type": "ephemeral_cluster",
"difficulty": "Hard",
"type": "task",
"weight": 5,
+1
View File
@@ -2,6 +2,7 @@
"id": "cks-audit-policy",
"title": "Write a Kubernetes Audit Policy",
"category": "Cluster Setup",
"isolation_type": "ephemeral_cluster",
"difficulty": "Hard",
"type": "task",
"weight": 7,
+1
View File
@@ -2,6 +2,7 @@
"id": "cks-automount-token",
"title": "Disable ServiceAccount Token",
"category": "Workload Security",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 3,
@@ -2,6 +2,7 @@
"id": "cks-drop-capabilities",
"title": "Drop Linux Capabilities",
"category": "Workload Security",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "task",
"weight": 5,
+1
View File
@@ -2,6 +2,7 @@
"id": "cks-egress-namespace",
"title": "Restrict Egress to Namespace",
"category": "Network Security",
"isolation_type": "shared_namespace",
"difficulty": "Hard",
"type": "task",
"weight": 6,
@@ -2,6 +2,7 @@
"id": "cks-image-pull-secret",
"title": "Private Registry Image Pull Secret",
"category": "Supply Chain Security",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 4,
+1
View File
@@ -2,6 +2,7 @@
"id": "cks-immutable-secret",
"title": "Immutable Secrets",
"category": "Cluster Security",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 3,
+1
View File
@@ -2,6 +2,7 @@
"id": "cks-mcq-api-server",
"title": "API Server Authentication",
"category": "Cluster Setup",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "mcq",
"weight": 2,
+1
View File
@@ -2,6 +2,7 @@
"id": "cks-mcq-audit-policy",
"title": "Kubernetes Audit Logging Levels",
"category": "Cluster Setup",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "mcq",
"weight": 3,
@@ -2,6 +2,7 @@
"id": "cks-mcq-image-footprint",
"title": "Minimizing Image Footprint",
"category": "Supply Chain Security",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "mcq",
"weight": 2,
+1
View File
@@ -2,6 +2,7 @@
"id": "cks-mcq-kubelet-auth",
"title": "Securing the Kubelet",
"category": "Cluster Setup",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "mcq",
"weight": 2,
@@ -2,6 +2,7 @@
"id": "cks-mcq-psp-replacement",
"title": "Pod Security Evolution",
"category": "Cluster Security",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "mcq",
"weight": 2,
@@ -2,6 +2,7 @@
"id": "cks-mcq-runtime-security",
"title": "Runtime Security Tools",
"category": "System Hardening",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "mcq",
"weight": 2,
+1
View File
@@ -2,6 +2,7 @@
"id": "cks-mcq-sandboxing",
"title": "Container Runtime Sandboxing",
"category": "System Hardening",
"isolation_type": "shared_namespace",
"difficulty": "Hard",
"type": "mcq",
"weight": 3,
@@ -2,6 +2,7 @@
"id": "cks-network-policy-metadata",
"title": "Block Cloud Metadata",
"category": "Network Security",
"isolation_type": "shared_namespace",
"difficulty": "Hard",
"type": "task",
"weight": 6,
+1
View File
@@ -2,6 +2,7 @@
"id": "cks-network-policy",
"title": "Default Deny Network Policy",
"category": "Network Security",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "task",
"weight": 5,
+1
View File
@@ -2,6 +2,7 @@
"id": "cks-non-root-enforce",
"title": "Enforce Non-Root Container",
"category": "Workload Security",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "task",
"weight": 5,
@@ -2,6 +2,7 @@
"id": "cks-pod-security-context",
"title": "Pod Security Context",
"category": "Workload Security",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "task",
"weight": 5,
+1
View File
@@ -2,6 +2,7 @@
"id": "cks-psa-namespace",
"title": "Pod Security Admission",
"category": "Cluster Security",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 4,
+1
View File
@@ -2,6 +2,7 @@
"id": "cks-rbac-clusterrole",
"title": "Cluster-Level RBAC",
"category": "Cluster Security",
"isolation_type": "ephemeral_cluster",
"difficulty": "Medium",
"type": "task",
"weight": 5,
@@ -2,6 +2,7 @@
"id": "cks-rbac-least-privilege",
"title": "RBAC Least Privilege",
"category": "Cluster Security",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "task",
"weight": 4,
@@ -2,6 +2,7 @@
"id": "cks-readonly-filesystem",
"title": "Read-Only Root Filesystem",
"category": "Workload Security",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 4,
+1
View File
@@ -2,6 +2,7 @@
"id": "cks-runtime-class",
"title": "RuntimeClass for Sandboxed Workloads",
"category": "System Hardening",
"isolation_type": "ephemeral_cluster",
"difficulty": "Medium",
"type": "task",
"weight": 5,
+1
View File
@@ -2,6 +2,7 @@
"id": "cks-seccomp-profile",
"title": "Seccomp Profiles",
"category": "Workload Security",
"isolation_type": "shared_namespace",
"difficulty": "Hard",
"type": "task",
"weight": 6,
+1
View File
@@ -2,6 +2,7 @@
"id": "cks-tls-ingress",
"title": "Secure Ingress with TLS",
"category": "Network Security",
"isolation_type": "shared_namespace",
"difficulty": "Hard",
"type": "task",
"weight": 7,
+1
View File
@@ -2,6 +2,7 @@
"id": "cluster-upgrade-mcq",
"title": "Cluster Upgrade Order",
"category": "Cluster Administration",
"isolation_type": "shared_namespace",
"difficulty": "Hard",
"type": "mcq",
"weight": 5,
+1
View File
@@ -2,6 +2,7 @@
"id": "configmap-volume",
"title": "Mount ConfigMap as Volume",
"category": "Configuration",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "task",
"weight": 7,
@@ -2,6 +2,7 @@
"id": "container-logging-mcq",
"title": "Container Logging and Monitoring",
"category": "Troubleshooting",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "mcq",
"weight": 3,
+1
View File
@@ -2,6 +2,7 @@
"id": "crashloop-fix",
"title": "Fix a CrashLoopBackOff Pod",
"category": "Troubleshooting",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "task",
"weight": 7,
+1
View File
@@ -2,6 +2,7 @@
"id": "cronjob-task",
"title": "Create a CronJob",
"category": "Workloads & Scheduling",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 4,
+1
View File
@@ -2,6 +2,7 @@
"id": "daemonset-mcq",
"title": "Understanding DaemonSets",
"category": "Workloads",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "mcq",
"weight": 3,
@@ -2,6 +2,7 @@
"id": "delete-by-label-basics",
"title": "Bulk Deletion by Label",
"category": "Core Concepts",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 3,
+1
View File
@@ -2,6 +2,7 @@
"id": "deploy-nginx",
"title": "Deploy and Expose Nginx",
"category": "Workloads",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 5,
+1
View File
@@ -2,6 +2,7 @@
"id": "deployment-rollback",
"title": "Rolling Back a Deployment",
"category": "Workloads & Scheduling",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 5,
@@ -2,6 +2,7 @@
"id": "deployment-strategy-mcq",
"title": "Deployment Update Strategies",
"category": "Workloads",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "mcq",
"weight": 3,
+1
View File
@@ -2,6 +2,7 @@
"id": "dns-resolution-mcq",
"title": "Kubernetes DNS Resolution",
"category": "Services & Networking",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "mcq",
"weight": 3,
@@ -2,6 +2,7 @@
"id": "dry-run-manifest-basics",
"title": "Generating YAML Manifests",
"category": "Core Concepts",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 4,
@@ -2,6 +2,7 @@
"id": "edit-deployment-basics",
"title": "Updating Deployments",
"category": "Workloads",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 4,
+1
View File
@@ -2,6 +2,7 @@
"id": "emptydir-pod",
"title": "Shared emptyDir Volume Between Containers",
"category": "Storage",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 5,
+1
View File
@@ -2,6 +2,7 @@
"id": "endpoint-fix-task",
"title": "Fix a Broken Service Selector",
"category": "Troubleshooting",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "task",
"weight": 7,
+1
View File
@@ -2,6 +2,7 @@
"id": "env-vars-configmap",
"title": "Environment Variables from ConfigMap",
"category": "Configuration",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 4,
+1
View File
@@ -2,6 +2,7 @@
"id": "etcd-backup-mcq",
"title": "etcd Backup and Restore",
"category": "Cluster Administration",
"isolation_type": "shared_namespace",
"difficulty": "Hard",
"type": "mcq",
"weight": 5,
+1
View File
@@ -2,6 +2,7 @@
"id": "exec-command-basics",
"title": "Execute Commands in Pods",
"category": "Troubleshooting",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 3,
@@ -2,6 +2,7 @@
"id": "expose-service-basics",
"title": "Exposing an Application",
"category": "Services & Networking",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 4,
+1
View File
@@ -2,6 +2,7 @@
"id": "extract-logs-basics",
"title": "Extract Container Logs",
"category": "Troubleshooting",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 4,
@@ -2,6 +2,7 @@
"id": "field-selector-basics",
"title": "Filtering Resources with Field Selectors",
"category": "Core Concepts",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 4,
+1
View File
@@ -2,6 +2,7 @@
"id": "ingress-task",
"title": "Create an Ingress Resource",
"category": "Services & Networking",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "task",
"weight": 7,
@@ -2,6 +2,7 @@
"id": "init-container-task-basics",
"title": "Pod with Init Container",
"category": "Core Concepts",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "task",
"weight": 5,
+1
View File
@@ -2,6 +2,7 @@
"id": "init-containers-mcq",
"title": "Init Containers Behaviour",
"category": "Core Concepts",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "mcq",
"weight": 3,
+1
View File
@@ -2,6 +2,7 @@
"id": "job-task",
"title": "Create a One-Shot Job",
"category": "Workloads & Scheduling",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 4,
+1
View File
@@ -2,6 +2,7 @@
"id": "jobs-cronjobs-mcq",
"title": "Jobs vs CronJobs",
"category": "Workloads",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "mcq",
"weight": 3,
+1
View File
@@ -2,6 +2,7 @@
"id": "jsonpath-basics",
"title": "JSONPath Data Extraction",
"category": "Core Concepts",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "task",
"weight": 5,
+1
View File
@@ -2,6 +2,7 @@
"id": "kubectl-cp-basics",
"title": "Copy Files to Containers",
"category": "Troubleshooting",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "task",
"weight": 4,
+1
View File
@@ -2,6 +2,7 @@
"id": "kubectl-diff-basics",
"title": "Previewing Changes with kubectl diff",
"category": "Core Concepts",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "task",
"weight": 4,
@@ -2,6 +2,7 @@
"id": "kubectl-essentials-mcq",
"title": "Essential kubectl Commands",
"category": "Core Concepts",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "mcq",
"weight": 2,
@@ -2,6 +2,7 @@
"id": "kubectl-explain-basics",
"title": "Exploring the API with kubectl explain",
"category": "Core Concepts",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 3,
@@ -2,6 +2,7 @@
"id": "kubectl-output-format-mcq",
"title": "kubectl Output Formats",
"category": "Core Concepts",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "mcq",
"weight": 2,
+1
View File
@@ -2,6 +2,7 @@
"id": "kubectl-patch-basics",
"title": "Patching Resources with kubectl patch",
"category": "Core Concepts",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "task",
"weight": 5,
+1
View File
@@ -2,6 +2,7 @@
"id": "labels-selectors-mcq",
"title": "Labels and Selectors",
"category": "Core Concepts",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "mcq",
"weight": 3,
+1
View File
@@ -2,6 +2,7 @@
"id": "limitrange-task",
"title": "LimitRange for Default Container Limits",
"category": "Workloads & Scheduling",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "task",
"weight": 6,
+1
View File
@@ -2,6 +2,7 @@
"id": "liveness-probe-task",
"title": "Configure a Liveness Probe",
"category": "Workloads",
"isolation_type": "shared_namespace",
"difficulty": "Medium",
"type": "task",
"weight": 6,
@@ -2,6 +2,7 @@
"id": "multi-container-pod-basics",
"title": "Create a Multi-Container Pod",
"category": "Core Concepts",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 5,
+1
View File
@@ -2,6 +2,7 @@
"id": "namespaces-basics",
"title": "Working with Namespaces",
"category": "Core Concepts",
"isolation_type": "shared_namespace",
"difficulty": "Easy",
"type": "task",
"weight": 4,
+1
View File
@@ -2,6 +2,7 @@
"id": "network-policy",
"title": "Isolate Traffic with NetworkPolicy",
"category": "Networking",
"isolation_type": "shared_namespace",
"difficulty": "Hard",
"type": "task",
"weight": 9,
+1
View File
@@ -2,6 +2,7 @@
"id": "networkpolicy-egress",
"title": "Egress NetworkPolicy",
"category": "Security",
"isolation_type": "shared_namespace",
"difficulty": "Hard",
"type": "task",
"weight": 8,
+1
View File
@@ -2,6 +2,7 @@
"id": "node-cordon-basics",
"title": "Cordoning and Uncordoning a Node",
"category": "Cluster Management",
"isolation_type": "ephemeral_cluster",
"difficulty": "Easy",
"type": "task",
"weight": 4,
+1
View File
@@ -2,6 +2,7 @@
"id": "node-label-selector",
"title": "Node Labels and nodeSelector",
"category": "Cluster Administration",
"isolation_type": "ephemeral_cluster",
"difficulty": "Easy",
"type": "task",
"weight": 5,
@@ -2,6 +2,7 @@
"id": "node-taint-toleration",
"title": "Taints and Tolerations",
"category": "Cluster Administration",
"isolation_type": "ephemeral_cluster",
"difficulty": "Medium",
"type": "task",
"weight": 7,

Some files were not shown because too many files have changed in this diff Show More