docs(platform): document ephemeral lab architecture
- docs/LAB_ARCHITECTURE.md — the full ephemeral-per-student design: two-VM production topology (Platform VM / Lab Host VM, correction A), lifecycle, data model, networking/egress model (correction B), resource model, routing/auth model, security boundaries, failure/recovery behavior, migration path from the current single container, and the lab-host abstraction (correction D). Status updated to reflect Phase 1 is now implemented, pointing at the two docs below for specifics. - docs/PHASE1_PLATFORM_FOUNDATION.md — the Phase 1 build plan: exact Postgres schema, service boundaries, API routes, lifecycle state machine, Docker/ firewall commands, security assumptions, file layout, and a 12-test plan. Updated with "Implementation notes" at the points where testing surfaced real deviations from the original design: the per-network (not static) egress-proxy address resolution, the COALESCE reaper fix, and the provisioning-failure cleanup fix. - docs/ENGINEERING_SOP.md — new §8 mapping the platform/provisioner/ and platform/egress-proxy/ components, what's built vs. not (no Gateway, no frontend wiring — internal API only), and where identity still lives (unchanged: Lab image's own SQLite). Roadmap item 5 (per-student isolation) updated from "not started" to "Phase 1 done, not yet cleared for minors." - README.md — Multi-Student Deployment section rewritten: the old per-student- volume persistence model is superseded by the Platform's registry (progress already lives outside the disposable container as of the prior curriculum/ roster work); points at docs/LAB_ARCHITECTURE.md and platform/provisioner/ for the real architecture instead of describing three delivery modes that were never actually built out. Co-Authored-By: Claude Sonnet 5 <[email protected]>
This commit is contained in:
Claude Sonnet 5
@@ -0,0 +1,188 @@
|
||||
# RootNode Academy — Engineering SOP
|
||||
|
||||
Living doc. Covers the "how" of building this platform — the "why" and business/curriculum
|
||||
process live in the Business Pack (`~/Documents/CDAA Technologies/07 RootNode academy/RootNode_Academy_Business_Pack/`),
|
||||
this doc doesn't duplicate that. Update this file in the same commit as the change it describes.
|
||||
|
||||
## 1. Where this actually lives
|
||||
|
||||
- **Repo:** `~/Dev/Gitea/ced4568/rootnode-academy` — remote is **Gitea**
|
||||
(`git.synthossystems.com/ced4568/rootnode-academy`), not GitHub.
|
||||
- **Gotcha:** `~/Dev/GitHub/ced4568/kubekosh` is a *different* clone of the same original
|
||||
upstream fork, still pointed at GitHub, and it's behind (missing the RootNode logo commit
|
||||
and the Real Ops enforce-limits scenario as of this writing). It's not deployed anywhere.
|
||||
Don't edit it by mistake — always confirm `git remote -v` shows `gitea:` before making changes.
|
||||
- Known-good baseline as of this doc: commit `6ddfb1c` ("add probe-port-mismatch and
|
||||
enforce-limits to rootnode-real-ops bundle"). Everything below builds on top of that,
|
||||
clean working tree.
|
||||
|
||||
## 2. Architecture at a glance
|
||||
|
||||
```
|
||||
backend/server.js Express API + SQLite (better-sqlite3) + WebSocket PTY terminal — the
|
||||
"Lab image," single-tenant, unmodified by the Platform work below
|
||||
frontend/ Vite + React SPA, built to frontend/dist and served by nginx
|
||||
scenarios/data/ One JSON file per scenario (task or MCQ)
|
||||
scenarios/bundles/ Flat scenario_id lists grouped by cert track (basics/appdev/admin/security/real-ops)
|
||||
scenarios/rootnode-path.json The RootNode guided learning path (staged, see §4)
|
||||
scripts/entrypoint.sh Container boot: starts k3s, nginx, node backend, sets the shell banner/PS1
|
||||
Dockerfile ubuntu:22.04 + k3s + node20 + nginx, single image, one port (80)
|
||||
|
||||
platform/provisioner/ Phase 1 Platform Provisioner — disposable per-student Lab lifecycle,
|
||||
PostgreSQL registry, egress firewall. See §8 and
|
||||
platform/provisioner/README.md.
|
||||
platform/egress-proxy/ Shared allowlisting egress proxy every Lab environment's traffic must
|
||||
pass through.
|
||||
```
|
||||
|
||||
Single Express process, single SQLite file at `/data/progress.db` (or `$PROGRESS_DB`),
|
||||
single embedded k3s cluster **per Lab environment**. The Lab image itself is still
|
||||
single-tenant by design — see §5 for identity, and §8 for how Phase 1 multiplies it into a
|
||||
real multi-student platform without modifying it.
|
||||
|
||||
## 3. Local dev / safe smoke-testing
|
||||
|
||||
Never test schema changes against the real `/data/progress.db` first. Point at a scratch file:
|
||||
|
||||
```bash
|
||||
cd backend && npm install # first time only
|
||||
node --check server.js # syntax check, catches typos fast
|
||||
PROGRESS_DB=/tmp/scratch/progress.db node server.js &
|
||||
curl -s http://localhost:4000/api/health
|
||||
# ...exercise the endpoints you changed...
|
||||
fuser -k 4000/tcp # stop it
|
||||
```
|
||||
|
||||
To test a migration against **real prior data** without touching prod: copy the actual
|
||||
`/data/progress.db` out of the running container/volume, point `PROGRESS_DB` at the copy,
|
||||
boot, and verify the data you expect is still there before ever pointing at the real file.
|
||||
|
||||
kubectl/k3s-backed endpoints (`/setup`, `/validate`, `/context`, the WS terminal) won't work
|
||||
outside the container — that's expected. API-and-DB-only endpoints (roster, login, progress,
|
||||
curriculum, bundles, scenarios list) all smoke-test fine on a bare host.
|
||||
|
||||
## 4. Curriculum content
|
||||
|
||||
Two separate structures, don't confuse them:
|
||||
|
||||
- **Bundles** (`scenarios/bundles/*.json`) — flat scenario_id lists, one per cert track.
|
||||
Loaded by `loadBundles()`/`GET /api/bundles`. Adding a bundle file here = adding a new
|
||||
cert-track library. Every scenario referenced must have a matching file in `scenarios/data/`.
|
||||
- **RootNode path** (`scenarios/rootnode-path.json`) — the guided onboarding sequence,
|
||||
separate from bundles on purpose (it's *staged*, bundles are *flat*, and the bundle loader
|
||||
would choke on a staged file if it were dropped in `scenarios/bundles/`). Loaded by
|
||||
`loadCurriculumPath()`/`GET /api/curriculum/path`. Stages are Foundation → Workloads →
|
||||
Networking → Configuration → Troubleshooting → Real Ops → Capstone, currently built by
|
||||
reusing existing `1-k8s-basics` and `rootnode-real-ops` scenarios grouped by their existing
|
||||
`category` field — no new scenario content was written for this. **Capstone stage is a
|
||||
locked placeholder** (`"locked": true`, empty `scenario_ids`) — the capstone engine itself
|
||||
is a separate, not-yet-built roadmap item (see §7, item 6). Do not put real scenarios in it
|
||||
until that engine exists, or you'll reproduce the "duplicate beginner scenarios as capstone"
|
||||
anti-pattern this path was built to avoid.
|
||||
|
||||
To add/reorder a stage: edit `rootnode-path.json` directly, referencing existing
|
||||
`scenarios/data/*.json` ids. No backend code change needed — it's read fresh via
|
||||
`POST /api/cache/reload` or on server restart.
|
||||
|
||||
## 5. Student identity — what's built, what isn't
|
||||
|
||||
**Built (backend only, as of this doc):**
|
||||
- `students` table: `student_id` (`student-NNN`, sequential), `display_name`, `cohort`,
|
||||
`created_at`, `last_active_at`.
|
||||
- Login model: **instructor roster, pick-your-name, no password.** `POST /api/roster` adds
|
||||
a student (instructor-only action, currently unauthenticated at the API level — see
|
||||
"not built" below). `GET /api/roster?cohort=X` lists names for a dropdown.
|
||||
`POST /api/login` just validates the id exists and returns it; the frontend is expected to
|
||||
hold onto `student_id` and send it back as an `X-Student-Id` header on every request.
|
||||
- `progress` table now keyed on `(student_id, scenario_id)` instead of `scenario_id` alone —
|
||||
each student's completion/attempts/hints are isolated. Verified by smoke test: two students
|
||||
answering the same scenario don't affect each other's state, and pre-migration data was
|
||||
preserved under `student_id = 'unassigned'`.
|
||||
- Exam `sessions` now carry `student_id` (set at session start), so exam history and the
|
||||
timer-reset-on-submit logic are correctly scoped per student instead of leaking across
|
||||
whoever's browser hits the API next.
|
||||
|
||||
**Not built yet — do not assume these exist:**
|
||||
- **No frontend UI.** No login screen, no name picker, no roster-add form. The API works;
|
||||
nothing calls it yet. This is the next slice.
|
||||
- **No auth on `/api/roster` POST.** Anyone who can reach the API can add a "student."
|
||||
Fine for a dev instance, not fine once this is reachable by actual students — needs an
|
||||
instructor-only gate before that happens (ties into the instructor dashboard, roadmap
|
||||
item 3).
|
||||
- **No concurrency / isolation.** Exam sessions are still global-single-active
|
||||
(`WHERE status='active' LIMIT 1` — see `server.js`), and there is one shared k3s cluster
|
||||
for the whole instance. Per-student identity ≠ per-student environment. **Multiple students
|
||||
cannot yet use this platform at the same time without stepping on each other.** That's
|
||||
roadmap item 4 (isolation) and it gates real classroom/minors use, same as the plan called
|
||||
out — this SOP is not claiming that gate is cleared.
|
||||
|
||||
## 6. Schema migration pattern
|
||||
|
||||
SQLite can't `ALTER` a primary key. The codebase's existing convention (see `progress` table
|
||||
history) is:
|
||||
|
||||
1. `PRAGMA table_info(<table>)` → check if the new column/shape is already there (idempotent,
|
||||
safe to run on every boot).
|
||||
2. Simple new nullable column → `ALTER TABLE ... ADD COLUMN`.
|
||||
3. Primary-key or structural change → build `<table>_new` with the new shape, `INSERT ...
|
||||
SELECT` the old data across with sane defaults for new required columns, `DROP TABLE`,
|
||||
`RENAME TO`. Never drop data — give it a legacy/default value (`'unassigned'`, etc.)
|
||||
instead of discarding rows.
|
||||
4. Test the migration against a **copy of real data** (§3) before it ever runs against
|
||||
`/data/progress.db` in the container.
|
||||
|
||||
## 7. Roadmap status
|
||||
|
||||
Order agreed 2026-08-17/18. Status as of this doc:
|
||||
|
||||
1. RootNode identity/branding pass — **partial.** Shell banner fixed
|
||||
(`⎈ RootNode Academy Lab — node: k8s-lab`). Upstream KubeKosh attribution left untouched in
|
||||
footer/README/LICENSE. Broader "make Real Ops first-class" UI work not started.
|
||||
2. RootNode curriculum mode — **backend done** (§4), frontend not started.
|
||||
3. Student accounts / persistent progress — **backend done** (§5), frontend not started,
|
||||
no instructor auth yet.
|
||||
4. Instructor dashboard — not started.
|
||||
5. Per-student isolation — **Phase 1 (platform foundation) done**, see §8. Ephemeral
|
||||
per-student Lab environments with Postgres-backed lifecycle, per-environment Docker
|
||||
networking, and firewall-enforced egress restriction are implemented and tested on a single
|
||||
dev machine. **Still not cleared for minors** — no Gateway/frontend wiring yet (environments
|
||||
are only reachable via the internal Provisioner API), no instructor auth on top of it, and
|
||||
the production two-VM Proxmox split this depends on doesn't exist yet. See
|
||||
`docs/PHASE1_PLATFORM_FOUNDATION.md` final report for exactly what's verified vs. not.
|
||||
6. Scenarios → teaching modules (incident context/investigation/prevention) — not started.
|
||||
7. Capstone/exam engine — not started (placeholder stage only, see §4).
|
||||
8. Progressive hint levels — not started.
|
||||
9. Production observability — not started.
|
||||
10. CI scenario testing — not started.
|
||||
11. Container/build optimization — explicitly deprioritized by design (platform works, not urgent).
|
||||
|
||||
Next slice: either the deferred frontend login (name picker → `X-Student-Id` header, curriculum
|
||||
view), or the Phase 2 platform work (Gateway, frontend wiring to the Provisioner, instructor
|
||||
auth) — see the Phase 1 final report for the concrete Phase 2 list. Not decided here.
|
||||
|
||||
## 8. Platform Provisioner (Phase 1)
|
||||
|
||||
Full design: `docs/LAB_ARCHITECTURE.md` + `docs/PHASE1_PLATFORM_FOUNDATION.md`. Operational
|
||||
reference (config, local dev, known limitations): `platform/provisioner/README.md`. This
|
||||
section is just the map.
|
||||
|
||||
- **What it is:** an internal-only service that provisions a disposable, fully isolated Lab
|
||||
environment per student — its own Docker network, its own container running the existing,
|
||||
unmodified Lab image, its own firewall-enforced egress. Postgres-backed registry
|
||||
(`environments`, `environment_events`, `lab_hosts`), idle reaper, startup reconciliation.
|
||||
- **What it is not (yet):** reachable by a student. There's no Gateway and no frontend wiring
|
||||
— it's exercised directly via its internal API (shared-secret auth) or
|
||||
`platform/provisioner/test/run.js`. Nothing here is a live feature.
|
||||
- **Where identity lives:** unchanged from §5 — `students`/`progress` are still on the Lab
|
||||
image's own SQLite. `environments.student_id` is deliberately opaque (no FK) until a later,
|
||||
separate migration puts roster/progress on the same Postgres instance as the environment
|
||||
registry.
|
||||
- **Local dev:** `platform/provisioner/docker-compose.dev.yml` runs Postgres + the egress
|
||||
proxy + the Provisioner on one machine — the documented Phase 1 dev exception. Production
|
||||
is a two-VM Proxmox split (Platform VM / Lab Host VM); the dev compose file is explicitly
|
||||
not that.
|
||||
- **Test procedure:** `platform/provisioner/test/run.js` — 12 automated integration tests
|
||||
against a live dev stack (isolation, firewall ordering, reset/destroy cleanup, audit
|
||||
events). Some properties (idle reaper timing, remote TLS lab-host, reconciliation-on-crash,
|
||||
real-image k3s behavior) were validated manually during implementation rather than
|
||||
automated — see the Phase 1 final report for what and why.
|
||||
Reference in New Issue
Block a user