rootnode-academy/docs/ENGINEERING_SOP.md

RootNode Academy — Engineering SOP

Living doc. Covers the "how" of building this platform — the "why" and business/curriculum process live in the Business Pack (~/Documents/CDAA Technologies/07 RootNode academy/RootNode_Academy_Business_Pack/), this doc doesn't duplicate that. Update this file in the same commit as the change it describes.

1. Where this actually lives

• Repo: ~/Dev/Gitea/ced4568/rootnode-academy — remote is Gitea (git.synthossystems.com/ced4568/rootnode-academy), not GitHub.
• Gotcha: ~/Dev/GitHub/ced4568/kubekosh is a different clone of the same original upstream fork, still pointed at GitHub, and it's behind (missing the RootNode logo commit and the Real Ops enforce-limits scenario as of this writing). It's not deployed anywhere. Don't edit it by mistake — always confirm git remote -v shows gitea: before making changes.
• Known-good baseline as of this doc: commit 6ddfb1c ("add probe-port-mismatch and enforce-limits to rootnode-real-ops bundle"). Everything below builds on top of that, clean working tree.

2. Architecture at a glance

backend/server.js      Express API + SQLite (better-sqlite3) + WebSocket PTY terminal — the
                        "Lab image," single-tenant, unmodified by the Platform work below
frontend/               Vite + React SPA, built to frontend/dist and served by nginx
scenarios/data/         One JSON file per scenario (task or MCQ)
scenarios/bundles/      Flat scenario_id lists grouped by cert track (basics/appdev/admin/security/real-ops)
scenarios/rootnode-path.json   The RootNode guided learning path (staged, see §4)
scripts/entrypoint.sh   Container boot: starts k3s, nginx, node backend, sets the shell banner/PS1
Dockerfile              ubuntu:22.04 + k3s + node20 + nginx, single image, one port (80)

platform/provisioner/   Phase 1 Platform Provisioner — disposable per-student Lab lifecycle,
                        PostgreSQL registry, egress firewall. See §8 and
                        platform/provisioner/README.md.
platform/egress-proxy/  Shared allowlisting egress proxy every Lab environment's traffic must
                        pass through.

Single Express process, single SQLite file at /data/progress.db (or $PROGRESS_DB), single embedded k3s cluster per Lab environment. The Lab image itself is still single-tenant by design — see §5 for identity, and §8 for how Phase 1 multiplies it into a real multi-student platform without modifying it.

3. Local dev / safe smoke-testing

Never test schema changes against the real /data/progress.db first. Point at a scratch file:

cd backend && npm install   # first time only
node --check server.js      # syntax check, catches typos fast
PROGRESS_DB=/tmp/scratch/progress.db node server.js &
curl -s http://localhost:4000/api/health
# ...exercise the endpoints you changed...
fuser -k 4000/tcp            # stop it

To test a migration against real prior data without touching prod: copy the actual /data/progress.db out of the running container/volume, point PROGRESS_DB at the copy, boot, and verify the data you expect is still there before ever pointing at the real file.

kubectl/k3s-backed endpoints (/setup, /validate, /context, the WS terminal) won't work outside the container — that's expected. API-and-DB-only endpoints (roster, login, progress, curriculum, bundles, scenarios list) all smoke-test fine on a bare host.

4. Curriculum content

Two separate structures, don't confuse them:

• Bundles (scenarios/bundles/*.json) — flat scenario_id lists, one per cert track. Loaded by loadBundles()/GET /api/bundles. Adding a bundle file here = adding a new cert-track library. Every scenario referenced must have a matching file in scenarios/data/.
• RootNode path (scenarios/rootnode-path.json) — the guided onboarding sequence, separate from bundles on purpose (it's staged, bundles are flat, and the bundle loader would choke on a staged file if it were dropped in scenarios/bundles/). Loaded by loadCurriculumPath()/GET /api/curriculum/path. Stages are Foundation → Workloads → Networking → Configuration → Troubleshooting → Real Ops → Capstone, currently built by reusing existing 1-k8s-basics and rootnode-real-ops scenarios grouped by their existing category field — no new scenario content was written for this. Capstone stage is a locked placeholder ("locked": true, empty scenario_ids) — the capstone engine itself is a separate, not-yet-built roadmap item (see §7, item 6). Do not put real scenarios in it until that engine exists, or you'll reproduce the "duplicate beginner scenarios as capstone" anti-pattern this path was built to avoid.

To add/reorder a stage: edit rootnode-path.json directly, referencing existing scenarios/data/*.json ids. No backend code change needed — it's read fresh via POST /api/cache/reload or on server restart.

5. Student identity — what's built, what isn't

Built (backend only, as of this doc):

• students table: student_id (student-NNN, sequential), display_name, cohort, created_at, last_active_at.
• Login model: instructor roster, pick-your-name, no password. POST /api/roster adds a student (instructor-only action, currently unauthenticated at the API level — see "not built" below). GET /api/roster?cohort=X lists names for a dropdown. POST /api/login just validates the id exists and returns it; the frontend is expected to hold onto student_id and send it back as an X-Student-Id header on every request.
• progress table now keyed on (student_id, scenario_id) instead of scenario_id alone — each student's completion/attempts/hints are isolated. Verified by smoke test: two students answering the same scenario don't affect each other's state, and pre-migration data was preserved under student_id = 'unassigned'.
• Exam sessions now carry student_id (set at session start), so exam history and the timer-reset-on-submit logic are correctly scoped per student instead of leaking across whoever's browser hits the API next.

Not built yet — do not assume these exist:

• No frontend UI. No login screen, no name picker, no roster-add form. The API works; nothing calls it yet. This is the next slice.
• No auth on /api/roster POST. Anyone who can reach the API can add a "student." Fine for a dev instance, not fine once this is reachable by actual students — needs an instructor-only gate before that happens (ties into the instructor dashboard, roadmap item 3).
• No concurrency / isolation. Exam sessions are still global-single-active (WHERE status='active' LIMIT 1 — see server.js), and there is one shared k3s cluster for the whole instance. Per-student identity ≠ per-student environment. Multiple students cannot yet use this platform at the same time without stepping on each other. That's roadmap item 4 (isolation) and it gates real classroom/minors use, same as the plan called out — this SOP is not claiming that gate is cleared.

6. Schema migration pattern

SQLite can't ALTER a primary key. The codebase's existing convention (see progress table history) is:

1. PRAGMA table_info(<table>) → check if the new column/shape is already there (idempotent, safe to run on every boot).
2. Simple new nullable column → ALTER TABLE ... ADD COLUMN.
3. Primary-key or structural change → build <table>_new with the new shape, INSERT ... SELECT the old data across with sane defaults for new required columns, DROP TABLE, RENAME TO. Never drop data — give it a legacy/default value ('unassigned', etc.) instead of discarding rows.
4. Test the migration against a copy of real data (§3) before it ever runs against /data/progress.db in the container.

7. Roadmap status

Order agreed 2026-08-17/18. Status as of this doc:

1. RootNode identity/branding pass — partial. Shell banner fixed (⎈ RootNode Academy Lab — node: k8s-lab). Upstream KubeKosh attribution left untouched in footer/README/LICENSE. Broader "make Real Ops first-class" UI work not started.
2. RootNode curriculum mode — backend done (§4), frontend not started.
3. Student accounts / persistent progress — backend done (§5), frontend not started, no instructor auth yet.
4. Instructor dashboard — not started.
5. Per-student isolation — Phase 1 (platform foundation) done, see §8. Ephemeral per-student Lab environments with Postgres-backed lifecycle, per-environment Docker networking, and firewall-enforced egress restriction are implemented and tested on a single dev machine. Still not cleared for minors — no Gateway/frontend wiring yet (environments are only reachable via the internal Provisioner API), no instructor auth on top of it, and the production two-VM Proxmox split this depends on doesn't exist yet. See docs/PHASE1_PLATFORM_FOUNDATION.md final report for exactly what's verified vs. not.
6. Scenarios → teaching modules (incident context/investigation/prevention) — not started.
7. Capstone/exam engine — not started (placeholder stage only, see §4).
8. Progressive hint levels — not started.
9. Production observability — not started.
10. CI scenario testing — not started.
11. Container/build optimization — explicitly deprioritized by design (platform works, not urgent).

Next slice: either the deferred frontend login (name picker → X-Student-Id header, curriculum view), or the Phase 2 platform work (Gateway, frontend wiring to the Provisioner, instructor auth) — see the Phase 1 final report for the concrete Phase 2 list. Not decided here.

8. Platform Provisioner (Phase 1)

Full design: docs/LAB_ARCHITECTURE.md + docs/PHASE1_PLATFORM_FOUNDATION.md. Operational reference (config, local dev, known limitations): platform/provisioner/README.md. This section is just the map.

• What it is: an internal-only service that provisions a disposable, fully isolated Lab environment per student — its own Docker network, its own container running the existing, unmodified Lab image, its own firewall-enforced egress. Postgres-backed registry (environments, environment_events, lab_hosts), idle reaper, startup reconciliation.
• What it is not (yet): reachable by a student. There's no Gateway and no frontend wiring — it's exercised directly via its internal API (shared-secret auth) or platform/provisioner/test/run.js. Nothing here is a live feature.
• Where identity lives: unchanged from §5 — students/progress are still on the Lab image's own SQLite. environments.student_id is deliberately opaque (no FK) until a later, separate migration puts roster/progress on the same Postgres instance as the environment registry.
• Local dev: platform/provisioner/docker-compose.dev.yml runs Postgres + the egress proxy + the Provisioner on one machine — the documented Phase 1 dev exception. Production is a two-VM Proxmox split (Platform VM / Lab Host VM); the dev compose file is explicitly not that.
• Test procedure: platform/provisioner/test/run.js — 12 automated integration tests against a live dev stack (isolation, firewall ordering, reset/destroy cleanup, audit events). Some properties (idle reaper timing, remote TLS lab-host, reconciliation-on-crash, real-image k3s behavior) were validated manually during implementation rather than automated — see the Phase 1 final report for what and why.