rootnode-academy/docs/ENGINEERING_SOP.md

RootNode Academy — Engineering SOP

Living doc. Covers the "how" of building this platform — the "why" and business/curriculum process live in the Business Pack (~/Documents/CDAA Technologies/07 RootNode academy/RootNode_Academy_Business_Pack/), this doc doesn't duplicate that. Update this file in the same commit as the change it describes.

1. Where this actually lives

• Repo: ~/Dev/Gitea/ced4568/rootnode-academy — remote is Gitea (git.synthossystems.com/ced4568/rootnode-academy), not GitHub.
• Gotcha: ~/Dev/GitHub/ced4568/kubekosh is a different clone of the same original upstream fork, still pointed at GitHub, and it's behind (missing the RootNode logo commit and the Real Ops enforce-limits scenario as of this writing). It's not deployed anywhere. Don't edit it by mistake — always confirm git remote -v shows gitea: before making changes.
• Known-good baseline as of this doc: commit 6ddfb1c ("add probe-port-mismatch and enforce-limits to rootnode-real-ops bundle"). Everything below builds on top of that, clean working tree.

2. Architecture at a glance

backend/server.js      Express API + SQLite (better-sqlite3) + WebSocket PTY terminal — the
                        "Lab image," single-tenant, unmodified by the Platform work below
frontend/               Vite + React SPA, built to frontend/dist and served by nginx
scenarios/data/         One JSON file per scenario (task or MCQ)
scenarios/bundles/      Flat scenario_id lists grouped by cert track (basics/appdev/admin/security/real-ops)
scenarios/rootnode-path.json   The RootNode guided learning path (staged, see §4)
scripts/entrypoint.sh   Container boot: starts k3s, nginx, node backend, sets the shell banner/PS1
Dockerfile              ubuntu:22.04 + k3s + node20 + nginx, single image, one port (80)

platform/provisioner/   Phase 1 Platform Provisioner — disposable per-student Lab lifecycle,
                        PostgreSQL registry, egress firewall. See §8 and
                        platform/provisioner/README.md.
platform/egress-proxy/  Shared allowlisting egress proxy every Lab environment's traffic must
                        pass through.

Single Express process, single SQLite file at /data/progress.db (or $PROGRESS_DB), single embedded k3s cluster per Lab environment. The Lab image itself is still single-tenant by design — see §5 for identity, and §8 for how Phase 1 multiplies it into a real multi-student platform without modifying it.

3. Local dev / safe smoke-testing

Never test schema changes against the real /data/progress.db first. Point at a scratch file:

cd backend && npm install   # first time only
node --check server.js      # syntax check, catches typos fast
PROGRESS_DB=/tmp/scratch/progress.db node server.js &
curl -s http://localhost:4000/api/health
# ...exercise the endpoints you changed...
fuser -k 4000/tcp            # stop it

To test a migration against real prior data without touching prod: copy the actual /data/progress.db out of the running container/volume, point PROGRESS_DB at the copy, boot, and verify the data you expect is still there before ever pointing at the real file.

kubectl/k3s-backed endpoints (/setup, /validate, /context, the WS terminal) won't work outside the container — that's expected. API-and-DB-only endpoints (roster, login, progress, curriculum, bundles, scenarios list) all smoke-test fine on a bare host.

4. Curriculum content

Two separate structures, don't confuse them:

• Bundles (scenarios/bundles/*.json) — flat scenario_id lists, one per cert track. Loaded by loadBundles()/GET /api/bundles. Adding a bundle file here = adding a new cert-track library. Every scenario referenced must have a matching file in scenarios/data/.
• RootNode path (scenarios/rootnode-path.json) — the guided onboarding sequence, separate from bundles on purpose (it's staged, bundles are flat, and the bundle loader would choke on a staged file if it were dropped in scenarios/bundles/). Loaded by loadCurriculumPath()/GET /api/curriculum/path (raw content) and, as of Phase 3B, computeCurriculumState()/GET /api/curriculum/state (rule-driven lock/complete/next-scenario state — see docs/PHASE3B_LEARNING_AND_ASSESSMENT.md §4). Stages are Foundation → Workloads → Networking → Configuration → Troubleshooting → Real Ops → Capstone, built by reusing existing 1-k8s-basics and rootnode-real-ops scenarios grouped by their existing category field — no new scenario content was written for this. Each stage carries a completion_rule (all_required/threshold/prerequisites). Capstone is no longer a hardcoded placeholder — it's "type": "capstone" with an "unlock_rule" and a "capstone_assessment_id" pointing at the rootnode-capstone assessment definition (Postgres, assessment_definitions table), which is where its actual scenario draw lives — see the Phase 3B doc §7. Don't put real scenarios directly into the JSON stage's scenario_ids for capstone; that field stays empty by design, the draw is defined in the assessment definition instead.

To add/reorder a stage: edit rootnode-path.json directly, referencing existing scenarios/data/*.json ids. No backend code change needed — it's read fresh via POST /api/cache/reload or on server restart.

5. Student identity — what's built, what isn't

Current state (Phase 5A — see docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md for the full design):

• students.student_id (student-NNN, sequential) is still the FK every other table hangs off of and never changes across an account's lifecycle, including conversion.
• Real credentials: email and login_identifier (independently optional/unique — email for older/self-paced students, an instructor-assigned login ID for younger ones), password_hash reusing instructors' exact scrypt machinery (src/auth/password.js). A student with neither field set is a plain pilot account, unchanged since Phase 2A: POST /api/login { student_id }, no password, instructor-roster-only.
• First login/reset is one shared, single-use, hashed, time-limited token mechanism (students-auth.js) — instructors never see or set a student's actual password.
• Sessions are the same Postgres-backed sessions table instructors use (opaque, HttpOnly, hashed, revocable, CSRF-protected); login is rate-limited the same Postgres-backed way as instructor login (Phase 4.1), now also applied to students.
• Per-student progress/scenario_attempts/exam sessions isolation (below) and real concurrent multi-student Lab isolation (Phase 1's Provisioner/k3s-per-student model) were both already in place well before this — Phase 5A only changed how a student proves who they are, not how their data or Labs are isolated.

Not built — deliberate non-goals, not gaps (Phase 5A §37):

• No public self-registration — accounts are still instructor-created/converted/imported only.
• No email delivery infrastructure — password reset works via an instructor-issued token today; self-service reset returns a real token only in non-production, by design.
• No Parent/Guardian portal — the schema doesn't block adding one later, but none exists.

Historical note, kept for context: earlier drafts of this section described a pre-Phase-2A state (X-Student-Id header, unauthenticated roster POST, one shared cluster) that no longer reflects any deployed version of this codebase; that content has been superseded by the above.

6. Schema migration pattern

SQLite can't ALTER a primary key. The codebase's existing convention (see progress table history) is:

1. PRAGMA table_info(<table>) → check if the new column/shape is already there (idempotent, safe to run on every boot).
2. Simple new nullable column → ALTER TABLE ... ADD COLUMN.
3. Primary-key or structural change → build <table>_new with the new shape, INSERT ... SELECT the old data across with sane defaults for new required columns, DROP TABLE, RENAME TO. Never drop data — give it a legacy/default value ('unassigned', etc.) instead of discarding rows.
4. Test the migration against a copy of real data (§3) before it ever runs against /data/progress.db in the container.

7. Roadmap status

Order agreed 2026-08-17/18. Status as of this doc:

1. RootNode identity/branding pass — partial. Shell banner fixed (⎈ RootNode Academy Lab — node: k8s-lab). Upstream KubeKosh attribution left untouched in footer/README/LICENSE. Broader "make Real Ops first-class" UI work not started.
2. RootNode curriculum mode — done. Rule-driven stage completion/unlock as of Phase 3B (§12), student-facing tabs (My Learning/Practice/Assessments/Capstone/Achievements) at frontend/src/platform/.
3. Student accounts / persistent progress — done, Postgres-authoritative as of Phase 2B (§9), sessions persistent as of Phase 3A (§10). Real student credentials (email or login-identifier + password, first-login/reset token flow) closed the remaining "pick your name" gap as of Phase 5A — see §5 above and docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md. Pilot (no-login) accounts still work unchanged for anyone not yet converted.
4. Instructor dashboard — done (§9/§10). Full admin portal at /admin: dashboard, cohort and student management, classroom Prepare/Destroy Labs, per-instructor accounts with owner/admin/instructor roles, audit log.
5. Per-student isolation — Phase 1 + Phase 2A + Phase 2B + Phase 3A done, see §8/§9/§10. Ephemeral per-student Lab environments, Gateway-mediated routing with persistent server-controlled sessions, a full instructor/admin control plane with real per-person accounts and RBAC, and a working entry→dashboard→launch→lab→progress flow are implemented and end-to-end tested on a single dev machine, including a 3-student cohort classroom workflow and a real Platform API restart mid-session. Still not cleared for minors — the production two-VM Proxmox split this depends on doesn't exist yet, and OIDC/SSO is a documented adapter, not a working integration (docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md §16).
6. Scenarios → teaching modules (incident context/investigation/prevention) — not started.
7. Capstone/exam engine — done as of Phase 3B (§12). RootNode Capstone (Real Ops-weighted, unlockable by curriculum-completion rule, not a permanent placeholder) plus a general exam engine (definitions, cohort assignment, random scenario draw, server-authoritative timer).
8. Progressive hint levels — done as of Phase 3B (§12). Server-enforced sequential reveal, tracked per attempt, policy-aware under assessment mode (free/disabled/flagged).
9. Production observability — not started.
10. CI scenario testing — not started.
11. Container/build optimization — explicitly deprioritized by design (platform works, not urgent).

Next slice: production OIDC/SSO, the production two-VM Proxmox deployment, and teaching-module scenario content are the biggest remaining items. See docs/PHASE3B_LEARNING_AND_ASSESSMENT.md §21 for the full Phase 4 boundary.

8. Platform Provisioner (Phase 1)

Full design: docs/LAB_ARCHITECTURE.md + docs/PHASE1_PLATFORM_FOUNDATION.md. Operational reference (config, local dev, known limitations): platform/provisioner/README.md. This section is just the map.

• What it is: an internal-only service that provisions a disposable, fully isolated Lab environment per student — its own Docker network, its own container running the existing, unmodified Lab image, its own firewall-enforced egress. Postgres-backed registry (environments, environment_events, lab_hosts), idle reaper, startup reconciliation.
• What it is not: directly reachable by a student or the Gateway anymore — as of Phase 2B, the Platform API (§9) is the only caller (student launch/reset AND admin prepare/destroy/ reset both go through it). Still exercised directly via its internal API (shared-secret auth) or platform/provisioner/test/run.js for isolated Provisioner testing.
• Where identity lives: as of Phase 2B, Postgres is authoritative — students/cohorts/progress/scenario_attempts live in the same rootnode_platform database as environments, owned by the Platform API's own migrations. See §9. As of Phase 3A, instructors/sessions/cohort_instructors/audit_events live there too. See §10. environments.student_id is still a plain opaque reference, not a real FK — that's a deliberate, documented choice (docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md §2), not an oversight.
• Local dev: platform/provisioner/docker-compose.dev.yml runs Postgres + the egress proxy + the Provisioner on one machine — the documented Phase 1 dev exception. Production is a two-VM Proxmox split (Platform VM / Lab Host VM); the dev compose file is explicitly not that.
• Test procedure: platform/provisioner/test/run.js — 12 automated integration tests against a live dev stack (isolation, firewall ordering, reset/destroy cleanup, audit events). Some properties (idle reaper timing, remote TLS lab-host, reconciliation-on-crash, real-image k3s behavior) were validated manually during implementation rather than automated — see the Phase 1 final report for what and why.

9. Platform API (Phase 2B)

Full design: docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md. Operational reference: platform/api/README.md.

• What it is: platform/api/ — a lightweight Node/Express service (no k3s, no --privileged, no Lab terminal runtime) owning students, cohorts, enrollments, curriculum assignments, progress, scenario attempts, and — as of Phase 3A — instructor accounts, persistent sessions, and the audit log (§10). The sole caller of the Provisioner for both the student launch/reset path and the instructor/admin control plane.
• What it is not: reachable directly by a browser — always through the Gateway (platform/gateway), same trust boundary the Provisioner has relative to this service.
• Local dev: part of the same platform/docker-compose.dev.yml stack as the Provisioner — no separate compose file.
• Test procedure: platform/test/e2e-phase2a.js (student flow) and platform/test/e2e-phase2b.js (classroom acceptance workflow) — both need a bootstrapped instructor account as of Phase 3A, see §10.

10. Identity and instructor access (Phase 3A)

Full design: docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md. Operational reference: platform/api/README.md (bootstrap, config).

• What it is: persistent Postgres-backed sessions (student + instructor, survive a Platform API restart), real per-instructor accounts (owner/admin/instructor roles, scrypt password hashing, per-instructor salt), server-enforced RBAC on every /api/admin/* route, cohort-scoped access for the plain instructor role, CSRF double-submit protection, login rate limiting, and a platform-level audit_events log.
• What it replaced: Phase 2B's in-memory session Maps (gone on every restart) and the single shared ADMIN_PASSWORD env var (no accountability, no roles, no revocation).
• Bootstrap: node platform/api/bootstrap-owner.js <email> <name> <password> — the only supported way to create the first instructor account; no manual database editing.
• Test procedure: platform/test/e2e-phase3a.js — 21 automated tests covering session persistence (including a real docker compose restart api), instructor auth, RBAC, cohort scoping, CSRF, rate limiting, password change, and audit logging.
• Known limitations, future OIDC path, production HTTPS expectations: see docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md §15–17.

11. Dependency lock convention

Every first-party Node service commits its package-lock.json: backend/, frontend/, platform/api/, platform/gateway/, platform/provisioner/. This wasn't consistently true before Phase 3B — platform/api/'s lockfile existed locally but was never committed, and platform/gateway//platform/provisioner/ never had one generated at all. Neither was a deliberate policy; both were oversights, fixed in Phase 3B (chore(deps) commits).

Going forward: after any npm install that changes a service's dependencies, the updated package-lock.json is part of the same commit as the package.json change — never left untracked, never regenerated-and-ignored. Validate a lockfile actually matches its package.json with npm ci (fails loudly on drift) before committing either.

12. Learning & Assessment Engine (Phase 3B)

Full design: docs/PHASE3B_LEARNING_AND_ASSESSMENT.md.

• What it is: the teaching/assessment layer on top of everything Phases 1–3A built — detailed per-attempt telemetry (scenario_attempts, evolved from Phase 2B's one-row-per-click shape into a real attempt lifecycle), server-enforced progressive hints, rule-driven curriculum stage/capstone unlock logic, a full exam + capstone assessment engine (assessment_definitions/assessment_sessions/assessment_session_scenarios), and persistent achievements records.
• Where the boundary moved: the Gateway now rewrites one Lab response (GET /lab/:token/api/scenarios/:scenarioId, stripping hints) so progressive hints can be genuinely server-gated rather than just UI-collapsed — the Lab image's own code/bytes are still untouched; this is a response rewrite at the Gateway boundary, same pattern as every other Lab-mediation decision made in earlier phases. The Lab's own frontend bundle did change this phase (ScenarioPanel.jsx, new ProgressiveHints/AssessmentBanner components) — Phase 3B explicitly authorized enhancing Lab UX, unlike prior phases.
• Test procedure: platform/test/e2e-phase3b.js — covers attempt telemetry, hint sequencing/persistence-through-Lab-reset, curriculum/capstone lock-unlock, the exam engine (random draw, timer, expiry, restart persistence), two-student isolation, and instructor analytics/RBAC.
• Known limitations: see the Phase 3B doc §20 — no automatic abandoned-attempt detection, no per-cohort needs-attention threshold configuration, raw-JSON pool_spec authoring in the admin UI (correct, not yet a guided form).

13. Production Infrastructure (Phase 4)

Full design: docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md, plus detail docs PRODUCTION_DEPLOYMENT.md, PRODUCTION_NETWORK_SECURITY.md, BACKUP_DR.md, OBSERVABILITY.md, CAPACITY_PLANNING.md, PRODUCTION_VALIDATION_CHECKLIST.md, and 12 RUNBOOK_*.md files, all under docs/. Everything lives in infra/ (Ansible, Terraform, Compose, firewall, monitoring, backup, CI, load-test tooling).

• What it is: the infrastructure-as-code needed to run rn-platform-01/rn-lab-host-01 on a real Hetzner/Proxmox host — buildable and locally testable now, even though the dedicated server hasn't been purchased. Status is explicitly PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE, not "deployed."
• Where the boundary held: the Platform VM / Lab Host VM security split from docs/LAB_ARCHITECTURE.md is codified in firewall rulesets and mutual-TLS Docker config, not weakened anywhere.
• Known gap surfaced, then resolved in Phase 4.1: the Gateway's Lab-reachability mechanism didn't survive the move to two real separate VMs — flagged explicitly rather than silently fixed, then actually built and simulation-tested (a Lab Host Router, platform/lab-router/) in Phase 4.1. See §13 below.
• Validated without the hardware: real tofu validate, ansible-playbook --syntax-check, docker compose config, caddy validate, promtool/amtool, and a genuine end-to-end backup→encrypt→restore→verify cycle against disposable local Postgres with the real production migrations applied. None of this proves the real Hetzner network boundary — that needs docs/PRODUCTION_VALIDATION_CHECKLIST.md walked against the real server.

13. Pre-Production Hardening (Phase 4.1)

Full design: docs/PHASE4_1_PREPRODUCTION_HARDENING.md, plus docs/PRODUCTION_NETWORK_SECURITY.md (routing resolution), docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md, docs/SECURITY_SCANNING.md.

• What it is: closes the Gateway-to-Lab cross-VM gap Phase 4 flagged (a Lab Host Router, platform/lab-router/, plus a routing abstraction in the Gateway, platform/gateway/src/lab-endpoint.js), then hardens the application layer found along the way: abandoned-attempt handling (scenario_attempts.status = 'abandoned', never a punitive failure), a Postgres-backed login rate limiter (survives a restart, ready to be shared/multi-instance), real pagination on audit/attempt/assessment-session history, a structured assessment authoring UI (owner-only raw JSON remains as a fallback), a minimal role_in_cohort enforcement (teacher/viewer), a /api/version endpoint, and admission control (a clean 503 "capacity full" instead of a 500 once a configured cap is reached).
• Tested, not just designed: a real two-Docker-network routing simulation (infra/scripts/simulate-two-host/, 13/13), a destructive Postgres restore drill (infra/backup/test-destructive-restore-drill.sh), a Lab Host loss/rebuild drill (infra/scripts/drills/test-lab-host-loss.js), a deployment rollback drill, a CI-fails-closed drill (which found and fixed a real pre-existing set -e bug in both smoke-test scripts — see that drill's own output), and a full fake-cohort dress rehearsal (infra/scripts/drills/test-fake-cohort-rehearsal.js) — all passing, all in infra/scripts/drills/ or infra/backup/.
• Known limitation carried forward: real remote/self-paced delivery needs actual student authentication (today's student_id-only login relies on instructors controlling who gets a valid id) — see docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md for the full accounting of what already works vs. what's genuinely Phase 5 scope. Closed in Phase 5A, below.

14. Student Identity & Remote Classroom (Phase 5A)

Full design: docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md, plus docs/RUNBOOK_STUDENT_ACCOUNT_LIFECYCLE.md and docs/RUNBOOK_REMOTE_CLASSROOM.md.

• What it is: two deliverables built on the Phase 4.1 milestone. (A) Real student accounts — email-or-login-identifier + password, reusing instructors' exact credential machinery, a single first-login/reset token mechanism, deactivation enforced per-request — replacing pilot "pick your name" identification without touching a single existing pilot student's progress. (B) A remote classroom control center — scheduled/live/ended sessions, derived (not stored) presence via heartbeat, a live instructor dashboard combining real Provisioner Lab state and real academic records, a help-request queue, guidance-only announcements/current-activity, and a generic external meeting-URL field. Explicitly not video/voice/chat/shell-takeover — RootNode owns identity and coordination, an external service owns the call.
• Tested, not just designed: platform/test/e2e-phase5a.js (21 checks: the two-student RBAC matrix and full password-security checklist, including a real Platform API restart proving rate-limit state survives it) and infra/scripts/drills/test-phase5a-rehearsal.js (15 checks: 5 pilot students with seeded real history, a live classroom, help requests, a real mid-rehearsal API restart proving classroom/session/presence recovery, then converting two students and confirming zero orphaned data) — both against a live docker-compose.dev.yml stack, not mocks. The full pre-existing regression suite was re-run given how much shared code this phase touched (server.js, admin-routes.js, sessions.js, rbac.js).

15. Self-Paced Learning & Enrollment Foundation (Phase 5B)

Full design: docs/PHASE5B_SELF_PACED_ENROLLMENT.md, plus docs/RUNBOOK_SELF_PACED_ENROLLMENT.md and docs/WEBSITE_ENROLLMENT_INTEGRATION.md.

• What it is: programs (a first-class learning product, distinct from curriculum JSON — curriculum_path is just what it points at) and enrollments (a student's academic access to a program, independent of cohort membership — a self-paced enrollment has no cohort at all). Server-enforced entitlement (enrollment.deriveEnrollmentAccess, mirroring classroom.js's derived-not-stored derivePresence pattern from Phase 5A) gates a new POST /api/student/programs/:id/launch in front of the exact same disposable-Lab sequence /api/environment/launch already used (extracted into environment-view.js so both share one implementation) — pending/future/expired/cancelled enrollments cannot launch, server-side, not just hidden in the UI. Completion is entirely server-derived from the existing capstone achievement mechanism (Phase 3B, unmodified) and idempotent. A new, separate POST /api/integrations/enrollments boundary (its own pre-shared-secret auth, no session cookie involved — same trust-boundary pattern as Gateway↔Provisioner's PROVISIONER_SHARED_SECRET) is what the separate public website will eventually call; idempotent via a DB unique constraint on the caller's own event id, proven safe under real concurrent duplicate requests, not just sequential retries.
• A real bug found and fixed: POST /api/progress/record predated the wrap() async-error-handling convention — an unexpected internal error there (a malformed environment_id) crashed the entire Platform API process, not just that one request. Fixed (wrapped + input-validated) and regression-tested.
• Tested, not just designed: platform/test/e2e-phase5b.js (32 checks) and infra/scripts/drills/test-phase5b-selfpaced-rehearsal.js (15 checks) — both against a live stack, including real Lab destroy/reaper-reconciliation/recreate persistence proofs, a full real capstone completion, integration-endpoint concurrency, and a real Platform API restart. Migration safety proven explicitly against a fresh Postgres seeded with realistic pre-5B data (see that doc §14).