Author SHA1 Message Date
ced4568 5bc0743b08 docs(phase4.1): pre-production hardening documentation
Release to Docker Hub / Build & Push Docker Image (push) Canceled after 0s
Release to Docker Hub / Create GitHub Release (push) Canceled after 0s
Master overview (PHASE4_1_PREPRODUCTION_HARDENING.md) covering all 27
substantive items, plus REMOTE_AND_SELF_PACED_ARCHITECTURE.md (reviews
what already works for remote/self-paced delivery - student-driven
provisioning, cohort-less students, idle-reaper-destroy-then-continue -
and the one real gap: student authentication, explicitly Phase 5 scope,
not built here).

Updated PRODUCTION_NETWORK_SECURITY.md (the Gateway-to-Lab gap is now
RESOLVED, with the full design writeup and a new control-plane-vs-data-plane
section), PRODUCTION_DEPLOYMENT.md (four-cert TLS bundle),
PRODUCTION_VALIDATION_CHECKLIST.md (Router mTLS check, admission control
checks), PHASE4_PRODUCTION_INFRASTRUCTURE.md and ENGINEERING_SOP.md and
LAB_ARCHITECTURE.md (stale 'known gap' pointers updated to point at the
resolution), BACKUP_DR.md (destructive drill), README.md and
platform/README.md (Phase 4.1 pointers, lab-router component, updated
known-limitations).

Also fixes a real pre-existing bug found while auditing what gets
committed: infra/.gitignore's patterns were written repo-root-relative
(e.g. "infra/terraform/...") but the file lives in infra/, so git resolved
them relative to THAT directory and none of them ever matched anything -
a real .terraform/ provider cache showed up as untracked when it should
have been ignored. Fixed by dropping the redundant leading "infra/" from
every pattern.

Status: PHASE 4.1 COMPLETE / PRE-PRODUCTION HARDENING COMPLETE / READY FOR
HARDWARE DEPLOYMENT.
2026-08-24 20:16:20 -05:00
ced4568 f3c356951c test(preprod): two-host routing simulation, drill suite, CI security scanning
- infra/scripts/simulate-two-host/: real two-Docker-network simulation
  (Gateway genuinely isolated from any Lab bridge network) proving the
  Lab Host Router fix - 13/13 real assertions (HTTP+WebSocket through the
  Router, Alice/Bob isolation, garbage/destroyed tokens, mTLS-only Router
  access).
- infra/scripts/drills/: Lab Host loss/rebuild (9/9), deployment rollback,
  CI-fails-closed (6 required failure modes), fake cohort dress rehearsal
  (14/14, one continuous narrative through the real app).
- infra/ci/gitea-workflows/build-and-test.yml: new security-scan job
  (npm audit, gitleaks, hadolint, shellcheck, trivy - all free/open-source),
  severity policy in docs/SECURITY_SCANNING.md (CRITICAL/HIGH blocks,
  MEDIUM/LOW reported only). Gates build-images alongside regression.

Found and fixed two real pre-existing bugs via the CI-fails-closed drill:
staging-smoke-test.sh and production-smoke-test.sh both had a classic
set -e gotcha (curl ... && echo "OK" as one statement - a failing command
that isn't last in an && chain doesn't trigger -e) that let an
unreachable/broken endpoint fall through silently instead of failing the
smoke test.

platform/test/e2e-phase3a.js updated for the new paginated
/api/admin/audit-events response shape (Phase 4.1 pagination work).
2026-08-24 20:16:00 -05:00
ced4568 f5110d3c1b test(dr): destructive Postgres restore drill
infra/backup/test-destructive-restore-drill.sh - realistic academy data
(owner, admin instructor, cohort, 3 students, progress, attempts, an
assessment definition, achievements) seeded through the REAL Platform API,
encrypted-backed-up, the source Postgres container genuinely destroyed
(docker rm, not just left running alongside a second one), a fresh
Postgres created from nothing, restored, and the real Platform API booted
again against the restored database and used to log in as the restored
admin instructor over HTTP. Never touches the developer's real database.
Confirmed passing.
2026-08-24 20:15:45 -05:00
ced4568 a8e543234c feat(platform): pagination, admission control, cohort roles, version endpoint, structured assessment editor
- Real server-side pagination (platform/api/src/pagination.js) on the audit
  log (was a flat LIMIT 200), student attempt history (was LIMIT 100), and
  assessment session history (was fully unbounded - the one genuine growth
  risk found). Frontend updated (AuditView, AssessmentsView, StudentsView).
- Admission control: resource-policy.yaml's new (disabled-by-default)
  admission_control.max_concurrent_labs, checked before touching Docker at
  all, returns a clean 503 CAPACITY_EXCEEDED instead of failing mid-provision.
  Relayed as a clean student-facing message, never a raw 500. New
  GET /capacity endpoint feeds the instructor dashboard.
- cohort_instructors.role_in_cohort (existed since Phase 2B, never read) is
  now enforced: teacher vs viewer, rbac.js's
  requireCohortWriteAccess/requireStudentCohortWriteAccess.
- GET /api/version - app version, deployed Git SHA, latest migration as a
  schema-version stand-in.
- Structured assessment authoring UI (AssessmentsView.jsx's PoolSpecEditor)
  replaces the raw-JSON pool_spec textarea, backed by a new
  GET /api/admin/scenario-catalog endpoint. Owner-only raw-JSON fallback
  remains for anything the structured editor doesn't cover.
- NEEDS_ATTENTION_FAIL_THRESHOLD/STALE_DAYS moved to env-var configuration.

Full regression suite (e2e-phase2a/2b/3a/3b, 67/67) re-run clean after
these changes; frontend production build confirmed clean.
2026-08-24 20:15:39 -05:00
ced4568 addf474b70 feat(security): Postgres-backed login rate limiting
Replaces the in-memory sliding-window Map (reset on every restart, not
shareable across instances) with a login_attempts table (migration 016).
No Redis added - login is not a hot path, and this is a single indexed
range-scan query. Restart behavior confirmed for real:
platform/test/e2e-phase3a.js's rate-limiting test passes after a genuine
Platform API restart in the same run.
2026-08-24 20:15:03 -05:00
ced4568 6dda13b837 feat(learning): abandoned attempt handling
scenario_attempts.status = 'abandoned' (a valid value since Phase 3B,
never previously set by anything) now gets set on: environment
destroy/reset, logout, session expiry (piggybacked on the existing hourly
session sweeper - no new background job), and switching to a different
scenario while one is still open. Never an academic failure marker -
result stays NULL, history is preserved. Call-site wiring in
server.js/admin-routes.js lands in the next commit alongside other
platform hardening in those same files.
2026-08-24 20:14:52 -05:00
ced4568 eae8e48d43 feat(routing): Lab Host Router closes the Gateway-to-Lab cross-VM gap
Adds platform/lab-router/ — a narrow service that runs on the Lab Host,
authenticates the Gateway via mTLS, and resolves an opaque routing token to
a local container purely from Docker label state (no Postgres access, no
callback to the Provisioner). Replaces the Gateway's env.internal_ip
assumption (only valid when Gateway and Lab share a Docker daemon) with a
data-driven direct/router decision per lab_host
(lab_hosts.routing_mode/router_endpoint, migration 004), resolved in one
place (platform/gateway/src/lab-endpoint.js's labTarget()) that every
proxy/fetch call site in server.js now goes through.

Extends the existing Docker TLS CA (infra/scripts/docker-tls/generate-docker-tls.sh)
with a second mTLS leaf pair (router-server/gateway-client) for this new
data-plane connection, deployed by the docker-tls Ansible role into a
separate directory from the control-plane pair. Firewall
(rn-lab-host-01.rules) replaces the old speculative wide port-range
allowance with one narrow rule: platform_ip -> the Router's port only.

dev-local stays routing_mode: direct — zero behavior change for local dev,
confirmed by the full existing regression suite passing unmodified.
2026-08-24 20:14:39 -05:00
ced4568 bf1e1b5c76 docs(phase4): production infrastructure documentation + runbooks
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs
(PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR,
OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST,
and 12 runbooks (first deployment, normal deploy, rollback, Postgres
restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner
recovery, stuck Lab cleanup, failed migration, backup failure, disk
failure, Lab Host resource pressure).

PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine
architectural gap this phase surfaced: the Gateway's existing
env.internal_ip mechanism for reaching a Lab container assumes Gateway and
Lab share a Docker daemon, which breaks once Labs run on a genuinely
separate rn-lab-host-01 VM. The network-policy half is built (a firewall
allowance for Gateway->Lab-Host on a published port range); the
application half (Provisioner/Gateway address handling) is an explicit,
reviewed, NOT-YET-BUILT follow-up flagged there rather than silently
decided.

Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and
platform/README.md with Phase 4 pointers, consistent with how every prior
phase updated these.

Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
2026-08-18 17:13:51 -05:00
ced4568 1d3afc736b feat(observability): add /healthz to Platform API and Gateway
Unauthenticated liveness/readiness probes for uptime monitoring and Ansible
deploy smoke tests (Phase 4). API's checks Postgres reachability via
SELECT 1; Gateway's answers locally without proxying anywhere, so it can
distinguish 'Gateway itself is down' from 'something behind it is down'.
No student/instructor data exposed.

Verified live against the real dev stack: both return 200 with the
expected JSON body. Full regression suite re-run after this change:
e2e-phase2a (10/10), e2e-phase2b (12/12), e2e-phase3a (21/21), e2e-phase3b
(24/24) - all passing, no regressions. Frontend build and the Lab image
(both the multi-service dev compose and the standalone single-container
Dockerfile) also confirmed still building/booting correctly.
2026-08-18 17:13:42 -05:00
ced4568 51a09241ad feat(loadtest): load-test tooling + capacity model
boot-labs.js drives the real admin Prepare-Labs endpoint (reusing the
existing session-helper.js from Phase 3A) and records boot time/failure
rate; sample-resources.sh samples real docker stats for running Labs.
capacity-model.js turns measured per-Lab CPU/memory into a recommended/
warning/hard-admission-control concurrency threshold - no concurrency
number is hardcoded as a marketing figure, every input is explicitly
placeholder data until measured against the real host.

Plus deploy/rollback/smoke-test scripts used by the CI/CD and deployment
runbooks. All syntax-checked; validate-compose.sh and capacity-model.js
actually run against real inputs.
2026-08-18 17:13:32 -05:00
ced4568 67e172c800 feat(cicd): Gitea Actions pipeline, deterministic image tagging
Automatic test/build/tag/staging-deploy on push to main; production
promotion stays a manual, human-run step (deploy-production.sh) - never
automated. SHA-tagged images always, semantic tags only on deliberate
release, never :latest alone in staging/production. No paid CI provider or
registry required (uses Gitea's own). Real regression job installs all
services, applies real migrations against a service Postgres, runs the
infra validation scripts.

Validated as real YAML (python3 yaml parse).
2026-08-18 17:13:25 -05:00
ced4568 b1b22b9285 feat(backup): encrypted Postgres backup/restore + off-host adapter
pg_dump -> gzip -> gpg AES-256 symmetric encryption, plaintext deleted
immediately, optional off-host rsync-over-SSH copy (Hetzner Storage Box or
any SSH target), retention pruning, Prometheus textfile-collector metrics
for backup success/age/size.

Genuinely tested end-to-end, not just written: real production migrations
(17 total, correct cross-service order) applied to a disposable Postgres,
sample data inserted, backed up, restored into a second disposable
Postgres, verified table-for-table and byte-for-byte. Exit 0.
2026-08-18 17:13:19 -05:00
ced4568 4e8fa1d413 feat(observability): Prometheus/Alertmanager/Grafana/Loki stack
Self-hosted, no paid SaaS. Host/container/Postgres metrics, uptime probes
against the new /healthz endpoints, alert rules split critical (pages) vs
warning (informational), one starter Grafana dashboard. Deliberately
separate from the application's instructor learning analytics (Phase 3B) -
different audience, different data.

Validated: promtool check config (valid), promtool check rules (11 rules,
valid), amtool check-config (1 inhibit rule, 2 receivers, valid).
2026-08-18 17:13:14 -05:00
ced4568 d2280d0f0d feat(security): production Compose topology, firewall rulesets, Docker mTLS
Named-bridge-network production Compose (not host networking, unlike dev)
for rn-platform-01; egress-proxy + agents for rn-lab-host-01. Caddy reverse
proxy, single public origin, Grafana restricted to management_cidr.
iptables-restore firewall rulesets for all three hosts (Proxmox host,
rn-platform-01, rn-lab-host-01) codifying the Platform/Lab-Host boundary:
no student-to-student traffic, no direct Proxmox exposure, no direct
Postgres/Provisioner exposure from the Lab Host.

Mutual TLS for the Provisioner<->Lab Host Docker connection: certificate
generation/verification scripts, dockerd TLS config, firewall-restricted
port 2376. Application support already existed since Phase 1; this is
purely the operational half.

Validated: docker compose config, caddy validate, firewall structural lint,
and a real openssl s_server/s_client mTLS handshake proving the cert chain
and mutual-auth rejection (legitimate client verified, rogue client
correctly rejected with verify code 19).
2026-08-18 17:13:09 -05:00
ced4568 507ac4c5d3 feat(ansible): idempotent role-based host config for both production VMs
common/firewall/docker-tls/platform/lab-host/monitoring-agent/backup roles,
orchestrated by site.yml with the ordering that matters (firewall/TLS land
before the services that depend on them). Ansible Vault for secrets, no new
tool introduced for this project's modest secret surface. Explicit
dev/staging/production environment separation, no secrets committed
(.gitignore covers vault plaintext, TLS keys, Terraform state, *.env).

Validated: ansible-playbook --syntax-check clean.
2026-08-18 17:13:01 -05:00
ced4568 cbf81b09e7 feat(proxmox): Proxmox VM template + Terraform/OpenTofu VM creation
Manual, documented bare-metal Proxmox install + cloud-init template build
(deliberately not automated — a fragile one-time step). Terraform/OpenTofu
config creates rn-platform-01/rn-lab-host-01 from that template via the
bpg/proxmox provider, with configurable (not hardcoded) VM sizing.

Validated: tofu fmt, tofu init -backend=false (real provider download),
tofu validate -> Success. No paid state backend required (local state).
2026-08-18 17:12:55 -05:00
ced4568 45a147a61f docs(phase3b): document the learning and assessment engine
New docs/PHASE3B_LEARNING_AND_ASSESSMENT.md covers the attempt/learning-record model,
progressive hints (including the Gateway response-rewrite mechanism), curriculum completion
rules, the assessment/exam engine and its timer/draw integrity guarantees, the RootNode
Capstone design, achievement records, the deliberately minimal versioning model, instructor
analytics, the Phase 3B RBAC matrix, security/integrity assumptions, dev/test commands, known
limitations, and the Phase 4 boundary.

Updates ENGINEERING_SOP.md (roadmap items 2/7/8 now done, new section 12, corrects a stale
capstone-placeholder description in section 4), LAB_ARCHITECTURE.md's status line,
README.md (corrects the 'Lab image unmodified' claim now that its frontend bundle carries
Phase 3B UX, plus new platform properties), platform/README.md, and platform/api/README.md.
2026-08-18 14:36:13 -05:00
ced4568 8956a95446 test(phase3b): add learning and assessment coverage
24 cases: attempt telemetry (start/fail/pass counters, duration), progressive hint sequencing
and exhaustion, hint persistence through a real Lab reset, curriculum stage lock/unlock,
capstone lock-before-prerequisites/unlock-after/random-draw-includes-required-Real-Ops/
cannot-double-start/completion-records-achievement, exam assignment/draw/timer/hint-policy,
cross-student session-read rejection (404, not 403 - consistent with the existing don't-confirm-
existence posture), two-student isolation, timer survives a Lab reset, an assessment session
survives a REAL docker compose restart api, and instructor analytics/needs-attention/RBAC.

Found and fixed two real product bugs in the process: result needed to become nullable (an
in-progress attempt has none yet) and hint_count needed a real DEFAULT 0 (Phase 2B left it
NULL/undefaulted since it wasn't measured yet - it is now, and NULL + 1 stays NULL in SQL).
2026-08-18 14:36:13 -05:00
ced4568 d4db8f99f5 feat(student): add learning tabs and Lab hint/assessment UX
StudentDashboard is now tabbed: My Learning (the original Phase 2A/2B flow, extracted
unchanged into MyLearningTab and extended with an overall percent-complete bar), Practice,
Assessments (available/active-with-live-countdown/history), Capstone (locked checklist ->
unlocked -> start/resume -> completed), Achievements (a flat dated list, not a leaderboard).

Inside the Lab UI: ProgressiveHints replaces the static hint array render when the Gateway has
stripped hints (detected via scenario.hints being undefined but hint_count present - standalone
dev mode never sees this, so it keeps the original behavior); a new top-level AssessmentBanner
shows a live countdown and completion count whenever a Platform assessment is running,
rendered alongside (not replacing) the pre-existing Lab-local exam-mode UI, which is untouched.
2026-08-18 14:35:59 -05:00
ced4568 7e5f357b6f feat(admin): add instructor learning analytics and assessment management
New /api/admin routes: cohort learning summary (completion by stage, active learners),
per-student learning detail (attempts, hint usage, assessment history, capstone state),
scenario analytics (completion rate, avg attempts, failure/hint frequency, avg time), and a
needs-attention view (repeated failures, hints used but still stuck, inactivity, an expired
assessment - deliberately instructional wording, not punitive, with an explicit test asserting
that). Assessment definition CRUD (admin/owner only), cohort assignment and session view/cancel
(any instructor role, cohort-scoped for a plain instructor via the existing Phase 3A RBAC
helpers - no new authorization primitives needed).

New Learning and Assessments tabs in the admin portal, visible to every instructor role;
definition-authoring controls are shown but the server 403s a create/edit from a non-admin/
owner role, same pattern CohortsView already uses for admin-only controls.
2026-08-18 14:35:59 -05:00
ced4568 4e3dacc043 feat(hints): wire student-facing learning API and progressive hint reveal
New learning-routes.js: scenario start/attempt-state/hint (sequential reveal, 409 once
exhausted, policy-aware under an active assessment), GET /api/curriculum/state (rule-driven,
alongside the unchanged /api/curriculum/path), assessments list/start/active/detail/submit,
GET /api/capstone, GET /api/achievements, GET /api/attempts/recent.

/api/progress/record now routes through attempts.recordValidation instead of a flat insert,
and, on a pass, checks curriculum/capstone achievement consequences and, if the scenario is
part of an active assessment, updates that session (assessment context is derived entirely
server-side from assessment_session_scenarios, never from anything the client asserts).

Gateway changes: two new relay routes (/lab/:token/api/scenarios/:id/start and .../hint,
mirroring the existing validate/answer pattern - pure Platform API operations, the Lab has no
attempt/hint concept) and a response rewrite on GET /lab/:token/api/scenarios/:id that strips
the Lab's own hints array (replacing it with a bare hint_count) so progressive reveal is
actually server-gated instead of just UI-collapsed - the Lab image's bytes stay untouched;
standalone/single-container dev mode never goes through the Gateway and keeps the full array
unaffected.
2026-08-18 14:35:44 -05:00
ced4568 0beb0e8f5e feat(learning): add persistent achievement records
New achievements table: stage_completed/curriculum_completed/assessment_completed/
capstone_completed, idempotent (WHERE NOT EXISTS guard, not a unique-index dance — simple and
correct at this write volume). Not a certificate — a durable internal record Phase 4/5 can
render one from later. recordAssessmentAchievement only fires on an actual pass, defensively,
even though today's only completion_rule already guarantees that.
2026-08-18 14:35:11 -05:00
ced4568 b908ade8e0 feat(assessment): add exam and capstone definition/session engine
New tables: assessment_definitions (reusable template — pool_spec, duration, hint_policy,
completion/unlock rules, a plain incrementing version bumped only on semantic edits),
assessment_cohort_assignments, assessment_sessions (one per student attempt, full snapshot of
what was drawn and under what rules — never affected by a later definition edit),
assessment_session_scenarios (the drawn set, ordered, with a real FK to scenario_attempts).
Deliberately does not touch/reuse exam_sessions (migration 006) — that was already scoped in
Phase 2B as historical-import-only, never the live mechanism; these tables are the real thing.

drawScenarios: every 'required' scenario id included, plus N random picks per pool group,
shuffled together (tiny seeded PRNG, mulberry32 — no new dependency; the seed is recorded for
audit, never shown to the student before the draw). One assessment at a time per student.
Timer is server-authoritative (expires_at fixed at draw time) and lazily expired — checked
whenever a session is touched, no background cron. Ships two seeded defaults: rootnode-capstone
(Real Ops scenarios required, weighted toward operational ability) and rootnode-quick-assessment
(a general example exam matching the spec's category-draw example).
2026-08-18 14:35:11 -05:00
ced4568 01a862e007 feat(learning): add scenario attempt telemetry and curriculum completion rules
Evolves scenario_attempts (Phase 2B: one row per validation click) into a real attempt
lifecycle — one row per continuous working session, with live counters (validation
attempts/passes/fails, hint reveals, duration) updated in place, not re-inserted. Backward
compatible: every new column is nullable/defaulted, existing rows backfilled under the new
semantics rather than dropped. hint_count (Phase 2B, always NULL — 'not measurable') finally
gets a real DEFAULT 0 and increments, since this is the phase that builds real hint tracking.

curriculum-rules.js computes rule-driven stage lock/complete/percent/next-scenario state from
scenarios/rootnode-path.json's new per-stage completion_rule (all_required/threshold/
prerequisites) against a student's progress. The capstone stage is no longer a hardcoded
locked placeholder — it carries an unlock_rule and a pointer at the assessment definition that
actually defines its scenario draw (see the assessment-engine commit).
2026-08-18 14:35:11 -05:00
ced4568 e793d9476b docs(deps): document the dependency-lock convention
Was ambiguous — backend/frontend committed lockfiles, api/gateway/provisioner didn't, with
no stated reason either way. Now explicit: every first-party Node service commits its
package-lock.json, validated with npm ci before each commit that touches it.
2026-08-18 13:50:38 -05:00
ced4568 957a99645a chore(deps): track gateway and provisioner package locks
backend/ and frontend/ already had committed lockfiles; platform/api/ just gained one.
gateway/ and provisioner/ never had one generated at all — not a deliberate policy, an
inconsistency. Generated via npm install --package-lock-only and validated with a clean
npm ci for both (72 and 153 packages respectively, no resolution surprises).
2026-08-18 13:50:13 -05:00
ced4568 60a1e79930 chore(deps): track platform API package lock
Was generated locally during Phase 3A work but never committed. Verified it corresponds
to the current package.json via a clean npm ci (118 packages, no resolution changes).
2026-08-18 13:49:35 -05:00
ced4568 40692926ee docs(phase3a): document production identity architecture
New docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md covers the schema, persistent session
design, instructor accounts, the full RBAC matrix, cohort scoping (including the documented
raw-environment-id simplification), student session hardening (explicitly still
classroom/pilot identification, not production auth), CSRF, rate limiting, audit logging,
the API inventory, a security hardening checklist, test results, known limitations, the
future OIDC path, and production HTTPS/cookie expectations — with an explicit BUILT NOW vs.
FUTURE split.

Updates ENGINEERING_SOP.md (new §9/§10, closing a stale '(§9)' forward-reference that dated
back to Phase 2B), LAB_ARCHITECTURE.md status line, README.md, platform/README.md, and
platform/api/README.md (bootstrap-owner usage, new config vars, no more ADMIN_PASSWORD).
2026-08-18 13:36:45 -05:00
ced4568 face0ae132 test(phase3a): add identity and RBAC end-to-end coverage
New e2e-phase3a.js (21 cases): no account enumeration, CSRF positive/negative, login rate
limiting, RBAC across all three roles, cohort-scoped instructor access (visibility + route
access), last-owner protection, live role/status re-evaluation (no separate revoke needed),
password change (old password rejected after, other sessions revoked), audit log content
(including a no-secrets-in-metadata assertion), student session RBAC/logout, and session
persistence across a REAL docker compose restart api mid-test.

New session-helper.js — a shared cookie-jar test helper (Node's fetch doesn't persist
cookies across calls) that also auto-attaches X-CSRF-Token from whichever rn_csrf_*
cookie it's holding, replacing the single-cookie-string trackers e2e-phase2a.js and
e2e-phase2b.js each had their own copy of.

e2e-phase2a.js and e2e-phase2b.js updated for the new auth model: admin login is now a real
bootstrapped OWNER_EMAIL/OWNER_PASSWORD account instead of the old shared ADMIN_PASSWORD.
All cases still passing: 10/10 and 12/12 respectively.
2026-08-18 13:36:45 -05:00
ced4568 1ca5dd3995 chore(platform): retire ADMIN_PASSWORD from the dev stack
No more shared instructor password env var — the api service now needs only
PROVISIONER_SHARED_SECRET. First-run setup is npm run bootstrap-owner instead.
2026-08-18 13:36:29 -05:00
ced4568 777a1ef880 feat(admin): add instructor management, audit, and account UI
Login form is now email+password (was a bare shared password field). Nav conditionally
shows Instructors (owner only) and Audit (owner/admin) tabs based on the signed-in
instructor's role. New InstructorsView (create/list/edit accounts, role/status controls,
session list + revoke-all, password reset — owner only), AuditView (read-only audit event
table), AccountView (change-password form for the signed-in instructor). CohortsView hides
the create-cohort form and curriculum-assignment button for the 'instructor' role — the
server already enforces this; this just avoids showing controls that would 403.
2026-08-18 13:36:29 -05:00
ced4568 7ce435ad32 feat(security): add CSRF double-submit protection and login rate limiting
Non-HttpOnly rn_csrf_s/rn_csrf_i cookies, set on login, echoed back as X-CSRF-Token on every
state-changing request — defense in depth on top of (not instead of) SameSite=Lax. In-memory
sliding-window rate limiting on instructor login (8 attempts / 5 min per ip:email, no new
dependency).

Real bug found while wiring this up: the Gateway's own progress-relay call (triggered when a
student solves a scenario) is a server-to-server POST to /api/progress/record relaying the
browser's session — it needs to forward the CSRF token too, and needs to forward BOTH the
rn_csrf_s cookie and a matching X-CSRF-Token header (forwarding only the header still 403'd,
since the double-submit check needs both on the request Platform API actually receives).

Also tightens the Gateway's session-resolution cache from 5000ms to 2000ms, so a
revoked/expired session stops working at the Gateway within a tighter, tested bound.

Frontend (admin + student API clients) now reads the relevant rn_csrf_* cookie via
document.cookie and attaches it on every POST/PATCH/DELETE.
2026-08-18 13:36:20 -05:00
ced4568 62726455d5 feat(rbac): enforce owner/admin/instructor permissions across the admin API
Three roles (owner > admin > instructor), enforced server-side on every /api/admin/* route
via reusable middleware (requireOwner/requireAdminOrOwner/requireAnyInstructor) — the
frontend hides controls a role can't use, but that's UX, never the actual boundary. A plain
'instructor' is further scoped to cohorts assigned via the new cohort_instructors table
(requireCohortAccess/requireStudentCohortAccess); owner/admin bypass that check entirely.

Owner-only: instructor account lifecycle, audit log. Admin/owner: cohort/student CRUD,
curriculum assignment. Any instructor role (cohort-scoped for 'instructor'): dashboard,
cohort/student views, Prepare/Destroy Labs, provisioning. Deliberately NOT cohort-scoped:
the raw /api/admin/environments/:id/* routes (no cohort context in a bare environment id
without an extra Provisioner round trip on every call) — a plain instructor still has full
Lab control via the cohort/student-scoped routes instead, which already carry the check.

server.js wires the new persistent-session middleware (studentSession/instructorSession —
re-fetches the instructor row from Postgres every request, so a role change or
deactivation takes effect on the very next request with no separate revoke needed) and adds
a last-resort error handler so a DB error in any async route returns 500 instead of
crashing the process or leaking a stack trace.
2026-08-18 13:36:11 -05:00
ced4568 37230d4ef1 feat(audit): add platform security audit log
New audit_events table, distinct from the Provisioner's environment_events (Lab-lifecycle
only). recordAudit() is called from the auth/RBAC/admin layers added in this phase — login
success/failure, logout, password change, instructor CRUD, role changes, session
revocation, cohort/roster changes, curriculum assignment, Prepare/Destroy Labs, and every
403 authorization denial (recorded centrally by the RBAC middleware, not per-route). Never
records plaintext passwords, raw session tokens, Provisioner secrets, or raw routing tokens.
2026-08-18 13:36:11 -05:00
ced4568 85fa920f96 feat(auth): add per-instructor accounts and local password login
Replaces the single shared ADMIN_PASSWORD (Phase 2B) with real accounts: instructors table
(email/display_name/role/status), scrypt password hashing with a fresh random salt PER
INSTRUCTOR (Phase 2B used one fixed salt because there was exactly one shared password —
that assumption is gone), auth_identities table so a future OIDC login can map to the same
instructor_id without a schema change.

New POST /api/auth/instructor/login, POST /api/auth/logout, GET /api/auth/session,
POST /api/auth/change-password (old password required, revokes other sessions). No account
enumeration: identical response whether the email doesn't exist or the password is wrong.

bootstrap-owner.js is the only supported way to create the first account — no manual
database editing to stand up a fresh deployment. Refuses to run if an active owner already
exists unless the target email is an existing instructor (zero-owners recovery path).
2026-08-18 13:35:54 -05:00
ced4568 52c17c1907 feat(identity): add persistent Postgres-backed session store
Replaces the in-memory studentSessions/adminSessions Maps (Phase 2B — gone on every
restart) with a sessions table. Browser holds an opaque CSPRNG bearer token; only its
SHA-256 hash is ever stored server-side, same principle as password hashing. Each session
has an absolute expiry (fixed at creation) and a sliding idle expiry (pushed forward on
every authenticated request, capped at the absolute one) — one atomic UPDATE...RETURNING
per resolve, no read-then-write race. Student and instructor sessions stay fully separate
token namespaces, matching Phase 2B's cookie-name separation.

cookies.js centralizes Set-Cookie construction so the Secure attribute (COOKIE_SECURE=1,
for real HTTPS deployments) is applied consistently instead of hand-built per call site.
2026-08-18 13:35:54 -05:00
ced4568 08ff5315de docs(phase2b): document admin and platform-data architecture
New docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md covers the Platform API
split, Postgres schema and migration guarantees/rollback, the session
cookie architecture (including the whoami-routing-bug fix), the admin
portal, the API/security boundary, known limitations, and Phase 3+
future work.

Updates ENGINEERING_SOP.md §8, LAB_ARCHITECTURE.md, README.md, and
platform/README.md to reflect Platform API as the sole Provisioner
caller and Postgres as authoritative for identity/progress as of
Phase 2B.
2026-08-18 12:23:17 -05:00
ced4568 6f1d202242 test(phase2b): add instructor classroom workflow coverage
Full acceptance test: admin auth (+ student-can't-call-admin check),
cohort create, 3-student add, curriculum assign, concurrent Prepare Labs
(3/3 ready, instant-reuse timing check), two students completing
different scenarios, dashboard reflecting completions, admin reset with
progress-intact assertion, admin destroy-labs with progress-intact
assertion for everyone, and a regression check that no Lab container
publishes a host port. 12/12 passing.
2026-08-18 12:23:17 -05:00
ced4568 9b28d786de chore(platform): wire platform-api into dev compose, retire identity service
Removes the old identity service/volume, adds the api service (host
networking, Postgres URL, Provisioner shared secret, ADMIN_PASSWORD,
both required via :?). gateway now points PLATFORM_API_URL at it instead
of the old IDENTITY_BACKEND_URL.
2026-08-18 12:23:17 -05:00
ced4568 8e4ef9ca53 feat(admin): add cohort and student management portal
New /admin route (frontend/src/admin/), gated by a separate admin
session (scrypt + timingSafeEqual against ADMIN_PASSWORD, no plaintext
storage, no new dependency). Covers cohort CRUD + roster + curriculum
assignment, and student CRUD + cohort assignment + environment/progress/
attempt views, all under /api/admin/* on the Platform API.

main.jsx now does a three-way path split (lab / admin / platform) so the
existing, unmodified Lab UI, the student dashboard, and the new admin
portal all ship in one Vite bundle without a router dependency.
2026-08-18 12:22:58 -05:00
ced4568 ae75bb4cbe feat(auth): add server-controlled student and admin sessions
Replaces the Phase 2A pilot ?sid= query param with an HttpOnly,
SameSite=Lax session cookie. Gateway now derives student identity from
the session (via Platform API's /api/session/whoami) instead of trusting
a client-suppliable id, and checks ownership on every proxied HTTP and
WebSocket Lab request, not just the initial navigation.

Fixes a routing bug found during testing: the fallback route was using
'does this session own a running environment' to decide whether an
unprefixed /api/* call belonged to the Lab UI or the dashboard — true in
both places. Replaced with an explicit LAB_OWNED_PATHS allowlist of the
exact unprefixed paths the existing Lab UI actually calls.

e2e-phase2a.js rewritten to use a cookie jar instead of headers/query
params, with explicit impersonation-rejection tests added. 10/10 passing.
2026-08-18 12:22:58 -05:00
ced4568 6b52c9e1d2 feat(platform-api): split lightweight platform service from Lab image
New platform/api/ service owns students, cohorts, curriculum, progress,
scenario attempts, and student/admin sessions. No k3s, no --privileged,
no terminal runtime — starts instantly, unlike the old identity service
which booted an unused k3s cluster just to serve roster/progress APIs.

Provisioner calls (student and admin) are consolidated here too, in
provisioner-client.js, so it's the only service that talks to the
Provisioner directly.
2026-08-18 12:22:58 -05:00
ced4568andClaude Sonnet 5 42dfb1c31b docs(phase2a): document gateway and student launch architecture
- docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md — the full Phase 2A record: node-pty
  root cause and fix, Gateway architecture, student flow (entry through
  reset/recreate), API routes added, HTTP/WebSocket routing design (and why
  the lab-session cookie exists), security model (what's checked, and the
  explicit statement that this is pilot identity, not production auth),
  reset/recreate and reaper-recovery semantics, and known limitations found
  during implementation (the identity instance's k3s resource cost and its
  network_mode: bridge requirement, the Gateway's in-memory cache scope, the
  ?sid= check's coverage). BUILT NOW vs. FUTURE (Phase 2B) called out
  explicitly per the governing requirement.
- docs/LAB_ARCHITECTURE.md, docs/ENGINEERING_SOP.md — status updated to
  reflect Phase 2A is implemented and tested, not just Phase 1; roadmap item 5
  updated accordingly; ENGINEERING_SOP's Provisioner section corrected (it's
  no longer un-fronted — the Gateway is the only caller now).
- README.md — Multi-Student Deployment section updated to describe the
  Gateway and student flow, not just the Provisioner; clarifies the
  node-pty fix is the only change to the Lab image itself.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-18 11:18:43 -05:00
ced4568andClaude Sonnet 5 05421bcb0e test(phase2a): add end-to-end student environment coverage
platform/test/e2e-phase2a.js — 15 assert-based tests against a live
platform/docker-compose.dev.yml stack, exercising the full student flow
through the Gateway exactly as a browser would: roster -> login ->
curriculum -> launch (real k3s boot) -> Gateway HTTP routing -> invalid-token
rejection -> cross-student ownership rejection (?sid= mismatch) ->
same-student access -> progress relay (solve in one student's lab, confirm
it shows in their curriculum and NOT the other student's) -> reset (stable
launch_path, progress survives).

All 15 pass. This is slow (each real environment launch is a genuine k3s
cold start) and deliberately not meant for every commit — see
platform/README.md for when to run it.

Does not cover what needs a browser/WebSocket client — real Lab image
terminal (node-pty, kubectl execution, scenario validation) and idle-reaper
timing were verified manually; see
docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md §8 for exactly what and how.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-18 11:16:44 -05:00
ced4568andClaude Sonnet 5 fb2945069c feat(student): add curriculum dashboard and lab launch/reset/recreate flow
StudentDashboard.jsx — "Up Next" curriculum view + Launch Lab flow, in one
component since they share state (environment status affects what curriculum
actions make sense).

- Renders GET /api/curriculum/path as-is: the existing RootNode path
  (Foundation -> Workloads -> Networking -> Configuration -> Troubleshooting
  -> Real Ops -> Capstone), no new structure invented. Capstone stays the
  locked placeholder from the earlier slice.
- Launch Lab -> POST /api/environment/launch, polls status every 2.5s while
  provisioning/resetting, transitions through none -> provisioning -> running
  -> error with a distinct UI state for each (EnvironmentPanel). On running,
  navigates to launch_path with the student's own id as ?sid= — a
  defense-in-depth ownership check the Gateway verifies on that specific
  navigation (see the gateway commit).
- One "Recreate Lab" control, calling POST /api/environment/reset — no raw
  destroy-without-recreate exposed to students, matching "don't expose raw
  destroy/admin controls beyond what is safe for a student." A visible note
  states progress is saved separately and survives.

Platform.module.css reuses the existing design tokens from index.css
(--green, --surface, --text, etc.) so this reads as the same product as the
Lab UI, not a bolted-on separate app.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-18 11:16:35 -05:00
ced4568andClaude Sonnet 5 5e4f5518fd feat(student): wire roster identity and student session
New Platform Shell entry point (frontend/src/platform/) alongside the
existing, completely unmodified Lab UI (App.jsx and everything under
frontend/src/components/). main.jsx now splits on the current URL path —
/lab/<token>/ renders the existing <App/> exactly as before; anything else
renders the new <PlatformShell/>. Both live in the same built bundle (it's
served both by the persistent identity instance and by every ephemeral Lab
container), split purely at runtime.

- EntryScreen.jsx — cohort dropdown -> name dropdown (existing
  GET /api/roster/cohorts, GET /api/roster?cohort=) -> POST /api/login. No
  password. Visibly labeled pilot identity, not production auth, per the
  governing requirement that this be stated explicitly, not just true in the
  architecture docs.
- api.js — student_id lives in localStorage only, sent as X-Student-Id on
  every authenticated request. Every authenticated call throws if no student
  is stored rather than silently going out anonymously — this is what
  "prevent accidental anonymous progress writes" means in practice, not just
  a comment.
- PlatformShell.jsx — entry screen if no stored session, dashboard otherwise.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-18 11:16:24 -05:00
ced4568andClaude Sonnet 5 ce6abfe6c7 feat(platform): relay scenario progress from ephemeral labs to identity backend
Real architectural gap found and fixed during Phase 2A testing, not part of
the original plan: scenario validate/answer requests proxied through
/lab/<token>/api/scenarios/:id/{validate,answer} were writing progress into
that ephemeral Lab container's own SQLite — which is destroyed on every
reset/reap. Confirmed by testing: solving a scenario never showed up on the
dashboard's curriculum view, and would have been silently lost on lab
recreate, directly contradicting "academic progress must survive lab
reset/recreate."

Root tension: scenario validation must run against the student's own live
cluster (only exists inside their Lab container) — it can't be re-run
anywhere else. But the canonical progress record needs to live on the
persistent identity instance, which never has the student's actual cluster
state applied to it.

Fix: this endpoint (POST /api/progress/record) records an outcome WITHOUT
re-running validation — it exists purely so the Gateway can relay a
validate/answer result it already obtained from the real Lab. The Gateway
forwards the real request to the Lab, returns that real response to the
browser unchanged, and separately calls this endpoint with the outcome. See
platform/gateway/src/server.js's relayScenarioOutcome().

Verified end-to-end: a scenario solved through a student's Lab shows up in
their curriculum immediately, survives their lab being destroyed and reset,
and is correctly attributed only to that student (not visible to a second
student's own curriculum view).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-18 11:16:08 -05:00
ced4568andClaude Sonnet 5 c89e39c5ee feat(gateway): add environment-aware HTTP and websocket routing
New internal component (platform/gateway/) — the only student-facing path to
disposable Lab environments. Browsers never reach a Lab container, the
Provisioner, or Postgres directly.

Routes:
- /api/environment/{status,launch,reset} — student-facing glue (X-Student-Id),
  calls the Provisioner internally (shared secret never reaches the browser).
  Reuses an existing non-terminal environment on launch rather than always
  provisioning a new one; reset destroys+recreates the same environment row.
- /lab/<token>/* — HTTP+WS proxy into that one student's Lab container,
  resolved by opaque routing token via a new Provisioner endpoint
  (GET /environments/by-token/:token). No raw container IP/port ever reaches
  the browser. Rejects invalid tokens (404), unready environments (202/503),
  and cross-student access via an optional ?sid= ownership check (403) — see
  docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md "Security" for why token-alone,
  student-id-alone, and env-id-alone are each individually insufficient and
  this is explicitly pilot-grade, not production auth.
- everything else — proxied to the persistent identity/curriculum backend
  (roster, login, curriculum, progress, platform-shell frontend), or to the
  student's own Lab if a valid lab-session cookie is present.

Routing design note: the existing Lab UI makes root-relative calls
(fetch('/api/scenarios'), the /shell-ws WebSocket) with no idea it might be
served under a /lab/<token>/ prefix — preserving that UI unmodified was a
hard requirement. Solved with an HttpOnly, SameSite=Lax cookie set on the
first /lab/<token>/ response: every later request from that browser carries
it, so the unmodified Lab UI's own unprefixed calls resolve correctly once a
student has navigated into a lab session, with zero changes to its code.
/api/environment/* is explicitly exempted from cookie-based routing so it
never gets hijacked by an active lab session.

Also added platform/docker-compose.dev.yml (the full local stack: Postgres +
egress-proxy + Provisioner + a persistent "identity" instance + Gateway) and
platform/README.md tying the components together. The "identity" instance
required an explicit `network_mode: bridge` — under Compose's own
auto-created bridge network, k3s failed at startup with "no default routes
found in /proc/net/route"; documented as a known limitation, not fully
root-caused.

Found and fixed during testing: express.json() applied globally broke
http-proxy's re-streaming of proxied POST bodies (e.g. POST /api/roster hung
forever, since the body had already been consumed by the parser before
proxy.web() tried to forward it). Removed — none of the Gateway's own routes
actually need a parsed body.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-18 11:15:55 -05:00
ced4568andClaude Sonnet 5 de03dbde9a fix(lab): restore node-pty terminal runtime
Root cause, confirmed empirically (not guessed):

1. node-gyp generates an incorrect build/config.gypi for this NodeSource-packaged
   Node 20 build — it reports node_shared: true (expects a shared libnode.so),
   but the actual installed Node is statically linked (no libnode.so.<abi>
   exists anywhere on the system; process.config.variables.node_shared on the
   real running Node is false). node-gyp trusted the wrong config and linked
   node-pty's native addon against -lnode, producing a binary that fails at
   require() time with "libnode.so.127: cannot open shared object file" —
   confirmed by loading the compiled .node file directly, bypassing node-pty's
   own path-search/error-masking logic (which collapses this into a generic
   "Cannot find module .../pty.node" across every path it checks).

2. A second, independent bug compounded it: no .dockerignore existed. A
   host-side `npm install` (run earlier for backend testing) left
   backend/node_modules on disk, built against the host's own environment. The
   Dockerfile's `COPY backend/ ./backend/` (placed after the npm-install RUN
   layer, for build-cache efficiency) silently overwrote the correctly-built
   node_modules with that host copy — including after a first .dockerignore
   fix using the pattern `node_modules/`, which only matches a node_modules
   directory at the build context ROOT in Docker's matching rules, not at any
   depth (unlike .gitignore, where the same-looking pattern does match any
   depth — a real, confirmed gotcha, not an assumption).

Fix:
- Dockerfile: `ENV npm_config_node_shared=false` (image-level, not a one-off
  RUN prefix, so it can't be silently lost if a native module ever gets
  rebuilt again for any reason) + a build-time `node -e "require(...)"` check
  that fails the build loudly if this regresses, instead of shipping a silent
  runtime failure.
- .dockerignore: `**/node_modules/` (recursive), not `node_modules/`.

No terminal architecture change, no fake shell, no changes to
backend/server.js's PTY handling. RootNode branding and upstream KubeKosh
attribution untouched. The single-container dev workflow
(`docker build -t rootnode-academy:test .`) is the same command as before —
it now just works.

Verified against the real image end-to-end, not a stand-in: build -> container
starts -> k3s reaches Ready -> backend starts, no crash -> nginx starts ->
browser UI 200s -> terminal WebSocket opens showing the RootNode banner ->
`kubectl get nodes` run through the terminal returns real cluster state ->
deploy-nginx scenario solved via the terminal -> all 5 validation checks pass.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-18 11:15:26 -05:00
241 changed files with 20163 additions and 72 deletions
+17
View File
@@ -0,0 +1,17 @@
# Root cause of the node-pty "Cannot find module .../pty.node" failure (Phase 2A): with no
# .dockerignore, a host-side `npm install` (e.g. for local testing) leaves backend/node_modules
# on disk, built against the host's own Node/libnode — not the container's. The Dockerfile's
# later `COPY backend/ ./backend/` (done after the npm-install RUN layer, for build caching)
# then silently overwrites the correctly-built-in-container node_modules with that host copy,
# which fails to load at runtime with an ABI/architecture mismatch. Native modules (node-pty,
# better-sqlite3) MUST always be built inside the image, never leaked in from the host.
#
# IMPORTANT: bare `node_modules/` only matches a node_modules directory at the build context
# ROOT in Docker's .dockerignore matching rules — it does NOT recursively match
# backend/node_modules or frontend/node_modules. This bit us once already (fixed here with
# `**/`): a first pass at this file used the bare pattern and the leak still happened.
**/node_modules/
frontend/dist/
.git/
*.db
*.db-journal
+1
View File
@@ -4,3 +4,4 @@ frontend/dist/
*.db-journal
/data/
.env
.ansible/
+22 -1
View File
@@ -22,6 +22,12 @@ RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
&& apt-get install -y nodejs \
&& rm -rf /var/lib/apt/lists/*
# node-gyp fix (Phase 2A root cause, see the note on the backend npm-install line below):
# set as a persistent image ENV, not a one-off RUN prefix, so it applies no matter when or how
# many times a native module gets (re)built in this image — inline `VAR=x npm install` only
# scopes the var to that one RUN command, which isn't a strong enough guarantee.
ENV npm_config_node_shared=false
# ── k3s (bundles kubectl, containerd, everything) ────────────────────────────
# We download the binary and the airgap images so the cluster starts offline.
RUN set -eux && \
@@ -41,8 +47,23 @@ RUN set -eux && \
WORKDIR /app
# Install backend dependencies
#
# npm_config_node_shared=false (set above, image-level) works around a real node-gyp bug
# affecting native modules (node-pty, better-sqlite3): node-gyp generates build/config.gypi
# from a Node headers/config reference it downloads for the target version, and for this
# NodeSource-packaged Node build that reference incorrectly reports node_shared=true (a shared
# libnode.so build) when the actual installed Node is statically linked (no libnode.so.<abi>
# exists at all). Left uncorrected, node-gyp links native addons against -lnode, producing a
# binary that fails at require() time with "libnode.so.<abi>: cannot open shared object file"
# — this is the prior node-pty runtime failure ("Cannot find module .../pty.node"),
# root-caused during Phase 2A: node-pty's own error handling collapses the real dlopen error
# into a generic "module not found" message across all the paths it checks, which is what made
# this look like a missing-file problem rather than a link-time misconfiguration. The
# `node -e require(...)` check below makes this a hard build failure if it ever regresses,
# instead of a silent runtime failure discovered later.
COPY backend/package.json ./backend/
RUN cd backend && npm install --production
RUN cd backend && npm install --production \
&& node -e "require('./node_modules/node-pty'); console.log('node-pty loads OK')"
# Build frontend
COPY frontend/package.json frontend/vite.config.js ./frontend/
+67 -17
View File
@@ -127,30 +127,80 @@ Everything runs inside one image managed by `scripts/entrypoint.sh`.
## Multi-Student Deployment (RootNode Academy addition)
The upstream project assumes one learner, one container — this image is still exactly that,
unmodified. RootNode Academy multiplies it into a real platform via a separate provisioning
layer rather than changing the image itself: see `docs/LAB_ARCHITECTURE.md` for the
architecture and `platform/provisioner/` for the Phase 1 implementation (Postgres-backed
environment registry, per-student Docker networking, firewall-enforced egress, idle reaper —
see `platform/provisioner/README.md` for how to run it).
The upstream project assumes one learner, one container — this image is still exactly that.
Its backend/entrypoint/Dockerfile are unmodified except for a node-pty build fix (see
`docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md` §1); its frontend bundle gained progressive-hint
controls and an assessment-mode indicator in Phase 3B (`docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`
§10–11) — additive UI, not a rewrite of the Lab experience. RootNode Academy multiplies the
image into a real platform via separate provisioning and gateway layers rather than changing
its core:
**Never publish the raw privileged container** — front it with isolation. The one-container
`docker run --privileged -p 7554:80 ...` command above stays supported for local development
and scenario authoring; it is not how this gets deployed to real students.
- `platform/provisioner/` — Postgres-backed environment registry, per-student Docker
networking, firewall-enforced egress, idle reaper (`docs/PHASE1_PLATFORM_FOUNDATION.md`).
- `platform/api/` — lightweight Platform API: students, cohorts, curriculum, progress, detailed
scenario attempt telemetry, progressive hints, assessment/capstone engine, achievements,
persistent student/instructor sessions, per-instructor accounts (RBAC: owner/admin/
instructor), audit log, instructor/admin control plane. No k3s, no `--privileged`
(`docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md`, `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md`,
`docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`).
- `platform/gateway/` — the only student-facing path to a Lab: resolves identity from a
server-controlled, persistent session cookie (not a client-suppliable header/param), proxies
HTTP/WebSocket into that one student's container, never a raw container IP/port, and
server-gates progressive hint reveal by stripping the Lab's own hint payload before it reaches
the browser (`docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md`, session model in
`docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md` §3, hint gating in
`docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` §3).
- The student flow (roster login → curriculum dashboard → Launch Lab → practice/assessments/
capstone → achievements) lives in `frontend/src/platform/`; the instructor/admin portal
(per-instructor login, cohorts, students, instructor accounts, audit log, classroom
"Prepare Labs," environment controls, learning analytics, assessment/capstone management)
lives in `frontend/src/admin/` at the `/admin` route — both additions alongside the existing
Lab UI, not a rewrite of it.
Key properties of the Phase 1 platform (not this image):
**Never publish the raw privileged container** — front it with isolation (that's what the
Gateway is). The one-container `docker run --privileged -p 7554:80 ...` command above stays
supported for local development and scenario authoring; it is not how this gets deployed to
real students.
Key properties of the platform (not this image):
- One container **per student**, its own Docker network, its own firewall chain — never a
shared cluster between learners.
- Progress/identity live outside the disposable container (currently this image's own SQLite,
moving to the Platform's PostgreSQL in a later phase) — destroying a student's environment
never loses their academic record.
shared cluster between learners. Verified with two simultaneous students: separate clusters,
separate progress, no cross-access.
- Progress/identity live outside the disposable container, in the Platform API's own
PostgreSQL (authoritative as of Phase 2B) — destroying or recreating a student's Lab never
loses their academic record. Verified with a full class-sized (3-student) cohort workflow:
provision, complete different scenarios, reset one student's Lab, destroy all Labs after
class — progress survives every step.
- Instructors sign in with their own account (no shared password) and manage cohorts, students,
curriculum assignment, and classroom Lab prep (provision an entire cohort before class, verify
readiness, destroy Labs after) from `/admin` — no manual database, Docker, or Kubernetes
access needed. Owners manage instructor accounts and roles; every instructor/admin action is
recorded in an audit log.
- Sessions (student and instructor) are Postgres-backed and survive a Platform API restart —
verified by actually restarting it mid-session and confirming access still works.
- Every validation attempt, hint reveal, curriculum stage completion, exam, and Capstone
attempt is recorded in Postgres, not the disposable Lab — a Lab reset or recreate never loses
attempt telemetry, hint progress, or an in-progress assessment's timer. Verified live:
reset a Lab mid-assessment, timer and drawn scenario set were unchanged afterward.
- Exams and the RootNode Capstone use a server-authoritative timer and a server-drawn,
server-stored scenario set — nothing the browser sends can change either.
- CPU/memory/PID/storage limits are enforced per container, config-driven.
- Egress is restricted to an allowlisted proxy at the firewall layer, not left open.
- Idle/abandoned environments are automatically cleaned up.
- Idle/abandoned environments are automatically cleaned up, and a student can launch a fresh
one and keep going without losing progress.
Production topology and what's built vs. not yet: `docs/LAB_ARCHITECTURE.md` and
`docs/PHASE1_PLATFORM_FOUNDATION.md`.
Production topology and what's built vs. not yet: `docs/LAB_ARCHITECTURE.md`,
`docs/PHASE1_PLATFORM_FOUNDATION.md`, `docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md`,
`docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md`, `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`.
The production infrastructure package (Proxmox/Ansible/Terraform/firewall/monitoring/backup/CI)
that runs all of this on real hardware lives in `infra/` — see
`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` and `docs/PHASE4_1_PREPRODUCTION_HARDENING.md` (the
Gateway-to-Lab cross-VM routing fix, plus application hardening — real pagination, abandonment
handling, a Postgres-backed rate limiter, a structured assessment editor, admission control).
Built and locally validated now; deployment is pending purchase of the target Hetzner dedicated
server.
---
+26
View File
@@ -886,6 +886,32 @@ app.post('/api/scenarios/:id/time', (req, res) => {
res.json({ ok: true });
});
// POST /api/progress/record — Phase 2A addition. Records a scenario outcome WITHOUT
// re-running validation. Exists because scenario validation itself must run against the
// student's own live cluster (inside their ephemeral Lab container), but the canonical
// progress record needs to live here, on the persistent identity instance — the Gateway
// calls this after a real validate/answer already succeeded against the student's Lab, to
// relay the outcome. See docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md "Progress relay."
app.post('/api/progress/record', (req, res) => {
const { scenario_id, status } = req.body || {};
if (!scenario_id || !['completed', 'in_progress'].includes(status)) {
return res.status(400).json({ error: 'scenario_id and a valid status are required' });
}
const studentId = getStudentId(req);
const progress = loadProgress(studentId);
const prev = progress[scenario_id] || { attempts: 0 };
progress[scenario_id] = {
...prev,
status,
attempts: (prev.attempts || 0) + 1,
last_validated: new Date().toISOString(),
completed_at: status === 'completed' ? new Date().toISOString() : prev.completed_at,
};
saveProgress(studentId, progress);
touchStudent(studentId);
res.json({ ok: true });
});
// GET /api/progress — full progress summary
app.get('/api/progress', (req, res) => {
const scenarios = loadScenarios();
+74
View File
@@ -0,0 +1,74 @@
# Backup & Disaster Recovery
## What's backed up (Tier 1) vs. not
| | Backed up | Why |
|---|---|---|
| Postgres data | Yes, daily | The only irreplaceable data — student records, cohort/session state, scenario attempts, assessments, achievements, audit events |
| `{{ rootnode_data_root }}` config/secrets (Vault files, env templates) | Yes, via the infra repo + Ansible Vault itself | Small, static, versioned |
| Infra code (`infra/`, Ansible, Terraform state) | Yes — it's in git; Terraform state should be backed up alongside Postgres if not using a remote backend | Needed to rebuild either VM from scratch |
| Disposable student Labs | **No, intentionally** | Ephemeral by design — a Lab is meant to be destroyed and recreated; backing it up would contradict the whole isolation/disposability model |
| Prometheus/Loki metrics & log history | No | Operational telemetry, not source-of-truth data; losing 30 days of dashboards on a rebuild is an inconvenience, not data loss |
## Backup mechanism
`infra/backup/backup-postgres.sh` (deployed to `/opt/rootnode/backup/` by the `backup` Ansible
role, run daily at 03:15 via cron):
1. `pg_dump --format=plain --no-owner --no-privileges | gzip -9`
2. Sanity check: reject if the compressed dump is under 1KB (almost certainly means Postgres
was unreachable or empty, not a real backup).
3. `gpg --symmetric --cipher-algo AES256` — encrypt, then delete the plaintext dump immediately.
4. Optional off-host copy via `rsync -az` over SSH — works for Hetzner Storage Box or any SSH
target; that's the extension point for S3-compatible alternatives (swap the rsync line for
`aws s3 cp`/`rclone` if a different target is chosen later).
5. Retention pruning (`find ... -mtime +{{ backup_retention_days }} -delete`, default 30 days).
6. Writes Prometheus textfile-collector metrics
(`rootnode_backup_last_run_success`, `..._last_success_timestamp_seconds`, `..._size_bytes`)
atomically (tmp file + rename) so a half-written metric file is never scraped.
## Restore
`infra/backup/restore-postgres.sh` — requires an explicit target DB URL (never infers or
defaults to production, to prevent an accidental overwrite), shows the target's current tables,
requires typing `RESTORE` to confirm, then decrypts and pipes into `psql`.
`infra/backup/verify-restore.sh` — confirms all 8 expected tables exist plus the
`schema_migrations` row count, so a "successful" restore that silently missed a table is caught.
**Genuinely tested**, not just written: `infra/backup/test-backup-restore.sh` runs the full
cycle against two disposable local Postgres containers, with the actual production migrations
applied (3 Provisioner + Platform API, in the correct cross-service order), a
sample row inserted, backed up, restored into the second container, and verified byte-for-byte.
Confirmed passing, exit 0.
**Phase 4.1's destructive drill goes one step further**
(`infra/backup/test-destructive-restore-drill.sh`): realistic data (owner, admin instructor,
cohort, 3 students, progress, attempts, an assessment definition, achievements) seeded through
the REAL Platform API, backed up, the source Postgres **container genuinely destroyed**
(`docker rm`, not just left running alongside a second one), a fresh Postgres created from
nothing, restored, and the real Platform API booted again against the restored database and
used to log in as the restored admin instructor over HTTP — proving the whole stack recovers,
not just the SQL. Never touches the developer's real persistent database. Confirmed passing.
## Disaster recovery scenarios
| Scenario | RPO | RTO (estimate) | What happens |
|---|---|---|---|
| Platform VM lost, disks intact | 0 | ~30 min | `docs/RUNBOOK_VM_REBUILD.md` — Ansible re-run against the same disks |
| Platform VM + its disks lost | Up to 24h (since last backup) | ~1-2h | Rebuild VM (Terraform), re-run Ansible, restore Postgres from the latest off-host backup (`docs/RUNBOOK_POSTGRES_RESTORE.md`) |
| Lab Host VM lost, any reason | 0 (intentional) | ~30 min | Rebuild VM, re-run Ansible — no data to restore, all Labs were disposable and get recreated on next student login |
| Single disk failure in a mirror | 0 | 0 (no outage) | `docs/RUNBOOK_DISK_FAILURE.md` — pool stays online DEGRADED, replace and resilver |
| Whole storage pool lost (both mirror sides) | Up to 24h | ~1-2h | Same as "Platform VM + disks lost" above — this is precisely the scenario the off-host copy exists for; a local-only backup on the same failed pool is useless here |
| Encryption passphrase lost | N/A | Unrecoverable | The single most important secret to protect and back up separately (e.g., in the same secrets manager as other Vault secrets, not only in `production.vault.yml`) — losing it makes every existing backup file permanently unreadable |
RPO is bounded by backup cadence (daily) — a mid-day incident can lose up to a day of student
activity. If that's ever unacceptable, the fix is a shorter cron interval or WAL archiving for
point-in-time recovery, not decided or built here since daily has not yet been proven
insufficient against real usage.
## What backup does NOT cover
Application code and container images — those live in Gitea and the image registry, both
already durable and versioned independently of this backup mechanism. TLS certificates — those
are regenerable (`docs/RUNBOOK_DOCKER_TLS_ROTATION.md`, `docs/RUNBOOK_TLS_RENEWAL.md`), not
backed up.
+59
View File
@@ -0,0 +1,59 @@
# Capacity Planning
How many concurrent student Labs `rn-lab-host-01` can actually run — determined from measured
data after deployment, not guessed or marketed now.
## The model
`infra/scripts/capacity-model.js`:
```
usable = hostTotal * (1 - overheadPct/100) - platformReserved
withMargin = usable * (1 - marginPct/100)
perLabLimit = floor(withMargin / perLabUsage)
recommended = min(cpuBoundLimit, memoryBoundLimit) # whichever runs out first
warning = floor(recommended * 0.8)
hardAdmission = recommended
```
Computed once for CPU, once for memory; the tighter (smaller) of the two constrains the actual
recommendation — a host can be memory-starved long before its CPU threads are exhausted, or
vice versa, and the model doesn't pretend otherwise.
**Inputs** (`infra/environments/production/capacity.yml.example` — explicitly placeholder data
until real numbers exist):
- `host_cpu_threads`, `host_memory_mb` — from the real Lab Host VM, once sized
- `overhead_pct` — Proxmox/hypervisor + OS overhead reserved off the top
- `platform_reserved_cpu`/`platform_reserved_memory_mb` — headroom for the egress proxy,
node-exporter, cadvisor, promtail themselves (they run on the Lab Host too)
- `per_lab_cpu`/`per_lab_memory_mb` — **measured**, from `infra/scripts/loadtest/`, not assumed
- `safety_margin_pct` — extra headroom below the raw math, for burst/variance
**Outputs**: recommended concurrent Labs, an 80%-of-recommended warning threshold (for
alerting — `LabHostHighMemory` in `docs/OBSERVABILITY.md`), and a hard admission-control
threshold the Provisioner's resource policy should enforce so it refuses to boot a Lab past that
point rather than degrading every running Lab.
## Load-test tooling that produces the measured inputs
`infra/scripts/loadtest/`:
- `boot-labs.js` — reuses `platform/test/session-helper.js` (built in Phase 3A) to drive the
real admin Prepare-Labs endpoint, booting N Labs and recording boot time/failure rate per Lab
into a JSON report.
- `sample-resources.sh` — samples real `docker stats` for every `rootnode-lab-*` container every
5 seconds to CSV (timestamp, name, CPU%, mem usage, mem%, net I/O, block I/O, PIDs).
Together these produce the `per_lab_cpu`/`per_lab_memory_mb` this model needs, plus data on how
boot time and failure rate change as concurrency rises — the dimensions this phase was told to
cover: single Lab boot, multiple simultaneous boots, steady-state active Labs, terminal
WebSocket load, scenario validation load, Gateway throughput, API request volume, Postgres
connection activity.
## What this document deliberately does not claim
No concurrency number in this repo (`capacity.yml.example`'s numbers included) is a real,
validated figure — every one is a placeholder until run against the real Lab Host VM with real
per-Lab measurements. Treating a placeholder as a marketing/sales number would be actively
misleading; the whole point of the model is that the real number gets filled in once, from
measurement, after deployment (`docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md` step 4 is the
trigger to re-measure and update it if the per-Lab footprint later changes).
+178 -31
View File
@@ -71,14 +71,18 @@ Two separate structures, don't confuse them:
- **RootNode path** (`scenarios/rootnode-path.json`) — the guided onboarding sequence,
separate from bundles on purpose (it's *staged*, bundles are *flat*, and the bundle loader
would choke on a staged file if it were dropped in `scenarios/bundles/`). Loaded by
`loadCurriculumPath()`/`GET /api/curriculum/path`. Stages are Foundation → Workloads →
Networking → Configuration → Troubleshooting → Real Ops → Capstone, currently built by
reusing existing `1-k8s-basics` and `rootnode-real-ops` scenarios grouped by their existing
`category` field — no new scenario content was written for this. **Capstone stage is a
locked placeholder** (`"locked": true`, empty `scenario_ids`) — the capstone engine itself
is a separate, not-yet-built roadmap item (see §7, item 6). Do not put real scenarios in it
until that engine exists, or you'll reproduce the "duplicate beginner scenarios as capstone"
anti-pattern this path was built to avoid.
`loadCurriculumPath()`/`GET /api/curriculum/path` (raw content) and, as of Phase 3B,
`computeCurriculumState()`/`GET /api/curriculum/state` (rule-driven lock/complete/next-scenario
state — see `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` §4). Stages are Foundation → Workloads →
Networking → Configuration → Troubleshooting → Real Ops → Capstone, built by reusing existing
`1-k8s-basics` and `rootnode-real-ops` scenarios grouped by their existing `category` field —
no new scenario content was written for this. Each stage carries a `completion_rule`
(`all_required`/`threshold`/`prerequisites`). **Capstone is no longer a hardcoded placeholder**
— it's `"type": "capstone"` with an `"unlock_rule"` and a `"capstone_assessment_id"` pointing
at the `rootnode-capstone` assessment definition (Postgres, `assessment_definitions` table),
which is where its actual scenario draw lives — see the Phase 3B doc §7. Don't put real
scenarios directly into the JSON stage's `scenario_ids` for capstone; that field stays empty
by design, the draw is defined in the assessment definition instead.
To add/reorder a stage: edit `rootnode-path.json` directly, referencing existing
`scenarios/data/*.json` ids. No backend code change needed — it's read fresh via
@@ -138,27 +142,38 @@ Order agreed 2026-08-17/18. Status as of this doc:
1. RootNode identity/branding pass — **partial.** Shell banner fixed
(`⎈ RootNode Academy Lab — node: k8s-lab`). Upstream KubeKosh attribution left untouched in
footer/README/LICENSE. Broader "make Real Ops first-class" UI work not started.
2. RootNode curriculum mode — **backend done** (§4), frontend not started.
3. Student accounts / persistent progress — **backend done** (§5), frontend not started,
no instructor auth yet.
4. Instructor dashboard — not started.
5. Per-student isolation — **Phase 1 (platform foundation) done**, see §8. Ephemeral
per-student Lab environments with Postgres-backed lifecycle, per-environment Docker
networking, and firewall-enforced egress restriction are implemented and tested on a single
dev machine. **Still not cleared for minors** — no Gateway/frontend wiring yet (environments
are only reachable via the internal Provisioner API), no instructor auth on top of it, and
the production two-VM Proxmox split this depends on doesn't exist yet. See
`docs/PHASE1_PLATFORM_FOUNDATION.md` final report for exactly what's verified vs. not.
2. RootNode curriculum mode — **done**. Rule-driven stage completion/unlock as of Phase 3B
(§12), student-facing tabs (My Learning/Practice/Assessments/Capstone/Achievements) at
`frontend/src/platform/`.
3. Student accounts / persistent progress — **done**, Postgres-authoritative as of Phase 2B
(§9), sessions persistent as of Phase 3A (§10). Roster name-picking is still classroom/pilot
identification, not strong student auth — documented, not silently glossed over
(docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md §7).
4. Instructor dashboard — **done** (§9/§10). Full admin portal at `/admin`: dashboard, cohort
and student management, classroom Prepare/Destroy Labs, per-instructor accounts with
owner/admin/instructor roles, audit log.
5. Per-student isolation — **Phase 1 + Phase 2A + Phase 2B + Phase 3A done**, see §8/§9/§10.
Ephemeral per-student Lab environments, Gateway-mediated routing with persistent
server-controlled sessions, a full instructor/admin control plane with real per-person
accounts and RBAC, and a working entry→dashboard→launch→lab→progress flow are implemented
and end-to-end tested on a single dev machine, including a 3-student cohort classroom
workflow and a real Platform API restart mid-session. **Still not cleared for minors** — the
production two-VM Proxmox split this depends on doesn't exist yet, and OIDC/SSO is a
documented adapter, not a working integration (docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md
§16).
6. Scenarios → teaching modules (incident context/investigation/prevention) — not started.
7. Capstone/exam engine — not started (placeholder stage only, see §4).
8. Progressive hint levels — not started.
7. Capstone/exam engine — **done** as of Phase 3B (§12). RootNode Capstone (Real Ops-weighted,
unlockable by curriculum-completion rule, not a permanent placeholder) plus a general exam
engine (definitions, cohort assignment, random scenario draw, server-authoritative timer).
8. Progressive hint levels — **done** as of Phase 3B (§12). Server-enforced sequential reveal,
tracked per attempt, policy-aware under assessment mode (free/disabled/flagged).
9. Production observability — not started.
10. CI scenario testing — not started.
11. Container/build optimization — explicitly deprioritized by design (platform works, not urgent).
Next slice: either the deferred frontend login (name picker → `X-Student-Id` header, curriculum
view), or the Phase 2 platform work (Gateway, frontend wiring to the Provisioner, instructor
auth) — see the Phase 1 final report for the concrete Phase 2 list. Not decided here.
Next slice: production OIDC/SSO, the production two-VM Proxmox deployment, and teaching-module
scenario content are the biggest remaining items. See
`docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` §21 for the full Phase 4 boundary.
## 8. Platform Provisioner (Phase 1)
@@ -170,13 +185,16 @@ section is just the map.
environment per student — its own Docker network, its own container running the existing,
unmodified Lab image, its own firewall-enforced egress. Postgres-backed registry
(`environments`, `environment_events`, `lab_hosts`), idle reaper, startup reconciliation.
- **What it is not (yet):** reachable by a student. There's no Gateway and no frontend wiring
— it's exercised directly via its internal API (shared-secret auth) or
`platform/provisioner/test/run.js`. Nothing here is a live feature.
- **Where identity lives:** unchanged from §5 — `students`/`progress` are still on the Lab
image's own SQLite. `environments.student_id` is deliberately opaque (no FK) until a later,
separate migration puts roster/progress on the same Postgres instance as the environment
registry.
- **What it is not:** directly reachable by a student or the Gateway anymore — as of Phase 2B,
the Platform API (§9) is the *only* caller (student launch/reset AND admin prepare/destroy/
reset both go through it). Still exercised directly via its internal API (shared-secret
auth) or `platform/provisioner/test/run.js` for isolated Provisioner testing.
- **Where identity lives:** as of Phase 2B, Postgres is authoritative —
`students`/`cohorts`/`progress`/`scenario_attempts` live in the same `rootnode_platform`
database as `environments`, owned by the Platform API's own migrations. See §9. As of Phase
3A, `instructors`/`sessions`/`cohort_instructors`/`audit_events` live there too. See §10.
`environments.student_id` is still a plain opaque reference, not a real FK — that's a
deliberate, documented choice (docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md §2), not an oversight.
- **Local dev:** `platform/provisioner/docker-compose.dev.yml` runs Postgres + the egress
proxy + the Provisioner on one machine — the documented Phase 1 dev exception. Production
is a two-VM Proxmox split (Platform VM / Lab Host VM); the dev compose file is explicitly
@@ -186,3 +204,132 @@ section is just the map.
events). Some properties (idle reaper timing, remote TLS lab-host, reconciliation-on-crash,
real-image k3s behavior) were validated manually during implementation rather than
automated — see the Phase 1 final report for what and why.
## 9. Platform API (Phase 2B)
Full design: `docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md`. Operational reference:
`platform/api/README.md`.
- **What it is:** `platform/api/` — a lightweight Node/Express service (no k3s, no
`--privileged`, no Lab terminal runtime) owning students, cohorts, enrollments, curriculum
assignments, progress, scenario attempts, and — as of Phase 3A — instructor accounts,
persistent sessions, and the audit log (§10). The sole caller of the Provisioner for both the
student launch/reset path and the instructor/admin control plane.
- **What it is not:** reachable directly by a browser — always through the Gateway
(`platform/gateway`), same trust boundary the Provisioner has relative to this service.
- **Local dev:** part of the same `platform/docker-compose.dev.yml` stack as the Provisioner —
no separate compose file.
- **Test procedure:** `platform/test/e2e-phase2a.js` (student flow) and
`platform/test/e2e-phase2b.js` (classroom acceptance workflow) — both need a bootstrapped
instructor account as of Phase 3A, see §10.
## 10. Identity and instructor access (Phase 3A)
Full design: `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md`. Operational reference:
`platform/api/README.md` (bootstrap, config).
- **What it is:** persistent Postgres-backed sessions (student + instructor, survive a Platform
API restart), real per-instructor accounts (`owner`/`admin`/`instructor` roles, scrypt
password hashing, per-instructor salt), server-enforced RBAC on every `/api/admin/*` route,
cohort-scoped access for the plain `instructor` role, CSRF double-submit protection, login
rate limiting, and a platform-level `audit_events` log.
- **What it replaced:** Phase 2B's in-memory session `Map`s (gone on every restart) and the
single shared `ADMIN_PASSWORD` env var (no accountability, no roles, no revocation).
- **Bootstrap:** `node platform/api/bootstrap-owner.js <email> <name> <password>` — the only
supported way to create the first instructor account; no manual database editing.
- **Test procedure:** `platform/test/e2e-phase3a.js` — 21 automated tests covering session
persistence (including a real `docker compose restart api`), instructor auth, RBAC, cohort
scoping, CSRF, rate limiting, password change, and audit logging.
- **Known limitations, future OIDC path, production HTTPS expectations:** see
`docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md` §15–17.
## 11. Dependency lock convention
Every first-party Node service commits its `package-lock.json`: `backend/`, `frontend/`,
`platform/api/`, `platform/gateway/`, `platform/provisioner/`. This wasn't consistently true
before Phase 3B — `platform/api/`'s lockfile existed locally but was never committed, and
`platform/gateway/`/`platform/provisioner/` never had one generated at all. Neither was a
deliberate policy; both were oversights, fixed in Phase 3B (`chore(deps)` commits).
Going forward: after any `npm install` that changes a service's dependencies, the updated
`package-lock.json` is part of the same commit as the `package.json` change — never left
untracked, never regenerated-and-ignored. Validate a lockfile actually matches its
`package.json` with `npm ci` (fails loudly on drift) before committing either.
## 12. Learning & Assessment Engine (Phase 3B)
Full design: `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`.
- **What it is:** the teaching/assessment layer on top of everything Phases 1–3A built —
detailed per-attempt telemetry (`scenario_attempts`, evolved from Phase 2B's one-row-per-click
shape into a real attempt lifecycle), server-enforced progressive hints, rule-driven
curriculum stage/capstone unlock logic, a full exam + capstone assessment engine
(`assessment_definitions`/`assessment_sessions`/`assessment_session_scenarios`), and
persistent `achievements` records.
- **Where the boundary moved:** the Gateway now rewrites one Lab response
(`GET /lab/:token/api/scenarios/:scenarioId`, stripping `hints`) so progressive hints can be
genuinely server-gated rather than just UI-collapsed — the Lab image's own code/bytes are
still untouched; this is a response rewrite at the Gateway boundary, same pattern as every
other Lab-mediation decision made in earlier phases. The Lab's own frontend bundle *did*
change this phase (`ScenarioPanel.jsx`, new `ProgressiveHints`/`AssessmentBanner` components)
— Phase 3B explicitly authorized enhancing Lab UX, unlike prior phases.
- **Test procedure:** `platform/test/e2e-phase3b.js` — covers attempt telemetry, hint
sequencing/persistence-through-Lab-reset, curriculum/capstone lock-unlock, the exam engine
(random draw, timer, expiry, restart persistence), two-student isolation, and instructor
analytics/RBAC.
- **Known limitations:** see the Phase 3B doc §20 — no automatic abandoned-attempt detection, no
per-cohort needs-attention threshold configuration, raw-JSON pool_spec authoring in the admin
UI (correct, not yet a guided form).
## 13. Production Infrastructure (Phase 4)
Full design: `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`, plus detail docs
`PRODUCTION_DEPLOYMENT.md`, `PRODUCTION_NETWORK_SECURITY.md`, `BACKUP_DR.md`,
`OBSERVABILITY.md`, `CAPACITY_PLANNING.md`, `PRODUCTION_VALIDATION_CHECKLIST.md`, and 12
`RUNBOOK_*.md` files, all under `docs/`. Everything lives in `infra/` (Ansible, Terraform,
Compose, firewall, monitoring, backup, CI, load-test tooling).
- **What it is:** the infrastructure-as-code needed to run rn-platform-01/rn-lab-host-01 on a
real Hetzner/Proxmox host — buildable and locally testable now, even though the dedicated
server hasn't been purchased. Status is explicitly **PHASE 4 BUILD COMPLETE / PRODUCTION
DEPLOYMENT PENDING HARDWARE**, not "deployed."
- **Where the boundary held:** the Platform VM / Lab Host VM security split from
`docs/LAB_ARCHITECTURE.md` is codified in firewall rulesets and mutual-TLS Docker config, not
weakened anywhere.
- **Known gap surfaced, then resolved in Phase 4.1:** the Gateway's Lab-reachability mechanism
didn't survive the move to two real separate VMs — flagged explicitly rather than silently
fixed, then actually built and simulation-tested (a Lab Host Router,
`platform/lab-router/`) in Phase 4.1. See §13 below.
- **Validated without the hardware:** real `tofu validate`, `ansible-playbook --syntax-check`,
`docker compose config`, `caddy validate`, `promtool`/`amtool`, and a genuine end-to-end
backup→encrypt→restore→verify cycle against disposable local Postgres with the real
production migrations applied. None of this proves the real Hetzner network boundary — that
needs `docs/PRODUCTION_VALIDATION_CHECKLIST.md` walked against the real server.
## 13. Pre-Production Hardening (Phase 4.1)
Full design: `docs/PHASE4_1_PREPRODUCTION_HARDENING.md`, plus
`docs/PRODUCTION_NETWORK_SECURITY.md` (routing resolution), `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`,
`docs/SECURITY_SCANNING.md`.
- **What it is:** closes the Gateway-to-Lab cross-VM gap Phase 4 flagged (a Lab Host Router,
`platform/lab-router/`, plus a routing abstraction in the Gateway,
`platform/gateway/src/lab-endpoint.js`), then hardens the application layer found along the
way: abandoned-attempt handling (`scenario_attempts.status = 'abandoned'`, never a punitive
failure), a Postgres-backed login rate limiter (survives a restart, ready to be
shared/multi-instance), real pagination on audit/attempt/assessment-session history, a
structured assessment authoring UI (owner-only raw JSON remains as a fallback), a minimal
`role_in_cohort` enforcement (`teacher`/`viewer`), a `/api/version` endpoint, and admission
control (a clean 503 "capacity full" instead of a 500 once a configured cap is reached).
- **Tested, not just designed:** a real two-Docker-network routing simulation
(`infra/scripts/simulate-two-host/`, 13/13), a destructive Postgres restore drill
(`infra/backup/test-destructive-restore-drill.sh`), a Lab Host loss/rebuild drill
(`infra/scripts/drills/test-lab-host-loss.js`), a deployment rollback drill, a CI-fails-closed
drill (which found and fixed a real pre-existing `set -e` bug in both smoke-test scripts —
see that drill's own output), and a full fake-cohort dress rehearsal
(`infra/scripts/drills/test-fake-cohort-rehearsal.js`) — all passing, all in
`infra/scripts/drills/` or `infra/backup/`.
- **Known limitation carried forward:** real remote/self-paced delivery needs actual student
authentication (today's `student_id`-only login relies on instructors controlling who gets a
valid id) — see `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md` for the full accounting of what
already works vs. what's genuinely Phase 5 scope.
+24 -6
View File
@@ -1,11 +1,17 @@
# RootNode Academy — Ephemeral Lab Architecture
Status: **Phase 1 (platform foundation) IMPLEMENTED 2026-08-18** — see
`docs/PHASE1_PLATFORM_FOUNDATION.md` for the concrete schema/API/test results, and
`platform/provisioner/README.md` for how to run it. This document remains the architecture
record for the full ephemeral-lab direction; later phases (Gateway, frontend wiring, instructor
dashboard, production Proxmox deployment — see the Phase 1 report's "Future Phase 2 items")
build on it but aren't built yet. Supersedes the namespace-isolation
Status: **Phase 1 (platform foundation) + Phase 2A (Gateway + student flow) + Phase 2B
(Platform API split, Postgres academic data, server sessions, admin portal) + Phase 3A
(persistent sessions, per-instructor accounts, RBAC, audit log) + Phase 3B (learning/assessment
engine: attempt telemetry, progressive hints, curriculum rules, exam + capstone engine,
achievements) IMPLEMENTED, 2026-08-18** — see `docs/PHASE1_PLATFORM_FOUNDATION.md`,
`docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md`, `docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md`,
`docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md`, and
`docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` for concrete schema/API/test results, and
`platform/README.md` for how to run the stack. This document remains the architecture record
for the full ephemeral-lab direction; production OIDC/SSO (an adapter interface exists, not a
working integration) and production Proxmox deployment are still future work (Phase 4 — see the
Phase 3B doc §21). Supersedes the namespace-isolation
(`shared_namespace`/`ephemeral_cluster`) approach that was in flight before this redirect — see
`scenarios/SCHEMA.md`'s `isolation_type` note for the pointer back here.
@@ -480,3 +486,15 @@ All four open questions this doc originally ended with are resolved — see the
at the top and §2–§5 above. Next: `docs/PHASE1_PLATFORM_FOUNDATION.md` — the concrete build
plan for the platform foundation (Postgres environment registry, provisioner skeleton,
networking, egress, resource policy, reaper) against these decisions.
## Phase 4: this boundary in production
Phase 4 (`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`) codifies the Platform/Lab-Host split this
document designed as two real, separately-firewalled Proxmox VMs (`rn-platform-01`,
`rn-lab-host-01`) connected by two separate mTLS channels (Provisioner↔dockerd CONTROL plane,
Gateway↔Lab Host Router DATA plane) and a tightly scoped firewall allowance — see
`docs/PRODUCTION_NETWORK_SECURITY.md`. Phase 4 surfaced a gap in the Gateway's `env.internal_ip`
mechanism for reaching a Lab container (it assumes Gateway and Lab share a Docker daemon, which
breaks once they're genuinely on separate VMs); Phase 4.1
(`docs/PHASE4_1_PREPRODUCTION_HARDENING.md`) built and simulation-tested the fix, a Lab Host
Router (`platform/lab-router/`).
+64
View File
@@ -0,0 +1,64 @@
# Observability
Self-hosted Prometheus + Grafana + Alertmanager + Loki/Promtail, deployed via
`infra/docker/platform/docker-compose.production.yml` on rn-platform-01, with lightweight agents
(node-exporter, cadvisor, promtail) also on rn-lab-host-01. No paid SaaS.
This is operational observability — host/container/database health, uptime, resource pressure.
It is deliberately separate from the application's own instructor learning analytics
(Phase 3B's cohort/scenario/student dashboards) — different audience (ops vs. instructors),
different data (infrastructure vs. learning outcomes), and mixing them would make both worse.
## What's real today
- **Host metrics** (both VMs): node-exporter — CPU, memory, disk, network.
- **Container metrics** (Lab Host): cadvisor — per-container CPU/memory/network, including
student Lab containers (`rootnode-lab-*`).
- **Postgres metrics**: postgres-exporter — connections, transaction rate, table sizes.
- **Uptime/liveness**: blackbox-exporter probing `/healthz` on the Gateway
(`platform/gateway/src/server.js`) and the Platform API (`platform/api/src/server.js`), both
added this phase specifically for this purpose, plus the Provisioner's pre-existing `/health`.
- **Logs**: Loki + Promtail, scraping Docker container logs and journald from both VMs, with a
redaction pipeline stage that strips `password|token|secret=`-pattern values before they're
stored (`infra/monitoring/promtail/`).
- **Dashboards**: Grafana, reachable only via `https://<origin>/grafana/` and only from
`management_cidr` (see `docs/PRODUCTION_NETWORK_SECURITY.md`). One starter dashboard,
`infra/monitoring/grafana/provisioning/dashboards/rootnode-overview.json` — 8 panels (Platform
+ Lab Host CPU/memory/disk, Postgres up/connections, running-Lab-count via cadvisor label
match [best-effort, caveated], uptime probes).
## What's NOT built yet (honest accounting, not silently assumed)
Application-level metrics — active student count, Lab provisioning success/failure rate,
Gateway request latency/error rate, per-endpoint API metrics — would need real instrumentation
added to the Platform API/Gateway/Provisioner (e.g. a `/metrics` endpoint via `prom-client`).
`infra/monitoring/prometheus/prometheus.yml.j2` has a commented-out scrape job as the landing
spot for this, explicitly marked as a follow-up, not wired to anything real yet.
## Alerting
`infra/monitoring/prometheus/rules/rootnode.yml` — two severities:
**Critical** (fires fast, pages): PlatformAPIDown, GatewayDown, PostgresDown, LabHostDown,
BackupFailed, DiskCriticallyFull.
**Warning** (informational, slower repeat): LabHostHighMemory, HighDiskUsage, BackupAging.
(RepeatedProvisioningFailures and ExcessiveFailedLogins are present as inert placeholders —
they need the not-yet-built application metrics above before they can fire on anything real.)
Never alerts on an individual student's scenario failure or a single Lab crash — those are
expected, routine outcomes of normal usage, not operational incidents.
`infra/monitoring/alertmanager/alertmanager.yml.j2` — critical alerts route separately with a
30-minute repeat interval, everything else at 4 hours, one webhook receiver
(`vault_alertmanager_webhook_url` — points at whatever channel gets configured at deploy time,
e.g. a Slack/Discord/Matrix webhook), and one inhibit rule so a LabHostDown critical suppresses
its own dependent warnings instead of paging twice for the same root cause.
## Validation performed (local, real tooling — not hardware-dependent)
`infra/monitoring/validate-monitoring.sh` wraps: `promtool check config` (Prometheus config),
`promtool check rules` (alert rules — 11 rules found, valid), `amtool check-config`
(Alertmanager — 1 inhibit rule, 2 receivers, valid). These prove syntactic/semantic correctness
of the configs, not that the real network paths between rn-platform-01 and rn-lab-host-01 work —
that requires the real hardware and is covered by `docs/PRODUCTION_VALIDATION_CHECKLIST.md`.
+292
View File
@@ -0,0 +1,292 @@
# RootNode Academy — Phase 2A: Gateway & Student Flow
Status: **IMPLEMENTED and end-to-end tested, 2026-08-18.** Builds on
`docs/LAB_ARCHITECTURE.md` and `docs/PHASE1_PLATFORM_FOUNDATION.md`
(Phase 1 milestone `a15897c78f7012c59ea3629a74dd8b1425f6f7f2`). This doc covers the three
Phase 2A workstreams: fixing the real Lab image, the Gateway, and the student frontend flow.
---
## 1. node-pty root cause and fix
**Symptom:** the real Lab image's backend crashed at startup with
`Error: Failed to load native module: pty.node, checked: build/Release, build/Debug,
prebuilds/linux-x64: Cannot find module './prebuilds/linux-x64//pty.node'`.
**What it looked like vs. what it was:** node-pty's own error handling collapses whatever
actually went wrong at each candidate path into one generic "module not found" message using
only the *last* path it tried — this made a link-time failure look like a missing-file problem.
**Root cause, confirmed empirically (not guessed):**
1. **node-gyp generates an incorrect `config.gypi`.** For this NodeSource-packaged Node 20
build, `node-gyp`'s downloaded headers/config reference reports `node_shared: true` (a
shared `libnode.so` build), but the actual installed Node is statically linked — there is no
`libnode.so.<abi>` anywhere on the system (confirmed: `find / -iname 'libnode*'` → nothing;
`process.config.variables.node_shared` on the real running Node → `false`). Node-gyp
trusted the wrong config and linked node-pty's native addon against `-lnode`, producing a
binary that fails at `require()` time with `libnode.so.127: cannot open shared object file`
— confirmed by loading the compiled `.node` file directly, bypassing node-pty's own
path-search/error-masking logic.
2. **A second, independent bug compounded it:** the repo had no `.dockerignore`. A host-side
`npm install` (run earlier in this project for unrelated backend testing) left a
`backend/node_modules/` on disk, built against the *host's* environment. The Dockerfile's
`COPY backend/ ./backend/` (placed after the npm-install `RUN` layer, for build-cache
efficiency) then silently overwrote the correctly-built-in-container `node_modules` with
that host copy — including after a first `.dockerignore` fix that used the pattern
`node_modules/`, which (a real gotcha) only matches a `node_modules` directory at the build
**context root** in Docker's matching rules, not `backend/node_modules` at any depth.
**Fix (`Dockerfile`, `.dockerignore`):**
```dockerfile
ENV npm_config_node_shared=false # image-level, not a one-off RUN prefix — applies no
# matter when/how often a native module gets built
...
RUN cd backend && npm install --production \
&& node -e "require('./node_modules/node-pty'); console.log('node-pty loads OK')"
```
The `node -e require(...)` line makes this a **hard build failure** if it ever regresses again,
instead of a silent runtime failure discovered later — not a workaround for the terminal, a
real build-time verification of a real native dependency.
```
# .dockerignore
**/node_modules/ # NOT `node_modules/` — that only matches the context root
frontend/dist/
.git/
*.db
*.db-journal
```
**Nothing else changed.** No terminal architecture change, no fake shell, no rewrite of
`backend/server.js`'s PTY handling. RootNode branding and upstream KubeKosh attribution
untouched. The single-container dev workflow (`docker build -t rootnode-academy:test .`) is
the same command as before — it now just works.
**Verified against the real image, not a stand-in** (see §8 for full results): image builds →
container starts → k3s reaches `Ready` → backend starts (`API server running on :4000`, no
crash) → nginx starts → browser UI returns 200 → terminal WebSocket opens and shows the
RootNode banner → `kubectl get nodes` run through the terminal returns real cluster state →
`deploy-nginx` scenario solved via the terminal and validated via
`POST /api/scenarios/deploy-nginx/validate` → all 5 checks pass.
---
## 2. Gateway architecture
New component: `platform/gateway/`. The only student-facing path to disposable Lab
environments — browsers never talk to a Lab container, the Provisioner, or Postgres directly.
```
Browser
│ (one stable origin — the Gateway)
▼
Gateway
├── /api/environment/* → calls the Provisioner internally (shared secret never
│ leaves the Gateway process)
├── /lab/<token>/* → HTTP + WS proxy into that specific student's Lab
│ container, resolved by opaque token via the Provisioner
└── everything else → proxied to the persistent "identity" backend (roster,
login, curriculum, progress, and the platform-shell
frontend) — OR to the student's own Lab, if a valid
lab-session cookie is present (see §5 for why)
```
**Why a persistent "identity" instance exists, and what it is:** the roster/curriculum/
progress SQLite data (built in the pre-Phase-2A slice) needs to be reachable *before* a student
has a Lab — they pick their name and see their curriculum before ever clicking Launch Lab. That
data can't live inside an ephemeral, disposable-by-design Lab container. Phase 2A runs it as
one plain, non-ephemeral instance of the existing, unmodified Lab image (`identity` in
`platform/docker-compose.dev.yml`) — same image, same code, just never destroyed/reset by the
Provisioner, the same way Postgres or the egress proxy are infrastructure rather than
per-student resources. This is a deliberate zero-new-code choice (see §9 known limitations for
its cost).
**Provisioner addition:** `GET /environments/by-token/:token` (small, internal-only, same
shared-secret auth as every other Provisioner route) — resolves a routing token to its
environment for the Gateway. This is the only change to `platform/provisioner/`.
**What the Gateway never does:** expose a Lab container's port publicly, expose the Docker
API, expose the Provisioner's shared secret to a browser, or trust a client-supplied
environment ID for anything without checking it against the Provisioner's own record.
---
## 3. Student flow
**A. Entry screen** (`frontend/src/platform/EntryScreen.jsx`) — cohort dropdown → name
dropdown (existing `GET /api/roster/cohorts` / `GET /api/roster?cohort=`) → `POST /api/login`.
No password. A visible note states this is pilot identity, not production auth.
**B. Session** — `student_id` stored in `localStorage` only (`frontend/src/platform/api.js`),
sent as `X-Student-Id` on every subsequent platform request. Every authenticated call throws
if no student is stored, rather than silently going out anonymously — this is what "prevent
accidental anonymous progress writes" means in practice.
**C. Dashboard / "Up Next"** (`StudentDashboard.jsx`) — renders `GET /api/curriculum/path`
as-is: the existing RootNode path (Foundation → Workloads → Networking → Configuration →
Troubleshooting → Real Ops → Capstone), no new structure invented. Capstone stays the locked
placeholder from the earlier slice.
**D. Launch Lab** — `POST /api/environment/launch` (Gateway): reuses a non-terminal
environment if one exists, otherwise provisions one via the Provisioner. UI states map directly
to environment status: none → provisioning/resetting (polled every 2.5s while in flight) →
running → error, each rendered distinctly (`EnvironmentPanel` in `StudentDashboard.jsx`). On
`running`, the browser navigates to `launch_path` (`/lab/<token>/`), with the student_id
attached as a `?sid=` query param for the Gateway's ownership check (see §5).
**E. Lab session** — the existing, **completely unmodified** Lab UI (`App.jsx` and everything
under `frontend/src/components/`). It has no student-identity awareness and needs none —
identity and progress stay entirely in the Platform layer, per the explicit requirement not to
inject student identity into the privileged Lab container.
**F. Reset/Recreate** — one button, "Recreate Lab," calling `POST /api/environment/reset`
(destroy + provision against the same environment row, per the Phase 1 lifecycle — same
`environment_id`, fresh container). No raw destroy-without-recreate control is exposed to
students; that keeps the surface to exactly what's safe, per the explicit requirement.
---
## 4. API routes added
All on the Gateway (`platform/gateway/src/server.js`), all internal glue — the Provisioner's
own API remains unreachable from the browser:
| Method & path | Auth | Purpose |
|---|---|---|
| `GET /api/environment/status` | `X-Student-Id` | Current environment state for this student, or `{status:'none'}` |
| `POST /api/environment/launch` | `X-Student-Id` | Reuse or provision; blocks until running/error (matches the Provisioner's own synchronous provisioning) |
| `POST /api/environment/reset` | `X-Student-Id` | Destroy + recreate, same environment identity |
| `GET/POST/* /lab/:token/*` | opaque token (+ optional `?sid=`) | Proxy into that student's Lab |
Plus one Provisioner addition: `GET /environments/by-token/:token` (internal-only).
Responses to the browser never include `internal_ip`, `container_id`, `network_id`,
`lab_host_id`, `environment_id`, or the raw routing token beyond its one legitimate use as a
navigation path segment (`launch_path`) — see §6.
---
## 5. Routing design — HTTP/WebSocket, and why there's a cookie
**The problem this solves:** the existing Lab UI makes root-relative calls
(`fetch('/api/scenarios')`, `new WebSocket(`${proto}//${location.host}/shell-ws`)`) — it has no
idea it might be served under a `/lab/<token>/` prefix, and it shouldn't have to (preserving
the existing architecture unmodified was an explicit requirement). Naively prefix-stripping
only the *first* request would leave every subsequent asset/API/WS call unrouted.
**Solution:** the Gateway sets an `HttpOnly`, `SameSite=Lax`, `Path=/` cookie (`rn_lab`,
8h max-age) on the first `/lab/<token>/` response. Every later request from that browser —
regardless of path — carries it automatically. Routing priority:
1. `/lab/<token>/*` (explicit) → resolve token, strip prefix, proxy, (re)set the cookie.
2. `/api/environment/*` → always the Gateway's own routes, **never** affected by the cookie —
this is what keeps "student is inside their lab" from accidentally hijacking the dashboard's
own environment-status calls.
3. Anything else, unprefixed → if the `rn_lab` cookie is present and resolves to a `running`
environment, proxy there (unprefixed) — this is what makes the *existing, unmodified* Lab
UI's own `/api/scenarios`, `/shell-ws`, static assets, etc. resolve correctly once a browser
has navigated into a lab session, with zero changes to that UI's code. Otherwise, proxy to
the identity backend (this is what serves `/`, the platform-shell bundle, `/api/roster`,
`/api/curriculum/path`, etc.).
WebSocket upgrades follow the identical priority (`server.on('upgrade', ...)` in
`gateway/src/server.js`), so the terminal's `/shell-ws` connection is routed the same way as
any other request.
**Stable URL, even across recreate:** the student-facing URL is always `/lab/<routing_token>/`
— the token lives on the `environments` row and does not change across `reset` (same row, new
container/network/IP). A bookmarked or previously-loaded lab URL keeps working after a
recreate without the student needing a new link.
---
## 6. Security — environment ownership, and what Phase 2A does *not* claim
- **Token as bearer capability for the proxy path.** A `/lab/<token>/` request is authorized by
possession of the token itself (CSPRNG, 32 bytes, never guessable) — the same trust model as
a signed download link. This is a deliberate, stated design choice, not an oversight.
- **`?sid=` as an additional check where it's available.** A full top-level browser navigation
cannot carry a custom header, so the Dashboard's "Launch/Continue Lab" link appends
`?sid=<student_id>` to the URL. If present, the Gateway compares it against the environment's
actual owning `student_id` (from the Provisioner) and rejects (403) on mismatch. This is
strictly additive security on top of token-alone — its absence (e.g., a bookmarked link
without the param) falls back to token-only trust, not a hole opened by omitting it.
- **`/api/environment/*` ownership is checked against the Provisioner's own record**, not
assumed from the header — `listForStudent`/`reset` always re-verify `env.student_id`.
- **Explicitly not production auth.** Per the governing requirement: routing-token-alone,
student-supplied-environment-ID-alone, and `X-Student-Id`-alone are each individually
insufficient and documented as such — Phase 2A's actual security property is the
*combination*, appropriate for a supervised pilot, not a claim of production-grade identity.
Production SSO/OIDC is explicitly future work (§9).
- **Nothing sensitive reaches the browser**: no internal Docker IPs, no Docker/Provisioner
endpoints, no shared secret, no Postgres credentials — verified by inspecting every Gateway
response shape in `server.js`'s `publicEnvView()`.
---
## 7. Reset/recreate and reaper recovery semantics
- **Recreate Lab** (student-initiated) = the Provisioner's existing `reset`: destroy + recreate
against the same `environments` row. Same `environment_id`/routing token, same
`/lab/<token>/` URL, fresh container/network/IP. Academic progress is untouched because it
never lived in the Lab container to begin with (Phase 1 property, unchanged).
- **Reaper-caused expiry** (idle timeout, no student action): the environment transitions to
`destroyed`. `GET /api/environment/status` correctly reports `none`/`destroyed` afterward —
the Dashboard's "Launch Lab" button provisions a **new** environment (new `environment_id`,
new token, new URL) rather than trying to resurrect the old row, matching the Provisioner's
actual lifecycle model. Progress survives because, again, it was never in the Lab.
- **Unexpected Lab death** (crash, not reaped): the Gateway's `isReachable()` check
(`status==='running' && internal_ip` present) fails closed — a dead container never gets
silently proxied to. The student sees the `error` state with a "Try Again" action.
---
## 8. Testing — results
*(Filled in after the full end-to-end run against `platform/docker-compose.dev.yml`; see the
Phase 2A final report for the authoritative results table. Summary below.)*
- Lab image: node-pty load, k3s Ready, terminal, kubectl, scenario validation — **all pass**,
§1 above.
- Gateway HTTP/WS proxying, two-student isolation, cross-student rejection, stable URL across
recreate, reaper recovery, existing single-container dev mode, scenario files untouched,
frontend/backend builds, Provisioner test suite — see final report for the full table and
exact commands run.
---
## 9. Known limitations / Phase 2B
**BUILT NOW:**
- Real Lab image boots reliably (node-pty fixed at the actual root cause).
- Gateway: HTTP + WS proxying, environment-aware routing, ownership checks, no public Lab
ports, no exposed Docker/Provisioner internals.
- Student entry (roster pick-name) → dashboard (curriculum + environment status) → Launch Lab
→ Lab session → Reset/Recreate, all wired through the Gateway as the single origin.
- Reaper-recovery and reset semantics correctly reflected in the student-facing states.
**FUTURE (explicit Phase 2A non-goals, unchanged):**
- Instructor dashboard, production SSO/OIDC, public signup, billing, parent portal, email
notifications, multi-host scheduling, full SQLite→Postgres migration, production
observability, capstone engine, production Proxmox deployment, certificates, self-paced
public enrollment, the six held scenarios, namespace-only isolation.
**Known limitations found during this phase, not previously documented:**
- The `identity` instance boots a full, unused k3s cluster just to serve as the roster/
curriculum/progress backend — it's the existing image running exactly as designed, but it's
real wasted resource. A Phase 2B optimization is a slimmed identity-only variant of the image
(skip k3s entirely) — not done here, to keep this phase at zero changes to the Lab image's
own runtime behavior beyond the node-pty fix.
- `identity` requires `network_mode: bridge` explicitly in `docker-compose.dev.yml` — under
Compose's own auto-created bridge network, k3s failed with "no default routes found in
/proc/net/route" at startup. Root cause not fully chased down (a Compose-network-vs-classic-
bridge routing-table timing/detection difference); the explicit `network_mode: bridge`
workaround is confirmed reliable and is what's shipped. Worth a deeper look before this
matters for a real deployment (bridge mode isn't compatible with a per-student isolated
network model anyway — the production identity service placement needs its own design pass).
- The Gateway's environment-lookup cache (5s TTL) is in-memory and per-process — fine for one
Gateway instance (Phase 2A's actual topology), would need a shared cache (or none) if the
Gateway is ever horizontally scaled.
- `?sid=` ownership verification only covers the initial `/lab/<token>/` navigation, not every
subsequent proxied request (by design, per §6) — acceptable for the stated pilot threat
model, revisit if this graduates past pilot use.
+213
View File
@@ -0,0 +1,213 @@
# RootNode Academy — Phase 2B: Admin Portal & Platform Data
Status: **IMPLEMENTED and end-to-end tested, 2026-08-18.** Builds on
`docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md` (Phase 2A milestone
`42dfb1c31b599d5e86b64915beb4b871b2e91055`). This doc covers: the platform/lab split, the
Postgres academic schema and SQLite migration, server-controlled sessions, and the
instructor/admin control plane.
---
## 1. Split: Platform API vs. Lab image
Phase 2A's "identity" instance was a full, `--privileged` Lab image booting an unused k3s
cluster just to serve roster/curriculum/progress APIs. Phase 2B replaces it with
`platform/api/` — a plain Node/Express service: no k3s, no `--privileged`, no terminal
runtime, no nginx (serves the built frontend directly via `express.static`). Startup is
instant instead of a 1-2 minute k3s cold start — directly observable in testing (see §8).
The Lab image (`Dockerfile`, `backend/server.js`, `scripts/entrypoint.sh`) is **completely
unchanged in this phase** — it's still the disposable-per-student, nested-k3s, terminal +
scenario-execution engine, exactly as Phase 1/2A built it. Platform API never runs
setup/validate/teardown commands itself; it only relays outcomes the Lab already produced
(see §7, "Progress relay," carried over from Phase 2A and extended).
Platform API owns: students, cohorts, enrollments, curriculum assignments, progress, scenario
attempts, student/admin session identity, and — new in Phase 2B — **all** Provisioner calls,
both student-initiated (launch/reset) and admin-initiated (prepare-labs, destroy-labs, reset,
destroy). The Gateway no longer calls the Provisioner directly at all; it proxies
`/api/environment/*` and `/api/admin/*` to Platform API, which is now the single place that
holds the Provisioner's shared secret.
---
## 2. PostgreSQL academic schema
New tables in the existing `rootnode_platform` database (same instance the Provisioner's
`lab_hosts`/`environments`/`environment_events` already live in — different migration
directory, same Postgres):
- `students` — `student_id` stays the same opaque TEXT identifier used everywhere else
(no breaking change to `environments.student_id`, which remains a plain reference with no
FK, exactly as Phase 1 designed it).
- `cohorts` / `cohort_enrollments` — replaces the old single `students.cohort` TEXT column
with a real many-to-many model (a student can move between cohorts over time; the old
model couldn't represent that at all).
- `curriculum_assignments` — records which curriculum path a cohort or student is on. Only
one path exists today (`rootnode-path`) — this table exists so assigning it is a real,
auditable instructor action, not because multiple paths exist yet.
- `progress` — same shape as the SQLite table it replaces.
- `scenario_attempts` — **new capability**, not a migration of anything that existed before.
One row per validate/answer outcome (result, validation count, timestamp, which
environment), giving real per-attempt history instead of just an aggregate status +
counter. `hint_count` is present but always NULL — the existing Lab engine's hints are
static UI content only (`frontend/src/components/ScenarioPanel.jsx`), nothing server-side
ever records a hint view. Documented here rather than inventing a number; the column is
ready if hint-view tracking is ever added to the Lab engine.
- `exam_sessions` — historical import of the old SQLite `sessions` table. **Scope note:**
exam mode's *live* session bookkeeping (start/shuffle/submit/abandon) stays Lab-image-local
for Phase 2B — productionizing it would need the same relay pattern progress got in Phase
2A, and it isn't exercised by the Phase 2B acceptance test. This table makes historical exam
records survive the migration and visible to instructors; it isn't written to by new exam
sessions yet. Flagged as a real gap, not silently ignored — see §13.
## 3. SQLite → Postgres migration
`platform/api/migrate-sqlite.js`. Run once as the cutover:
```bash
SQLITE_PATH=/path/to/progress.db PLATFORM_DB_URL=postgres://... node migrate-sqlite.js
```
- **Backs up first, unconditionally** — copies the SQLite file to
`<path>.backup-<timestamp>` before reading anything. The backup is never deleted or
modified by the script.
- **Idempotent** — every write uses `ON CONFLICT`; re-running re-syncs from the SQLite source
rather than duplicating or erroring.
- **Never silently drops data** — prints and writes a JSON report
(`migration-report-<timestamp>.json`) with exact counts (students, progress rows,
exam_sessions, cohorts created, enrollments created) and anything that couldn't be read
(e.g. a missing table), with the reason.
- **Preserves the old `students.cohort` column** as a real cohort + enrollment row instead of
discarding it.
- **Rollback:** Postgres only becomes authoritative once this has run and the Platform API is
pointed at it — nothing here modifies or deletes the SQLite source. To roll back: stop
Platform API, point student/instructor traffic back at a SQLite-backed service (the old
Phase 2A "identity" image is unmodified and still buildable/runnable — see
`docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md`), and restore from the backup file if the live
SQLite file was somehow touched (it shouldn't have been — the migration is read-only against
it).
See the final report for actual migrated record counts from this implementation's test run.
---
## 3. Student session — productionized ownership
Phase 2A's `?sid=` was explicitly pilot-only: a client-supplied query parameter, trivially
changeable. Phase 2B replaces it with a server-controlled session:
- `POST /api/login {student_id}` (unchanged trigger — pick-your-name, no password, still
pilot-grade identity, stated as such) creates a session server-side
(`platform/api/src/sessions.js`, in-memory, 8h TTL) and returns an opaque, CSPRNG token as
an `HttpOnly`, `SameSite=Lax` cookie (`rn_session`). The browser cannot read or fabricate
this value.
- Platform API derives `student_id` **only** from this cookie
(`studentSession` middleware) — never from a header or query parameter a client controls.
- The Gateway resolves "who is this browser" by calling Platform API's
`GET /api/session/whoami` (forwarding the cookie), then compares that student_id against
the environment's actual owning `student_id` (from the Provisioner, via Platform API) —
**on every proxied Lab request, HTTP and WebSocket, not just the initial navigation.** A
short-TTL cache (`SESSION_CACHE_TTL_MS`, 5s) keeps the extra round trip cheap.
- Changing the token, URL, query parameter, or environment ID in the browser has **no effect**
on who the Gateway thinks is asking — that's determined entirely by the unforgeable cookie.
Verified by testing: a valid session for Student A requesting Student B's `/lab/<token>/`
gets **403**, not just an unset-parameter fallback.
**Explicitly not production SSO/OIDC.** Sessions are in-memory (a Platform API restart logs
everyone out — acceptable for a pilot/classroom tool, not silently ignored, see §13), and
login is still pick-your-name with no password. Production OIDC/Google Workspace SSO remains
Phase 3+ (§13/§14).
### Routing-hint cookie — a real bug found while building this
The Gateway's unprefixed fallback route (needed so the *unmodified* Lab UI's root-relative
`/api/scenarios`, `/shell-ws`, etc. resolve correctly once a browser is inside `/lab/<token>/`
— see Phase 2A's cookie design) initially used "does this session currently own a running
environment" as the signal for "should I try routing this to the Lab." That's wrong: it's
true on the *dashboard* too, not just inside the lab page, and it hijacked
`GET /api/session/whoami` (and would have hijacked `/api/environment/*`, `/api/admin/*`,
`/api/roster*`) to the student's Lab container — which doesn't have those routes at all —
whenever a student had ever launched a lab. Confirmed by testing (whoami returned the SPA's
`index.html` instead of JSON).
Fixed with an explicit allowlist (`LAB_OWNED_PATHS` in `platform/gateway/src/server.js`) of
the specific unprefixed paths the Lab UI actually calls
(`/api/scenarios*`, `/api/bundles`, `/api/sessions*`, `/api/cache*`, `/api/health`,
`/shell-ws`, `/assets/*`) — everything else always goes to Platform API regardless of whether
the student has a running lab. `LAB_ACTIVE_COOKIE` (set only on an actual `/lab/<token>/`
visit) is a secondary gate on top of that, not the sole signal.
---
## 4-9. Admin/Instructor portal
New `/admin` route in the same frontend bundle (`frontend/src/admin/`) — a separate React tree
from the student Platform Shell, gated by its own login (`rn_admin` session cookie, completely
separate cookie/token namespace from `rn_session` — a student session can never call an admin
route and vice versa).
**Dashboard** (`AdminDashboard.jsx`) — active cohorts, total students, Labs by state
(running/provisioning/error/idle), recent completions, recent environment failures. No
Prometheus/Grafana — this is "is class ready / is anything on fire," not an observability
platform (explicit non-goal).
**Cohorts** (`CohortsView.jsx`) — create/edit, roster (add/view students), curriculum
assignment, per-stage progress bars aggregated across the cohort, and the classroom
preparation workflow:
- **Prepare Labs** — finds enrolled active students, provisions anyone without a usable
environment. Runs concurrently (`Promise.allSettled`) — sequential would make a real cohort
prep take the better part of an hour, since each provision is a genuine k3s cold start.
Returns a per-student readiness report.
- **Destroy Cohort Labs** — destroys every enrolled student's environment. Academic progress
lives entirely in Postgres, untouched by this — by construction, not by a separate
safeguard, since the destroy path never touches `students`/`progress`/`scenario_attempts`.
**Students** (`StudentsView.jsx`) — per-student curriculum progress, recent attempts
(scenario, result, validation count, timestamp), current environment status, and controls:
provision / reset-recreate / destroy / view event history.
**Admin auth** (`platform/api/src/admin-auth.js`) — a single instructor credential
(`ADMIN_PASSWORD` env var, the same secret-injection channel every other component here
uses), verified via `scrypt` + `timingSafeEqual` rather than a plain string comparison — no
plaintext password ever sits in a database or gets logged. Explicitly not OIDC; documented as
Phase 3+ future work.
---
## 10-12. API routes, security boundary
See the route table in the final report. Boundary preserved throughout:
```
Student browser → Gateway → Platform API → Provisioner → Lab Host
Admin browser → Gateway → Platform API → Provisioner → Lab Host
```
Browser JavaScript never reaches the Provisioner directly in either path — Platform API is
now the *only* thing holding `PROVISIONER_SHARED_SECRET`.
---
## 13. Known limitations
- **Sessions are in-memory, single-process.** A Platform API restart invalidates every student
and admin session. Acceptable at pilot/classroom scale; would need a shared store (Postgres
or Redis) before this runs across multiple Platform API instances.
- **Exam mode is not fully productionized** — live exam sessions stay Lab-local; only
historical records migrate to Postgres. See §2.
- **`identity`'s Phase 2A `network_mode: bridge` requirement no longer applies** — Platform
API replaces it entirely and doesn't run k3s, so that whole class of problem is gone, not
just documented around.
- **Admin credential is a single shared password**, not per-instructor accounts — fine for a
single-instructor pilot, not a multi-instructor deployment model.
- **`?sid=` is gone entirely** (Phase 2A's mechanism) — nothing reads it anymore; a stale
bookmarked URL containing it is harmless, the parameter is simply ignored.
## Future (Phase 3+)
Production OIDC/SSO, per-instructor accounts, full students/progress historical fidelity for
exam mode, shared session store for horizontal scaling, everything already listed as
out-of-scope in Phase 2A/2B (parent portal, billing, capstone engine, production Proxmox,
full observability, multi-host scheduling, public enrollment).
@@ -0,0 +1,344 @@
# Phase 3A — Identity and Instructor Access
Productionizes identity for RootNode Academy: persistent sessions, real per-instructor accounts
with roles, server-enforced RBAC, CSRF/rate-limiting hardening, and a platform-level audit log.
Builds on `docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md` (Platform API split, Postgres migration,
first-generation session cookies, admin portal) — read that first if you haven't.
## 1. What changed, in one paragraph
Phase 2B's sessions lived in an in-memory `Map` (gone on every restart) and instructor access
was one shared `ADMIN_PASSWORD` env var (no accountability, no roles, no revocation). Phase 3A
replaces both: sessions are now rows in Postgres (`sessions` table, survive a restart, are
individually revocable), and instructors are real accounts (`instructors` table) with a role —
`owner`, `admin`, or `instructor` — enforced server-side on every `/api/admin/*` route, with a
plain `instructor` further scoped to the cohorts they're assigned to. Every security-relevant
action is now recorded in a platform-level audit log (`audit_events`).
## 2. Schema added
```
instructors instructor_id, email (unique), display_name, password_hash, role, status,
created_at, updated_at, last_login_at
auth_identities auth_identity_id, instructor_id, provider ('local' today), provider_subject,
provider_email, created_at, last_login_at — one 'local' row per instructor;
a future OIDC login adds a second row for the SAME instructor_id, no schema
change needed then.
sessions session_id, session_type ('student'|'instructor'), student_id, instructor_id,
token_hash (SHA-256 of the bearer token — the raw token is NEVER stored),
created_at, last_seen_at, expires_at (absolute), idle_expires_at (sliding,
capped at expires_at), revoked_at, metadata
cohort_instructors cohort_id, instructor_id, role_in_cohort — scopes the plain 'instructor'
role; owner/admin never consult this table (global visibility by role alone)
audit_events audit_event_id, actor_type, actor_id, action, resource_type, resource_id,
result, source_ip, metadata, created_at
```
All five are new migrations (`platform/api/migrations/007`–`011`), idempotent (the existing
`schema_migrations` tracking table already guarantees that), and applied automatically on
Platform API startup — no manual migration step. None of Phase 2B's tables
(`students`/`cohorts`/`progress`/`scenario_attempts`/etc.) were touched; all Phase 2B data is
preserved exactly as-is. `sessions.instructor_id` and `cohort_instructors.instructor_id` are
real foreign keys to `instructors` (both tables live in the same Platform API-owned migration
set) — unlike `environments.student_id` in the Provisioner's schema, there's no
cross-service-database reason to keep this one an opaque reference.
## 3. Persistent session design
Browser holds one opaque, CSPRNG bearer token (32 random bytes, base64url) per session, in an
`HttpOnly`, `SameSite=Lax` cookie (`Secure` too, once `COOKIE_SECURE=1`). The server stores only
`SHA-256(token)` — a Postgres dump or backup can't be replayed as a working session, same
principle as password hashing.
Two independent lifecycles (`sessions.session_type`), separate cookie names
(`rn_session`/`rn_instructor`), separate token namespaces — a bug can't confuse a student
session for an instructor one. Each session has:
- **Absolute expiry** (`expires_at`, fixed at creation) — 8h student / 12h instructor by default,
configurable via `STUDENT_SESSION_TTL_HOURS` / `INSTRUCTOR_SESSION_TTL_HOURS`.
- **Idle expiry** (`idle_expires_at`, slides forward on every authenticated request, never past
`expires_at`) — 2h default for both, configurable via `*_SESSION_IDLE_HOURS`.
Resolving a session (`sessions.js` `resolveSession`) is one atomic `UPDATE ... RETURNING` that
checks `revoked_at IS NULL AND expires_at > now() AND idle_expires_at > now()` and slides the
idle window forward in the same round trip — no separate read-then-write race.
**Restart behavior:** a Platform API restart no longer logs anyone out — verified in
`platform/test/e2e-phase3a.js` by actually running `docker compose restart api` mid-test and
confirming both an instructor session and a student session still resolve afterward.
**Role/status changes take effect immediately, no revocation needed:** the instructor session
middleware re-fetches the instructor row from Postgres on *every* request rather than trusting
anything cached in the session token — so demoting a role or deactivating an account is live on
the very next request that account makes, with no separate "walk every session and revoke"
step required. Explicit revocation (owner-triggered "revoke all sessions," and automatic on
password change) exists too, for the case where you want to kill an active session immediately
rather than wait for its next request.
## 4. Instructor accounts
`instructors.password_hash` is `scrypt` with a **per-instructor random salt** (Phase 2B's
`admin-auth.js` used one fixed salt because there was exactly one shared password — that's gone,
this is real per-row salting now), stored as `"<salt-hex>:<hash-hex>"`, verified with
`crypto.timingSafeEqual`. No plaintext password ever touches a log line, a database column, or
an API response body (`publicInstructor()` strips `password_hash` before any JSON response).
Bootstrap: `node platform/api/bootstrap-owner.js <email> <name> [password]` — the only supported
way to create the first account, so a fresh deployment never requires editing Postgres by hand.
Refuses to run if an active owner already exists unless the target email is an existing
instructor (the intended zero-owners recovery path).
## 5. Role model (RBAC matrix)
| Action | owner | admin | instructor |
|---|---|---|---|
| Manage instructor accounts (create/edit/deactivate/reset password/revoke sessions) | ✅ | ❌ | ❌ |
| View audit log | ✅ | ✅ | ❌ |
| Create/edit cohorts, create/edit students, assign curriculum | ✅ | ✅ | ❌ |
| Assign an instructor to a cohort | ✅ | ✅ | ❌ |
| View dashboard, cohort list, cohort/student detail, progress | ✅ (global) | ✅ (global) | ✅ (**assigned cohorts only**) |
| Prepare Labs / Destroy Cohort Labs, provision a student's Lab | ✅ (any cohort) | ✅ (any cohort) | ✅ (**assigned cohorts only**) |
| Reset/destroy/view-events by raw environment id | ✅ | ✅ | ❌ — see §6 |
Enforced by `src/rbac.js`, applied per-route in `src/admin-routes.js` — never in the frontend
alone (`frontend/src/admin/*.jsx` hides controls a role can't use, but that's UX; every route
re-checks server-side regardless of what the browser sends or hides). A denied request gets a
flat `403 {"error":"forbidden"}` — no distinction in the response between "wrong role" and
"right role, wrong cohort," and no distinction between "resource doesn't exist" and "you can't
see it," so a 403 never confirms anything about what exists to someone probing.
An unauthenticated request gets `401`, not `403` — the two are kept distinct throughout.
## 6. Cohort-scoped instructor access
`cohort_instructors` maps `instructor_id → cohort_id` (assigned by an owner/admin, never
self-service). A plain `instructor` role's requests to cohort- or student-keyed routes go
through `requireCohortAccess`/`requireStudentCohortAccess`, which check this table;
owner/admin bypass it entirely (global visibility by role alone).
**Deliberate simplification:** the raw `/api/admin/environments/:id/{reset,DELETE,events}`
routes (operate directly on a Provisioner environment id, no cohort context in the URL) stay
owner/admin-only rather than resolving cohort scope from a bare environment id — that resolution
would need an extra Provisioner round trip on every single call. A plain instructor still has
full Lab control, just via the cohort/student-scoped routes (`/cohorts/:id/prepare-labs`,
`/students/:id/environment/provision`, etc.), which already carry the scope check. Upgrade path
if this ever needs to change: add a `student_id`/`cohort_id` lookup to the Provisioner's
`GET /environments/:id` response and extend `requireCohortAccess` to resolve through it.
## 7. Student session hardening
The pick-your-name roster flow is unchanged UX — **this is classroom/pilot identification, not
production authentication.** There is still no password or possession factor at the identity
step; anyone who knows a display name (or can see the roster dropdown) can "become" that
student. What Phase 3A hardens is everything *after* that pick:
- The resulting session is server-controlled and now **persistent** (survives a Platform API
restart — verified, see §3).
- A student cannot switch identity by editing client state — there's nothing client-side to
edit; the cookie is `HttpOnly` and opaque.
- Logout revokes the session server-side (`sessions.destroyStudentSession`), not just a
client-side cookie clear.
- Every Gateway-proxied Lab request (HTTP and WebSocket) re-verifies ownership against the
session on every request, not just the initial navigation — unchanged from Phase 2B, still
true here.
- A student session can never call `/api/admin/*` — those routes require an `rn_instructor`
session in a completely separate cookie/token namespace.
- CSRF (see §8) now also protects student mutating routes (`/api/environment/launch`, `/reset`,
`/api/progress/record`, `/api/logout`) the same way it protects instructor ones.
**Future upgrade path that doesn't require touching the session system:** one-time student
enrollment codes, parent-issued codes, or institutional SSO would all still terminate in the
exact same `sessions` table and `rn_session` cookie — only the *identity verification step*
before `createStudentSession()` is called would change. This is why the session store was worth
building as a shared, session-type-agnostic module rather than something bespoke to the current
roster-picker.
## 8. CSRF protection
Double-submit cookie pattern, on top of (not instead of) `SameSite=Lax`. On login, a
**non-HttpOnly** cookie is set — `rn_csrf_s` (student) / `rn_csrf_i` (instructor) — deliberately
readable by the frontend's own JS via `document.cookie`. Every state-changing request
(`POST`/`PATCH`/`DELETE`) must echo that value back as an `X-CSRF-Token` header; the server
checks the header matches the cookie it's holding for that same request. A cross-origin
attacker's page can trigger a cookie-carrying request but can't read the cookie's value (browser
same-origin policy) or attach a matching custom header without CORS, which isn't granted to
other origins — so it can never produce a request where the two match.
`GET`/`HEAD`/`OPTIONS` are exempt (nothing to protect). Applied uniformly, including to
`/api/logout` and `/api/auth/logout` — a forged logout is low-severity, but consistent
enforcement was simpler to reason about than a per-route exception.
**One real bug found while testing this:** the Gateway's own progress-relay call
(`platform/gateway/src/server.js` `relayScenarioOutcome`, triggered when a student solves a
scenario inside their Lab) is a server-to-server call to `POST /api/progress/record`, not a
browser one — but it's relaying the browser's own session, so it needs to forward the CSRF
token too. The first fix attempt forwarded only the `X-CSRF-Token` header and still 403'd,
because the double-submit check needs BOTH the cookie *and* a matching header on the request
Platform API actually receives — the fix forwards both.
## 9. Login rate limiting
In-memory sliding window (`src/security.js`), keyed by `ip:email`, max 8 attempts per 5 minutes,
`429` past that. No new dependency — small enough to hand-roll. **Known limitation:** in-memory
means it resets on a Platform API restart and doesn't share state across multiple instances; a
Postgres- or Redis-backed limiter would be the upgrade if this ever needs to survive a restart
or scale beyond one process. Not needed at pilot scale.
## 10. Audit log
`audit_events`, distinct from the Provisioner's `environment_events` (Lab-lifecycle-only).
Records: instructor login success/failure, logout, password change, instructor
created/updated/deactivated, role changes, session revocation, cohort create/update, roster
changes, curriculum assignment, Prepare Labs, Lab reset/destroy, and every `403` authorization
denial (recorded centrally by the RBAC middleware itself, not sprinkled per-route). Never
records plaintext passwords, raw session tokens, Provisioner secrets, or raw routing tokens —
verified by an explicit test assertion (`e2e-phase3a.js` "audit log" case scans recorded
metadata for anything password-shaped). Viewable at `/admin` → Audit (owner/admin only), latest
200 events, no pagination UI yet — pilot-scale, not a SIEM.
## 11. API inventory (new/changed)
```
POST /api/auth/instructor/login email+password login, rate-limited, no enumeration
POST /api/auth/logout revokes the current instructor session
GET /api/auth/session current instructor identity (role, etc.)
POST /api/auth/change-password requires current password; revokes other sessions
POST /api/login UNCHANGED route, now backed by persistent sessions
POST /api/logout UNCHANGED route, now CSRF-checked + persistent revoke
GET /api/session/whoami UNCHANGED — Gateway's ownership-check dependency
GET /api/admin/instructors owner only
POST /api/admin/instructors owner only
GET /api/admin/instructors/:id owner only — includes session list
PATCH /api/admin/instructors/:id owner only — role/status/name; blocks removing the last active owner
POST /api/admin/instructors/:id/revoke-sessions owner only
POST /api/admin/instructors/:id/reset-password owner only — also revokes that instructor's sessions
GET /api/admin/audit-events admin/owner
POST /api/admin/cohorts/:id/instructors owner/admin — assigns an instructor to a cohort
```
Every pre-existing `/api/admin/*` route from Phase 2B still exists at the same path — behavior
unchanged for owner/admin, newly scoped for the `instructor` role (see §5/§6). `/api/admin/login`,
`/api/admin/logout`, `/api/admin/whoami` (the old shared-password routes) are gone — replaced by
the `/api/auth/instructor/*` routes above.
## 12. Frontend/admin changes
- `AdminPortal.jsx` — email+password login form (was a bare password field); nav now
conditionally shows Instructors (owner) and Audit (owner/admin) tabs; adds an Account tab.
- `InstructorsView.jsx` (new) — create/list/edit instructors, role/status controls, session list
+ revoke-all, password reset.
- `AuditView.jsx` (new) — read-only audit event table.
- `AccountView.jsx` (new) — change-password form for the signed-in instructor.
- `CohortsView.jsx` — create-cohort form and curriculum-assignment button hidden for the
`instructor` role (server already enforces this; this just avoids showing controls that would
403).
- `frontend/src/admin/api.js` / `frontend/src/platform/api.js` — both now read the relevant
`rn_csrf_*` cookie and attach `X-CSRF-Token` on every mutating call.
## 13. Security hardening review (what was checked)
- **Cookie settings** — `HttpOnly` on session cookies, `SameSite=Lax`, `Secure` gated behind
`COOKIE_SECURE=1` (off by default for local HTTP dev).
- **CSRF** — §8.
- **Login brute force** — §9.
- **Session fixation** — every login issues a brand-new session row (never reuses/upgrades an
existing token), so there's nothing to "fix" a pre-auth session into.
- **Password handling** — §4.
- **Account enumeration** — identical response for unknown-email and wrong-password (tested).
- **Authorization bypass** — every `/api/admin/*` route re-checks server-side (§5); tested with
both role-based and cohort-scoped negative cases.
- **Inactive accounts** — rejected at the session-middleware level on every request, not just at
login (a deactivation takes effect immediately, §3).
- **Revoked sessions** — `resolveSession`'s `WHERE revoked_at IS NULL` makes a revoked session
fail identically to an expired one.
- **Role downgrade/upgrade session invalidation** — handled by the live re-fetch design (§3),
not a separate revoke pass.
- **WebSocket authentication** — unchanged from Phase 2A/2B: the Gateway's WS upgrade handler
re-verifies ownership via the same session-resolution path as HTTP.
- **Gateway ownership checks** — unchanged mechanism, tightened cache TTL (5000ms → 2000ms) so a
revoked session stops working at the Gateway within a tighter, tested bound.
- **`trust proxy`** — off by default, opt-in via `TRUST_PROXY=1`, so `req.ip` (used for audit
`source_ip` and rate-limit keys) isn't trivially spoofable via `X-Forwarded-For` unless a real
trusted proxy is actually in front.
## 14. Test results
- `platform/provisioner/test/run.js` (Phase 1, isolated stack) — **not re-run this phase**: zero
source changes to `platform/provisioner/` (confirmed via `git diff`), and its own dev compose
stack collides on port 5433 with the shared `platform/docker-compose.dev.yml` stack used for
everything else in this phase. Provisioner was still exercised heavily and indirectly through
every e2e suite below (provision/reset/destroy, concurrent Prepare Labs, cohort-scoped
operations) — all passing.
- `platform/test/e2e-phase2a.js` — **10/10 passed** (student flow, now under persistent sessions).
- `platform/test/e2e-phase2b.js` — **12/12 passed** (classroom workflow, now under real
instructor accounts instead of a shared password).
- `platform/test/e2e-phase3a.js` — **21/21 passed**: no-enumeration, CSRF positive+negative,
rate limiting, role creation, RBAC denial (instructor→owner-only, instructor→admin-only,
admin→owner-only), cohort scoping (visibility + access), last-owner protection, live
role/status re-evaluation, password change + old-password rejection + other-session
revocation, audit log content (including the no-secrets assertion), student session RBAC,
student logout revocation, and session persistence across a real `docker compose restart api`.
- Real Lab image — rebuilt clean; standalone `--privileged` single-container mode boots and
serves with zero Platform API/Postgres/Provisioner dependency (verified live).
- Frontend production build — clean, no errors.
- Scenario content / Real Ops bundle / Lab `Dockerfile`/`backend/`/`scripts/` — `git diff`
against the Phase 2B milestone SHA is empty for all of them: untouched.
## 15. Known limitations
- **Login rate limiting is in-memory, single-process** (§9) — resets on restart, doesn't share
state across multiple Platform API instances. Fine at pilot scale.
- **Gateway session cache is a 2-second window** — a revoked/expired session can still work at
the Gateway for up to ~2s after revocation (tightened from 5s in Phase 2B). Platform API
itself always reflects revocation immediately; this bound is specific to the Gateway's
whoami-result cache, which exists purely to avoid a Platform API round trip on every single
proxied Lab byte.
- **No distributed session revocation push** — revoking a session updates Postgres immediately;
anything holding a *cached* copy of "this session is valid" (only the Gateway does this, and
only for 2s) catches up on its next cache miss, not instantly.
- **Audit log has no pagination UI** — latest 200 events only. Add pagination if usage ever
exceeds that being useful.
- **Roster name-picking is still not strong student identity** (§7) — explicitly documented,
not silently glossed over.
- **`cohort_instructors` role_in_cohort field exists but isn't consulted for finer-grained
permissions yet** — currently any assignment grants the full `instructor` role's cohort-scoped
actions; a co-teacher vs. lead-teacher distinction within a cohort would consume this field but
isn't built.
- **Exam mode's live bookkeeping is still Lab-local** — unchanged from Phase 2B, still a real gap
for anyone relying on cross-restart exam state.
## 16. Future OIDC path
`auth_identities` (§2) already supports it structurally: a Google/OIDC login would create a
second `auth_identities` row (`provider='google'`, `provider_subject=<their sub>`) pointing at
an existing `instructor_id`, or create a new `instructors` row + `auth_identities` row for a
first-time SSO login. The missing piece is the actual OIDC client flow (discovery, token
exchange, JWT verification) — deliberately **not built this phase**, per the explicit scope
instruction to stop at a documented adapter interface rather than pull in an OIDC client
library and expand scope. `instructors.password_hash` staying nullable (already the case) is
what allows an SSO-only account to exist with no local password at all, whenever that flow is
built.
## 17. Production HTTPS / cookie expectations
Local dev and the current single-VM pilot deployment run over plain HTTP — `COOKIE_SECURE` is
unset (cookies lack `Secure`) and `TRUST_PROXY` is unset by default. Before any real-HTTPS
production deployment: set `COOKIE_SECURE=1` (cookies become HTTPS-only) and, if fronted by a
reverse proxy/load balancer, set `TRUST_PROXY=1` so `req.ip` reflects the real client IP instead
of the proxy's.
## 18. BUILT NOW vs. FUTURE
**Built now:** persistent Postgres-backed sessions (student + instructor), per-instructor
accounts with scrypt password hashing, three-role RBAC enforced server-side, cohort-scoped
instructor access, CSRF double-submit protection, login rate limiting, platform audit log,
owner-safety invariant (never zero active owners), password change with other-session
revocation, `auth_identities` table ready for future OIDC mapping.
**Future (not built):** real OIDC/SSO client flow, multi-instance-safe rate limiting, audit log
pagination/export, cohort_instructors role-in-cohort permission granularity, distributed
(non-Gateway-cache) session revocation push, one-time student enrollment codes /
parent-issued codes, everything already on the Phase 2B "DO NOT BUILD YET" list (capstone
engine, billing, production Proxmox deployment, etc.) — unchanged, still not in scope.
+370
View File
@@ -0,0 +1,370 @@
# Phase 3B — Learning & Assessment Engine
Turns the platform infrastructure built in Phases 1–3A (identity, sessions, RBAC, disposable
Labs, Gateway routing, admin portal) into a real teaching and assessment system: detailed
attempt telemetry, progressive hints, rule-driven curriculum progression, a proper exam engine,
a RootNode Capstone, and persistent achievement records. Builds on
`docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md` — read that first if you haven't.
## 1. What changed, in one paragraph
Phase 2B's `scenario_attempts` table was one row per validation click, written atomically at
completion, with `hint_count` deliberately left unmeasured. Phase 3B evolves it into a real
attempt lifecycle — one row per continuous working session on a scenario, with live counters
(validation attempts/passes/fails, hint reveals, duration) updated in place as the student
works. On top of that: a rule-driven curriculum-progression engine that actually locks/unlocks
stages instead of a permanent capstone placeholder; a progressive hint system that's genuinely
server-enforced (not just collapsed-by-default in the UI); a full exam/capstone assessment
engine with server-authoritative timers and random scenario draws; and persistent achievement
records for stage/curriculum/assessment/capstone completion.
## 2. Learning record model
`scenario_attempts` (migration `012`) gained: `cohort_id`, `curriculum_stage` (both snapshotted
at attempt-start, so a later cohort/curriculum change never rewrites history), `status`
(`in_progress`/`completed`/`abandoned`), `validation_attempt_count`, `validation_pass_count`,
`validation_fail_count`, `highest_hint_level`, `duration_seconds`, `assessment_session_id`.
`result` (Phase 2B) is now nullable — meaningful only once `status = 'completed'`.
`hint_count` (Phase 2B, always `NULL`) finally gets a `DEFAULT 0` and real increments.
This is an **evolution, not a new table** — Phase 2B's historical rows are backfilled (one
validation performed, matching their original one-row-per-click semantics), not dropped or
reshaped. `platform/api/src/attempts.js` owns the lifecycle:
- `getOrCreateOpenAttempt` — finds the open (`in_progress`) attempt for
`(student, scenario, environment)`, or starts one. Environment-scoped so a destroyed/recreated
Lab never silently resumes an attempt tied to compute that no longer exists.
- `recordValidation` — increments the attempt/pass/fail counters; closes the attempt
(`status='completed'`, `duration_seconds` computed) only on a pass.
- `recordHintReveal` — sequential only: level *N* can't be revealed before *N-1* has been.
Returns a 409 once every hint is exhausted.
No telemetry is invented that the Lab doesn't actually emit — `duration_seconds` is measured
server-side from `started_at` to completion (not client-reported), and hint reveals only exist
because Phase 3B builds a real reveal endpoint (§3), not because the old static hint array was
reinterpreted as "measured."
## 3. Validation telemetry & progressive hints
`POST /api/scenarios/:id/start` — fire-and-forget from the Lab UI when a scenario becomes
active (both task and MCQ types); creates/reopens the attempt with a real `started_at`.
`POST /api/progress/record` (Gateway-relayed from the Lab's existing validate/answer calls,
unchanged mechanism from Phase 2A/2B) now routes through `attempts.recordValidation` instead of
a flat insert, and — on a pass — checks curriculum/capstone/assessment consequences (§4, §6).
`POST /api/scenarios/:id/hint` reveals hints **one at a time, server-enforced**. This required a
real architectural change: Phase 2B's Lab backend (unmodified, per the "Lab image stays
unmodified" convention through Phase 3A) returns the scenario's full `hints` array in
`GET /api/scenarios/:id` — always did, since the Lab has no student/attempt concept at all. A
client could already see hint 3 in the raw response even with the UI collapsed. Making the gate
real meant the Gateway now **strips `hints` from that response** (replacing it with a bare
`hint_count`) for exactly the multi-student, Platform-mediated deployment
(`platform/gateway/src/server.js`, `GET /lab/:token/api/scenarios/:scenarioId`) — the Lab
image's own bytes are still untouched; this is a response rewrite at the boundary, the same
pattern already used for the `hints`-stripping shape as everything else the Gateway mediates.
Standalone/single-container dev mode never goes through the Gateway, so it keeps the original
fully-preloaded array unchanged — the frontend detects which mode it's in
(`scenario.hints === undefined && typeof scenario.hint_count === 'number'`) and switches to
`ProgressiveHints` (`frontend/src/components/ProgressiveHints.jsx`) only in the gated case.
Instructor-visible attempt summary matches the shape in the Phase 3B spec directly:
```
Student: Alice Scenario: Probe Port Mismatch Attempts: 2
Validation: Failed 2, Passed 1 Hints used: 1/3 Time: 14m 22s Final: Completed
```
— all straight off `scenario_attempts` columns, no derived/estimated numbers, and explicitly
**not** turned into a score. See §12 for where instructors see this.
## 4. Curriculum completion rules
`scenarios/rootnode-path.json` gained a `version` field and, per stage, a `completion_rule`:
```json
{ "type": "all_required" } // every scenario_id in the stage must be completed (default)
{ "type": "threshold", "percent": 80 } // at least N% completed
{ "type": "prerequisites", "scenario_ids": [...] } // only these specific ids must be completed
```
The capstone stage is no longer a hardcoded `"locked": true` placeholder — it carries
`"type": "capstone"`, an `"unlock_rule"` (`{"type":"stages_complete","stage_ids":[...]}`), and a
`"capstone_assessment_id"` pointing at the assessment definition that drives it (§6/§7).
`platform/api/src/curriculum-rules.js`'s `computeCurriculumState(path, progress, capstoneStatus)`
is the rule engine — pure, synchronous, no DB access — answering: per-stage locked/complete/
percent/next-scenario, the student's current stage, overall percent complete, and the capstone's
unlock status (server route supplies `capstoneStatus` after checking for a
`capstone_completed` achievement, since that's Postgres state this module doesn't touch).
Exposed at `GET /api/curriculum/state` (new; `GET /api/curriculum/path` from Phase 2A/2B is
unchanged and still used by the pre-existing dashboard summary).
Stage/curriculum completion also triggers achievement recording (§8) automatically, from inside
`POST /api/progress/record` — no separate "check for completion" step the student or frontend
has to remember to call.
## 5 & 6. Assessment model and exam engine
`PRACTICE` is just... practice: the existing curriculum flow, hints freely available (subject to
the scenario actually having any), unrestricted retries, no timer, no session concept.
`ASSESSMENT` (exam or capstone — same underlying engine, §7 covers what makes a capstone
different) is a **definition** (the reusable template: pool, duration, hint policy, completion
rule) producing **sessions** (one per student attempt, immutable snapshot of what was drawn and
under what rules). New tables (migration `013`):
```
assessment_definitions slug PK, type ('exam'|'capstone'), title, duration_minutes,
hint_policy ('free'|'disabled'|'flagged'), pool_spec JSONB,
completion_rule JSONB, unlock_rule JSONB (capstone only),
version INTEGER, status ('active'|'inactive')
assessment_cohort_assignments which cohorts a definition is available to
assessment_sessions one per student attempt — snapshots type/title/duration_minutes/
hint_policy from the definition at draw time, plus
definition_version, started_at, server-computed expires_at,
submitted_at, random_seed, result JSONB
assessment_session_scenarios the drawn set, ordered, with per-scenario status + a real FK
back to the scenario_attempts row once worked
```
`exam_sessions` (Phase 1/2B, historical-import-only from the old Lab-local SQLite exam
bookkeeping) is **left untouched** — it was already scoped as dead historical data, never the
live mechanism, so there was nothing to migrate into the tables above without inventing data
that was never real. This is a deliberate non-duplication: the assessment tables above *are*
the real thing Phase 2B's note said didn't exist yet.
`pool_spec` shape:
```json
{ "required": ["scenario-id", ...], "random": [{ "count": 1, "from": ["a", "b", "c"] }, ...] }
```
`platform/api/src/assessments.js`'s `drawScenarios` includes every `required` id, plus `count`
random picks from each `random` group, shuffled together into presentation order. The draw uses
a tiny deterministic PRNG (mulberry32, seeded from `crypto.randomBytes` — no new dependency)
purely so the seed can be recorded for audit; a student never sees the seed or the undrawn pool
before starting.
**One assessment at a time per student** — `startSession` refuses if the student already has a
`not_started`/`in_progress` session for *any* definition. This keeps "which scenario is this
validate call happening inside" unambiguous for hint-policy enforcement (§19) without needing
the client to declare a mode.
**Timer**: `expires_at` is computed once, server-side, at `started_at + duration_minutes`, and
never recomputed from anything the client sends. Expiry is checked **lazily** — whenever a
session is read or written, if `now() > expires_at` and it's still `in_progress`, it flips to
`expired` right then. No background cron (explicitly out of scope this phase — see the
non-goals list). Verified live: a 1-minute test definition correctly transitioned to `expired`
on the first request after the minute passed, and unblocked starting a new assessment.
## 7. RootNode Capstone
A capstone is an `assessment_definitions` row with `type='capstone'` — same engine as an exam,
different framing and unlock gate. The shipped default (seeded in migration `013`, idempotent
`INSERT ... ON CONFLICT DO NOTHING`) is `rootnode-capstone`:
- **Required** (always drawn): all four RootNode Real Ops scenarios
(`rootnode-runaway-monitor`, `broken-deployment`, `rootnode-probe-port-mismatch`,
`rootnode-enforce-limits`) — "feature prominently," per the spec, satisfied by making them
non-optional rather than just likely.
- **Random**: one Workloads task, one Configuration task, one Troubleshooting task — task-type
scenarios preferred over MCQ, since the point is operational ability, not recall.
- 90 minutes, `hint_policy: 'flagged'` (available, but every use is recorded — appropriate for
a capstone that should mostly reflect independent work without being punitive about needing a
nudge).
- `unlock_rule`: every non-capstone curriculum stage complete (Foundation through Real Ops).
`GET /api/capstone` is a student-friendly view combining `computeCurriculumState`'s capstone
block with the definition and the student's own session history —
`locked` → `unlocked` → `completed`, with a prerequisite checklist while locked. Verified live,
end to end: locked with all six blocking stage ids, unlocked to zero blocking ids the moment the
last stage completed, a real draw (7 scenarios, all four Real Ops present), completion of all
seven finalizing the session and recording a `capstone_completed` achievement.
## 8. Achievement / completion records
`achievements` (migration `014`): `student_id`, `achievement_type`
(`stage_completed`/`curriculum_completed`/`assessment_completed`/`capstone_completed`),
`curriculum_path_id`/`curriculum_version` (for stage/curriculum types),
`assessment_session_id` (for assessment/capstone types), `achieved_at`, `metadata`.
`recordAchievement` is idempotent via a `WHERE NOT EXISTS` guard (not a unique-index/
`ON CONFLICT` dance — simple and correct at this write volume) — calling it again for something
already recorded is a safe no-op. Triggered automatically: `checkStageAndCurriculumAchievements`
runs inside `POST /api/progress/record` after every pass; `recordAssessmentAchievement` runs
after `assessments.recordScenarioOutcome`/`submitSession` finalizes a session, and only on an
actual pass (defensive — today's only `completion_rule` is `all_scenarios_passed`, which already
guarantees a `completed` session passed, but a future threshold-based rule shouldn't silently
grant achievements for a partial result).
**Not a certificate.** No public/shareable artifact is generated — this is explicitly a Phase
4/5 concern (see the non-goals list); a certificate can be rendered from these records later
without needing new data.
## 9. Assessment/exam Lab behavior
An assessment doesn't provision a separate Lab — the student works through their drawn
scenarios inside the **same** disposable environment their curriculum work already uses. This
is what makes "Lab reset doesn't erase assessment state" nearly free: assessment state
(`assessment_sessions`, `assessment_session_scenarios`, `scenario_attempts`) lives entirely in
Platform Postgres; the Lab environment is purely compute. Resetting/recreating it changes the
container, never the rows. Verified live: reset an environment mid-assessment,
`expires_at`/`status` on the session were unchanged afterward.
**Assessment-mode detection is entirely server-derived** — `findActiveAssessmentContextForScenario`
looks up whether the scenario a `validate`/`hint` call is happening against is currently in the
student's own active session's drawn set. Nothing the client sends declares "I'm in assessment
mode"; the server's own `assessment_session_scenarios` table is the only source of truth (see
§19).
If Lab infrastructure fails mid-assessment, the assessment session itself is unaffected — an
instructor can reset (or, if truly needed, destroy/reprovision) the student's Lab via the
existing admin environment controls (Phase 2B) without touching assessment state at all; there
is no code path that resets an assessment as a side effect of a Lab operation.
## 10 & 11. Student UI and Lab UX
`frontend/src/platform/StudentDashboard.jsx` is now tabbed: **My Learning** (the original
Phase 2A/2B "Up Next" + Launch/Reset Lab flow, extracted unchanged into `MyLearningTab.jsx` and
extended with an overall percent-complete bar from the new rule-driven curriculum state),
**Practice** (scenario/stage status + recent attempt history), **Assessments** (available/
active-with-live-countdown/history), **Capstone** (locked checklist → unlocked → start/resume →
completed, with attempt history), **Achievements** (a flat, dated list — not a leaderboard).
Inside the Lab UI itself (`frontend/src/components/ScenarioPanel.jsx`, `App.jsx`):
`ProgressiveHints` replaces the static hint array render when the Gateway has stripped hints
(§3); an `AssessmentBanner` (new, top-level, polls `/api/assessments/active`) shows a live
countdown and per-scenario completion count whenever an assessment is running — distinct from,
and rendered alongside, the pre-existing Lab-local exam-mode UI (`ExamTimer`/`ExamSession` etc.,
upstream KubeKosh feature, untouched). Both render nothing in standalone/single-container dev
mode (no Platform API to poll there), so neither regresses that deployment path.
## 12. Instructor learning analytics
`/admin` → **Learning**: cohort summary (student count, 7-day active-learner count, average
completion, per-stage completion counts), a **Needs Attention** list, and a per-student
drill-down (curriculum state, full attempt history with validation/hint counts, assessment
history). `/admin` → **Assessments**: create/edit definitions (admin/owner), assign to a cohort,
view/cancel sessions — visible to every instructor role, cohort-scoped server-side for the plain
`instructor` role the same way Phase 3A's other cohort-scoped routes are.
`GET /api/admin/scenario-analytics` (optional `?cohort_id=`): completions, distinct students,
average validation attempts, total failures, total hints used, average completion time — one
`GROUP BY scenario_id` query over `scenario_attempts`, nothing pre-aggregated or cached.
## 13. "Needs Attention"
`GET /api/admin/cohorts/:id/needs-attention` computes four signals per student, on read (no
stored table, no background job): **Repeated Attempts** (an open attempt with ≥3 failed
validations), **May Need Help** (an open attempt where a hint has been used but the scenario
isn't done), **Needs Attention — inactive** (no activity in 7 days), and **Needs Attention —
incomplete assessment** (their most recent assessment session expired without completing). All
four thresholds are named constants in `admin-learning-routes.js`
(`NEEDS_ATTENTION_FAIL_THRESHOLD`, `NEEDS_ATTENTION_STALE_DAYS`), not hardcoded inline. Wording
is deliberately instructional (`Needs Attention`, `May Need Help`, `Repeated Attempts`) — an
explicit test asserts none of the returned labels match punitive language.
## 14 & 15. Assessment/capstone management (admin)
A structured form (`frontend/src/admin/AssessmentsView.jsx`) — id, title, type, duration, hint
policy, and a raw JSON `pool_spec` textarea. Not a drag-and-drop authoring tool, per the explicit
scope line; the JSON textarea is honest about what's actually being edited rather than hiding it
behind partial UI that can't express every pool shape. Editing a definition's
duration/hint_policy/pool_spec/completion_rule bumps its `version` automatically
(`admin-learning-routes.js` diffs the incoming fields against the current row) — cosmetic edits
(title, description) don't. Create/edit is admin/owner only server-side; assign-to-cohort and
session view/cancel are available to a scoped `instructor` too, matching the RBAC matrix (§18).
## 16. Versioning
Deliberately minimal, per the explicit "don't overbuild full CMS versioning" instruction:
- **Curriculum path**: a plain `"version": 1` integer in `rootnode-path.json` itself, recorded
on `stage_completed`/`curriculum_completed` achievements as a snapshot.
- **Assessment definitions**: a plain incrementing `version` integer, bumped only on a semantic
change (duration/hint_policy/pool_spec/completion_rule — anything that would change what a
session is graded against). `assessment_sessions.definition_version` snapshots which version
was live at draw time, alongside a full redundant snapshot of the fields themselves
(title/duration/hint_policy) — so a later definition edit, or even deletion, can never change
what an already-started session is measured against.
No full version-history table exists for either — if that's ever needed, `version` is already a
column to build on, not a retrofit.
## 17. RBAC matrix (Phase 3B additions)
| Action | owner | admin | instructor | student |
|---|---|---|---|---|
| Create/edit assessment definitions | ✅ | ✅ | ❌ | ❌ |
| Assign an assessment to a cohort | ✅ | ✅ | ✅ (own cohorts) | ❌ |
| View/cancel assessment sessions | ✅ (any) | ✅ (any) | ✅ (own cohorts) | ❌ |
| View cohort learning summary / needs-attention / scenario analytics | ✅ (global) | ✅ (global) | ✅ (own cohorts) | ❌ |
| View a student's learning detail | ✅ | ✅ | ✅ (if enrolled in an owned cohort) | own only |
| Start/work an assessment, view own attempts/hints/achievements | ❌ | ❌ | ❌ | ✅ (own only) |
Enforced entirely server-side, reusing Phase 3A's `rbac.js` (`requireAdminOrOwner`,
`requireAnyInstructor`, `requireCohortAccess`, `requireStudentCohortAccess`) — no new
authorization primitives were needed, only new routes wired through the existing ones.
## 18. Security / integrity
- **Timer is server-authoritative** (§6) — `expires_at` fixed at draw time, never accepts a
client value, checked lazily on every touch.
- **Scenario draw is server-stored** — `assessment_session_scenarios` is the only source of
truth for what a student is supposed to be working on; nothing the client sends can add,
remove, or reorder it.
- **A student cannot submit/read another student's session** — every assessment route filters
by `student_id = req.studentId` server-side; a foreign session id returns 404, not 403 (same
"don't confirm existence" posture as Phase 3A's RBAC denials) — verified by an explicit test.
- **Gateway ownership** (Phase 2A/2B/3A) is unchanged and still enforced on every proxied Lab
request, HTTP and WebSocket, including the new `/start` and `/hint` relays.
- **Lab recreation never resets attempt/exam state** (§9) — verified live.
- **Hidden solution/pool data**: undrawn pool alternatives are never returned to the student
(only the actual draw); hints beyond the currently-unlocked level are never sent (§3, the
actual mechanism, not just a UI convention).
- **No invasive anti-cheat** — no browser surveillance, no lockdown mode. Explicitly out of
scope, per the non-goals list.
## 19. Dev/test commands
```bash
cd platform && docker compose -f docker-compose.dev.yml up -d
(cd api && PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5433/rootnode_platform \
node bootstrap-owner.js [email protected] "Your Name" "a-real-password-12-chars-plus")
OWNER_EMAIL=[email protected] OWNER_PASSWORD=... node test/e2e-phase2a.js
OWNER_EMAIL=[email protected] OWNER_PASSWORD=... node test/e2e-phase2b.js
OWNER_EMAIL=[email protected] OWNER_PASSWORD=... node test/e2e-phase3a.js
OWNER_EMAIL=[email protected] OWNER_PASSWORD=... node test/e2e-phase3b.js
```
`e2e-phase3b.js` additionally shells out to `docker compose restart api` for its restart-
persistence case (set `SKIP_RESTART_TEST=1` to skip that one case if running somewhere that
isn't safe to restart).
## 20. Known limitations
- **No automatic "abandoned" detection** — an attempt a student never returns to stays
`in_progress` forever; there's no reaper tying Lab destruction back to attempt state. A
documented, deliberate scope line (would need wiring to Provisioner `environment_events`,
more machinery than the instructional-visibility goal needs).
- **`cohort_instructors.role_in_cohort`** (Phase 3A) still isn't consulted for finer-grained
assessment permissions — any cohort assignment grants the full scoped-instructor action set.
- **Needs-attention thresholds are fixed constants**, not per-cohort configurable yet.
- **No pagination** on the audit log (Phase 3A limitation, unchanged) or on scenario analytics.
- **Assessment pool_spec authoring is raw JSON** in the admin UI — correct and complete, but not
a guided form; a validation-assist UI is a natural but unbuilt next step.
- **One assessment at a time per student** — intentional (§6), but means a student can't have a
capstone and an exam "in flight" simultaneously even if that were ever desired.
## 21. Phase 4 boundary
Everything in this document runs on a single dev machine against
`platform/docker-compose.dev.yml`. **Not built, and explicitly out of scope this phase:**
production Hetzner/Proxmox deployment, production OIDC provider integration, a parent portal,
payments, public signup, automated email, equipment inventory, a multi-host scheduler,
Grafana/Prometheus production dashboards, public certificates, a marketplace/self-paced commerce
layer, the six held KubeKosh scenarios, invasive proctoring, and AI-driven grading. All of that
stays exactly where the Phase 3A doc's non-goals list and this document's §21 leave it — Phase 4
territory.
+254
View File
@@ -0,0 +1,254 @@
# Phase 4.1 — Pre-Production Hardening
**Status: PHASE 4.1 COMPLETE. PRE-PRODUCTION HARDENING COMPLETE. READY FOR HARDWARE DEPLOYMENT.**
Phase 4 built the production infrastructure package with the real Hetzner/Proxmox hardware not
yet purchased, and flagged one genuine architectural gap: the Gateway's mechanism for reaching
a student Lab assumed Gateway and Lab shared a Docker daemon, which is false the moment Labs run
on a genuinely separate `rn-lab-host-01` VM. Phase 4.1's job was to close that gap and harden
everything else discovered along the way, still without the real hardware.
## 1. The core fix: a Lab Host Router
**Before:** Gateway → `http://${env.internal_ip}:${env.internal_port}` directly — a Docker
bridge-local address, unreachable across a real VM boundary.
**After:** Gateway → (mode-aware) → either `direct` (unchanged, dev/single-host) or `router`
(production): a small new service, `platform/lab-router/`, runs on the Lab Host itself,
authenticates the Gateway via mTLS (the same private CA the Docker TLS control plane already
used, extended with two new leaf certs), resolves an opaque routing token to a local container
purely from `docker ps` label state (no Postgres access, no call back to the Provisioner), and
reverse-proxies HTTP and WebSocket traffic to it.
The decision of which mode a given `lab_host` uses is data-driven
(`lab_hosts.routing_mode`/`router_endpoint`, `platform/provisioner/migrations/004_lab_hosts_routing.sql`),
resolved once per request in `platform/gateway/src/lab-endpoint.js`'s `labTarget()` — the only
place in the Gateway that knows this distinction exists. Every proxy/fetch call site in
`platform/gateway/src/server.js` goes through it.
**Exact `env.internal_ip` replacement:** `labTarget(env, token)` returns `{ url, headers, agent,
mode }`. In `direct` mode, `url` is exactly the old `http://internal_ip:internal_port`. In
`router` mode, `url` is the lab_host's `router_endpoint`, `headers` carries
`X-RootNode-Route-Token: <token>` (computed server-side from the Provisioner-verified
environment row — never trusted from the browser), and `agent` is an `https.Agent` configured
with the Gateway's mTLS client identity.
## 2. No per-student host port
Deliberately not solved by publishing a host port per Lab (the originally-flagged, rejected
approach). The Router resolves entirely from Docker's own label state on its own host — no
port-per-student anything, no dynamic firewall rule per Lab.
## 3. Lab Host Router — design
See `platform/lab-router/src/server.js`'s header comment for the full "narrow by design" list:
one route (proxy-by-token), requires mTLS (no bearer-token fallback), never exposes Docker API/
container IDs/network info to its caller, never provides student-to-student routing (each
request resolves to exactly the one container matching its own token).
## 4. Multi-Lab-Host ready
Routing is resolved per `lab_host_id` — adding `rn-lab-host-02` later is one new `lab_hosts` row
(its own `routing_mode`/`router_endpoint`/TLS material), no Gateway code change. No scheduler
was built (explicitly out of scope) — the Provisioner still picks "the one enabled lab_host"
(`selectLabHost`), unchanged from Phase 1.
## 5. Local development compatibility
`dev-local` (the Unix-socket lab_host used by `docker-compose.dev.yml`) stays `routing_mode:
direct` — zero behavior change for the T480/any local dev setup, confirmed by the full existing
regression suite passing unmodified (§29 below) and by every new drill in this phase running
against the same dev stack. The standalone single-container Lab image build/boot is unaffected
(it never touches the Gateway or Provisioner at all).
## 6-9. Simulation, HTTP/WS validation, security tests, routing token security
`infra/scripts/simulate-two-host/` — a real two-Docker-network simulation (not mocks): Gateway
attached ONLY to an isolated bridge with no route to any Lab's bridge network, Lab Host Router
host-networked (mirrors `egress-proxy`'s existing pattern) so it can reach whatever per-Lab
bridge the Provisioner creates. 13/13 real assertions pass, including:
- Gateway genuinely cannot open a TCP connection to a Lab's Docker-internal IP (proven from
inside the Gateway container's own network namespace via a raw connect attempt).
- Gateway reaches the Lab correctly through the Router — HTTP AND a real WebSocket terminal
session running a genuine `kubectl get nodes` inside the Lab.
- Alice/Bob isolation (own lab ALLOW, other's DENY — HTTP and WebSocket both), garbage tokens,
anonymous requests, and destroyed-environment tokens are all rejected through the real code
path.
- The Router refuses any connection with no client certificate — mTLS is the real
authentication mechanism.
Routing token security reviewed: already CSPRNG since Phase 1 (`generateRoutingToken`), never
logged in plaintext (the Router's `tokenFingerprint` truncates), redacted in every API response
except the two narrow internal reads that legitimately need it, and — critically — the header
the Router trusts is computed server-side from the Provisioner-verified environment row, never
read from the browser's own request, so a browser cannot forge routing by supplying its own
`X-RootNode-Route-Token`.
## 10. Control plane vs data plane
Kept strictly separate — see `docs/PRODUCTION_NETWORK_SECURITY.md`'s dedicated section. Student
browser traffic is never proxied through the Docker API at any point.
## 11. Firewall update
`infra/firewall/rn-lab-host-01.rules`'s old speculative wide port-range allowance replaced with
one narrow rule: `platform_ip` → the Router's port only (`8443` default, configurable).
## 12. Self-paced / remote delivery architecture check
`docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`. Short version: student-driven provisioning,
cohort-less students, and "idle reaper destroys → progress survives → student returns → fresh
Lab → curriculum continues" all already worked before this phase and are now proven end-to-end
by two new drills. The one real gap for actual remote/self-paced delivery is student
authentication (today's bare `student_id` login assumes an instructor-controlled roster) — not
built here, explicitly Phase 5 scope.
## 13. Abandoned attempt cleanup
`scenario_attempts.status = 'abandoned'` (already a valid value since Phase 3B, never
previously set) now gets set on: environment destroy/reset (student- or instructor-triggered),
logout, session expiry (piggybacked on the existing hourly session sweeper — no new background
job), and switching to a different scenario while one is still open. Never a punitive marker —
`result` stays NULL, history is preserved, instructors see an honest "abandoned (reason)"
instead of a phantom "still in progress."
## 14. Gateway session cache
Reviewed — already a tight 2-second TTL since Phase 3A, specifically to bound the
revocation-propagation window. Decision: **keep it, don't add push-based invalidation** — the
added coupling (Platform API would need to know how to reach the Gateway) isn't justified by
shrinking an already-2-second window further. Documented reasoning, not a silently-accepted gap.
## 15. Rate limiting
Login rate limiting moved from an in-memory `Map` (reset on every restart, not shareable across
instances) to Postgres (`login_attempts` table, `migrations/016_login_attempts.sql`) —
`platform/api/src/security.js`'s `tooManyAttempts()`. No Redis added. Restart behavior
confirmed for real: `platform/test/e2e-phase3a.js`'s rate-limiting test passes after a genuine
Platform API restart in the same run.
## 16. Pagination
Real server-side pagination (`platform/api/src/pagination.js`) on: the audit log (was a flat
`LIMIT 200`), student attempt history (was `LIMIT 100`), and assessment session history (was
**fully unbounded** — the one genuine growth risk found). Scenario analytics was deliberately
left as-is — grouped by scenario id, naturally bounded around ~100 rows, no real growth risk to
paginate against. Frontend (`AuditView.jsx`, `AssessmentsView.jsx`, `StudentsView.jsx`) updated
to match; `frontend` build confirmed clean.
## 17. Needs-attention configuration
`NEEDS_ATTENTION_FAIL_THRESHOLD`/`NEEDS_ATTENTION_STALE_DAYS` moved from hardcoded constants to
env-var-configurable (matching the existing session-TTL pattern), sensible defaults unchanged.
## 18. Assessment authoring UX
Raw-JSON `pool_spec` textarea replaced with a structured editor
(`frontend/src/admin/AssessmentsView.jsx`'s `PoolSpecEditor`): required scenarios and random
pools picked from a real scenario catalog (`GET /api/admin/scenario-catalog`, new), grouped by
curriculum stage, via native multi-select — no new UI framework. An owner-only "advanced" raw
JSON view remains for anything the structured editor doesn't cover.
## 19. Cohort role-in-cohort
`cohort_instructors.role_in_cohort` existed since Phase 2B but was never read. Implemented
minimally: `teacher` (unchanged full capability) vs `viewer` (read-only — dashboards/analytics
still visible, but Prepare/Destroy Labs and provisioning a student's environment are now 403'd
server-side, `rbac.js`'s `requireCohortWriteAccess`/`requireStudentCohortWriteAccess`).
## 20. Backup / restore destructive drill
`infra/backup/test-destructive-restore-drill.sh` — real owner/admin/cohort/3-students/progress/
attempts/assessment/achievements seeded through the real Platform API, encrypted-backed-up, the
disposable Postgres container **genuinely destroyed** (not just restored into a second
container), a fresh Postgres created from nothing, restored, and the real Platform API booted
again against the restored database and used to log in as the restored admin over HTTP. All
data verified present; never touches the developer's real database.
## 21. Lab Host rebuild drill
`infra/scripts/drills/test-lab-host-loss.js` — a student's Lab container is destroyed out from
under the Provisioner (no clean destroy call, simulating total Lab Host loss). Proves: Platform
stays up throughout, the stale environment is never presented as healthy once the Provisioner's
real `reconcileOnStartup` reconciliation runs, academic progress is untouched, and the student
can launch a genuinely new Lab and continue.
## 22. Deployment rollback test
`infra/scripts/drills/test-deployment-rollback.sh` — builds a known-good and a deliberately
broken gateway image, "deploys" each via the real deterministic-tag mechanism, proves the smoke
test catches the regression, computes the rollback target with the exact same
`deploy-history.log` logic `rollback-production.sh` uses, and proves the rollback recovers.
Documents explicitly that image rollback never implies a DB migration rollback.
## 23. CI/CD failure tests
`infra/scripts/drills/test-ci-fails-closed.sh` — proves all six required failure modes (test/
syntax failure, frontend build failure, image build failure, config validation failure,
migration preflight failure, staging smoke test failure) genuinely exit nonzero. **Found and
fixed two real pre-existing bugs**: `staging-smoke-test.sh` and `production-smoke-test.sh` both
had a classic `set -e` gotcha (`curl ... && echo "OK"` as one statement — a failing command
that isn't the last in an `&&` chain doesn't trigger `-e`) that let an unreachable/broken
endpoint fall through silently instead of failing the smoke test. Fixed in both files.
## 24. Security scanning
`infra/ci/gitea-workflows/build-and-test.yml`'s new `security-scan` job — npm audit, gitleaks
(secrets), hadolint (Dockerfiles), shellcheck (shell scripts), trivy (container image CVEs).
All free/open-source. Severity policy documented in `docs/SECURITY_SCANNING.md`: CRITICAL/HIGH
blocks the build, MEDIUM/LOW is reported only. `security-scan` gates `build-images` the same
way `regression` does.
## 25. Version / release visibility
`GET /api/version` (Platform API) — app version (`package.json`), deployed Git SHA
(`GIT_SHA` env var, set to the same value as the image tag in the production compose file), and
the most recently applied migration as a schema-version stand-in. Unauthenticated, minimal, no
infrastructure detail.
## 26. Admission control failure mode
`resource-policy.yaml`'s new (disabled-by-default) `admission_control.max_concurrent_labs` —
when set, the Provisioner checks it BEFORE touching Docker at all, so a full host fails cleanly
(503, `CAPACITY_EXCEEDED`) instead of failing mid-provision. Relayed by the Platform API as a
clean "Lab capacity is temporarily full. Try again shortly." — never a raw 500. A new
`GET /capacity` Provisioner endpoint feeds the instructor dashboard's capacity display.
## 27. Fake cohort dress rehearsal
`infra/scripts/drills/test-fake-cohort-rehearsal.js` — one continuous narrative: instructor
login, cohort + 4 students, curriculum assignment, Prepare Labs, students working different
scenarios, hints, a deliberate failed validation, instructor analytics, a Lab reset, a Lab
reaped (same consequence as the real idle reaper — see the drill's own comment on why the real
multi-minute timer isn't waited out), the student returning and continuing, an exam
assigned/taken, and end-of-class Lab destruction — progress verified intact after every
destructive step. 14/14 passing.
## 28. Documentation
This file, `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`, `docs/SECURITY_SCANNING.md` (new).
Updated: `docs/LAB_ARCHITECTURE.md`, `docs/PRODUCTION_NETWORK_SECURITY.md`,
`docs/PRODUCTION_DEPLOYMENT.md`, `docs/PRODUCTION_VALIDATION_CHECKLIST.md`, `docs/BACKUP_DR.md`,
`docs/ENGINEERING_SOP.md`, `README.md`, `platform/README.md`.
## 29. Regression
Re-ran the full existing suite after every application-code change in this phase:
`e2e-phase2a.js` (10/10), `e2e-phase2b.js` (12/12), `e2e-phase3a.js` (21/21 — including a real
audit-log pagination-shape fix and confirming rate-limit persistence across a genuine restart),
`e2e-phase3b.js` (24/24). Frontend production build clean. Real Lab image builds/boots (both
dev-compose and standalone single-container). `tofu validate`, `ansible-playbook
--syntax-check`, production Compose config (both platform and lab-host), `caddy validate`,
`promtool`/`amtool`, and the backup/restore mechanism all re-validated clean after the
infrastructure changes in this phase (new `lab-router` service, new firewall rule, new Ansible
tasks). No knowingly-unexplained regression was left in this codebase at the end of this phase.
## What Phase 4.1 deliberately did NOT build
Production deployment itself, Hetzner/Proxmox installation, public course signup, payments,
subscriptions, parent portal, marketing site, full LMS, video hosting, email automation,
certificates, course marketplace, mobile app, a multi-host scheduler, an AI tutor, invasive
proctoring, held scenario content, student email/password authentication (documented as the one
real gap for self-paced delivery, not built). All explicit non-goals per this phase's
authorization.
+187
View File
@@ -0,0 +1,187 @@
# Phase 4 — Production Infrastructure Package
**Status: PHASE 4 BUILD COMPLETE. PRODUCTION DEPLOYMENT PENDING HARDWARE.**
The dedicated Hetzner server this is designed for hasn't been purchased yet. Everything in this
document and the `infra/` tree it describes is buildable, testable (against local
Docker/Postgres/disposable certs), and reviewable **now** — none of it requires the real
hardware to exist. What genuinely can't happen until the hardware exists is called out
explicitly throughout, never silently assumed to already work.
This document is the map. Detail docs: `docs/PRODUCTION_DEPLOYMENT.md` (step-by-step),
`docs/PRODUCTION_NETWORK_SECURITY.md` (network/firewall/TLS design), `docs/BACKUP_DR.md`
(backup/restore/disaster recovery), `docs/OBSERVABILITY.md` (metrics/logs/alerts),
`docs/CAPACITY_PLANNING.md` (sizing model), and the `docs/RUNBOOK_*.md` files (14 topics,
command-oriented).
## 1. What was built
```
infra/
├── proxmox/ manual bare-metal install docs + cloud-init template definition
├── terraform/ VM creation (rn-platform-01, rn-lab-host-01) — real `tofu validate` pass
├── ansible/ all host configuration — real `ansible-playbook --syntax-check` pass
│ ├── roles/ common, firewall, docker-tls, platform, lab-host, monitoring-agent, backup
│ ├── group_vars/ production.yml, staging.yml, vault template
│ └── site.yml top-level playbook, correct role ordering
├── docker/ production Compose files, Caddyfile, Postgres tuning — real `docker
│ compose config` + `caddy validate` pass
├── firewall/ iptables-restore rulesets for all 3 hosts — structural lint pass
├── monitoring/ Prometheus/Alertmanager/Loki/Promtail/Grafana config — real `promtool`
│ + `amtool` validation pass
├── backup/ Postgres backup/restore/verify scripts — REAL end-to-end test against
│ disposable local Postgres, including the actual production migrations
├── scripts/ deploy/rollback/smoke-test/CI/loadtest/capacity-model tooling
├── ci/ Gitea Actions workflow (build/test/stage automatic, prod manual)
└── environments/ dev/staging/production config templates, no secrets committed
```
18 new docs (this file + 5 detail docs + 12 runbooks), 4 updated (`LAB_ARCHITECTURE.md`,
`ENGINEERING_SOP.md`, `README.md`, `platform/README.md`).
## 2. Architecture (unchanged from the design, now codified)
```
Hetzner Dedicated Server
└── Proxmox VE
├── rn-platform-01 (frontend, Platform API, Gateway, Provisioner, Postgres,
│ Prometheus/Alertmanager/Grafana/Loki, backup cron, Caddy)
└── rn-lab-host-01 (Docker Engine w/ TLS, egress proxy, disposable student Labs,
node-exporter/cadvisor/promtail)
```
The Platform VM / Lab Host VM security boundary from `docs/LAB_ARCHITECTURE.md` is the one
thing this phase was explicitly told never to weaken — it's codified, not just described, in
`infra/firewall/` and `infra/ansible/roles/docker-tls/`.
## 3. VM specifications
| | rn-platform-01 | rn-lab-host-01 |
|---|---|---|
| Baseline (default in `infra/terraform/variables.tf`) | 4 cores, 16 GB RAM, 100 GB disk | 24 cores, 96 GB RAM, 500 GB disk |
| Recommended (once real host is ~16c/32t/128GB) | same — Postgres/API/Gateway/Provisioner don't need much | majority of remaining host resources |
| Max-growth | scale RAM with Postgres `shared_buffers` if instructor/cohort count grows a lot | scale with measured concurrent-Lab capacity (`docs/CAPACITY_PLANNING.md`) |
| Configurable via | `terraform.tfvars`' `platform_vm`/`lab_host_vm` objects | same |
NUMA/CPU pinning: not configured — not useful at this scale (a handful of VMs on one physical
host, not a dozen contending for cache locality). Revisit only if real load testing shows a
reason to.
## 4. Storage decision framework
Not a final answer — `infra/environments/production/provisioner-config/README.md` and this
section exist so the final ZFS layout is a deployment-time decision made against real disks,
not guessed now. Candidate layouts (`infra/terraform/variables.tf`'s `vm_disk_datastore` names
whichever one gets chosen):
- **4× NVMe**: mirrored vdevs (RAID10-equivalent) — two mirror pairs, one for
Postgres/platform data (I/O-sensitive, small), one for Lab Host disposable data
(throughput-sensitive, large, never needs redundancy since nothing on it is precious).
- **2× SSD/NVMe**: one mirrored vdev, logical dataset separation (ZFS datasets, not separate
pools) between platform (`{{ rootnode_data_root }}/postgres`, `/backups`) and Lab Host data.
- **Mixed SSD/HDD**: only if the purchased server's actual disks require it — HDD for
disposable Lab scratch space is acceptable (it's disposable), never for Postgres.
**Never single-disk for Postgres data**, in any layout — a mirror is the floor, not a nice-to-have.
Pre-deployment disk inspection checklist (model, size, SMART, power-on hours, wear/endurance,
sector errors, temperature, firmware): `docs/RUNBOOK_DISK_FAILURE.md`'s inspection section —
run once, against the real drives, before committing to a layout.
## 5-8. Network, firewall, Docker TLS, public ingress
Full detail: `docs/PRODUCTION_NETWORK_SECURITY.md`. Summary: two private VM networks, Docker TLS
(mutual, already application-supported since Phase 1 — this phase only needed the certificate
generation/deployment automation, not new app code), a single public origin (Caddy, automatic
HTTPS) proxying only to the Gateway, and one genuinely new architectural item this phase
surfaced and addressed at the network-policy layer: **Gateway-to-Lab reachability across two
real VMs** — see that doc's dedicated section for what's built vs. what still needs a small,
reviewed Provisioner/Gateway code change before real go-live.
## 9-10. Production auth/cookie config, secrets
`infra/environments/production/*.env.template` set `COOKIE_SECURE=1`/`TRUST_PROXY=1` explicitly
(dev's unsafe defaults are never inherited). Secrets: Ansible Vault (already bundled with
Ansible — no new tool to install/trust for this project's secret surface), template at
`infra/ansible/group_vars/production.vault.yml.template`. Nothing is committed —
`infra/.gitignore` covers `*.env`, TLS private key material, vault plaintext, Terraform state.
## 11-13. Postgres, backup, DR
Full detail: `docs/BACKUP_DR.md`. Summary: conservative tuning baseline (not hand-tuned before
real RAM is known), daily encrypted backups with an off-host adapter point (Storage Box/S3/any
SSH target), and a **real, tested** backup→encrypt→restore→verify cycle
(`infra/backup/test-backup-restore.sh`) — run against disposable local Postgres with the actual
production migrations applied, not a mock.
## 14-17. Observability, logging, alerting, uptime
Full detail: `docs/OBSERVABILITY.md`. Prometheus + Grafana + Loki/Promtail (the user's stated
Prometheus/Grafana preference, honored) + Alertmanager, all self-hosted, no paid dependency.
Deliberately does not duplicate the application's own instructor learning analytics
(Phase 3B) — operational observability and learning analytics stay separate concerns with
separate audiences, per the explicit instruction. Application-level metrics (active
students/Labs, provisioning failures, Gateway error rate) are a documented, **not-yet-built**
follow-up — see that doc's honest accounting of what's real today (host/container/Postgres
metrics, HTTP healthz probes) vs. what needs app instrumentation later.
## 18-19. CI/CD, image strategy
`infra/ci/gitea-workflows/build-and-test.yml` — automatic test/build/tag/staging-deploy,
**manual** production promotion (`infra/scripts/deploy-production.sh`), matching the stated
preference. SHA-tagged images always, semantic tags on deliberate releases, never `:latest` in
staging/production. Full detail: `infra/docker/IMAGE_STRATEGY.md`.
## 20-21. Ansible, Terraform
`infra/ansible/site.yml` — idempotent roles, doesn't assume the VMs already exist (Terraform
creates them first). Real `ansible-playbook --syntax-check` pass. `infra/terraform/` — real
`tofu validate` pass against the actual `bpg/proxmox` provider schema. Neither installs Proxmox
itself or builds the base VM template — both stay deliberate manual, documented, one-time steps
(`infra/proxmox/README.md`), per the explicit "don't over-automate the fragile one-time step"
instruction.
## 22-23. Environment config, staging
`infra/environments/{dev,staging,production}/` — explicit separation, production never
inherits dev defaults. Staging is designed to run in the homelab or a resource-constrained
slice of the real server later — no second dedicated server required.
## 24-25. Load testing, capacity model
`infra/scripts/loadtest/` (real Prepare-Labs-driven boot test + `docker stats` sampling) and
`infra/scripts/capacity-model.js` (a real, working worksheet — tested against example inputs,
produces CPU-vs-memory-bound recommended/warning/hard-admission-control thresholds). No
concurrency number is hard-coded as a marketing figure; the example config in
`infra/environments/production/capacity.yml.example` is explicitly flagged as placeholder data.
## 26-27. Validation checklist, runbooks
`docs/PRODUCTION_VALIDATION_CHECKLIST.md` — one master checklist spanning hardware through
academy-workflow verification. 12 runbooks (`docs/RUNBOOK_*.md`) — first deployment, normal
deploy, rollback, Postgres restore, VM rebuild (both hosts), Docker TLS rotation, TLS renewal,
owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host
resource pressure.
## What this phase deliberately did NOT touch
Learning system design, curriculum content, RBAC model, session architecture — all Phase 3A/3B,
unchanged. No parent portal, billing, held scenarios, or production OIDC provider integration.
RootNode production infrastructure is dedicated to RootNode Academy — never folded into
existing Synthos production servers, and canonical source stays Gitea, never GitHub `kubekosh`.
## Known gap surfaced during this phase (not silently papered over) — RESOLVED in Phase 4.1
Building the real production Compose/network config surfaced that the Gateway's existing
mechanism for reaching a Lab container (`env.internal_ip`, a Docker-bridge-local address) only
works when Gateway and Lab share a Docker daemon — true in every dev/single-host setup so far,
**not** true once Labs run on a genuinely separate `rn-lab-host-01`. At the time this phase
closed, the network-policy half was built but the application half was flagged as an explicit,
not-yet-built follow-up rather than guessed at mid-infra-phase.
**Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1) built and proved the application
half** — a Lab Host Router (`platform/lab-router/`) plus a `labTarget()` routing abstraction in
the Gateway, real-tested against a genuine two-network Docker simulation
(`infra/scripts/simulate-two-host/`), not just designed. See
`docs/PRODUCTION_NETWORK_SECURITY.md`'s "Gateway-to-Lab reachability" section for the full
resolution. This item no longer blocks real two-VM go-live.
+90
View File
@@ -0,0 +1,90 @@
# Production Deployment
Step-by-step, for when the real Hetzner host exists. See
`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` for the architecture this executes.
## 0. One-time, manual, not automated (deliberately)
1. Install Proxmox VE on the bare-metal host — `infra/proxmox/README.md`.
2. Inspect the real disks, pick a ZFS layout — `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4,
`docs/RUNBOOK_DISK_FAILURE.md`'s inspection checklist.
3. Build the cloud-init VM template Terraform will clone —
`infra/proxmox/cloud-init/user-data.yaml` + `infra/proxmox/README.md`.
4. Fill in `infra/terraform/terraform.tfvars` (copy from `.tfvars.example`) with the real
Proxmox API endpoint, node name, storage/datastore names, and chosen VM sizes.
5. Fill in `infra/ansible/inventory/production.ini` with the real VM IPs, and
`infra/ansible/group_vars/production.yml` with `platform_ip`, `lab_host_ip`,
`management_cidr`, `proxmox_mgmt_cidr`.
6. Copy `infra/ansible/group_vars/production.vault.yml.template` →
`production.vault.yml`, fill in every `vault_*` secret with real generated values, then
`ansible-vault encrypt group_vars/production.vault.yml`. Never commit the plaintext.
## 1. Create the VMs
```bash
cd infra/terraform
tofu init
tofu plan # review before applying against real hardware
tofu apply
```
Creates `rn-platform-01` and `rn-lab-host-01` from the cloud-init template. Confirm both boot
and are SSH-reachable from the management network before continuing.
## 2. Generate TLS certificates (offline, once)
```bash
export ROOTNODE_TLS_BUNDLE_DIR=/secure/offline/path # never inside the repo
./infra/scripts/docker-tls/generate-docker-tls.sh
./infra/scripts/docker-tls/verify-docker-tls.sh
```
Generates FOUR leaf certs under one CA (Phase 4.1 extended this from two): the CONTROL-plane
pair (`server-*`/`provisioner-client-*`, for Provisioner↔dockerd) and the DATA-plane pair
(`router-server-*`/`gateway-client-*`, for Gateway↔Lab Host Router — see
`docs/PRODUCTION_NETWORK_SECURITY.md`'s "control plane vs data plane" section). The CA private
key stays in `ROOTNODE_TLS_BUNDLE_DIR` and never touches either VM — only the CA cert (public)
and each host's own leaf cert+key go to each host, both pairs deployed by the same `docker-tls`
Ansible role into two separate directories (`/etc/rootnode/docker-tls/` and
`/etc/rootnode/lab-router-tls/`).
## 3. Configure both hosts
```bash
cd infra/ansible
ansible-playbook -i inventory/production.ini site.yml --ask-vault-pass
```
Role order matters and is fixed in `site.yml`: `common` → `firewall` → `docker-tls` →
`platform`/`lab-host` → `monitoring-agent` → `backup`. Firewall and TLS land before anything
that depends on them.
## 4. First application deploy
```bash
./infra/scripts/deploy-production.sh <image-tag>
```
Pulls the tagged images, runs the compose stack, applies Postgres migrations (automatic, on
Platform API/Provisioner startup — see `docs/RUNBOOK_FAILED_MIGRATION.md` if one fails).
## 5. Verify
```bash
./infra/scripts/production-smoke-test.sh
```
Curls `/healthz` on the Gateway and Platform API, confirms Postgres is reachable, confirms
Prometheus/Grafana/Alertmanager are up. Then walk
`docs/PRODUCTION_VALIDATION_CHECKLIST.md` in full before calling the deployment live.
## Normal (non-first) deploys
`docs/RUNBOOK_NORMAL_DEPLOY.md`. Rollback: `docs/RUNBOOK_ROLLBACK.md`.
## What this document does NOT cover
DNS/Cloudflare setup for the public origin (a one-time manual step outside this repo's scope —
point the A/AAAA record at the real host and confirm Caddy issues its certificate). Which
storage layout to pick (decision framework only, §4 of the master doc — the real answer depends
on the real disks).
+130
View File
@@ -0,0 +1,130 @@
# Production Network & Security
The Platform VM / Lab Host VM boundary is the core production control. This document is where
every network decision that touches it gets written down. Nothing here weakens it.
## Topology
```
Internet ──HTTPS(443)──▶ Caddy (rn-platform-01) ──▶ Gateway (internal bridge)
│
management_cidr ──SSH──▶ rn-platform-01 │
management_cidr ──SSH──▶ rn-lab-host-01 │
proxmox_mgmt_cidr ──22/8006──▶ Proxmox host │
│
rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd) [CONTROL plane]
rn-platform-01 (Gateway) ──mTLS:8443──▶ rn-lab-host-01 (Lab Host Router) [DATA plane]
rn-lab-host-01 (promtail) ──3100──▶ rn-platform-01 (Loki)
rn-lab-host-01 (node-exporter/cadvisor) ◀──scrape── rn-platform-01 (Prometheus)
```
Firewall source of truth: `infra/firewall/{proxmox-host,rn-platform-01,rn-lab-host-01}.rules`
(iptables-restore format, deployed by the `firewall` Ansible role). Validate with
`infra/firewall/validate-firewall.sh`.
## What is explicitly denied
- Student Lab → another student Lab (no student-to-student traffic, ever).
- Student Lab → Proxmox management (both the host firewall and the `lab_subnet_range` DENY
rule in `rn-lab-host-01.rules`).
- Student Lab → Platform Postgres / Platform API / Gateway internals (logged DENY in
`rn-platform-01.rules` for anything from `lab_host_ip` to ports 5432/4100).
- rn-lab-host-01 → Postgres (same rule — the Lab Host itself has no legitimate reason to reach
Postgres; only the Provisioner on rn-platform-01 does, and it talks to Postgres over the
internal Docker bridge, never across the VM boundary).
- Any student egress except through the approved egress proxy (enforced by the Provisioner's
existing per-Lab `DOCKER-USER` chain manipulation from Phase 1 — unchanged by this phase).
- Direct public exposure of the Provisioner, Postgres, Docker TLS port, Proxmox UI, or Grafana
(Grafana is reachable only via Caddy's `/grafana/*` path, itself restricted to
`management_cidr` by a `caddy` matcher — see `infra/docker/platform/Caddyfile.j2`).
## Control plane vs data plane (Phase 4.1 §10)
Two separate, never-conflated mTLS connections between the two VMs, deliberately kept apart
both conceptually and operationally:
| | Control plane | Data plane |
|---|---|---|
| What | Provisioner manages Lab containers | Gateway serves student browser traffic |
| Path | Provisioner → dockerd (port 2376) | Gateway → Lab Host Router (port 8443) |
| Who initiates | Provisioner only | Gateway only |
| What it can do | Create/start/stop/remove containers, full Docker API | Reverse-proxy one HTTP/WS request to one already-running container by opaque token — nothing else |
| Certs | `provisioner-client-*` / `server-*` | `gateway-client-*` / `router-server-*` |
| Exposes Docker API to caller? | Yes (that's its job — Provisioner IS a Docker client) | **Never** — the Router never exposes Docker API, container IDs, or network info to its caller |
The student browser's traffic is proxied at the Gateway/Router layer and never touches the
Docker API at any point — the Router deliberately has no route that echoes back anything
Docker-internal (see `platform/lab-router/src/server.js`'s header comment for the full "narrow
by design" list).
## Docker TLS (Provisioner ↔ Lab Host)
Mutual TLS on port 2376, restricted at the firewall to `platform_ip` only. Application support
already existed since Phase 1 (`platform/provisioner/src/docker.js`'s `buildClient()`); this
phase added the operational half: `infra/scripts/docker-tls/generate-docker-tls.sh` (CA +
server + client cert generation), the `docker-tls` Ansible role (deploys certs, configures
`dockerd`'s `daemon.json`), and `docs/RUNBOOK_DOCKER_TLS_ROTATION.md` for renewal. The CA
private key is generated offline and never committed or deployed to either VM — only the CA
cert (to verify the peer) and each host's own leaf cert+key.
## Public ingress
One origin, Caddy, automatic HTTPS. `reverse_proxy gateway:4200` — Caddy talks only to the
Gateway; nothing else is proxied. WebSocket upgrade works automatically (Caddy's default
`reverse_proxy` behavior, no special config needed for the terminal WebSocket path). Whether to
front this with Cloudflare or expose the origin directly is left as a deployment-time decision
(both work with Caddy's ACME flow; Cloudflare adds DDoS absorption at the cost of Cloudflare
being able to see plaintext unless "Full (strict)" mode is used) — not decided here since it
doesn't affect anything else in this document.
## Gateway-to-Lab reachability across two real VMs — RESOLVED in Phase 4.1
Phase 4 flagged this as the single most important open item; Phase 4.1
(`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1) closed it. Kept here as the historical record of
what the gap was and how it was closed, since the reasoning matters for anyone extending this
later (e.g. adding a second Lab Host).
**The gap, as originally found:** the Gateway used to reach a running Lab container via
`env.internal_ip` — a Docker bridge-local address recorded by the Provisioner when it creates
the container. That only works when Gateway and the Lab share a Docker daemon, true in dev and
any single-host setup, false the moment Labs run on a genuinely separate `rn-lab-host-01`.
**The fix — a Lab Host Router (`platform/lab-router/`):** a small, narrow service that runs on
rn-lab-host-01 (host-networked, like `egress-proxy`, so it can reach whatever per-Lab bridge
network the Provisioner creates) and does exactly one thing: given an opaque routing token over
mTLS, find the ONE local container labeled with that token (`docker ps --filter
label=rootnode.routing-token=<token>` — resolved purely from local Docker state, no Postgres,
no call back to the Provisioner) and reverse-proxy the request to it, HTTP and WebSocket both.
The Gateway now decides `direct` vs `router` per `lab_host` (`lab_hosts.routing_mode` +
`router_endpoint`, `platform/provisioner/migrations/004_lab_hosts_routing.sql`) via
`platform/gateway/src/lab-endpoint.js` — a single function, `labTarget(env, token)`, that every
proxy/fetch call site in `server.js` goes through instead of building
`http://${env.internal_ip}:${env.internal_port}` itself. `dev-local` stays `direct` (T480/local
dev unaffected, zero behavior change there); a real `rn-lab-host-01` is configured `router`.
**Firewall (`infra/firewall/rn-lab-host-01.rules`):** the old speculative
`lab_gateway_port_range` allowance (which assumed option (a), publishing per-Lab ports) was
replaced with a single, narrow rule: `platform_ip` → port `{{ lab_router_port | default('8443')
}}` only. No wide port range, no per-Lab port publishing at all — the Router is the only thing
the Gateway ever talks to on rn-lab-host-01 for Lab traffic.
**mTLS:** the Gateway↔Router pair uses the SAME private CA as the Provisioner↔dockerd pair
(`infra/scripts/docker-tls/generate-docker-tls.sh`, extended in Phase 4.1 to also issue
`router-server`/`gateway-client` leaf certs), deployed to a separate directory
(`/etc/rootnode/lab-router-tls/`) by the same `docker-tls` Ansible role.
**Proven, not just designed:** `infra/scripts/simulate-two-host/` builds a real two-network
Docker simulation (Gateway genuinely isolated on its own bridge, no route to any Lab's bridge)
and runs 13 real assertions against it — Gateway cannot reach a Lab's Docker-internal IP
directly, but reaches it correctly through the Router (HTTP and a real WebSocket
`kubectl get nodes`), Alice/Bob isolation holds, garbage/destroyed tokens are rejected, and the
Router itself refuses any connection without the Gateway's client certificate. See that
directory's README for exactly what is and isn't proven by a same-daemon simulation versus real
separate hardware.
## Cookie / auth production settings
`infra/environments/production/*.env.template`: `COOKIE_SECURE=1`, `TRUST_PROXY=1`. Dev's
insecure defaults (`COOKIE_SECURE` unset, no proxy trust) are never inherited — production env
files are separate files, not a dev file with overrides layered on.
+58
View File
@@ -0,0 +1,58 @@
# Production Validation Checklist
Walk this once, top to bottom, before calling a production deployment live. Every item here
needs the real hardware — this is the "DEPLOY LATER" half of Phase 4, not something this repo
can prove in advance.
## Hardware / Proxmox
- [ ] Disk inspection done, layout chosen and matches what's actually in the server (`docs/RUNBOOK_DISK_FAILURE.md` inspection checklist, `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4)
- [ ] Proxmox VE installed, reachable only from `management_cidr`/`proxmox_mgmt_cidr`
- [ ] VM template built from real cloud-init (`infra/proxmox/README.md`)
- [ ] `tofu apply` created both VMs; both boot and are SSH-reachable
## Platform VM
- [ ] `ansible-playbook site.yml` completed clean, all roles
- [ ] `docker compose ps` shows every service healthy (postgres, provisioner, api, gateway, reverse-proxy, prometheus, alertmanager, grafana, loki, promtail, exporters)
- [ ] `curl https://<origin>/healthz` (via Gateway) returns 200
- [ ] `curl http://localhost:4300/healthz` (Platform API, from the VM) returns `{"ok":true,"db":"ok"}`
- [ ] TLS certificate issued and valid for the real domain
## Lab Host VM
- [ ] Docker daemon TLS verified: `docker --tlsverify --tlscacert=... -H tcp://<lab_host_ip>:2376 info` succeeds from the Platform VM, fails from anywhere else
- [ ] Lab Host Router mTLS verified: a request to `https://<lab_host_ip>:8443` with no client cert is rejected; with the Gateway's real client cert it isn't (mirrors `infra/scripts/simulate-two-host/test-remote-routing.js`'s "direct public access: NOT AVAILABLE" case, now against the real host)
- [ ] Provisioner successfully creates a real Lab against the real Lab Host
- [ ] Gateway can actually reach that Lab's terminal (HTTP UI load + WebSocket `kubectl` command) — application mechanism built and simulation-tested in Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1/§6/§7); this line is the real-hardware confirmation that the same result holds across the actual two VMs, not two Docker networks on one daemon
- [ ] Egress proxy correctly restricts a Lab's outbound traffic (Provisioner's per-Lab `DOCKER-USER` chain)
## Security
- [ ] `infra/firewall/validate-firewall.sh` run against the real hosts with real `iptables-restore --test`, not just the structural lint
- [ ] Confirm from an unauthorized IP: Postgres, Provisioner, Docker TLS port, Proxmox UI are all unreachable
- [ ] Confirm Student Lab → another Lab is blocked; confirm Student Lab → Proxmox is blocked
- [ ] `COOKIE_SECURE`/`TRUST_PROXY` active (check a real Set-Cookie header over HTTPS)
## Academy (application-level smoke test)
- [ ] Owner bootstrap login works
- [ ] Instructor can create a cohort, roster a student
- [ ] Student can log in, boot a Lab, complete a scenario, see progress
- [ ] An assessment can be assigned, taken, and graded end-to-end
- [ ] Instructor analytics dashboards populate with real data
## Backup
- [ ] Manual `backup-postgres.sh` run succeeds against real Postgres, real off-host target reachable
- [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness — Phase 4.1's `infra/backup/test-destructive-restore-drill.sh` proved the full mechanism (realistic data, real storage destruction, restore, real Platform API boot) locally; this line is the real-hardware confirmation
- [ ] Cron job confirmed scheduled (`crontab -l`)
## Observability
- [ ] Grafana dashboard shows real data for both VMs
- [ ] Trigger one alert deliberately (e.g. stop a service) and confirm it fires and reaches the webhook
- [ ] Loki has logs flowing from both VMs
## Pre-production hardening (Phase 4.1 — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md)
- [ ] `resource-policy.yaml`'s `admission_control.max_concurrent_labs` set from a real measured
`infra/scripts/capacity-model.js` run against the real Lab Host, not left disabled
- [ ] A deliberate over-capacity attempt returns the clean 503 "Lab capacity is temporarily
full" message, not a 500 — confirms the real host's admission control matches the local test
- [ ] `docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md`'s steps still make sense once capacity numbers are real
Only once every box above is checked against the real server does the deployment graduate from
"infrastructure package built" to "actually in production."
@@ -0,0 +1,85 @@
# Remote & Self-Paced Delivery Architecture
Phase 4.1 §12. This is an architecture review, not a new product — no self-registration,
payments, or public enrollment is built here (explicit non-goal). The question this document
answers: does the infrastructure and application assume "an instructor always provisions the
Lab," and if so, where, and what would need to change for self-paced delivery to work.
## The short answer: less of that assumption exists than expected
RootNode Academy's Lab lifecycle was built student-driven from Phase 1 — `POST
/api/environment/launch` is called by the STUDENT'S OWN session, not by an instructor action.
"Prepare Labs" (`POST /api/admin/cohorts/:id/prepare-labs`) is a classroom **convenience** that
front-loads several students' launches at once before a synchronous class session, not the only
path to a running Lab. This was true before Phase 4.1 and confirmed still true by this phase's
own regression suites (`platform/test/e2e-phase2a.js` roster's two students with `cohort_id:
null` and takes them through login → curriculum → launch → scenario work with zero cohort
involvement) and by two new Phase 4.1 drills built specifically to exercise the remote/returning
-student narrative:
- `infra/scripts/drills/test-lab-host-loss.js` — student launches, makes progress, their Lab is
destroyed out from under them (standing in for an idle-reaper destroy), Platform stays up,
progress is untouched, they launch again unprompted and continue.
- `infra/scripts/drills/test-fake-cohort-rehearsal.js` — the same "returns later, fresh Lab,
curriculum continues" narrative as part of a fuller classroom rehearsal.
Both pass today, with zero application code changes needed for the self-paced-relevant parts of
this flow specifically — see the Phase 4.1 report for the full run results.
## What already works, unmodified
- **Student-initiated provisioning** — no instructor step is required between "student wants to
work" and "Lab exists." (`platform/api/src/server.js`'s `/api/environment/launch`.)
- **Cohort-less students** — `cohort_id` is nullable on student creation
(`POST /api/admin/students`); a student with no cohort logs in, sees the curriculum, launches
Labs, and makes progress identically to a cohorted one. Confirmed by inspection AND by the
existing regression suite, not just by schema reading.
- **Idle reaper → academic state survives → return → continue** — the exact lifecycle this
section describes as the target self-paced experience is what Phase 1's reaper +
Phase 2B's Postgres-backed academic data (separate from the disposable Lab) were already
designed to produce. Phase 4.1 proved it end-to-end, twice (see above).
- **Curriculum content itself is not cohort-gated** — `GET /api/curriculum/path` reads the one
curriculum path file directly (`curriculum.loadCurriculumPath()`); it never checks
`curriculum_assignments` or cohort membership at all. A self-paced student sees the full
curriculum with no instructor action required to "unlock" it.
## What's genuinely missing for a real self-paced product (Phase 5, not built here)
1. **No self-registration / public enrollment.** A student row is created only by an instructor
(`POST /api/admin/students`, admin/owner RBAC-gated). There is no public "sign up" surface.
This is almost certainly the right call to leave alone for now — it's explicitly a Phase 5+
product/business decision (course commerce, is it free, who's allowed to enroll), not an
infrastructure gap.
2. **No real student authentication.** A student session is created from a bare `student_id`
(`POST /api/login`, no password) — deliberately, since today every `student_id` is issued and
controlled by an instructor via the roster, not chosen or known by the public. A genuinely
remote/self-paced product where students access their account without an instructor in the
loop needs real credentials (email+password, or a magic link) — reusing the same
`password_hash`/`scrypt` machinery `instructors` already has (`src/auth/password.js`) would
be the natural extension, not a new auth system. **This is the one item on this list that
would actually block self-paced delivery today** — everything else already works or is a
product decision, not an infrastructure one.
3. **`curriculum_assignments.student_id` is schema-ready but unwired.** The table has supported
a direct per-student curriculum assignment (independent of any cohort) since Phase 2B
(`CONSTRAINT curriculum_assignment_target CHECK (cohort_id IS NOT NULL OR student_id IS NOT
NULL)`) — but nothing in the application ever reads or writes that column today; there's only
one curriculum path, so nothing has needed to differentiate yet. Not a gap that blocks
self-paced delivery now (every student already sees the one path with no gate), but the
moment a second curriculum path exists, this is where per-individual assignment lands — the
schema was designed for exactly this and doesn't need a migration to support it.
4. **No delivery-mode concept anywhere.** In-person, remote, and self-paced sessions aren't
distinguished in any data model today, and nothing currently needs them to be — capacity
planning, the reaper, and academic records all behave identically regardless of how a student
is attending. Worth being aware this is unmodeled, not worth building a taxonomy for before a
real product reason exists (YAGNI).
## What Phase 4.1 deliberately did NOT do here
No signup flow, no student password/email auth implementation, no per-student curriculum
assignment route, no payments, no public course catalog, no held/premium scenario content. All
explicit non-goals per this phase's authorization. This document exists so Phase 5 planning
starts from an accurate picture of the gap (one real item: student authentication) instead of a
guess.
+31
View File
@@ -0,0 +1,31 @@
# Runbook: Backup Failure
Triggered by the `BackupFailed`/`BackupAging` alerts (`infra/monitoring/prometheus/rules/rootnode.yml`).
```bash
# 1. See what actually happened — the cron wrapper logs to syslog
journalctl -u rootnode-backup --since "24 hours ago" -o cat
# or, if the cron job never even ran:
grep rootnode-postgres-backup /var/log/syslog
# 2. Run it manually to see the failure live
source /etc/rootnode/backup.env
/opt/rootnode/backup/backup-postgres.sh
```
**Common causes:**
- Disk full on `{{ rootnode_data_root }}/backups` — check `df -h`, prune more aggressively
(`BACKUP_RETENTION_DAYS`) or the ZFS dataset needs more space (see `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4).
- Off-host target unreachable (`BACKUP_OFFSITE_TARGET`) — check the SSH key/credentials
haven't expired, the target host/bucket is reachable: `ssh <offsite-target-host> echo ok` or
the equivalent for whatever adapter is configured (`docs/BACKUP_DR.md`).
- `BACKUP_ENCRYPTION_PASSPHRASE` missing/wrong in `/etc/rootnode/backup.env` — re-check against
the value in Ansible Vault (`vault_backup_encryption_passphrase`); a mismatch here means even
a "successful" old backup becomes unrestorable, so treat this specific failure mode as urgent.
- The dump itself is suspiciously small (script's own sanity check tripped) — Postgres may be
unhealthy; check `docker exec rootnode-platform-postgres-1 pg_isready` first.
**After fixing:** run the backup manually once to confirm success (metric updates immediately,
alert clears within one Prometheus scrape interval), then confirm the cron job is still
correctly scheduled (`crontab -l` as root, or re-run the `backup` Ansible role if the cron entry
itself is suspect).
+44
View File
@@ -0,0 +1,44 @@
# Runbook: Disk Failure
Applies once real hardware/ZFS exists — see `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4 for
the storage decision framework this assumes (mirrored vdevs, never single-disk for Postgres).
```bash
# 1. Confirm the failure
zpool status -v
smartctl -a /dev/sdX # the suspect drive
# 2. If it's a mirrored vdev (the required layout for anything holding Postgres data — see §4),
# the pool stays online in DEGRADED state. No immediate data-loss risk, but no redundancy
# either until replaced — treat DEGRADED as urgent, not routine.
zpool status # confirm DEGRADED, not FAULTED/UNAVAIL (those mean act immediately, possibly
# restore from backup if the redundant mirror side is also compromised)
# 3. Replace the physical drive, then:
zpool replace <pool> <old-device> <new-device>
zpool status # watch the resilver progress
# 4. If a whole pool was lost (both mirror sides failed simultaneously — the scenario mirroring
# exists to make vanishingly unlikely, but plan for it anyway):
# - rn-lab-host-01's disposable data: nothing to recover, just rebuild the VM
# (docs/RUNBOOK_VM_REBUILD.md)
# - rn-platform-01's Postgres data: restore from the latest off-host backup
# (docs/RUNBOOK_POSTGRES_RESTORE.md) — this is exactly the scenario off-host backup
# copies exist for; a local-only backup on the SAME failed pool would be useless here
```
**Pre-deployment disk inspection checklist** (run once when the real drives are known, before
they go into service — `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4):
```bash
for d in /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1; do # adjust to real device names
echo "== $d =="
smartctl -i "$d" | grep -E 'Model|Serial|Capacity|Firmware'
smartctl -A "$d" | grep -iE 'power_on_hours|reallocated|pending|uncorrectable|wear|percentage.used|temperature'
smartctl -H "$d" | grep -i overall
done
```
Record model, size, power-on hours, wear/endurance percentage, reallocated/pending sector
counts, temperature, and firmware version for every drive before committing to a ZFS layout —
a drive already showing wear or reallocated sectors shouldn't be trusted with Postgres data
even in a mirror.
+31
View File
@@ -0,0 +1,31 @@
# Runbook: Docker TLS Rotation
Rotate on a schedule (recommend yearly, well before the ~2.25-year cert lifetime
`generate-docker-tls.sh` defaults to) or immediately if a client cert may have leaked.
```bash
# 1. Generate a fresh bundle (new CA too — simplest, avoids any question about whether the old
# CA's private key was ever exposed)
cd infra/scripts/docker-tls
./generate-docker-tls.sh --out-dir ./out-$(date +%Y%m%d) --lab-host-ip <real-ip> --lab-host-dns rn-lab-host-01.internal
./verify-docker-tls.sh ./out-$(date +%Y%m%d)
# 2. Redeploy — this restarts dockerd on rn-lab-host-01 (brief interruption to any in-flight
# Docker API call from the Provisioner, NOT to already-running student Labs — their
# containers keep running through a dockerd restart)
cd ../../ansible
ROOTNODE_TLS_BUNDLE_DIR=../scripts/docker-tls/out-$(date +%Y%m%d) \
ansible-playbook site.yml -i inventory/production.ini --tags docker-tls --ask-vault-pass
# 3. Verify the Provisioner can still reach rn-lab-host-01
curl -s http://localhost:4100/health # from rn-platform-01
```
**Troubleshooting a failed rotation:**
- `x509: certificate signed by unknown authority` — the new ca.pem wasn't deployed to both
sides, or an old one is cached; re-run the docker-tls role on BOTH hosts.
- Provisioner logs `ECONNREFUSED` on port 2376 — dockerd didn't actually restart with the new
daemon.json; check `systemctl status docker` on rn-lab-host-01 and the ExecStart override
(`infra/ansible/roles/docker-tls/templates/docker-override.conf.j2`) actually applied.
- Firewall — confirm `infra/firewall/rn-lab-host-01.rules`' 2376 rule still matches
rn-platform-01's actual IP (didn't change as part of this rotation, but worth a sanity check).
+35
View File
@@ -0,0 +1,35 @@
# Runbook: Failed Migration
Both Platform API and Provisioner apply their own migrations automatically on startup
(`platform/api/src/db.js`, `platform/provisioner/src/db.js`), file-tracked in
`schema_migrations`, wrapped in a transaction per file — a migration either fully applies or
fully rolls back, it can't leave a table half-altered.
**If a deploy fails because a migration errored:**
```bash
# 1. Check what actually happened
docker logs rootnode-platform-api-1 --tail 50 # the error is right there — migration file + message
docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
"SELECT * FROM schema_migrations ORDER BY applied_at DESC LIMIT 10;"
# 2. Since the failing migration's transaction rolled back, the schema is still in its
# PREVIOUS valid state — the app just won't start until the migration succeeds. Fix the
# root cause (usually a real data issue the migration didn't anticipate, or the
# cross-service ordering dependency — Provisioner's migrations must run before Platform
# API's, since scenario_attempts has a real FK to environments; see
# docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md).
# 3. Redeploy once fixed — the migration runner is idempotent and will just pick up where it
# left off; already-applied migrations are skipped.
./infra/scripts/deploy-production.sh <same-or-fixed-tag>
```
**If you need to roll back the CODE but the migration already succeeded** (schema changed,
code that depended on it is being reverted): rolling back application images
(`docs/RUNBOOK_ROLLBACK.md`) does NOT undo a Postgres migration — there is no automatic
down-migration mechanism (deliberate, matches the existing check-before-alter, additive-only
migration convention documented in `docs/ENGINEERING_SOP.md` §6). If the old code genuinely
can't run against the new schema, restore from the pre-migration backup instead
(`docs/RUNBOOK_POSTGRES_RESTORE.md`) rather than trying to hand-write a down-migration under
pressure.
+34
View File
@@ -0,0 +1,34 @@
# Runbook: First Production Deployment
Prerequisites: Proxmox installed (`infra/proxmox/README.md`), VMs created
(`infra/terraform/`), DNS pointed at rn-platform-01's public IP.
```bash
# 1. Generate Docker TLS material (offline, on your own machine — never on either VM)
cd infra/scripts/docker-tls
./generate-docker-tls.sh --out-dir ./out --lab-host-ip <real-lab-host-ip> --lab-host-dns rn-lab-host-01.internal
./verify-docker-tls.sh ./out
# 2. Fill in real values
cd ../../ansible
cp group_vars/production.vault.yml.template group_vars/production.vault.yml
$EDITOR group_vars/production.vault.yml # real passwords/secrets — openssl rand -hex 32 etc.
ansible-vault encrypt group_vars/production.vault.yml
$EDITOR group_vars/production.yml # real IPs, domain, CIDRs
$EDITOR inventory/production.ini # real IPs
# 3. Deploy
ROOTNODE_TLS_BUNDLE_DIR=../scripts/docker-tls/out \
ansible-playbook site.yml -i inventory/production.ini --ask-vault-pass
# 4. Bootstrap the owner account (once, by a human)
ssh rootnode-deploy@rn-platform-01
docker exec -it rootnode-platform-api-1 node bootstrap-owner.js [email protected] "Your Name"
# (password comes from BOOTSTRAP_OWNER_PASSWORD in the environment, or the CLI prompt — see
# platform/api/README.md)
# 5. Verify
../../infra/scripts/production-smoke-test.sh
```
Then work through `docs/PRODUCTION_VALIDATION_CHECKLIST.md` in full before calling this done.
@@ -0,0 +1,29 @@
# Runbook: High Lab Host Resource Pressure
Triggered by the `LabHostHighMemory`/`HighDiskUsage` warning alerts.
```bash
# 1. How many Labs are actually running, and against what admission-control threshold?
curl -s http://localhost:4100/environments | jq '[.[] | select(.status=="running")] | length'
node infra/scripts/capacity-model.js # recommended/warning/hard thresholds
# 2. If genuinely over capacity: this is expected pressure from real usage, not a bug.
# - Destroy idle/abandoned Labs early via /admin (per-cohort "Destroy Labs" after a class
# session ends, rather than waiting for the idle reaper's default window)
# - If a specific cohort's class just ended, use /admin -> Cohorts -> [cohort] -> Destroy
# Cohort Labs immediately rather than waiting
# 3. If pressure is high with FEWER Labs running than capacity-model.js's threshold: something
# is consuming more per-Lab resources than the model assumes (a scenario doing something
# unusually heavy, a memory leak in a long-running Lab, zombie processes from a crashed
# reset). Check per-container usage:
docker stats --no-stream | grep rootnode-lab-
# 4. Update infra/environments/production/capacity.yml with fresh measured numbers
# (infra/scripts/loadtest/) if the per-Lab footprint has genuinely changed, so the
# admission-control threshold reflects reality going forward.
```
Do not silently raise resource limits in `provisioner-config/resource-policy.yaml` to make an
alert go away without understanding why usage is higher than modeled — that just moves the
same pressure onto host RAM/CPU directly instead of onto individual Lab limits.
+17
View File
@@ -0,0 +1,17 @@
# Runbook: Normal Deploy
For a routine code change already merged to `main` on the canonical Gitea repo.
1. CI (`infra/ci/gitea-workflows/build-and-test.yml`) runs automatically: tests, image build/tag/push, staging deploy, staging smoke test.
2. Review the staging smoke test result and, ideally, poke around `https://staging.<domain>` yourself.
3. Promote to production (manual, deliberate):
```bash
cd infra/scripts
./deploy-production.sh sha-<12-char-sha> # or a release tag, e.g. v1.3.0
./production-smoke-test.sh
```
4. Watch Grafana (`/grafana/`) for 10-15 minutes — error rates, Postgres connections, Lab Host
memory — before considering the deploy "settled."
5. If anything looks wrong: `docs/RUNBOOK_ROLLBACK.md`.
+27
View File
@@ -0,0 +1,27 @@
# Runbook: Owner Account Recovery
**Locked out with zero working owner accounts** (all deactivated, forgot the only owner's
password, etc.):
```bash
ssh rootnode-deploy@rn-platform-01
docker exec -it rootnode-platform-api-1 sh
node bootstrap-owner.js [email protected] "Your Name"
# bootstrap-owner.js refuses to run if an active owner already exists UNLESS the target email
# belongs to an EXISTING instructor row — in that case it promotes them to owner. Point it at
# any known instructor email to recover through them, or a brand-new email to create a fresh
# owner outright if the instructors table itself is in an unknown state.
```
**Forgot password, account still active and you have SSH access:**
```bash
docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
"SELECT instructor_id, email, role, status FROM instructors WHERE role='owner';"
# Then use another owner's account to reset via /admin -> Instructors -> [account] -> Reset
# Password, OR re-run bootstrap-owner.js targeting that email (also resets the password).
```
**Never** hand-edit `password_hash` in the database — it's `scrypt(salt):hash` in a specific
format (`platform/api/src/auth/password.js`); a malformed value just breaks login rather than
setting a usable password. Always go through `bootstrap-owner.js` or the admin UI's reset flow.
+45
View File
@@ -0,0 +1,45 @@
# Runbook: Postgres Restore
**When to use:** Postgres corruption/data loss on rn-platform-01, or verifying a backup is
actually good (do this periodically even without an incident — an untested backup is a
hypothesis, not a backup).
```bash
# 1. Find the backup to restore
ls -la /srv/rootnode-data/backups/ # or wherever backup_offsite_target points, if local copies are gone
# 2. Create a FRESH target database — never restore over the live one directly
docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
"CREATE DATABASE rootnode_platform_restore_test;"
# 3. Restore into it
source /etc/rootnode/backup.env # for BACKUP_ENCRYPTION_PASSPHRASE
/opt/rootnode/backup/restore-postgres.sh \
/srv/rootnode-data/backups/rootnode-platform-<timestamp>.sql.gz.gpg \
"postgres://rootnode_platform:$PGPASS@localhost:5432/rootnode_platform_restore_test"
# type RESTORE when prompted
# 4. Verify
/opt/rootnode/backup/verify-restore.sh \
"postgres://rootnode_platform:$PGPASS@localhost:5432/rootnode_platform_restore_test"
# 5a. If this was a drill: drop the test database
docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
"DROP DATABASE rootnode_platform_restore_test;"
# 5b. If this was a REAL incident: stop the app stack, rename the broken DB, rename the
# restored one into its place, restart.
systemctl stop rootnode-platform
docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
"ALTER DATABASE rootnode_platform RENAME TO rootnode_platform_broken_$(date +%s);"
docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
"ALTER DATABASE rootnode_platform_restore_test RENAME TO rootnode_platform;"
systemctl start rootnode-platform
```
Local, non-production dry run of this exact flow: `infra/backup/test-backup-restore.sh` (uses
disposable containers, safe to run anytime, no real data involved).
**Recovery point objective:** up to 24 hours of academic data (daily backup cadence) — see
`docs/BACKUP_DR.md` for the full RPO/RTO discussion and why that's the honest number, not an
aspirational one.
+18
View File
@@ -0,0 +1,18 @@
# Runbook: Rollback
```bash
cd infra/scripts
./rollback-production.sh # rolls back to the previous deployed tag automatically
# or, to go back further than one step:
./rollback-production.sh sha-<older-sha>
./production-smoke-test.sh
```
This rolls back **application images only**. If the bad deploy included a database migration
that needs undoing too, schema changes aren't automatically reversible — see
`docs/RUNBOOK_FAILED_MIGRATION.md` instead (or in addition).
After rolling back: file an issue / note in the deploy log (`/opt/rootnode/deploy-history.log`
already has the rollback recorded) about what went wrong before re-attempting the forward
deploy.
+40
View File
@@ -0,0 +1,40 @@
# Runbook: Stuck Lab Cleanup
A Lab that's stuck `provisioning`/`resetting` past its expected boot window, or a container
that outlived its Postgres `environments` row (crash during teardown, etc.).
**Via the admin UI (preferred, no host access needed):** `/admin` -> Students -> [student] ->
Environment -> Destroy, then Provision again if they need a fresh one immediately.
**Via the API directly** (same effect, useful for scripting a batch cleanup):
```bash
curl -s -b <owner-session-cookie> -X DELETE https://academy.example.com/api/admin/environments/<environment_id>
```
**A container the Provisioner itself has lost track of** (rare — e.g. a crash during
provisioning left an orphaned container/network/firewall chain with no matching `environments`
row, or vice versa):
```bash
# On rn-lab-host-01 — list what's actually running
docker ps -a --format '{{.Names}}\t{{.Status}}' | grep '^rootnode-lab-'
# Cross-reference against the Provisioner's own view
curl -s http://localhost:4100/environments | jq '.[] | {environment_id, status, container_name}'
# Provisioner already does this automatically on its own restart (reconcileOnStartup,
# platform/provisioner/src/lifecycle.js) — a simple `systemctl restart rootnode-platform`
# (which restarts the provisioner container as part of the stack) is often the fastest fix for
# a batch of orphans rather than hand-cleaning each one.
# Manual last resort — remove the container/network/firewall chain directly:
docker rm -f rootnode-lab-<id>
docker network rm rootnode-lab-net-<id>
iptables -D DOCKER-USER -s <subnet> -j ROOTNODE-ENV-<id> # remove the jump rule FIRST
iptables -F ROOTNODE-ENV-<id> && iptables -X ROOTNODE-ENV-<id>
```
Never manually delete an `environments` row from Postgres directly unless you've confirmed the
container/network/firewall chain are already gone — the Provisioner treats that table as
authoritative and will behave unpredictably against real Docker/iptables state it doesn't know
matches.
+19
View File
@@ -0,0 +1,19 @@
# Runbook: Public TLS Certificate Renewal
If using Caddy's built-in Let's Encrypt (the default in `infra/docker/platform/Caddyfile.j2`):
**nothing to do** — Caddy renews automatically well before expiry, as long as port 80 stays
reachable for the ACME HTTP-01 challenge (`infra/firewall/rn-platform-01.rules` already allows
it). `infra/scripts/production-smoke-test.sh` checks the live expiry date on every run as an
early warning if auto-renewal ever silently breaks.
If using Cloudflare origin certs instead (see `docs/PRODUCTION_NETWORK_SECURITY.md` §8 for
that decision): Cloudflare-issued origin certs are typically 15-year validity — this is a
once-per-network-topology-change concern, not a recurring one. Re-issue from the Cloudflare
dashboard (SSL/TLS -> Origin Server) and redeploy the cert/key via the `platform` Ansible role
if it ever needs replacing (domain change, suspected compromise).
**Manual check:**
```bash
echo | openssl s_client -connect academy.example.com:443 -servername academy.example.com 2>/dev/null \
| openssl x509 -noout -enddate
```
+31
View File
@@ -0,0 +1,31 @@
# Runbook: Platform VM / Lab Host VM Rebuild
Both VMs are designed to be disposable and rebuildable from code — nothing irreplaceable
should ever live only on either VM's disk outside of what's backed up (`docs/BACKUP_DR.md`).
## rn-platform-01 lost
Academic data survives (it's in Postgres backups, off-host). Instructor/student accounts and
progress are recoverable; anything since the last backup is not (`docs/BACKUP_DR.md`'s RPO).
```bash
cd infra/terraform && terraform apply # recreates the VM from the template
cd ../ansible && ansible-playbook site.yml -i inventory/production.ini --limit platform --ask-vault-pass
# Then restore Postgres — docs/RUNBOOK_POSTGRES_RESTORE.md, step 5b variant (fresh DB this time,
# no need to rename since it's a fresh instance)
./infra/scripts/production-smoke-test.sh
```
## rn-lab-host-01 lost
**No academic data is lost** — this host only ever holds disposable Lab containers by design
(`docs/BACKUP_DR.md`'s Tier distinction). Every enrolled student's Lab needs re-provisioning
after this, which is a normal "Prepare Labs" click, not a recovery operation.
```bash
cd infra/terraform && terraform apply
cd ../ansible && ansible-playbook site.yml -i inventory/production.ini --limit lab_host --ask-vault-pass
# Re-generate Docker TLS material for the new host if its identity (IP/DNS) changed —
# docs/RUNBOOK_DOCKER_TLS_ROTATION.md
# From /admin: Prepare Labs again for any cohort that needs it.
```
+47
View File
@@ -0,0 +1,47 @@
# Security Scanning (Phase 4.1 §24)
`infra/ci/gitea-workflows/build-and-test.yml`'s `security-scan` job. All free, open-source,
no account/signup required for any of them — no paid scanner was added.
## What's scanned
| Tool | Checks | Scope |
|---|---|---|
| `npm audit --audit-level=high` | Known-vulnerable dependencies | Every first-party Node service |
| [gitleaks](https://github.com/gitleaks/gitleaks) | Committed secrets (keys, tokens, credentials) | Whole repo |
| [hadolint](https://github.com/hadolint/hadolint) | Dockerfile best-practice/security lint | Every first-party Dockerfile |
| [shellcheck](https://www.shellcheck.net/) | Shell script correctness/security | `infra/**/*.sh` |
| [trivy](https://github.com/aquasecurity/trivy) | OS/library CVEs in a built container image | `platform-api` image (representative — same base/layers as the other Node services) |
## Severity policy
- **CRITICAL / HIGH** — fails the build (`security-scan` blocks `build-images`, which blocks
`deploy-staging`, which blocks production promotion — see the job dependency chain in the
workflow file).
- **MEDIUM / LOW** — reported in the job output, never blocks. A build that's permanently red
over a low-severity transitive dependency finding nobody can immediately fix trains everyone
to ignore red builds, which is worse than not scanning at all.
`npm audit --audit-level=high` and trivy's `--severity HIGH,CRITICAL` both implement this
directly at the tool level, not via manual triage.
## What's NOT in the automated job (and why)
- **Dedicated IaC security linting** (checkov/tflint) — not added. `infra/scripts/validate-compose.sh`,
`infra/firewall/validate-firewall.sh`, `ansible-playbook --syntax-check`, and `tofu validate`
(all in the `regression` job) already catch structural/syntax problems in the infra code;
a dedicated security-policy linter for Terraform/Ansible is a reasonable Phase 5 addition once
there's a real cloud footprint to misconfigure, not before.
- **DAST/penetration testing** — out of scope for CI; this is a build-time gate, not a running-system
test. Covered separately by `docs/PRODUCTION_VALIDATION_CHECKLIST.md`'s security section once
real hardware exists.
- **SAST beyond hadolint/shellcheck** — the codebase is small enough (a handful of first-party
Node services) that `npm audit` + secret scanning + Dockerfile/shell lint cover the realistic
risk surface; a dedicated SAST tool (Semgrep, CodeQL) is a reasonable addition once the
codebase or team grows enough to justify the added CI time.
## Local reproduction
Every check above can be run locally with the exact same free tools — no CI-only dependency.
See the `security-scan` job's steps in `infra/ci/gitea-workflows/build-and-test.yml` for the
exact commands (they're copy-pasteable).
+2
View File
@@ -8,6 +8,7 @@ import ExamTimer from './components/ExamTimer'
import ExamReport from './components/ExamReport'
import ExamStartModal from './components/ExamStartModal'
import ExamHistory from './components/ExamHistory'
import AssessmentBanner from './components/AssessmentBanner'
import styles from './App.module.css'
const MIN_SIDEBAR_W = 180
@@ -303,6 +304,7 @@ export default function App() {
return (
<div className={styles.app}>
<Header clusterReady={clusterReady} onShowHistory={() => setShowHistory(true)} />
<AssessmentBanner />
{/* Bundle navigation bar */}
<BundleNav
+39
View File
@@ -0,0 +1,39 @@
import { useState } from 'react'
import { adminApi } from './api'
import styles from './Admin.module.css'
export default function AccountView({ instructor }) {
const [oldPassword, setOldPassword] = useState('')
const [newPassword, setNewPassword] = useState('')
const [error, setError] = useState(null)
const [notice, setNotice] = useState(null)
const [busy, setBusy] = useState(false)
const submit = async (e) => {
e.preventDefault()
setBusy(true); setError(null); setNotice(null)
try {
await adminApi.changePassword(oldPassword, newPassword)
setOldPassword(''); setNewPassword('')
setNotice('Password changed. Any other sessions you had open elsewhere were signed out.')
} catch (e) { setError(e.message) } finally { setBusy(false) }
}
return (
<div className={styles.card}>
<h2>Account</h2>
<p className={styles.muted}>{instructor.display_name} · {instructor.email} · role: {instructor.role}</p>
<h3 style={{ marginTop: '1.5rem' }}>Change password</h3>
<form onSubmit={submit} className={styles.formRow}>
{error && <div className={styles.errorBox}>{error}</div>}
<div><label className={styles.label}>Current password</label>
<input className={styles.input} type="password" value={oldPassword} onChange={(e) => setOldPassword(e.target.value)} autoComplete="current-password" required /></div>
<div><label className={styles.label}>New password (12+ chars)</label>
<input className={styles.input} type="password" value={newPassword} onChange={(e) => setNewPassword(e.target.value)} autoComplete="new-password" minLength={12} required /></div>
<div><button className={styles.btn} disabled={busy}>{busy ? 'Changing…' : 'Change password'}</button></div>
</form>
{notice && <p className={styles.muted}>{notice}</p>}
</div>
)
}
+62
View File
@@ -0,0 +1,62 @@
/* Admin portal — visually distinct from the student shell (a muted accent instead of
--green) so instructors and students never mistake which mode they're in, while still
reusing the same design tokens for everything else. */
.wrap { min-height: 100vh; background: var(--bg); color: var(--text); font-family: var(--sans); }
.loginWrap { height: 100vh; display: flex; align-items: center; justify-content: center; }
.loginCard { width: 360px; background: var(--surface); border: 1px solid var(--border); border-radius: var(--radius-lg); padding: 2rem; }
.brand { font-family: var(--mono); font-size: 1.2rem; font-weight: 600; color: var(--blue); display: flex; align-items: center; gap: 0.5rem; }
.subtitle { color: var(--text-2); font-size: 0.85rem; margin: 0.4rem 0 1.25rem; }
.input, .select {
width: 100%; padding: 0.55rem 0.6rem; margin-top: 0.3rem;
background: var(--surface2); border: 1px solid var(--border2); border-radius: var(--radius);
color: var(--text); font-family: var(--sans); font-size: 0.9rem;
}
.label { display: block; font-size: 0.8rem; color: var(--text-2); margin-top: 0.9rem; }
.btn {
padding: 0.55rem 0.9rem; background: var(--blue); color: #04141c; border: none;
border-radius: var(--radius); font-weight: 600; font-size: 0.85rem; cursor: pointer;
}
.btn:disabled { opacity: 0.5; cursor: not-allowed; }
.btnSecondary { padding: 0.5rem 0.8rem; background: transparent; color: var(--text); border: 1px solid var(--border2); border-radius: var(--radius); font-size: 0.85rem; cursor: pointer; }
.btnDanger { padding: 0.5rem 0.8rem; background: transparent; color: var(--red); border: 1px solid var(--red); border-radius: var(--radius); font-size: 0.85rem; cursor: pointer; }
.header { display: flex; align-items: center; justify-content: space-between; padding: 1rem 2rem; border-bottom: 1px solid var(--border); }
.nav { display: flex; gap: 1.25rem; }
.navLink { color: var(--text-2); cursor: pointer; font-size: 0.9rem; background: none; border: none; padding: 0.3rem 0; }
.navLinkActive { color: var(--blue); border-bottom: 2px solid var(--blue); }
.content { max-width: 1000px; margin: 1.5rem auto; padding: 0 1.5rem; }
.statGrid { display: grid; grid-template-columns: repeat(auto-fit, minmax(140px, 1fr)); gap: 0.75rem; margin-bottom: 1.5rem; }
.statCard { background: var(--surface); border: 1px solid var(--border); border-radius: var(--radius); padding: 1rem; }
.statValue { font-size: 1.6rem; font-weight: 700; }
.statLabel { color: var(--text-2); font-size: 0.78rem; margin-top: 0.2rem; }
.card { background: var(--surface); border: 1px solid var(--border); border-radius: var(--radius-lg); padding: 1.25rem; margin-bottom: 1.25rem; }
.card h2, .card h3 { margin-bottom: 0.75rem; font-size: 1rem; }
table { width: 100%; border-collapse: collapse; font-size: 0.85rem; }
th { text-align: left; color: var(--text-2); font-weight: 600; padding: 0.4rem 0.6rem; border-bottom: 1px solid var(--border2); font-size: 0.75rem; text-transform: uppercase; }
td { padding: 0.5rem 0.6rem; border-bottom: 1px solid var(--border); }
tr:hover td { background: var(--surface2); }
.pill { display: inline-block; padding: 0.15rem 0.55rem; border-radius: 999px; font-size: 0.75rem; font-family: var(--mono); }
.pillOk { background: var(--green-dim); color: var(--green); }
.pillWarn { background: var(--amber-dim); color: var(--amber); }
.pillErr { background: var(--red-dim); color: var(--red); }
.pillMuted { background: var(--surface3); color: var(--text-2); }
.formRow { display: flex; gap: 0.5rem; align-items: end; margin-bottom: 1rem; flex-wrap: wrap; }
.formRow > div { flex: 1; min-width: 140px; }
.linkBtn { background: none; border: none; color: var(--blue); cursor: pointer; font-size: 0.85rem; padding: 0; text-decoration: underline; }
.errorBox { background: var(--red-dim); color: var(--red); border: 1px solid var(--red); border-radius: var(--radius); padding: 0.6rem 0.8rem; font-size: 0.85rem; margin-bottom: 1rem; }
.muted { color: var(--text-2); font-size: 0.85rem; }
/* Phase 4.1 §16 — shared pagination control (AuditView, StudentsView attempts). */
.pager { display: flex; align-items: center; gap: 0.75rem; margin-top: 0.75rem; }
+68
View File
@@ -0,0 +1,68 @@
import { useState, useEffect } from 'react'
import { adminApi } from './api'
import styles from './Admin.module.css'
// Operational overview — active cohorts, students, Lab states, recent activity. Deliberately
// lean (no Prometheus/Grafana here, see docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md §9) — this is
// "is class ready / is anything on fire," not a metrics platform.
export default function AdminDashboard() {
const [data, setData] = useState(null)
const [error, setError] = useState(null)
useEffect(() => {
adminApi.dashboard().then(setData).catch((e) => setError(e.message))
}, [])
if (error) return <div className={styles.errorBox}>{error}</div>
if (!data) return <p className={styles.muted}>Loading…</p>
return (
<>
<div className={styles.statGrid}>
<Stat label="Active cohorts" value={data.active_cohorts} />
<Stat label="Total students" value={data.total_students} />
<Stat label="Labs running" value={data.labs.running} />
<Stat label="Labs provisioning" value={data.labs.provisioning} />
<Stat label="Labs in error" value={data.labs.error} />
<Stat label="Labs idle/destroyed" value={data.labs.idle_or_destroyed} />
</div>
<div className={styles.card}>
<h3>Recent completions</h3>
{data.recent_completions.length === 0 ? <p className={styles.muted}>None yet.</p> : (
<table>
<thead><tr><th>Student</th><th>Scenario</th><th>Completed</th></tr></thead>
<tbody>
{data.recent_completions.map((c, i) => (
<tr key={i}><td>{c.display_name}</td><td>{c.scenario_id}</td><td>{new Date(c.completed_at).toLocaleString()}</td></tr>
))}
</tbody>
</table>
)}
</div>
<div className={styles.card}>
<h3>Recent environment failures</h3>
{data.recent_failures.length === 0 ? <p className={styles.muted}>None.</p> : (
<table>
<thead><tr><th>Student</th><th>Reason</th><th>When</th></tr></thead>
<tbody>
{data.recent_failures.map((f, i) => (
<tr key={i}><td>{f.student_id}</td><td>{f.error_reason || '—'}</td><td>{f.started_at ? new Date(f.started_at).toLocaleString() : '—'}</td></tr>
))}
</tbody>
</table>
)}
</div>
</>
)
}
function Stat({ label, value }) {
return (
<div className={styles.statCard}>
<div className={styles.statValue}>{value}</div>
<div className={styles.statLabel}>{label}</div>
</div>
)
}
+107
View File
@@ -0,0 +1,107 @@
import { useState, useEffect } from 'react'
import { adminApi } from './api'
import AdminDashboard from './AdminDashboard'
import CohortsView from './CohortsView'
import StudentsView from './StudentsView'
import InstructorsView from './InstructorsView'
import AuditView from './AuditView'
import AccountView from './AccountView'
import LearningView from './LearningView'
import AssessmentsView from './AssessmentsView'
import styles from './Admin.module.css'
// Instructor/Admin portal — clearly separate from the student experience (/admin route,
// separate rn_instructor session cookie, separate login). A student session cannot reach any
// of these views or their underlying /api/admin/* routes. Per-person accounts replace Phase
// 2B's single shared ADMIN_PASSWORD — see docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md.
// Learning/Assessments (Phase 3B) are visible to every instructor role — cohort scoping for a
// plain 'instructor' happens server-side per route, not by hiding the whole tab.
const NAV = [
{ key: 'dashboard', label: 'Dashboard' },
{ key: 'cohorts', label: 'Cohorts' },
{ key: 'students', label: 'Students' },
{ key: 'learning', label: 'Learning' },
{ key: 'assessments', label: 'Assessments' },
{ key: 'instructors', label: 'Instructors', roles: ['owner'] },
{ key: 'audit', label: 'Audit', roles: ['owner', 'admin'] },
]
export default function AdminPortal() {
const [instructor, setInstructor] = useState(null) // null = checking, false = need login, object = signed in
const [view, setView] = useState('dashboard')
useEffect(() => {
adminApi.session().then((r) => setInstructor(r.instructor)).catch(() => setInstructor(false))
}, [])
if (instructor === null) return null
if (!instructor) return <InstructorLogin onLogin={setInstructor} />
const visibleNav = NAV.filter((item) => !item.roles || item.roles.includes(instructor.role))
return (
<div className={styles.wrap}>
<header className={styles.header}>
<div className={styles.brand}>⚙ RootNode Admin</div>
<nav className={styles.nav}>
{visibleNav.map((item) => (
<button key={item.key} className={`${styles.navLink} ${view === item.key ? styles.navLinkActive : ''}`} onClick={() => setView(item.key)}>
{item.label}
</button>
))}
<button className={`${styles.navLink} ${view === 'account' ? styles.navLinkActive : ''}`} onClick={() => setView('account')}>
{instructor.display_name}
</button>
<button className={styles.navLink} onClick={() => adminApi.logout().then(() => setInstructor(false))}>Sign out</button>
</nav>
</header>
<div className={styles.content}>
{view === 'dashboard' && <AdminDashboard />}
{view === 'cohorts' && <CohortsView role={instructor.role} />}
{view === 'students' && <StudentsView role={instructor.role} />}
{view === 'learning' && <LearningView />}
{view === 'assessments' && <AssessmentsView role={instructor.role} />}
{view === 'instructors' && instructor.role === 'owner' && <InstructorsView currentInstructorId={instructor.instructor_id} />}
{view === 'audit' && ['owner', 'admin'].includes(instructor.role) && <AuditView />}
{view === 'account' && <AccountView instructor={instructor} />}
</div>
</div>
)
}
function InstructorLogin({ onLogin }) {
const [email, setEmail] = useState('')
const [password, setPassword] = useState('')
const [error, setError] = useState(null)
const [busy, setBusy] = useState(false)
const submit = async (e) => {
e.preventDefault()
setBusy(true); setError(null)
try {
const { instructor } = await adminApi.login(email, password)
onLogin(instructor)
} catch (e) {
setError(e.message || 'Sign-in failed.')
} finally {
setBusy(false)
}
}
return (
<div className={styles.loginWrap}>
<form className={styles.loginCard} onSubmit={submit}>
<div className={styles.brand}>⚙ RootNode Admin</div>
<p className={styles.subtitle}>Instructor sign-in. Not connected to student accounts.</p>
{error && <div className={styles.errorBox}>{error}</div>}
<label className={styles.label}>Email</label>
<input className={styles.input} type="email" value={email} onChange={(e) => setEmail(e.target.value)} autoFocus autoComplete="username" />
<label className={styles.label} style={{ marginTop: '0.75rem' }}>Password</label>
<input className={styles.input} type="password" value={password} onChange={(e) => setPassword(e.target.value)} autoComplete="current-password" />
<button className={styles.btn} style={{ marginTop: '1.25rem', width: '100%' }} disabled={busy || !email || !password}>
{busy ? 'Signing in…' : 'Sign in'}
</button>
</form>
</div>
)
}
+258
View File
@@ -0,0 +1,258 @@
import { useState, useEffect, useCallback } from 'react'
import { adminApi } from './api'
import styles from './Admin.module.css'
// Exam and Capstone definition/session management. Definition authoring (create/edit) is
// admin/owner only server-side (RBAC) — a plain instructor can still assign an existing
// definition to their own cohort and view/cancel sessions within it, so the form below is
// always shown but the server will 403 a create/edit attempt from an instructor role; this
// mirrors CohortsView's approach of not pre-emptively hiding every control, since assessments
// are viewed by all roles and only definition-authoring is restricted.
export default function AssessmentsView({ role }) {
const [assessments, setAssessments] = useState(null)
const [cohorts, setCohorts] = useState(null)
const [selected, setSelected] = useState(null)
const [error, setError] = useState(null)
const canAuthor = role === 'owner' || role === 'admin'
const reload = useCallback(() => {
adminApi.assessments().then(setAssessments).catch((e) => setError(e.message))
adminApi.cohorts().then(setCohorts).catch(() => {})
}, [])
useEffect(reload, [reload])
if (selected) return <AssessmentDetail definition={selected} cohorts={cohorts} onBack={() => { setSelected(null); reload() }} />
return (
<>
{error && <div className={styles.errorBox}>{error}</div>}
{canAuthor && (
<div className={styles.card}>
<h2>Create assessment</h2>
<CreateAssessmentForm onCreated={reload} role={role} />
</div>
)}
<div className={styles.card}>
<h2>Exams &amp; Capstones</h2>
{!assessments ? <p className={styles.muted}>Loading…</p> : (
<table>
<thead><tr><th>Title</th><th>Type</th><th>Duration</th><th>Hints</th><th>Status</th><th /></tr></thead>
<tbody>
{assessments.map((a) => (
<tr key={a.assessment_definition_id}>
<td>{a.title}</td>
<td><span className={styles.pill}>{a.type}</span></td>
<td>{a.duration_minutes} min</td>
<td>{a.hint_policy}</td>
<td><span className={`${styles.pill} ${a.status === 'active' ? styles.pillOk : styles.pillMuted}`}>{a.status}</span></td>
<td><button className={styles.linkBtn} onClick={() => setSelected(a)}>Open</button></td>
</tr>
))}
</tbody>
</table>
)}
</div>
</>
)
}
// Phase 4.1 §18 — pool_spec is built from structured picks (required scenarios + named random
// pools), not hand-typed JSON. `catalog` ({ stages: [{ id, name, scenarios: [{id,title}] }] })
// comes from GET /api/admin/scenario-catalog. An owner can still drop into a raw-JSON view for
// anything the structured editor doesn't cover — see the toggle below.
function PoolSpecEditor({ catalog, required, setRequired, pools, setPools }) {
const allScenarios = (catalog?.stages || []).flatMap((s) => s.scenarios.map((sc) => ({ ...sc, stageName: s.name })))
const titleFor = (id) => allScenarios.find((s) => s.id === id)?.title || id
const addPool = () => setPools([...pools, { count: 1, from: [] }])
const removePool = (i) => setPools(pools.filter((_, idx) => idx !== i))
const updatePool = (i, patch) => setPools(pools.map((p, idx) => (idx === i ? { ...p, ...patch } : p)))
return (
<div>
<label className={styles.label}>Required scenarios (always included)</label>
<select className={styles.select} multiple size={6} style={{ width: '100%' }}
value={required} onChange={(e) => setRequired([...e.target.selectedOptions].map((o) => o.value))}>
{catalog?.stages.map((stage) => (
<optgroup key={stage.id} label={stage.name}>
{stage.scenarios.map((sc) => <option key={sc.id} value={sc.id}>{sc.title}</option>)}
</optgroup>
))}
</select>
<p className={styles.muted}>{required.length ? `${required.length} selected: ${required.map(titleFor).join(', ')}` : 'None selected — cmd/ctrl-click to pick multiple.'}</p>
<label className={styles.label} style={{ marginTop: '1rem', display: 'block' }}>Random pools (draw N at random from a set, every attempt)</label>
{pools.map((pool, i) => (
<div key={i} className={styles.card} style={{ marginBottom: '0.5rem' }}>
<div className={styles.formRow}>
<div style={{ maxWidth: '100px' }}><label className={styles.label}>Draw count</label>
<input className={styles.input} type="number" min="1" value={pool.count}
onChange={(e) => updatePool(i, { count: Number(e.target.value) })} /></div>
<div><button type="button" className={styles.linkBtn} onClick={() => removePool(i)}>Remove pool</button></div>
</div>
<label className={styles.label}>Draw from</label>
<select className={styles.select} multiple size={5} style={{ width: '100%' }}
value={pool.from} onChange={(e) => updatePool(i, { from: [...e.target.selectedOptions].map((o) => o.value) })}>
{catalog?.stages.map((stage) => (
<optgroup key={stage.id} label={stage.name}>
{stage.scenarios.map((sc) => <option key={sc.id} value={sc.id}>{sc.title}</option>)}
</optgroup>
))}
</select>
{pool.count > pool.from.length && <p className={styles.muted} style={{ color: 'var(--amber, #b58900)' }}>Draw count ({pool.count}) exceeds the pool size ({pool.from.length}) — every draw will use every scenario, not a random subset.</p>}
</div>
))}
<button type="button" className={styles.linkBtn} onClick={addPool}>+ Add random pool</button>
</div>
)
}
function CreateAssessmentForm({ onCreated, role }) {
const [form, setForm] = useState({ id: '', type: 'exam', title: '', duration_minutes: 60, hint_policy: 'disabled' })
const [catalog, setCatalog] = useState(null)
const [required, setRequired] = useState([])
const [pools, setPools] = useState([])
const [advanced, setAdvanced] = useState(false)
const [rawPoolSpec, setRawPoolSpec] = useState('')
const [error, setError] = useState(null)
const [busy, setBusy] = useState(false)
useEffect(() => { adminApi.scenarioCatalog().then(setCatalog).catch((e) => setError(e.message)) }, [])
const submit = async (e) => {
e.preventDefault()
setBusy(true); setError(null)
try {
// Owner's raw-JSON view, when open, is the source of truth — otherwise built from the
// structured pickers above.
const pool_spec = advanced
? JSON.parse(rawPoolSpec || '{"required":[],"random":[]}')
: { required, random: pools.filter((p) => p.count > 0 && p.from.length > 0) };
if (!advanced && !required.length && !pool_spec.random.length) {
throw new Error('Pick at least one required scenario or one random pool.')
}
await adminApi.createAssessment({ ...form, duration_minutes: Number(form.duration_minutes), pool_spec })
setForm({ id: '', type: 'exam', title: '', duration_minutes: 60, hint_policy: 'disabled' })
setRequired([]); setPools([]); setRawPoolSpec('')
onCreated()
} catch (e) { setError(e.message?.includes('JSON') ? 'Raw pool_spec must be valid JSON' : e.message) } finally { setBusy(false) }
}
return (
<form onSubmit={submit}>
{error && <div className={styles.errorBox}>{error}</div>}
<div className={styles.formRow}>
<div><label className={styles.label}>ID (slug)</label>
<input className={styles.input} value={form.id} onChange={(e) => setForm({ ...form, id: e.target.value })} required /></div>
<div><label className={styles.label}>Title</label>
<input className={styles.input} value={form.title} onChange={(e) => setForm({ ...form, title: e.target.value })} required /></div>
<div><label className={styles.label}>Type</label>
<select className={styles.select} value={form.type} onChange={(e) => setForm({ ...form, type: e.target.value })}>
<option value="exam">exam</option><option value="capstone">capstone</option>
</select></div>
<div><label className={styles.label}>Duration (min)</label>
<input className={styles.input} type="number" value={form.duration_minutes} onChange={(e) => setForm({ ...form, duration_minutes: e.target.value })} required /></div>
<div><label className={styles.label}>Hint policy</label>
<select className={styles.select} value={form.hint_policy} onChange={(e) => setForm({ ...form, hint_policy: e.target.value })}>
<option value="disabled">disabled</option><option value="flagged">flagged</option><option value="free">free</option>
</select></div>
</div>
{role === 'owner' && (
<p className={styles.muted}>
<button type="button" className={styles.linkBtn} onClick={() => setAdvanced(!advanced)}>
{advanced ? '← Back to the structured editor' : 'Advanced: edit pool_spec as raw JSON →'}
</button>
</p>
)}
{advanced ? (
<>
<label className={styles.label}>Pool spec (JSON)</label>
<textarea className={styles.input} style={{ width: '100%', fontFamily: 'var(--mono)', minHeight: '4rem' }}
placeholder='{"required":["scenario-id"],"random":[{"count":1,"from":["a","b"]}]}'
value={rawPoolSpec} onChange={(e) => setRawPoolSpec(e.target.value)} required />
</>
) : !catalog ? <p className={styles.muted}>Loading scenario catalog…</p> : (
<PoolSpecEditor catalog={catalog} required={required} setRequired={setRequired} pools={pools} setPools={setPools} />
)}
<button className={styles.btn} style={{ marginTop: '0.75rem' }} disabled={busy}>{busy ? 'Creating…' : 'Create'}</button>
</form>
)
}
function AssessmentDetail({ definition, cohorts, onBack }) {
// Phase 4.1 §16 — assessmentSessions is now paginated (was unbounded); `result` holds the
// full { items, page, total, total_pages } envelope, `sessions` stays the plain array for
// the table below so its render logic doesn't need to change.
const [result, setResult] = useState(null)
const [page, setPage] = useState(1)
const [error, setError] = useState(null)
const [notice, setNotice] = useState(null)
const [cohortId, setCohortId] = useState(cohorts?.[0]?.cohort_id || '')
const [busy, setBusy] = useState(false)
const reload = useCallback(() => {
adminApi.assessmentSessions(definition.assessment_definition_id, page).then(setResult).catch((e) => setError(e.message))
}, [definition, page])
useEffect(reload, [reload])
const sessions = result?.items || null
const assign = async () => {
setBusy(true); setError(null); setNotice(null)
try { await adminApi.assignAssessment(definition.assessment_definition_id, cohortId); setNotice('Assigned.') }
catch (e) { setError(e.message) } finally { setBusy(false) }
}
const cancelSession = async (sessionId) => {
if (!window.confirm('Cancel this session?')) return
try { await adminApi.cancelAssessmentSession(sessionId); reload() } catch (e) { setError(e.message) }
}
return (
<>
<button className={styles.linkBtn} onClick={onBack}>&larr; Back to assessments</button>
<h2 style={{ margin: '1rem 0' }}>{definition.title} <span className={styles.muted}>({definition.type})</span></h2>
{error && <div className={styles.errorBox}>{error}</div>}
{notice && <p className={styles.muted}>{notice}</p>}
<div className={styles.card}>
<h3>Assign to cohort</h3>
<div className={styles.formRow}>
<div><select className={styles.select} value={cohortId} onChange={(e) => setCohortId(e.target.value)}>
{cohorts?.map((c) => <option key={c.cohort_id} value={c.cohort_id}>{c.name}</option>)}
</select></div>
<div><button className={styles.btn} disabled={busy || !cohortId} onClick={assign}>Assign</button></div>
</div>
</div>
<div className={styles.card}>
<h3>Sessions</h3>
{!sessions ? <p className={styles.muted}>Loading…</p> : sessions.length === 0 ? <p className={styles.muted}>None yet.</p> : (
<table>
<thead><tr><th>Student</th><th>Status</th><th>Started</th><th>Result</th><th /></tr></thead>
<tbody>
{sessions.map((s) => (
<tr key={s.assessment_session_id}>
<td>{s.display_name}</td>
<td><span className={`${styles.pill} ${s.status === 'completed' ? styles.pillOk : s.status === 'expired' ? styles.pillErr : styles.pillWarn}`}>{s.status}</span></td>
<td>{s.started_at ? new Date(s.started_at).toLocaleString() : '—'}</td>
<td>{s.result ? `${s.result.scenarios_passed}/${s.result.scenarios_total}` : '—'}</td>
<td>{['not_started', 'in_progress'].includes(s.status) && <button className={styles.linkBtn} onClick={() => cancelSession(s.assessment_session_id)}>Cancel</button>}</td>
</tr>
))}
</tbody>
</table>
)}
{result && result.total_pages > 1 && (
<div className={styles.pager}>
<button disabled={page <= 1} onClick={() => setPage((p) => p - 1)}>Previous</button>
<span className={styles.muted}>Page {result.page} of {result.total_pages} ({result.total} total)</span>
<button disabled={page >= result.total_pages} onClick={() => setPage((p) => p + 1)}>Next</button>
</div>
)}
</div>
</>
)
}
+49
View File
@@ -0,0 +1,49 @@
import { useState, useEffect } from 'react'
import { adminApi } from './api'
import styles from './Admin.module.css'
// admin/owner only (enforced server-side too). Phase 4.1 §16 — real server-side pagination
// (platform/api/src/pagination.js) replaces the old flat "latest 200, nothing older" cap.
export default function AuditView() {
const [page, setPage] = useState(1)
const [result, setResult] = useState(null)
const [error, setError] = useState(null)
useEffect(() => {
setResult(null)
adminApi.auditEvents(page).then(setResult).catch((e) => setError(e.message))
}, [page])
const events = result?.items || []
return (
<div className={styles.card}>
<h2>Audit log</h2>
<p className={styles.muted}>{result ? `${result.total} total events` : 'Security/admin events, most recent first.'}</p>
{error && <div className={styles.errorBox}>{error}</div>}
{!result ? <p className={styles.muted}>Loading…</p> : (
<>
<table>
<thead><tr><th>When</th><th>Actor</th><th>Action</th><th>Resource</th><th>Result</th></tr></thead>
<tbody>
{events.map((e) => (
<tr key={e.audit_event_id}>
<td>{new Date(e.created_at).toLocaleString()}</td>
<td>{e.actor_type}{e.actor_id ? ` · ${e.actor_id.slice(0, 8)}` : ''}</td>
<td>{e.action}</td>
<td>{e.resource_type ? `${e.resource_type}${e.resource_id ? ` (${e.resource_id})` : ''}` : '—'}</td>
<td><span className={`${styles.pill} ${e.result === 'success' ? styles.pillOk : styles.pillErr}`}>{e.result}</span></td>
</tr>
))}
</tbody>
</table>
<div className={styles.pager}>
<button disabled={page <= 1} onClick={() => setPage((p) => p - 1)}>Previous</button>
<span className={styles.muted}>Page {result.page} of {result.total_pages}</span>
<button disabled={page >= result.total_pages} onClick={() => setPage((p) => p + 1)}>Next</button>
</div>
</>
)}
</div>
)
}
+221
View File
@@ -0,0 +1,221 @@
import { useState, useEffect, useCallback } from 'react'
import { adminApi } from './api'
import styles from './Admin.module.css'
export default function CohortsView({ role }) {
const [cohorts, setCohorts] = useState(null)
const [selected, setSelected] = useState(null)
const [error, setError] = useState(null)
const canManage = role !== 'instructor' // owner/admin only — a plain instructor works within cohorts already assigned to them
const reload = useCallback(() => {
adminApi.cohorts().then(setCohorts).catch((e) => setError(e.message))
}, [])
useEffect(reload, [reload])
if (selected) return <CohortDetail cohortId={selected} canManage={canManage} onBack={() => { setSelected(null); reload() }} />
return (
<>
{error && <div className={styles.errorBox}>{error}</div>}
{canManage && (
<div className={styles.card}>
<h2>Create cohort</h2>
<CreateCohortForm onCreated={reload} />
</div>
)}
<div className={styles.card}>
<h2>Cohorts</h2>
{!cohorts ? <p className={styles.muted}>Loading…</p> : cohorts.length === 0 ? <p className={styles.muted}>None yet.</p> : (
<table>
<thead><tr><th>Name</th><th>Tier</th><th>Students</th><th>Status</th><th /></tr></thead>
<tbody>
{cohorts.map((c) => (
<tr key={c.cohort_id}>
<td>{c.name}</td>
<td>{c.tier || '—'}</td>
<td>{c.student_count}</td>
<td><span className={`${styles.pill} ${c.status === 'active' ? styles.pillOk : styles.pillMuted}`}>{c.status}</span></td>
<td><button className={styles.linkBtn} onClick={() => setSelected(c.cohort_id)}>Open</button></td>
</tr>
))}
</tbody>
</table>
)}
</div>
</>
)
}
function CreateCohortForm({ onCreated }) {
const [form, setForm] = useState({ cohort_id: '', name: '', tier: '', start_date: '', end_date: '' })
const [error, setError] = useState(null)
const [busy, setBusy] = useState(false)
const submit = async (e) => {
e.preventDefault()
setBusy(true); setError(null)
try {
await adminApi.createCohort(form)
setForm({ cohort_id: '', name: '', tier: '', start_date: '', end_date: '' })
onCreated()
} catch (e) { setError(e.message) } finally { setBusy(false) }
}
return (
<form onSubmit={submit}>
{error && <div className={styles.errorBox}>{error}</div>}
<div className={styles.formRow}>
<div><label className={styles.label}>Cohort ID (slug)</label>
<input className={styles.input} placeholder="root-fall-2026" value={form.cohort_id} onChange={(e) => setForm({ ...form, cohort_id: e.target.value })} required /></div>
<div><label className={styles.label}>Name</label>
<input className={styles.input} placeholder="Root Fall 2026" value={form.name} onChange={(e) => setForm({ ...form, name: e.target.value })} required /></div>
<div><label className={styles.label}>Tier</label>
<input className={styles.input} placeholder="root" value={form.tier} onChange={(e) => setForm({ ...form, tier: e.target.value })} /></div>
<div><label className={styles.label}>Start date</label>
<input className={styles.input} type="date" value={form.start_date} onChange={(e) => setForm({ ...form, start_date: e.target.value })} /></div>
<div><label className={styles.label}>End date</label>
<input className={styles.input} type="date" value={form.end_date} onChange={(e) => setForm({ ...form, end_date: e.target.value })} /></div>
<div><button className={styles.btn} disabled={busy}>{busy ? 'Creating…' : 'Create cohort'}</button></div>
</div>
</form>
)
}
function CohortDetail({ cohortId, canManage, onBack }) {
const [students, setStudents] = useState(null)
const [progress, setProgress] = useState(null)
const [prepResults, setPrepResults] = useState(null)
const [error, setError] = useState(null)
const [busy, setBusy] = useState(false)
const [newStudentName, setNewStudentName] = useState('')
const reload = useCallback(() => {
adminApi.cohortStudents(cohortId).then(setStudents).catch((e) => setError(e.message))
adminApi.cohortProgress(cohortId).then(setProgress).catch(() => {})
}, [cohortId])
useEffect(reload, [reload])
const addStudent = async (e) => {
e.preventDefault()
if (!newStudentName.trim()) return
setBusy(true); setError(null)
try {
await adminApi.createStudent(newStudentName.trim(), cohortId)
setNewStudentName('')
reload()
} catch (e) { setError(e.message) } finally { setBusy(false) }
}
const prepareLabs = async () => {
setBusy(true); setError(null); setPrepResults(null)
try {
const { results } = await adminApi.prepareLabs(cohortId)
setPrepResults(results)
} catch (e) { setError(e.message) } finally { setBusy(false) }
}
const destroyLabs = async () => {
if (!window.confirm(`Destroy all Labs for this cohort? Academic progress is not affected.`)) return
setBusy(true); setError(null)
try {
const { results } = await adminApi.destroyLabs(cohortId)
setPrepResults(results)
} catch (e) { setError(e.message) } finally { setBusy(false) }
}
const assignCurriculum = async () => {
setBusy(true)
try { await adminApi.assignCohortCurriculum(cohortId, 'rootnode-path') } catch (e) { setError(e.message) } finally { setBusy(false) }
}
const readyCount = prepResults?.filter((r) => r.status === 'running').length
return (
<>
<button className={styles.linkBtn} onClick={onBack}>&larr; Back to cohorts</button>
<h2 style={{ margin: '1rem 0' }}>{cohortId}</h2>
{error && <div className={styles.errorBox}>{error}</div>}
<div className={styles.card}>
<h3>Roster ({students?.length ?? '…'})</h3>
{canManage && (
<form onSubmit={addStudent} className={styles.formRow}>
<div><input className={styles.input} placeholder="New student name" value={newStudentName} onChange={(e) => setNewStudentName(e.target.value)} /></div>
<div><button className={styles.btn} disabled={busy}>Add student</button></div>
</form>
)}
{students && (
<table>
<thead><tr><th>Name</th><th>Status</th><th>Last active</th></tr></thead>
<tbody>
{students.map((s) => (
<tr key={s.student_id}>
<td>{s.display_name}</td>
<td><span className={`${styles.pill} ${s.status === 'active' ? styles.pillOk : styles.pillMuted}`}>{s.status}</span></td>
<td>{s.last_active_at ? new Date(s.last_active_at).toLocaleString() : 'never'}</td>
</tr>
))}
</tbody>
</table>
)}
</div>
{canManage && (
<div className={styles.card}>
<h3>Curriculum</h3>
<p className={styles.muted}>RootNode Learning Path (Foundation → Workloads → Networking → Configuration → Troubleshooting → Real Ops → Capstone)</p>
<button className={styles.btnSecondary} onClick={assignCurriculum} disabled={busy}>Assign RootNode Path to cohort</button>
</div>
)}
{progress && (
<div className={styles.card}>
<h3>Cohort progress ({progress.student_count} students)</h3>
{progress.stages.map((s) => (
<div key={s.id} style={{ marginBottom: '0.5rem' }}>
<div style={{ display: 'flex', justifyContent: 'space-between', fontSize: '0.85rem' }}>
<span>{s.name}</span><span className={styles.muted}>{s.pct}%</span>
</div>
<div style={{ background: 'var(--surface3)', borderRadius: 4, height: 6 }}>
<div style={{ background: 'var(--green)', borderRadius: 4, height: 6, width: `${s.pct}%` }} />
</div>
</div>
))}
</div>
)}
<div className={styles.card}>
<h3>Classroom preparation</h3>
<div className={styles.formRow}>
<button className={styles.btn} onClick={prepareLabs} disabled={busy}>{busy ? 'Working…' : 'Prepare Labs'}</button>
<button className={styles.btnDanger} onClick={destroyLabs} disabled={busy}>Destroy Cohort Labs</button>
</div>
{prepResults && (
<>
{readyCount !== undefined && <p className={styles.muted}>{readyCount}/{prepResults.length} READY</p>}
<table>
<thead><tr><th>Student</th><th>Status</th></tr></thead>
<tbody>
{prepResults.map((r) => (
<tr key={r.student_id}>
<td>{r.display_name || r.student_id}</td>
<td><StatusPill status={r.status} /></td>
</tr>
))}
</tbody>
</table>
</>
)}
</div>
</>
)
}
export function StatusPill({ status }) {
const cls = status === 'running' ? styles.pillOk
: status === 'error' ? styles.pillErr
: ['provisioning', 'requested', 'resetting'].includes(status) ? styles.pillWarn
: styles.pillMuted
return <span className={`${styles.pill} ${cls}`}>{status}</span>
}
+180
View File
@@ -0,0 +1,180 @@
import { useState, useEffect, useCallback } from 'react'
import { adminApi } from './api'
import styles from './Admin.module.css'
// Owner-only — the server enforces this too (requireOwner on every /api/admin/instructors*
// route); this component is only ever mounted for an owner, but the backend never trusts that.
export default function InstructorsView({ currentInstructorId }) {
const [instructors, setInstructors] = useState(null)
const [selected, setSelected] = useState(null)
const [error, setError] = useState(null)
const reload = useCallback(() => {
adminApi.instructors().then(setInstructors).catch((e) => setError(e.message))
}, [])
useEffect(reload, [reload])
if (selected) return <InstructorDetail instructorId={selected} isSelf={selected === currentInstructorId} onBack={() => { setSelected(null); reload() }} />
return (
<>
{error && <div className={styles.errorBox}>{error}</div>}
<div className={styles.card}>
<h2>Add instructor</h2>
<CreateInstructorForm onCreated={reload} />
</div>
<div className={styles.card}>
<h2>Instructors</h2>
{!instructors ? <p className={styles.muted}>Loading…</p> : (
<table>
<thead><tr><th>Name</th><th>Email</th><th>Role</th><th>Status</th><th>Last login</th><th /></tr></thead>
<tbody>
{instructors.map((i) => (
<tr key={i.instructor_id}>
<td>{i.display_name}</td>
<td>{i.email}</td>
<td><span className={styles.pill}>{i.role}</span></td>
<td><span className={`${styles.pill} ${i.status === 'active' ? styles.pillOk : styles.pillMuted}`}>{i.status}</span></td>
<td>{i.last_login_at ? new Date(i.last_login_at).toLocaleString() : 'never'}</td>
<td><button className={styles.linkBtn} onClick={() => setSelected(i.instructor_id)}>Open</button></td>
</tr>
))}
</tbody>
</table>
)}
</div>
</>
)
}
function CreateInstructorForm({ onCreated }) {
const [form, setForm] = useState({ email: '', display_name: '', role: 'instructor', password: '' })
const [error, setError] = useState(null)
const [busy, setBusy] = useState(false)
const submit = async (e) => {
e.preventDefault()
setBusy(true); setError(null)
try {
await adminApi.createInstructor(form)
setForm({ email: '', display_name: '', role: 'instructor', password: '' })
onCreated()
} catch (e) { setError(e.message) } finally { setBusy(false) }
}
return (
<form onSubmit={submit}>
{error && <div className={styles.errorBox}>{error}</div>}
<div className={styles.formRow}>
<div><label className={styles.label}>Name</label>
<input className={styles.input} value={form.display_name} onChange={(e) => setForm({ ...form, display_name: e.target.value })} required /></div>
<div><label className={styles.label}>Email</label>
<input className={styles.input} type="email" value={form.email} onChange={(e) => setForm({ ...form, email: e.target.value })} required /></div>
<div><label className={styles.label}>Role</label>
<select className={styles.select} value={form.role} onChange={(e) => setForm({ ...form, role: e.target.value })}>
<option value="instructor">instructor</option>
<option value="admin">admin</option>
<option value="owner">owner</option>
</select></div>
<div><label className={styles.label}>Temporary password (12+ chars)</label>
<input className={styles.input} type="password" value={form.password} onChange={(e) => setForm({ ...form, password: e.target.value })} required minLength={12} /></div>
<div><button className={styles.btn} disabled={busy}>{busy ? 'Adding…' : 'Add instructor'}</button></div>
</div>
</form>
)
}
function InstructorDetail({ instructorId, isSelf, onBack }) {
const [instructor, setInstructor] = useState(null)
const [error, setError] = useState(null)
const [notice, setNotice] = useState(null)
const [busy, setBusy] = useState(false)
const [newPassword, setNewPassword] = useState('')
const reload = useCallback(() => {
adminApi.instructor(instructorId).then(setInstructor).catch((e) => setError(e.message))
}, [instructorId])
useEffect(reload, [reload])
const updateRole = async (role) => {
setBusy(true); setError(null); setNotice(null)
try { await adminApi.updateInstructor(instructorId, { role }); reload() }
catch (e) { setError(e.message) } finally { setBusy(false) }
}
const toggleStatus = async () => {
const status = instructor.status === 'active' ? 'inactive' : 'active'
setBusy(true); setError(null); setNotice(null)
try { await adminApi.updateInstructor(instructorId, { status }); reload() }
catch (e) { setError(e.message) } finally { setBusy(false) }
}
const revokeSessions = async () => {
setBusy(true); setError(null)
try { await adminApi.revokeInstructorSessions(instructorId); setNotice('All sessions revoked.'); reload() }
catch (e) { setError(e.message) } finally { setBusy(false) }
}
const resetPassword = async (e) => {
e.preventDefault()
setBusy(true); setError(null); setNotice(null)
try { await adminApi.resetInstructorPassword(instructorId, newPassword); setNewPassword(''); setNotice('Password reset. All of their sessions were revoked.') }
catch (e) { setError(e.message) } finally { setBusy(false) }
}
if (!instructor) return error ? <div className={styles.errorBox}>{error}</div> : <p className={styles.muted}>Loading…</p>
return (
<>
<button className={styles.linkBtn} onClick={onBack}>&larr; Back to instructors</button>
<h2 style={{ margin: '1rem 0' }}>{instructor.display_name} <span className={styles.muted}>({instructor.email})</span></h2>
{error && <div className={styles.errorBox}>{error}</div>}
{notice && <p className={styles.muted}>{notice}</p>}
<div className={styles.card}>
<h3>Role &amp; status</h3>
<div className={styles.formRow}>
<div><label className={styles.label}>Role</label>
<select className={styles.select} value={instructor.role} onChange={(e) => updateRole(e.target.value)} disabled={busy}>
<option value="instructor">instructor</option>
<option value="admin">admin</option>
<option value="owner">owner</option>
</select></div>
<div><button className={instructor.status === 'active' ? styles.btnDanger : styles.btnSecondary} onClick={toggleStatus} disabled={busy}>
{instructor.status === 'active' ? 'Deactivate' : 'Activate'}
</button></div>
</div>
{isSelf && <p className={styles.muted}>This is your own account — the platform refuses to remove the last active owner, whichever account that is.</p>}
</div>
<div className={styles.card}>
<h3>Sessions</h3>
<p className={styles.muted}>Last login: {instructor.last_login_at ? new Date(instructor.last_login_at).toLocaleString() : 'never'}</p>
{instructor.sessions?.length ? (
<table>
<thead><tr><th>Created</th><th>Last active</th><th>Expires</th><th>Status</th></tr></thead>
<tbody>
{instructor.sessions.map((s) => (
<tr key={s.session_id}>
<td>{new Date(s.created_at).toLocaleString()}</td>
<td>{new Date(s.last_seen_at).toLocaleString()}</td>
<td>{new Date(s.expires_at).toLocaleString()}</td>
<td>{s.revoked_at ? <span className={`${styles.pill} ${styles.pillMuted}`}>revoked</span> : <span className={`${styles.pill} ${styles.pillOk}`}>active</span>}</td>
</tr>
))}
</tbody>
</table>
) : <p className={styles.muted}>No sessions on record.</p>}
<button className={styles.btnDanger} style={{ marginTop: '0.75rem' }} onClick={revokeSessions} disabled={busy}>Revoke all sessions</button>
</div>
<div className={styles.card}>
<h3>Reset password</h3>
<form onSubmit={resetPassword} className={styles.formRow}>
<div><input className={styles.input} type="password" placeholder="New password (12+ chars)" value={newPassword} onChange={(e) => setNewPassword(e.target.value)} minLength={12} required /></div>
<div><button className={styles.btn} disabled={busy}>Set new password</button></div>
</form>
</div>
</>
)
}
+160
View File
@@ -0,0 +1,160 @@
import { useState, useEffect, useCallback } from 'react'
import { adminApi } from './api'
import styles from './Admin.module.css'
// Instructor learning analytics — cohort summary, "needs attention," per-scenario analytics,
// and a student drill-down. Deliberately no leaderboards or scoring language — this is
// instructional visibility, not a ranking (docs/PHASE3B_LEARNING_AND_ASSESSMENT.md §13).
export default function LearningView() {
const [cohorts, setCohorts] = useState(null)
const [cohortId, setCohortId] = useState(null)
const [summary, setSummary] = useState(null)
const [needsAttention, setNeedsAttention] = useState(null)
const [scenarioStats, setScenarioStats] = useState(null)
const [selectedStudent, setSelectedStudent] = useState(null)
const [error, setError] = useState(null)
useEffect(() => { adminApi.cohorts().then((c) => { setCohorts(c); if (c[0]) setCohortId(c[0].cohort_id) }).catch((e) => setError(e.message)) }, [])
const reload = useCallback(() => {
if (!cohortId) return
adminApi.cohortLearning(cohortId).then(setSummary).catch((e) => setError(e.message))
adminApi.needsAttention(cohortId).then(setNeedsAttention).catch((e) => setError(e.message))
adminApi.scenarioAnalytics(cohortId).then(setScenarioStats).catch((e) => setError(e.message))
}, [cohortId])
useEffect(reload, [reload])
if (selectedStudent) return <StudentLearningDetail studentId={selectedStudent} onBack={() => setSelectedStudent(null)} />
return (
<>
{error && <div className={styles.errorBox}>{error}</div>}
<div className={styles.card}>
<label className={styles.label}>Cohort</label>
<select className={styles.select} value={cohortId || ''} onChange={(e) => setCohortId(e.target.value)}>
{cohorts?.map((c) => <option key={c.cohort_id} value={c.cohort_id}>{c.name}</option>)}
</select>
</div>
{summary && (
<div className={styles.card}>
<h2>Cohort Summary</h2>
<div className={styles.statGrid}>
<div className={styles.statCard}><div className={styles.statValue}>{summary.student_count}</div><div className={styles.statLabel}>Students</div></div>
<div className={styles.statCard}><div className={styles.statValue}>{summary.active_learners}</div><div className={styles.statLabel}>Active (7d)</div></div>
<div className={styles.statCard}><div className={styles.statValue}>{summary.overall_completion_pct}%</div><div className={styles.statLabel}>Avg. Completion</div></div>
</div>
<table>
<thead><tr><th>Stage</th><th>Students Complete</th></tr></thead>
<tbody>
{summary.stages.map((s) => (
<tr key={s.id}><td>{s.name}</td><td>{s.completedBy}/{summary.student_count} ({s.pct}%)</td></tr>
))}
</tbody>
</table>
</div>
)}
<div className={styles.card}>
<h2>Needs Attention</h2>
{!needsAttention ? <p className={styles.muted}>Loading…</p> : needsAttention.length === 0 ? <p className={styles.muted}>No signals right now.</p> : (
<table>
<thead><tr><th>Student</th><th>Signal</th><th>Detail</th><th /></tr></thead>
<tbody>
{needsAttention.map((s, i) => (
<tr key={i}>
<td>{s.display_name}</td>
<td><span className={`${styles.pill} ${styles.pillWarn}`}>{s.label}</span></td>
<td className={styles.muted}>{s.detail}</td>
<td><button className={styles.linkBtn} onClick={() => setSelectedStudent(s.student_id)}>View</button></td>
</tr>
))}
</tbody>
</table>
)}
</div>
<div className={styles.card}>
<h2>Scenario Analytics</h2>
{!scenarioStats ? <p className={styles.muted}>Loading…</p> : (
<table>
<thead><tr><th>Scenario</th><th>Completions</th><th>Avg Attempts</th><th>Failures</th><th>Hints Used</th><th>Avg Time</th></tr></thead>
<tbody>
{scenarioStats.map((s) => (
<tr key={s.scenario_id}>
<td>{s.scenario_id}</td><td>{s.completions}</td><td>{s.avg_attempts}</td>
<td>{s.total_failures}</td><td>{s.total_hints_used}</td>
<td>{s.avg_duration_seconds ? `${Math.round(s.avg_duration_seconds / 60)}m` : '—'}</td>
</tr>
))}
</tbody>
</table>
)}
</div>
</>
)
}
function StudentLearningDetail({ studentId, onBack }) {
const [data, setData] = useState(null)
const [error, setError] = useState(null)
useEffect(() => { adminApi.studentLearning(studentId).then(setData).catch((e) => setError(e.message)) }, [studentId])
if (error) return <div className={styles.errorBox}>{error}</div>
if (!data) return <p className={styles.muted}>Loading…</p>
return (
<>
<button className={styles.linkBtn} onClick={onBack}>&larr; Back</button>
<h2 style={{ margin: '1rem 0' }}>{studentId}</h2>
{data.curriculum && (
<div className={styles.card}>
<h3>Curriculum ({data.curriculum.percent_complete}%)</h3>
{data.curriculum.stages.map((s) => (
<div key={s.id} style={{ fontSize: '0.85rem', padding: '0.2rem 0' }}>
{s.complete ? '✅' : s.locked ? '🔒' : '🟡'} {s.name} — {s.completed}/{s.total}
</div>
))}
</div>
)}
<div className={styles.card}>
<h3>Attempts</h3>
<table>
<thead><tr><th>Scenario</th><th>Status</th><th>Validations</th><th>Hints (max level)</th><th>Duration</th></tr></thead>
<tbody>
{data.attempts.map((a, i) => (
<tr key={i}>
<td>{a.scenario_id}</td>
<td><span className={`${styles.pill} ${a.status === 'completed' ? styles.pillOk : styles.pillWarn}`}>{a.status}</span></td>
<td>{a.validation_pass_count} passed, {a.validation_fail_count} failed</td>
<td>{a.hint_count} ({a.highest_hint_level})</td>
<td>{a.duration_seconds ? `${Math.round(a.duration_seconds / 60)}m` : '—'}</td>
</tr>
))}
</tbody>
</table>
</div>
<div className={styles.card}>
<h3>Assessment History</h3>
{data.assessment_sessions.length === 0 ? <p className={styles.muted}>None yet.</p> : (
<table>
<thead><tr><th>Assessment</th><th>Status</th><th>Result</th></tr></thead>
<tbody>
{data.assessment_sessions.map((s) => (
<tr key={s.assessment_session_id}>
<td>{s.title}</td>
<td><span className={`${styles.pill} ${s.status === 'completed' ? styles.pillOk : styles.pillMuted}`}>{s.status}</span></td>
<td>{s.result ? `${s.result.scenarios_passed}/${s.result.scenarios_total}` : '—'}</td>
</tr>
))}
</tbody>
</table>
)}
</div>
</>
)
}
+136
View File
@@ -0,0 +1,136 @@
import { useState, useEffect, useCallback } from 'react'
import { adminApi } from './api'
import { StatusPill } from './CohortsView'
import styles from './Admin.module.css'
export default function StudentsView() {
const [students, setStudents] = useState(null)
const [selected, setSelected] = useState(null)
const [error, setError] = useState(null)
const reload = useCallback(() => {
adminApi.students().then(setStudents).catch((e) => setError(e.message))
}, [])
useEffect(reload, [reload])
if (selected) return <StudentDetail studentId={selected} onBack={() => { setSelected(null); reload() }} />
return (
<div className={styles.card}>
<h2>Students ({students?.length ?? '…'})</h2>
{error && <div className={styles.errorBox}>{error}</div>}
{students && (
<table>
<thead><tr><th>Name</th><th>ID</th><th>Status</th><th>Last active</th><th /></tr></thead>
<tbody>
{students.map((s) => (
<tr key={s.student_id}>
<td>{s.display_name}</td>
<td className={styles.muted}>{s.student_id}</td>
<td><span className={`${styles.pill} ${s.status === 'active' ? styles.pillOk : styles.pillMuted}`}>{s.status}</span></td>
<td>{s.last_active_at ? new Date(s.last_active_at).toLocaleString() : 'never'}</td>
<td><button className={styles.linkBtn} onClick={() => setSelected(s.student_id)}>Open</button></td>
</tr>
))}
</tbody>
</table>
)}
</div>
)
}
function StudentDetail({ studentId, onBack }) {
const [progress, setProgress] = useState(null)
const [attempts, setAttempts] = useState(null)
const [env, setEnv] = useState(null)
const [events, setEvents] = useState(null)
const [error, setError] = useState(null)
const [busy, setBusy] = useState(false)
const reload = useCallback(() => {
adminApi.studentProgress(studentId).then(setProgress).catch(() => {})
// Phase 4.1 §16 — studentAttempts is now paginated; this panel only ever shows the most
// recent page (labeled "Recent attempts," not a full history browser — see
// /api/admin/students/:id/attempts's own pagination for the underlying capability).
adminApi.studentAttempts(studentId).then((r) => setAttempts(r.items)).catch(() => {})
adminApi.studentEnvironment(studentId).then((rows) => setEnv(rows?.find((e) => e.status !== 'destroyed') || null)).catch(() => {})
}, [studentId])
useEffect(reload, [reload])
const withBusy = (fn) => async () => {
setBusy(true); setError(null)
try { await fn(); reload() } catch (e) { setError(e.message) } finally { setBusy(false) }
}
const loadEvents = async () => {
if (!env) return
try { setEvents(await adminApi.environmentEvents(env.environment_id)) } catch (e) { setError(e.message) }
}
return (
<>
<button className={styles.linkBtn} onClick={onBack}>&larr; Back to students</button>
<h2 style={{ margin: '1rem 0' }}>{studentId}</h2>
{error && <div className={styles.errorBox}>{error}</div>}
<div className={styles.card}>
<h3>Environment</h3>
{!env ? <p className={styles.muted}>No Lab provisioned.</p> : (
<>
<p>Status: <StatusPill status={env.status} /> — Lab host: {env.lab_host_id} — created {new Date(env.created_at).toLocaleString()}</p>
{env.last_active_at && <p className={styles.muted}>Last active: {new Date(env.last_active_at).toLocaleString()}</p>}
</>
)}
<div className={styles.formRow} style={{ marginTop: '0.75rem' }}>
{!env && <button className={styles.btn} disabled={busy} onClick={withBusy(() => adminApi.provisionStudentLab(studentId))}>Provision Lab</button>}
{env && <button className={styles.btnSecondary} disabled={busy} onClick={withBusy(() => adminApi.resetEnvironment(env.environment_id))}>Reset/Recreate</button>}
{env && (
<button className={styles.btnDanger} disabled={busy} onClick={withBusy(async () => {
if (window.confirm('Destroy this Lab? Progress is not affected.')) await adminApi.destroyEnvironment(env.environment_id)
})}>Destroy Lab</button>
)}
{env && <button className={styles.linkBtn} onClick={loadEvents}>View event history</button>}
</div>
{events && (
<table style={{ marginTop: '0.75rem' }}>
<thead><tr><th>Event</th><th>Actor</th><th>When</th></tr></thead>
<tbody>
{events.map((e) => <tr key={e.id}><td>{e.event_type}</td><td>{e.actor}</td><td>{new Date(e.created_at).toLocaleString()}</td></tr>)}
</tbody>
</table>
)}
</div>
<div className={styles.card}>
<h3>Curriculum progress</h3>
{!progress ? <p className={styles.muted}>Loading…</p> : progress.stages.map((s) => (
<div key={s.id} style={{ marginBottom: '0.5rem' }}>
<strong>{s.name}</strong>
<div className={styles.muted}>
{s.scenarios.filter((sc) => sc.status === 'completed').length}/{s.scenarios.length} complete
</div>
</div>
))}
</div>
<div className={styles.card}>
<h3>Recent attempts</h3>
{!attempts ? <p className={styles.muted}>Loading…</p> : attempts.length === 0 ? <p className={styles.muted}>None yet.</p> : (
<table>
<thead><tr><th>Scenario</th><th>Result</th><th>Checks</th><th>When</th></tr></thead>
<tbody>
{attempts.map((a) => (
<tr key={a.attempt_id}>
<td>{a.scenario_id}</td>
<td><span className={`${styles.pill} ${a.result === 'passed' ? styles.pillOk : styles.pillWarn}`}>{a.result}</span></td>
<td className={styles.muted}>{a.validation_count ?? '—'}</td>
<td>{new Date(a.created_at).toLocaleString()}</td>
</tr>
))}
</tbody>
</table>
)}
</div>
</>
)
}
+98
View File
@@ -0,0 +1,98 @@
// Admin API client. Uses the rn_instructor session cookie exclusively (same-origin, sent
// automatically) — completely separate credential/cookie from student sessions (see
// platform/api/src/sessions.js). No admin data is ever cached client-side beyond the current
// React state; nothing here reads or writes localStorage.
//
// rn_csrf_i is a deliberately non-HttpOnly cookie (double-submit CSRF pattern, mirrored on the
// student side in frontend/src/platform/api.js) — read here and echoed back as a header on
// every state-changing request.
const CSRF_COOKIE = 'rn_csrf_i'
function getCookie(name) {
return document.cookie.split('; ').find((c) => c.startsWith(`${name}=`))?.split('=')[1]
}
async function json(res) {
if (!res.ok) {
const body = await res.json().catch(() => ({}))
throw new Error(body.message || body.error || `Request failed (${res.status})`)
}
return res.json()
}
function withCsrf(init) {
return { ...init, headers: { ...init.headers, 'X-CSRF-Token': getCookie(CSRF_COOKIE) || '' } }
}
function post(path, body) {
return fetch(path, withCsrf({ method: 'POST', headers: { 'Content-Type': 'application/json' }, body: body ? JSON.stringify(body) : undefined })).then(json)
}
function patch(path, body) {
return fetch(path, withCsrf({ method: 'PATCH', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) })).then(json)
}
function del(path) {
return fetch(path, withCsrf({ method: 'DELETE' })).then(json)
}
function get(path) {
return fetch(path).then(json)
}
export const adminApi = {
// No CSRF token yet at login (nothing authenticated to forge into).
login: (email, password) => fetch('/api/auth/instructor/login', {
method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ email, password }),
}).then(json),
logout: () => post('/api/auth/logout'),
session: () => get('/api/auth/session'),
changePassword: (old_password, new_password) => post('/api/auth/change-password', { old_password, new_password }),
dashboard: () => get('/api/admin/dashboard'),
students: () => get('/api/admin/students'),
createStudent: (display_name, cohort_id) => post('/api/admin/students', { display_name, cohort_id }),
updateStudent: (id, fields) => patch(`/api/admin/students/${id}`, fields),
studentProgress: (id) => get(`/api/admin/students/${id}/progress`),
studentAttempts: (id, page = 1) => get(`/api/admin/students/${id}/attempts?page=${page}`), // Phase 4.1 §16 — paginated, see pagination.js
studentEnvironment: (id) => get(`/api/admin/students/${id}/environment`),
provisionStudentLab: (id) => post(`/api/admin/students/${id}/environment/provision`),
assignStudentCohort: (id, cohort_id) => post(`/api/admin/students/${id}/cohort`, { cohort_id }),
removeStudentCohort: (id, cohortId) => del(`/api/admin/students/${id}/cohort/${cohortId}`),
cohorts: () => get('/api/admin/cohorts'),
createCohort: (fields) => post('/api/admin/cohorts', fields),
updateCohort: (id, fields) => patch(`/api/admin/cohorts/${id}`, fields),
cohortStudents: (id) => get(`/api/admin/cohorts/${id}/students`),
cohortProgress: (id) => get(`/api/admin/cohorts/${id}/progress`),
assignCohortCurriculum: (id, curriculum_path) => post(`/api/admin/cohorts/${id}/curriculum`, { curriculum_path }),
assignCohortInstructor: (id, instructor_id, role_in_cohort) => post(`/api/admin/cohorts/${id}/instructors`, { instructor_id, role_in_cohort }),
prepareLabs: (id) => post(`/api/admin/cohorts/${id}/prepare-labs`),
destroyLabs: (id) => post(`/api/admin/cohorts/${id}/destroy-labs`),
resetEnvironment: (id) => post(`/api/admin/environments/${id}/reset`),
destroyEnvironment: (id) => del(`/api/admin/environments/${id}`),
environmentEvents: (id) => get(`/api/admin/environments/${id}/events`),
instructors: () => get('/api/admin/instructors'),
createInstructor: (fields) => post('/api/admin/instructors', fields),
instructor: (id) => get(`/api/admin/instructors/${id}`),
updateInstructor: (id, fields) => patch(`/api/admin/instructors/${id}`, fields),
revokeInstructorSessions: (id) => post(`/api/admin/instructors/${id}/revoke-sessions`),
resetInstructorPassword: (id, new_password) => post(`/api/admin/instructors/${id}/reset-password`, { new_password }),
auditEvents: (page = 1) => get(`/api/admin/audit-events?page=${page}`), // Phase 4.1 §16 — paginated, see pagination.js
// ── Phase 3B: learning analytics ────────────────────────────────────────────────────────
cohortLearning: (id) => get(`/api/admin/cohorts/${id}/learning`),
studentLearning: (id) => get(`/api/admin/students/${id}/learning`),
scenarioAnalytics: (cohortId) => get(`/api/admin/scenario-analytics${cohortId ? `?cohort_id=${cohortId}` : ''}`),
needsAttention: (cohortId) => get(`/api/admin/cohorts/${cohortId}/needs-attention`),
scenarioCatalog: () => get('/api/admin/scenario-catalog'), // Phase 4.1 §18 — structured assessment editor
// ── Phase 3B: assessment (exam + capstone) management ───────────────────────────────────
assessments: () => get('/api/admin/assessments'),
createAssessment: (fields) => post('/api/admin/assessments', fields),
updateAssessment: (id, fields) => patch(`/api/admin/assessments/${id}`, fields),
assignAssessment: (id, cohort_id) => post(`/api/admin/assessments/${id}/assign`, { cohort_id }),
assessmentSessions: (id, page = 1) => get(`/api/admin/assessments/${id}/sessions?page=${page}`), // Phase 4.1 §16 — paginated, see pagination.js
cancelAssessmentSession: (sessionId) => post(`/api/admin/assessment-sessions/${sessionId}/cancel`),
}
@@ -0,0 +1,47 @@
import { useState, useEffect } from 'react'
// Phase 3B — shows when the student has an active Platform-level assessment (exam or capstone)
// running, distinct from the pre-existing Lab-local exam mode (ExamTimer/ExamSession above,
// which predates this and stays untouched). Polls /api/assessments/active, which 404s/errors
// harmlessly in standalone/single-container dev mode (no Platform API there at all) — so this
// component renders nothing in that mode, by design.
export default function AssessmentBanner() {
const [active, setActive] = useState(null)
const [remaining, setRemaining] = useState(null)
useEffect(() => {
let cancelled = false
function poll() {
fetch('/api/assessments/active').then((r) => r.json()).then((d) => {
if (!cancelled) setActive(d.active ? d : null)
}).catch(() => { if (!cancelled) setActive(null) })
}
poll()
const id = setInterval(poll, 15000)
return () => { cancelled = true; clearInterval(id) }
}, [])
useEffect(() => {
if (!active) return
const tick = () => setRemaining(Math.max(0, new Date(active.expires_at) - Date.now()))
tick()
const id = setInterval(tick, 1000)
return () => clearInterval(id)
}, [active])
if (!active) return null
const m = Math.floor(remaining / 60000), s = Math.floor((remaining % 60000) / 1000)
const completedCount = active.scenarios.filter((sc) => sc.status === 'completed').length
return (
<div style={{
display: 'flex', alignItems: 'center', gap: '1rem', padding: '0.5rem 1rem',
background: 'var(--amber-dim)', color: 'var(--amber)', fontSize: '0.85rem', fontFamily: 'var(--mono)',
borderBottom: '1px solid var(--border)',
}}>
<span>🎯 Assessment Mode: {active.title}</span>
<span>{completedCount}/{active.scenarios.length} scenarios complete</span>
<span>⏱ {m}m {s}s remaining</span>
</div>
)
}
@@ -0,0 +1,91 @@
import { useState, useEffect, useCallback } from 'react'
import ReactMarkdown from 'react-markdown'
import remarkGfm from 'remark-gfm'
import styles from './ScenarioPanel.module.css'
// Phase 3B progressive hints — used when the Gateway has stripped the full `hints` array from
// the scenario payload (the multi-student, Platform-mediated deployment; see
// platform/gateway/src/server.js's GET /lab/:token/api/scenarios/:scenarioId). Hints are
// fetched one at a time from the Platform API, server-enforced sequential (can't skip to hint
// 3), and every reveal is recorded against the student's attempt — none of that is possible
// with the old fully-preloaded array this replaces for this deployment mode.
//
// In standalone/single-container dev mode there's no Gateway rewrite, so `scenario.hints` is
// still the full array and ScenarioPanel never renders this component at all — see the
// `hintCount != null` check at the call site.
export default function ProgressiveHints({ scenarioId, totalHints }) {
const [revealed, setRevealed] = useState([]) // [{level, title, body, command}]
const [loading, setLoading] = useState(false)
const [error, setError] = useState(null)
const [copiedIdx, setCopiedIdx] = useState(null)
const [assessmentHintPolicy, setAssessmentHintPolicy] = useState(null) // null = not in an assessment; 'free'|'disabled'|'flagged'
// A scenario/environment change resets local reveal state — the server is still the source
// of truth (GET /attempt would show the real highest_hint_level), this is just what's
// rendered locally this session. Also checks whether this scenario is part of an active
// Platform assessment, purely to show the policy indicator pre-emptively (the server enforces
// the policy regardless, on every reveal call — this is UX, not the security boundary).
useEffect(() => {
setRevealed([]); setError(null); setAssessmentHintPolicy(null)
fetch('/api/assessments/active').then((r) => r.json()).then((d) => {
if (d.active && d.scenarios?.some((s) => s.scenario_id === scenarioId)) setAssessmentHintPolicy(d.hint_policy)
}).catch(() => {})
}, [scenarioId])
const revealNext = useCallback(async () => {
setLoading(true); setError(null)
try {
const r = await fetch(`api/scenarios/${scenarioId}/hint`, { method: 'POST' })
const d = await r.json()
if (!r.ok) { setError(d.message || d.error || 'Could not reveal a hint.'); return }
setRevealed((h) => [...h, { level: d.level, ...d.hint }])
} catch {
setError('Could not reach the platform to reveal a hint.')
} finally {
setLoading(false)
}
}, [scenarioId])
function copyCmd(cmd, idx) {
if (navigator.clipboard && window.isSecureContext) {
navigator.clipboard.writeText(cmd).then(() => { setCopiedIdx(idx); setTimeout(() => setCopiedIdx(null), 1800) })
}
}
if (totalHints === 0) return <div className={styles.noHints}>No hints available for this scenario.</div>
if (assessmentHintPolicy === 'disabled' && revealed.length === 0) {
return <div className={styles.noHints}>Hints Disabled — this assessment does not allow hints.</div>
}
return (
<div>
{assessmentHintPolicy === 'flagged' && <div className={styles.setupNote} style={{ marginBottom: '0.75rem' }}>Using a hint will be recorded.</div>}
{revealed.map((hint, i) => (
<div key={i} className={styles.hintCard}>
<div className={styles.hintHeader}>
<div className={styles.hintLeft}>
<span className={styles.hintNum}>Hint {hint.level}</span>
<span className={styles.hintTitle}>{hint.title}</span>
</div>
</div>
<div className={styles.hintBody}>
<p className={styles.hintText}><ReactMarkdown remarkPlugins={[remarkGfm]}>{hint.body}</ReactMarkdown></p>
{hint.command && (
<div className={styles.cmdBlock}>
<pre className={styles.cmdPre}>{hint.command}</pre>
<button className={styles.copyBtn} onClick={() => copyCmd(hint.command, i)}>{copiedIdx === i ? '✓ Copied' : 'Copy'}</button>
</div>
)}
</div>
</div>
))}
{error && <div className={styles.validateError}>⚠ {error}</div>}
{revealed.length < totalHints && (
<button className={styles.setupBtn} disabled={loading} onClick={revealNext}>
{loading ? 'Revealing…' : `Reveal Hint ${revealed.length + 1} of ${totalHints}`}
</button>
)}
</div>
)
}
+22 -1
View File
@@ -3,6 +3,7 @@ import ReactMarkdown from 'react-markdown'
import remarkGfm from 'remark-gfm'
import styles from './ScenarioPanel.module.css'
import { useConfirm } from './useConfirm'
import ProgressiveHints from './ProgressiveHints'
// Inline markdown: renders without a wrapping <p> — safe for buttons/spans
const inlineComponents = {
@@ -48,6 +49,15 @@ export default function ScenarioPanel({ scenario, onProgressUpdate, onScenarioSt
setHintsRevealed([])
}, [scenario?.id])
// Phase 3B attempt telemetry — fire-and-forget "scenario started" so the Platform can record
// a real started_at for this working session. Only meaningful under the Platform-multiplied
// deployment (routed through the Gateway); 404s harmlessly in standalone/single-container dev
// mode, where there's no Platform API to reach at all.
useEffect(() => {
if (!scenario?.id) return
fetch(`api/scenarios/${scenario.id}/start`, { method: 'POST' }).catch(() => {})
}, [scenario?.id])
const timeSpentRef = useRef(0)
const lastScenarioIdRef = useRef(null)
@@ -341,7 +351,7 @@ export default function ScenarioPanel({ scenario, onProgressUpdate, onScenarioSt
onClick={() => setTab(t)}
>
{t === 'problem' ? '📄 Problem'
: t === 'hints' ? `💡 Hints (${scenario.hints?.length || 0})`
: t === 'hints' ? `💡 Hints (${scenario.hints?.length ?? scenario.hint_count ?? 0})`
: '✓ Validate'}
</button>
))}
@@ -448,6 +458,15 @@ export default function ScenarioPanel({ scenario, onProgressUpdate, onScenarioSt
{/* HINTS TAB */}
{tab === 'hints' && (
<div className={styles.tabPane} style={{ animation: 'fadeIn 0.2s ease' }}>
{/* Phase 3B: the Gateway strips `hints` from the payload under the Platform-
multiplied deployment (see platform/gateway/src/server.js) — hint_count survives
as the signal to switch to server-gated progressive reveal instead of the old
fully-preloaded array. Standalone/dev mode still has the full array (no Gateway
rewrite happens there) and keeps the original behavior untouched below. */}
{scenario.hints === undefined && typeof scenario.hint_count === 'number' ? (
<ProgressiveHints scenarioId={scenario.id} totalHints={scenario.hint_count} />
) : (
<>
{scenario.hints?.length === 0 && (
<div className={styles.noHints}>No hints available for this scenario.</div>
)}
@@ -483,6 +502,8 @@ export default function ScenarioPanel({ scenario, onProgressUpdate, onScenarioSt
</div>
)
})}
</>
)}
</div>
)}
+12 -1
View File
@@ -1,14 +1,25 @@
import React from 'react'
import ReactDOM from 'react-dom/client'
import App from './App.jsx'
import PlatformShell from './platform/PlatformShell.jsx'
import AdminPortal from './admin/AdminPortal.jsx'
import './index.css'
// Apply saved theme before first paint to avoid flash
const savedTheme = localStorage.getItem('kubekosh-theme') || 'dark'
document.documentElement.setAttribute('data-theme', savedTheme)
// This bundle is served both by the lightweight Platform API (root paths + /admin) and by
// every ephemeral per-student Lab container (reached via the Gateway at /lab/<token>/) — same
// build, same code, split purely by the URL path at runtime. See
// docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md "Student/admin flow" for why: it lets the existing
// Lab UI (<App/>) stay completely unmodified while the Platform Shell and Admin Portal live
// alongside it in the same bundle.
const path = window.location.pathname
const view = path.startsWith('/lab/') ? 'lab' : path.startsWith('/admin') ? 'admin' : 'platform'
ReactDOM.createRoot(document.getElementById('root')).render(
<React.StrictMode>
<App />
{view === 'lab' ? <App /> : view === 'admin' ? <AdminPortal /> : <PlatformShell />}
</React.StrictMode>
)
+40
View File
@@ -0,0 +1,40 @@
import { useState, useEffect } from 'react'
import { api } from './api'
import styles from './Platform.module.css'
const LABELS = {
stage_completed: 'Stage completed', curriculum_completed: 'Curriculum completed',
assessment_completed: 'Assessment completed', capstone_completed: 'Capstone completed',
}
const ICONS = {
stage_completed: '✅', curriculum_completed: '🎓', assessment_completed: '📝', capstone_completed: '🏁',
}
// Not a public certificate — a durable internal record. Phase 4/5 can render a certificate from
// these later (docs/PHASE3B_LEARNING_AND_ASSESSMENT.md §16); nothing here is shareable yet.
export default function AchievementsTab() {
const [achievements, setAchievements] = useState(null)
const [error, setError] = useState(null)
useEffect(() => {
api.achievements().then(setAchievements).catch((e) => setError(e.message))
}, [])
return (
<section className={styles.card}>
<h2>Achievements &amp; History</h2>
{error && <div className={styles.errorBox}>{error}</div>}
{!achievements ? <p className={styles.muted}>Loading…</p> : achievements.length === 0 ? <p className={styles.muted}>Nothing yet — complete a stage to see it here.</p> : (
achievements.map((a) => (
<div key={a.achievement_id} style={{ display: 'flex', alignItems: 'center', gap: '0.6rem', padding: '0.5rem 0', borderBottom: '1px solid var(--border)' }}>
<span style={{ fontSize: '1.2rem' }}>{ICONS[a.achievement_type] || '🏅'}</span>
<div>
<div>{LABELS[a.achievement_type] || a.achievement_type}{a.stage_id ? `: ${a.stage_id}` : ''}{a.metadata?.title ? `: ${a.metadata.title}` : ''}</div>
<div className={styles.muted} style={{ fontSize: '0.78rem' }}>{new Date(a.achieved_at).toLocaleString()}</div>
</div>
</div>
))
)}
</section>
)
}
+113
View File
@@ -0,0 +1,113 @@
import { useState, useEffect, useCallback } from 'react'
import { api } from './api'
import styles from './Platform.module.css'
// Available exams (assigned to the student's cohort), an active session if one is running (with
// a live countdown), and completed history. Starting an exam draws its scenario set
// server-side — the student works through it inside their existing Lab, the same one My
// Learning launches (see docs/PHASE3B_LEARNING_AND_ASSESSMENT.md §9 for why a separate Lab
// per assessment isn't needed).
export default function AssessmentsTab() {
const [assessments, setAssessments] = useState(null)
const [active, setActive] = useState(null)
const [error, setError] = useState(null)
const [busy, setBusy] = useState(false)
const reload = useCallback(() => {
api.assessments().then(setAssessments).catch((e) => setError(e.message))
api.activeAssessment().then(setActive).catch(() => {})
}, [])
useEffect(reload, [reload])
const handleStart = async (definitionId) => {
setBusy(true); setError(null)
try { await api.startAssessment(definitionId); reload() }
catch (e) { setError(e.message) } finally { setBusy(false) }
}
return (
<>
{error && <div className={styles.errorBox}>{error}</div>}
{active?.active && (
<section className={styles.card}>
<h2>Active Assessment</h2>
<p><strong>{active.title}</strong></p>
<CountdownLine expiresAt={active.expires_at} />
<table className={styles.dataTable} style={{ marginTop: '0.75rem' }}>
<thead><tr><th>#</th><th>Scenario</th><th>Status</th></tr></thead>
<tbody>
{active.scenarios.map((s) => (
<tr key={s.scenario_id}>
<td>{s.position + 1}</td><td>{s.scenario_id}</td>
<td>
{s.status === 'completed' ? <span className={`${styles.pill} ${styles.pillOk}`}>completed</span>
: <span className={`${styles.pill} ${styles.pillMuted}`}>{s.status}</span>}
</td>
</tr>
))}
</tbody>
</table>
<p className={styles.hint} style={{ marginTop: '0.75rem' }}>Launch your Lab from My Learning and work through these scenarios there — this list stays in sync automatically.</p>
</section>
)}
<section className={styles.card}>
<h2>Available Assessments</h2>
{!assessments ? <p className={styles.muted}>Loading…</p> : assessments.length === 0 ? <p className={styles.muted}>None assigned to your cohort yet.</p> : (
assessments.map((a) => {
const completedSessions = a.sessions.filter((s) => s.status === 'completed')
return (
<div key={a.assessment_definition_id} style={{ padding: '0.75rem 0', borderBottom: '1px solid var(--border)' }}>
<p><strong>{a.title}</strong> <span className={styles.muted}>· {a.duration_minutes} min</span></p>
{a.description && <p className={styles.muted} style={{ fontSize: '0.85rem' }}>{a.description}</p>}
{completedSessions.length > 0 && (
<p className={styles.hint}>Completed {completedSessions.length}× — best result: {completedSessions.some((s) => s.result?.passed) ? 'passed' : 'not yet passed'}</p>
)}
{!active?.active && (
<button className={styles.secondaryBtn} disabled={busy} onClick={() => handleStart(a.assessment_definition_id)}>
{busy ? 'Starting…' : 'Start'}
</button>
)}
</div>
)
})
)}
</section>
<section className={styles.card}>
<h2>History</h2>
{!assessments ? <p className={styles.muted}>Loading…</p> : (
<table className={styles.dataTable}>
<thead><tr><th>Assessment</th><th>Started</th><th>Status</th><th>Result</th></tr></thead>
<tbody>
{assessments.flatMap((a) => a.sessions).filter((s) => s.status !== 'not_started').sort((a, b) => new Date(b.started_at) - new Date(a.started_at)).map((s) => (
<tr key={s.assessment_session_id}>
<td>{assessments.find((a) => a.sessions.includes(s))?.title}</td>
<td>{s.started_at ? new Date(s.started_at).toLocaleString() : '—'}</td>
<td><StatusPill status={s.status} /></td>
<td>{s.result ? `${s.result.scenarios_passed}/${s.result.scenarios_total}` : '—'}</td>
</tr>
))}
</tbody>
</table>
)}
</section>
</>
)
}
function StatusPill({ status }) {
const cls = status === 'completed' ? styles.pillOk : status === 'expired' || status === 'cancelled' ? styles.pillErr : styles.pillWarn
return <span className={`${styles.pill} ${cls}`}>{status}</span>
}
function CountdownLine({ expiresAt }) {
const [remaining, setRemaining] = useState(() => Math.max(0, new Date(expiresAt) - Date.now()))
useEffect(() => {
const id = setInterval(() => setRemaining(Math.max(0, new Date(expiresAt) - Date.now())), 1000)
return () => clearInterval(id)
}, [expiresAt])
const m = Math.floor(remaining / 60000), s = Math.floor((remaining % 60000) / 1000)
return <p className={styles.statusLine}><span className={styles.spinner} /> Time remaining: {m}m {s}s</p>
}
+89
View File
@@ -0,0 +1,89 @@
import { useState, useEffect, useCallback } from 'react'
import { api } from './api'
import styles from './Platform.module.css'
const STAGE_NAMES = {
foundation: 'Foundation', workloads: 'Workloads', networking: 'Networking',
configuration: 'Configuration', troubleshooting: 'Troubleshooting', 'real-ops': 'RootNode: Real Ops',
}
export default function CapstoneTab() {
const [capstone, setCapstone] = useState(null)
const [error, setError] = useState(null)
const [busy, setBusy] = useState(false)
const reload = useCallback(() => {
api.capstone().then(setCapstone).catch((e) => setError(e.message))
}, [])
useEffect(reload, [reload])
const handleStart = async () => {
setBusy(true); setError(null)
try { await api.startAssessment(capstone.assessment_definition_id); reload() }
catch (e) { setError(e.message) } finally { setBusy(false) }
}
if (error) return <div className={styles.errorBox}>{error}</div>
if (!capstone) return <p className={styles.muted}>Loading…</p>
if (capstone.status === 'locked') {
return (
<section className={styles.card}>
<h2>Capstone</h2>
<div className={styles.lockedBox}>
<p style={{ fontSize: '2rem' }}>🔒</p>
<p>Complete the rest of the curriculum to unlock the RootNode Capstone.</p>
</div>
<h3 style={{ marginTop: '1rem', fontSize: '0.9rem' }}>Prerequisites</h3>
{capstone.blocking_stage_ids.map((id) => (
<div key={id} className={styles.checklistItem}><span>⬜</span> {STAGE_NAMES[id] || id}</div>
))}
</section>
)
}
const activeSession = capstone.sessions.find((s) => s.status === 'in_progress');
const completedSession = capstone.sessions.find((s) => s.status === 'completed' && s.result?.passed);
return (
<section className={styles.card}>
<h2>Capstone {capstone.status === 'completed' ? <span className={`${styles.pill} ${styles.pillOk}`} style={{ marginLeft: '0.5rem' }}>completed</span> : <span className={`${styles.pill} ${styles.pillOk}`} style={{ marginLeft: '0.5rem' }}>unlocked</span>}</h2>
{capstone.definition && (
<>
<p><strong>{capstone.definition.title}</strong> <span className={styles.muted}>· {capstone.definition.duration_minutes} min</span></p>
<p className={styles.muted} style={{ fontSize: '0.85rem' }}>{capstone.definition.description}</p>
</>
)}
{completedSession ? (
<p className={styles.hint} style={{ marginTop: '1rem' }}>
Completed {new Date(completedSession.submitted_at).toLocaleString()} — {completedSession.result.scenarios_passed}/{completedSession.result.scenarios_total} scenarios passed.
</p>
) : activeSession ? (
<p className={styles.hint} style={{ marginTop: '1rem' }}>Your Capstone is in progress — launch your Lab from My Learning to continue.</p>
) : (
<button className={styles.primaryBtn} style={{ marginTop: '1rem' }} disabled={busy} onClick={handleStart}>
{busy ? 'Starting…' : 'Start Capstone'}
</button>
)}
{capstone.sessions.length > 0 && (
<>
<h3 style={{ marginTop: '1.5rem', fontSize: '0.9rem' }}>Attempt History</h3>
<table className={styles.dataTable}>
<thead><tr><th>Started</th><th>Status</th><th>Result</th></tr></thead>
<tbody>
{capstone.sessions.map((s) => (
<tr key={s.assessment_session_id}>
<td>{s.started_at ? new Date(s.started_at).toLocaleString() : '—'}</td>
<td>{s.status}</td>
<td>{s.result ? `${s.result.scenarios_passed}/${s.result.scenarios_total}` : '—'}</td>
</tr>
))}
</tbody>
</table>
</>
)}
</section>
)
}
+72
View File
@@ -0,0 +1,72 @@
import { useState, useEffect } from 'react'
import { api } from './api'
import styles from './Platform.module.css'
// A. Entry screen — RootNode Academy branding, choose your name from roster, no password.
// Explicitly pilot/dev identity, not production auth — see
// docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md "Student identity limitation."
export default function EntryScreen({ onLogin }) {
const [cohorts, setCohorts] = useState([])
const [cohort, setCohort] = useState('')
const [roster, setRoster] = useState([])
const [studentId, setStudentId] = useState('')
const [loading, setLoading] = useState(false)
const [error, setError] = useState(null)
useEffect(() => {
api.cohorts().then(setCohorts).catch((e) => setError(e.message))
}, [])
useEffect(() => {
if (!cohort) { setRoster([]); return }
api.roster(cohort).then(setRoster).catch((e) => setError(e.message))
}, [cohort])
const handleLogin = async () => {
if (!studentId) return
setLoading(true)
setError(null)
try {
const student = await api.login(studentId)
onLogin(student)
} catch (e) {
setError(e.message)
} finally {
setLoading(false)
}
}
return (
<div className={styles.entryWrap}>
<div className={styles.entryCard}>
<div className={styles.brand}>
<span className={styles.brandIcon}>⎈</span> RootNode Academy
</div>
<p className={styles.tagline}>Real-world Kubernetes labs. Pick your name to continue.</p>
<p className={styles.pilotNote}>Pilot identity — no password. Not production auth.</p>
{error && <div className={styles.errorBox}>{error}</div>}
<label className={styles.label}>Cohort</label>
<select className={styles.select} value={cohort} onChange={(e) => { setCohort(e.target.value); setStudentId('') }}>
<option value="">Select a cohort…</option>
{cohorts.map((c) => <option key={c} value={c}>{c}</option>)}
</select>
{cohort && (
<>
<label className={styles.label}>Your name</label>
<select className={styles.select} value={studentId} onChange={(e) => setStudentId(e.target.value)}>
<option value="">Select your name…</option>
{roster.map((s) => <option key={s.student_id} value={s.student_id}>{s.display_name}</option>)}
</select>
</>
)}
<button className={styles.primaryBtn} disabled={!studentId || loading} onClick={handleLogin}>
{loading ? 'Signing in…' : 'Continue'}
</button>
</div>
</div>
)
}
+135
View File
@@ -0,0 +1,135 @@
import { useState, useEffect, useCallback, useRef } from 'react'
import { api } from './api'
import styles from './Platform.module.css'
const POLL_MS = 2500
// The original Phase 2A/2B "Up Next" + Launch Lab / Reset flow, now extended with overall
// percent-complete (Phase 3B's rule-driven curriculum state, GET /api/curriculum/state, instead
// of the older /api/curriculum/path stats-only shape — same stage list, richer per-stage data).
export default function MyLearningTab() {
const [curriculum, setCurriculum] = useState(null)
const [envStatus, setEnvStatus] = useState(null)
const [error, setError] = useState(null)
const [busy, setBusy] = useState(false)
const pollRef = useRef(null)
const loadCurriculum = useCallback(() => {
api.curriculumState().then(setCurriculum).catch((e) => setError(e.message))
}, [])
const loadEnvStatus = useCallback(() => {
api.environmentStatus().then(setEnvStatus).catch((e) => setError(e.message))
}, [])
useEffect(() => { loadCurriculum(); loadEnvStatus() }, [loadCurriculum, loadEnvStatus])
useEffect(() => {
const inFlight = envStatus && ['requested', 'provisioning', 'resetting'].includes(envStatus.status)
if (inFlight && !pollRef.current) {
pollRef.current = setInterval(loadEnvStatus, POLL_MS)
} else if (!inFlight && pollRef.current) {
clearInterval(pollRef.current)
pollRef.current = null
}
return () => { if (pollRef.current) { clearInterval(pollRef.current); pollRef.current = null } }
}, [envStatus, loadEnvStatus])
const handleLaunch = async () => {
setBusy(true); setError(null)
try {
const result = await api.launchEnvironment()
setEnvStatus(result)
if (result.status === 'running' && result.launch_path) window.location.href = result.launch_path
} catch (e) { setError(e.message) } finally { setBusy(false) }
}
const handleReset = async () => {
setBusy(true); setError(null)
try { setEnvStatus(await api.resetEnvironment()) } catch (e) { setError(e.message) } finally { setBusy(false) }
}
const currentStage = curriculum?.stages?.find((s) => s.id === curriculum.current_stage)
return (
<>
{error && <div className={styles.errorBox}>{error}</div>}
<section className={styles.card}>
<h2>Current Stage</h2>
{!curriculum ? <p className={styles.muted}>Loading curriculum…</p> : (
<>
<div style={{ marginBottom: '0.75rem' }}>
<div style={{ display: 'flex', justifyContent: 'space-between', fontSize: '0.85rem', marginBottom: '0.3rem' }}>
<span>Overall progress</span><span className={styles.muted}>{curriculum.percent_complete}%</span>
</div>
<div className={styles.progressBarTrack}><div className={styles.progressBarFill} style={{ width: `${curriculum.percent_complete}%` }} /></div>
</div>
<div className={styles.stageRow}>
{curriculum.stages.map((s) => (
<div key={s.id} className={`${styles.stagePill} ${s.id === curriculum.current_stage ? styles.stagePillActive : ''} ${s.locked ? styles.stagePillLocked : ''}`}>
<span>{s.icon}</span> {s.name}
<span className={styles.stageCount}>{s.completed}/{s.total}</span>
</div>
))}
</div>
{currentStage ? (
<p className={styles.nextUp}>
Up next: <strong>{currentStage.name}</strong>
{currentStage.next_scenario && <> — {currentStage.next_scenario}</>}
</p>
) : (
<p className={styles.nextUp}>All curriculum stages complete — your Capstone is waiting.</p>
)}
</>
)}
</section>
<section className={styles.card}>
<h2>Your Lab</h2>
<EnvironmentPanel status={envStatus} busy={busy} onLaunch={handleLaunch} onReset={handleReset} />
</section>
</>
)
}
function EnvironmentPanel({ status, busy, onLaunch, onReset }) {
const s = status?.status || 'none'
if (s === 'none' || s === 'destroyed') {
return (
<>
<p className={styles.muted}>No lab running yet.</p>
<button className={styles.primaryBtn} disabled={busy} onClick={onLaunch}>
{busy ? 'Launching…' : 'Launch Lab'}
</button>
</>
)
}
if (['requested', 'provisioning', 'resetting'].includes(s)) {
return <p className={styles.statusLine}><span className={styles.spinner} /> {s === 'resetting' ? 'Recreating your lab…' : 'Starting your lab…'}</p>
}
if (s === 'running') {
return (
<>
<p className={styles.statusLine}><span className={styles.dotOk} /> Lab ready.</p>
<div className={styles.btnRow}>
<a className={styles.primaryBtn} href={`${status.launch_path}`}>Continue Lab</a>
<button className={styles.secondaryBtn} disabled={busy} onClick={onReset}>
{busy ? 'Working…' : 'Recreate Lab'}
</button>
</div>
<p className={styles.hint}>Recreating gives you a clean environment. Your progress is saved separately and is never lost.</p>
</>
)
}
if (s === 'error') {
return (
<>
<p className={styles.statusLine}><span className={styles.dotErr} /> {status.error_reason || 'Your lab hit a problem.'}</p>
<button className={styles.primaryBtn} disabled={busy} onClick={onLaunch}>
{busy ? 'Launching…' : 'Try Again'}
</button>
</>
)
}
return <p className={styles.muted}>Unknown state ({s}).</p>
}
+183
View File
@@ -0,0 +1,183 @@
/* Platform Shell styling — reuses the existing design tokens from index.css (--bg, --surface,
--green, --text, etc.) so the entry/dashboard screens feel like the same product as the Lab
UI, not a bolted-on separate app. */
.entryWrap {
height: 100vh;
display: flex;
align-items: center;
justify-content: center;
background: var(--bg);
font-family: var(--sans);
color: var(--text);
}
.entryCard {
width: 380px;
background: var(--surface);
border: 1px solid var(--border);
border-radius: var(--radius-lg);
padding: 2rem;
}
.brand {
font-family: var(--mono);
font-size: 1.3rem;
font-weight: 600;
color: var(--green);
display: flex;
align-items: center;
gap: 0.5rem;
}
.brandIcon { font-size: 1.4rem; }
.tagline { color: var(--text-2); margin: 0.5rem 0 0.25rem; font-size: 0.9rem; }
.pilotNote { color: var(--text-3); font-size: 0.75rem; margin-bottom: 1.25rem; }
.label { display: block; font-size: 0.8rem; color: var(--text-2); margin: 0.9rem 0 0.3rem; }
.select {
width: 100%;
padding: 0.55rem 0.6rem;
background: var(--surface2);
border: 1px solid var(--border2);
border-radius: var(--radius);
color: var(--text);
font-family: var(--sans);
font-size: 0.9rem;
}
.primaryBtn {
margin-top: 1.25rem;
width: 100%;
padding: 0.65rem 1rem;
background: var(--green);
color: #05130a;
border: none;
border-radius: var(--radius);
font-weight: 600;
font-size: 0.9rem;
cursor: pointer;
text-align: center;
text-decoration: none;
display: inline-block;
}
.primaryBtn:disabled { opacity: 0.5; cursor: not-allowed; }
.secondaryBtn {
padding: 0.65rem 1rem;
background: transparent;
color: var(--text);
border: 1px solid var(--border2);
border-radius: var(--radius);
font-size: 0.9rem;
cursor: pointer;
}
.secondaryBtn:disabled { opacity: 0.5; cursor: not-allowed; }
.linkBtn {
background: none;
border: none;
color: var(--text-2);
font-size: 0.85rem;
cursor: pointer;
text-decoration: underline;
}
.errorBox {
background: var(--red-dim);
color: var(--red);
border: 1px solid var(--red);
border-radius: var(--radius);
padding: 0.6rem 0.8rem;
font-size: 0.85rem;
margin-bottom: 1rem;
}
/* ── Dashboard ─────────────────────────────────────────────────────────── */
.dashWrap {
min-height: 100vh;
background: var(--bg);
color: var(--text);
font-family: var(--sans);
padding: 0 0 3rem;
}
.dashHeader {
display: flex;
align-items: center;
justify-content: space-between;
padding: 1.1rem 2rem;
border-bottom: 1px solid var(--border);
}
.headerRight { display: flex; align-items: center; gap: 1rem; }
.studentName { font-weight: 600; }
.card {
max-width: 760px;
margin: 1.5rem auto 0;
background: var(--surface);
border: 1px solid var(--border);
border-radius: var(--radius-lg);
padding: 1.5rem;
}
.card h2 { font-size: 1rem; margin-bottom: 1rem; color: var(--text); }
.muted { color: var(--text-2); font-size: 0.9rem; }
.stageRow { display: flex; flex-wrap: wrap; gap: 0.5rem; margin-bottom: 0.75rem; }
.stagePill {
display: flex; align-items: center; gap: 0.4rem;
padding: 0.4rem 0.7rem;
background: var(--surface2);
border: 1px solid var(--border2);
border-radius: 999px;
font-size: 0.8rem;
color: var(--text-2);
}
.stagePillActive { border-color: var(--green); color: var(--text); }
.stagePillLocked { opacity: 0.5; }
.stageCount { color: var(--text-3); font-family: var(--mono); font-size: 0.75rem; }
.nextUp { font-size: 0.9rem; color: var(--text); }
.statusLine { display: flex; align-items: center; gap: 0.5rem; font-size: 0.9rem; margin-bottom: 0.9rem; }
.btnRow { display: flex; gap: 0.75rem; }
.hint { color: var(--text-3); font-size: 0.78rem; margin-top: 0.75rem; }
.dotOk, .dotErr { width: 8px; height: 8px; border-radius: 50%; display: inline-block; }
.dotOk { background: var(--green); }
.dotErr { background: var(--red); }
.spinner {
width: 12px; height: 12px;
border: 2px solid var(--border2);
border-top-color: var(--green);
border-radius: 50%;
display: inline-block;
animation: spin 0.7s linear infinite;
}
@keyframes spin { to { transform: rotate(360deg); } }
/* ── Learning tabs (Phase 3B) ─────────────────────────────────────────── */
.tabNav { display: flex; gap: 1.25rem; padding: 0 2rem; border-bottom: 1px solid var(--border); }
.tab { color: var(--text-2); cursor: pointer; font-size: 0.9rem; background: none; border: none; padding: 0.75rem 0; }
.tabActive { color: var(--green); border-bottom: 2px solid var(--green); }
.progressBarTrack { background: var(--surface3); border-radius: 4px; height: 8px; overflow: hidden; }
.progressBarFill { background: var(--green); height: 8px; }
table.dataTable { width: 100%; border-collapse: collapse; font-size: 0.85rem; }
.dataTable th { text-align: left; color: var(--text-2); font-weight: 600; padding: 0.4rem 0.5rem; border-bottom: 1px solid var(--border2); font-size: 0.72rem; text-transform: uppercase; }
.dataTable td { padding: 0.5rem 0.5rem; border-bottom: 1px solid var(--border); }
.pill { display: inline-block; padding: 0.15rem 0.55rem; border-radius: 999px; font-size: 0.75rem; font-family: var(--mono); }
.pillOk { background: var(--green-dim); color: var(--green); }
.pillWarn { background: var(--amber-dim); color: var(--amber); }
.pillErr { background: var(--red-dim); color: var(--red); }
.pillMuted { background: var(--surface3); color: var(--text-2); }
.lockedBox { text-align: center; padding: 2rem 1rem; color: var(--text-2); }
.checklistItem { display: flex; align-items: center; gap: 0.5rem; padding: 0.35rem 0; font-size: 0.88rem; }
+28
View File
@@ -0,0 +1,28 @@
import { useState, useEffect } from 'react'
import { api, getCachedDisplay } from './api'
import EntryScreen from './EntryScreen'
import StudentDashboard from './StudentDashboard'
// Top-level Platform Shell: entry screen if no valid session, dashboard otherwise. Session
// validity is decided by the server (GET /api/session/whoami against the rn_session cookie),
// not by anything read from localStorage/sessionStorage — those only cache the display name
// for a snappier first paint, per api.js.
export default function PlatformShell() {
const [student, setStudent] = useState(null)
const [checking, setChecking] = useState(true)
useEffect(() => {
const cached = getCachedDisplay()
if (cached) setStudent(cached) // optimistic paint while we confirm with the server
api.whoami()
.then(({ student_id }) => {
if (!cached || cached.student_id !== student_id) setStudent({ student_id, display_name: cached?.display_name || student_id })
})
.catch(() => setStudent(null))
.finally(() => setChecking(false))
}, [])
if (checking) return null
if (!student) return <EntryScreen onLogin={setStudent} />
return <StudentDashboard student={student} onLogout={() => setStudent(null)} />
}
+77
View File
@@ -0,0 +1,77 @@
import { useState, useEffect } from 'react'
import { api } from './api'
import styles from './Platform.module.css'
// Available/completed scenarios (from curriculum state) + recent attempt history
// (GET /api/attempts/recent) — purely informational, no controls here; the actual work happens
// inside the Lab itself once launched from My Learning.
export default function PracticeTab() {
const [curriculum, setCurriculum] = useState(null)
const [attempts, setAttempts] = useState(null)
const [error, setError] = useState(null)
useEffect(() => {
api.curriculumState().then(setCurriculum).catch((e) => setError(e.message))
api.recentAttempts().then(setAttempts).catch((e) => setError(e.message))
}, [])
const allScenarios = curriculum?.stages.flatMap((s) => Array(s.total).fill(null).map((_, i) => i)) || []
const completedCount = curriculum?.stages.reduce((n, s) => n + s.completed, 0) || 0
const totalCount = allScenarios.length
return (
<>
{error && <div className={styles.errorBox}>{error}</div>}
<section className={styles.card}>
<h2>Scenarios</h2>
{!curriculum ? <p className={styles.muted}>Loading…</p> : (
<table className={styles.dataTable}>
<thead><tr><th>Stage</th><th>Completed</th><th>Status</th></tr></thead>
<tbody>
{curriculum.stages.map((s) => (
<tr key={s.id}>
<td>{s.icon} {s.name}</td>
<td>{s.completed}/{s.total}</td>
<td>
{s.locked ? <span className={`${styles.pill} ${styles.pillMuted}`}>locked</span>
: s.complete ? <span className={`${styles.pill} ${styles.pillOk}`}>complete</span>
: <span className={`${styles.pill} ${styles.pillWarn}`}>in progress</span>}
</td>
</tr>
))}
</tbody>
</table>
)}
{curriculum && <p className={styles.hint} style={{ marginTop: '0.75rem' }}>{completedCount}/{totalCount} scenarios completed overall.</p>}
</section>
<section className={styles.card}>
<h2>Recent Attempts</h2>
{!attempts ? <p className={styles.muted}>Loading…</p> : attempts.length === 0 ? <p className={styles.muted}>No attempts yet — launch your Lab and pick a scenario.</p> : (
<table className={styles.dataTable}>
<thead><tr><th>Scenario</th><th>Status</th><th>Validations</th><th>Hints</th><th>Duration</th></tr></thead>
<tbody>
{attempts.map((a, i) => (
<tr key={i}>
<td>{a.scenario_id}</td>
<td>
{a.status === 'completed' ? <span className={`${styles.pill} ${styles.pillOk}`}>completed</span>
: <span className={`${styles.pill} ${styles.pillWarn}`}>in progress</span>}
</td>
<td>{a.validation_pass_count} passed, {a.validation_fail_count} failed</td>
<td>{a.hint_count || 0}</td>
<td>{a.duration_seconds ? formatDuration(a.duration_seconds) : '—'}</td>
</tr>
))}
</tbody>
</table>
)}
</section>
</>
)
}
function formatDuration(seconds) {
const m = Math.floor(seconds / 60), s = seconds % 60
return `${m}m ${s}s`
}
@@ -0,0 +1,50 @@
import { useState } from 'react'
import { api } from './api'
import styles from './Platform.module.css'
import MyLearningTab from './MyLearningTab'
import PracticeTab from './PracticeTab'
import AssessmentsTab from './AssessmentsTab'
import CapstoneTab from './CapstoneTab'
import AchievementsTab from './AchievementsTab'
const TABS = [
{ key: 'learning', label: 'My Learning' },
{ key: 'practice', label: 'Practice' },
{ key: 'assessments', label: 'Assessments' },
{ key: 'capstone', label: 'Capstone' },
{ key: 'achievements', label: 'Achievements' },
]
// Student home — tabbed as of Phase 3B (docs/PHASE3B_LEARNING_AND_ASSESSMENT.md §10): My
// Learning (the original Phase 2A/2B "Up Next" + Lab launch flow, unchanged in behavior, just
// extracted into its own component), Practice, Assessments, Capstone, Achievements/History.
export default function StudentDashboard({ student, onLogout }) {
const [tab, setTab] = useState('learning')
const handleLogout = () => { api.logout().then(onLogout) }
return (
<div className={styles.dashWrap}>
<header className={styles.dashHeader}>
<div className={styles.brand}><span className={styles.brandIcon}>⎈</span> RootNode Academy</div>
<div className={styles.headerRight}>
<span className={styles.studentName}>{student.display_name}</span>
<button className={styles.linkBtn} onClick={handleLogout}>Sign out</button>
</div>
</header>
<nav className={styles.tabNav}>
{TABS.map((t) => (
<button key={t.key} className={`${styles.tab} ${tab === t.key ? styles.tabActive : ''}`} onClick={() => setTab(t.key)}>
{t.label}
</button>
))}
</nav>
{tab === 'learning' && <MyLearningTab />}
{tab === 'practice' && <PracticeTab />}
{tab === 'assessments' && <AssessmentsTab />}
{tab === 'capstone' && <CapstoneTab />}
{tab === 'achievements' && <AchievementsTab />}
</div>
)
}
+86
View File
@@ -0,0 +1,86 @@
// Platform Shell API client. Identity is a server-controlled session cookie (rn_session,
// HttpOnly — this code can't read it, and doesn't need to; the browser sends it automatically
// on every same-origin request). Roster name-picking is classroom/pilot identification, not
// production authentication (see docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md §5) — what's
// hardened here is the session that results from it: persistent (survives an API restart),
// unforgeable, and checked on every request.
//
// display_name is still cached in sessionStorage purely for UI convenience (showing "Hi,
// Alice" without an extra round trip) — it carries no authority; every actual API call is
// authorized by the cookie alone, not by anything read from here.
//
// rn_csrf_s is a deliberately non-HttpOnly cookie (double-submit CSRF pattern) — this code
// reads it and echoes it back as a header on every state-changing request, proving the request
// came from our own origin's JS (a cross-origin attacker can trigger the cookie but can't read
// its value or attach a matching custom header without CORS, which we don't grant elsewhere).
const DISPLAY_CACHE_KEY = 'rootnode_student_display'
const CSRF_COOKIE = 'rn_csrf_s'
export function getCachedDisplay() {
try { return JSON.parse(sessionStorage.getItem(DISPLAY_CACHE_KEY) || 'null') } catch { return null }
}
function cacheDisplay(student) {
sessionStorage.setItem(DISPLAY_CACHE_KEY, JSON.stringify(student))
}
function clearCachedDisplay() {
sessionStorage.removeItem(DISPLAY_CACHE_KEY)
}
function getCookie(name) {
return document.cookie.split('; ').find((c) => c.startsWith(`${name}=`))?.split('=')[1]
}
async function json(res) {
if (!res.ok) {
const body = await res.json().catch(() => ({}))
throw new Error(body.message || body.error || `Request failed (${res.status})`)
}
return res.json()
}
function mutate(path, body) {
return fetch(path, {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'X-CSRF-Token': getCookie(CSRF_COOKIE) || '' },
body: body ? JSON.stringify(body) : undefined,
}).then(json)
}
export const api = {
cohorts: () => fetch('/api/roster/cohorts').then(json),
roster: (cohort) => fetch(`/api/roster?cohort=${encodeURIComponent(cohort)}`).then(json),
login: async (student_id) => {
// No CSRF token yet at login (nothing authenticated to forge into) — same reasoning as the
// instructor login route.
const student = await fetch('/api/login', {
method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ student_id }),
}).then(json)
cacheDisplay(student)
return student
},
logout: async () => {
await mutate('/api/logout').catch(() => {})
clearCachedDisplay()
},
whoami: () => fetch('/api/session/whoami').then(json),
curriculumPath: () => fetch('/api/curriculum/path').then(json),
curriculumState: () => fetch('/api/curriculum/state').then(json),
environmentStatus: () => fetch('/api/environment/status').then(json),
launchEnvironment: () => mutate('/api/environment/launch'),
resetEnvironment: () => mutate('/api/environment/reset'),
recentAttempts: () => fetch('/api/attempts/recent').then(json),
assessments: () => fetch('/api/assessments').then(json),
activeAssessment: () => fetch('/api/assessments/active').then(json),
assessmentDetail: (sessionId) => fetch(`/api/assessments/${sessionId}`).then(json),
startAssessment: (definitionId) => mutate(`/api/assessments/${definitionId}/start`),
submitAssessment: (sessionId) => mutate(`/api/assessments/${sessionId}/submit`),
capstone: () => fetch('/api/capstone').then(json),
achievements: () => fetch('/api/achievements').then(json),
}
+34
View File
@@ -0,0 +1,34 @@
# Never commit filled secrets — only *.env.template files are tracked.
*.env
!*.env.template
# Docker TLS material — CA/server/client certs and keys, generated by
# infra/scripts/docker-tls/generate-ca.sh and friends. Templates/READMEs stay tracked.
docker/tls/*.pem
docker/tls/*.key
docker/tls/*.crt
docker/tls/*.srl
!docker/tls/**/README.md
# Ansible Vault password files and any decrypted vars
ansible/.vault_pass
ansible/**/vault_plain.yml
# Terraform/OpenTofu state and generated vars (may contain resource IDs/IPs)
# Phase 4.1: these patterns were previously written repo-root-relative (e.g.
# "infra/terraform/...") inside THIS file, which git resolves relative to the .gitignore's own
# directory (infra/) — so they silently matched nothing at all. Found via `git status` showing
# a real .terraform/ provider cache as untracked when it should have been ignored. Fixed by
# dropping the redundant leading "infra/" from every pattern below.
terraform/**/*.tfstate
terraform/**/*.tfstate.*
terraform/**/.terraform/
terraform/**/terraform.tfvars
!terraform/**/terraform.tfvars.example
# Backup output and restore scratch space
backup/output/
backup/restore-test/
# Load test results (local artifacts, not source)
scripts/loadtest/results/
+10
View File
@@ -0,0 +1,10 @@
[defaults]
inventory = inventory/production.ini
roles_path = roles
host_key_checking = True
retry_files_enabled = False
stdout_callback = yaml
[privilege_escalation]
become = True
become_method = sudo
@@ -0,0 +1,22 @@
# Ansible Vault template — copy to production.vault.yml and encrypt it, NEVER commit the
# plaintext or the encrypted file to git (infra/.gitignore already excludes
# **/vault_plain.yml; the encrypted production.vault.yml itself is small/safe to commit IF
# encrypted — see docs/PRODUCTION_DEPLOYMENT.md §10 for the actual decision made). Ansible
# Vault chosen over SOPS/age here because it's already bundled with Ansible (no new tool to
# install/trust) and this project's secret surface is small enough that Vault's simplicity
# outweighs SOPS's per-key-value granularity — revisit if that changes.
#
# cp production.vault.yml.template production.vault.yml
# # fill in real values below
# ansible-vault encrypt production.vault.yml
#
# Reference a vaulted value from group_vars/production.yml or a role like:
# postgres_password: "{{ vault_postgres_password }}"
vault_postgres_password: __REPLACE__
vault_provisioner_shared_secret: __REPLACE_openssl_rand_hex_32__
vault_owner_bootstrap_password: __REPLACE__ # used once, by a human, per docs/RUNBOOK_FIRST_DEPLOYMENT.md
vault_postgres_exporter_password: __REPLACE__
vault_backup_encryption_passphrase: __REPLACE_openssl_rand_base64_32__
vault_backup_offsite_credentials: __REPLACE_e.g._Hetzner_Storage_Box_ssh_key_passphrase_or_S3_secret_key__
vault_grafana_admin_password: __REPLACE__
+33
View File
@@ -0,0 +1,33 @@
# Production variables — referenced by every {{ }} placeholder in infra/docker/, infra/firewall/,
# and the roles below. Fill in the __PLACEHOLDER__ values once the real server exists
# (docs/PRODUCTION_DEPLOYMENT.md §1). Nothing here is a secret — real secrets live in
# infra/environments/production/*.env (gitignored) and Ansible Vault (see
# infra/ansible/group_vars/production.vault.yml.template), never in this plain-vars file.
env: production
academy_domain: __REPLACE_e.g._academy.rootnode.example__
# ── Network topology (Phase 4 §5 — placeholders until real hardware exists) ────────────────
platform_ip: __PLATFORM_VM_IP__
lab_host_ip: __LAB_HOST_VM_IP__
management_cidr: __YOUR_ADMIN_ACCESS_CIDR__ # e.g. a VPN/WireGuard subnet, never 0.0.0.0/0
proxmox_mgmt_cidr: __PROXMOX_HOST_MGMT_CIDR__
lab_subnet_range: "10.42.0.0/16" # matches NO_PROXY_CIDRS already used by the app
lab_router_port: "8443" # Gateway -> Lab Host Router (data plane) — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3
# ── Storage (Phase 4 §4 — a decision framework, not a final layout; see docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §4)
rootnode_data_root: /srv/rootnode-data # expected to be a ZFS dataset mountpoint on the real host
# ── Registry / images (infra/docker/IMAGE_STRATEGY.md) ─────────────────────────────────────
registry: git.synthossystems.com
image_tag: __GIT_SHA_OR_RELEASE_TAG__
postgres_exporter_user: rootnode_exporter
# ── Observability (infra/monitoring/) ───────────────────────────────────────────────────────
prometheus_retention_days: 30
alertmanager_webhook_url: __REPLACE_e.g._uptime_kuma_or_ntfy_webhook__
# ── Backup (infra/backup/) ───────────────────────────────────────────────────────────────────
backup_retention_days: 30
backup_offsite_target: __REPLACE_e.g._Hetzner_Storage_Box_path_or_S3_bucket__
+22
View File
@@ -0,0 +1,22 @@
# Staging variables — same shape as production.yml, lower stakes. See
# infra/environments/staging/README.md.
env: staging
academy_domain: __REPLACE_e.g._staging.academy.rootnode.example_or_unset_for_no_public_TLS__
platform_ip: __STAGING_PLATFORM_IP__
lab_host_ip: __STAGING_LAB_HOST_IP__
management_cidr: __YOUR_ADMIN_ACCESS_CIDR__
proxmox_mgmt_cidr: __PROXMOX_HOST_MGMT_CIDR__
lab_subnet_range: "10.42.0.0/16"
rootnode_data_root: /srv/rootnode-data-staging
registry: git.synthossystems.com
image_tag: __STAGING_TAG_e.g._staging-<sha>__
prometheus_retention_days: 7
alertmanager_webhook_url: __OPTIONAL_staging_can_skip_alerting__
backup_retention_days: 7
backup_offsite_target: __OPTIONAL_staging_backups_are_lower_priority__
+16
View File
@@ -0,0 +1,16 @@
# Production inventory. IPs/hostnames are placeholders — fill in once the real VMs exist
# (docs/PRODUCTION_DEPLOYMENT.md §1). This file is safe to commit (no secrets, just topology).
[platform]
rn-platform-01 ansible_host=__PLATFORM_VM_IP__
[lab_host]
rn-lab-host-01 ansible_host=__LAB_HOST_VM_IP__
[rootnode:children]
platform
lab_host
[rootnode:vars]
ansible_user=rootnode-deploy
ansible_ssh_common_args='-o StrictHostKeyChecking=yes'
+16
View File
@@ -0,0 +1,16 @@
# Staging inventory — same group shape as production, pointed at homelab VMs/containers or a
# resource-constrained slice of the dedicated server once it exists (infra/environments/staging/README.md).
[platform]
rn-platform-staging ansible_host=__STAGING_PLATFORM_IP__
[lab_host]
rn-lab-host-staging ansible_host=__STAGING_LAB_HOST_IP__
[rootnode:children]
platform
lab_host
[rootnode:vars]
ansible_user=rootnode-deploy
env=staging
+63
View File
@@ -0,0 +1,63 @@
---
# Deploys the backup script + a daily cron job on rn-platform-01. The script itself
# (infra/backup/backup-postgres.sh) is identical to what infra/backup/test-backup-restore.sh
# exercises locally — same code path, just pointed at real production values here.
- name: Install postgresql-client + gpg (backup script dependencies)
ansible.builtin.apt:
name: [postgresql-client, gnupg]
state: present
- name: Create backup directories
ansible.builtin.file:
path: "{{ item }}"
state: directory
owner: root
group: root
mode: "0700"
loop:
- /opt/rootnode/backup
- "{{ rootnode_data_root }}/backups"
- name: Deploy backup script
ansible.builtin.copy:
src: "{{ role_path }}/../../../backup/backup-postgres.sh"
dest: /opt/rootnode/backup/backup-postgres.sh
mode: "0750"
- name: Deploy restore + verify scripts (not run by cron, kept alongside for the runbook)
ansible.builtin.copy:
src: "{{ role_path }}/../../../backup/{{ item }}"
dest: "/opt/rootnode/backup/{{ item }}"
mode: "0750"
loop:
- restore-postgres.sh
- verify-restore.sh
- name: Deploy the backup env file (PLATFORM_DB_URL, encryption passphrase, offsite target)
ansible.builtin.template:
src: backup.env.j2
dest: /etc/rootnode/backup.env
owner: root
group: root
mode: "0600"
- name: Deploy cron wrapper (sources the env file, writes to syslog too)
ansible.builtin.template:
src: run-backup.sh.j2
dest: /opt/rootnode/backup/run-backup.sh
mode: "0750"
- name: Install daily backup cron job
ansible.builtin.cron:
name: rootnode-postgres-backup
user: root
minute: "15"
hour: "3"
job: /opt/rootnode/backup/run-backup.sh
- name: Ensure node-exporter's textfile collector directory exists (backup metrics land here)
ansible.builtin.file:
path: /var/lib/node-exporter/textfile
state: directory
mode: "0755"
@@ -0,0 +1,6 @@
PLATFORM_DB_URL=postgres://rootnode_platform:{{ vault_postgres_password }}@localhost:5432/rootnode_platform
BACKUP_DIR={{ rootnode_data_root }}/backups
BACKUP_ENCRYPTION_PASSPHRASE={{ vault_backup_encryption_passphrase }}
BACKUP_RETENTION_DAYS={{ backup_retention_days }}
BACKUP_OFFSITE_TARGET={{ backup_offsite_target }}
METRICS_TEXTFILE_DIR=/var/lib/node-exporter/textfile
@@ -0,0 +1,9 @@
#!/usr/bin/env bash
# Cron wrapper — sources the deployed env file and runs the real backup script, logging to
# syslog (visible via journalctl and shipped to Loki by promtail like every other unit) in
# addition to whatever cron itself captures.
set -euo pipefail
set -a
source /etc/rootnode/backup.env
set +a
exec /opt/rootnode/backup/backup-postgres.sh 2>&1 | systemd-cat -t rootnode-backup
@@ -0,0 +1,3 @@
---
common_timezone: UTC
common_unattended_upgrades: true
@@ -0,0 +1,35 @@
---
# Shared Docker Engine install — used by both platform and lab-host roles (import_tasks from
# common, not duplicated). Follows Docker's own official apt-repo installation method.
- name: Add Docker's GPG key
ansible.builtin.get_url:
url: https://download.docker.com/linux/ubuntu/gpg
dest: /etc/apt/keyrings/docker.asc
mode: "0644"
register: docker_gpg
retries: 3
delay: 5
until: docker_gpg is succeeded
- name: Add Docker apt repository
ansible.builtin.apt_repository:
repo: "deb [arch={{ ansible_architecture | replace('x86_64','amd64') }} signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable"
state: present
filename: docker
- name: Install Docker Engine + Compose plugin
ansible.builtin.apt:
name:
- docker-ce
- docker-ce-cli
- containerd.io
- docker-compose-plugin
state: present
update_cache: true
- name: Enable and start Docker
ansible.builtin.systemd:
name: docker
enabled: true
state: started
+58
View File
@@ -0,0 +1,58 @@
---
# Baseline hardening applied to both rn-platform-01 and rn-lab-host-01 before anything
# RootNode-specific — idempotent, safe to re-run (Phase 4 §20 requirement).
- name: Set timezone
community.general.timezone:
name: "{{ common_timezone }}"
- name: Install base packages
ansible.builtin.apt:
name:
- curl
- ca-certificates
- gnupg
- iptables
- iptables-persistent
- unattended-upgrades
- fail2ban
- jq
state: present
update_cache: true
cache_valid_time: 3600
- name: Enable unattended security upgrades
ansible.builtin.copy:
dest: /etc/apt/apt.conf.d/20auto-upgrades
content: |
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
mode: "0644"
when: common_unattended_upgrades
- name: Create the rootnode-deploy service account (if not the connecting user already)
ansible.builtin.user:
name: rootnode-deploy
shell: /bin/bash
groups: docker
append: true
create_home: true
register: deploy_user_result
failed_when: false # non-fatal — some hosts may already use this as ansible_user
- name: Ensure fail2ban is enabled for SSH
ansible.builtin.service:
name: fail2ban
state: started
enabled: true
- name: Create /opt/rootnode and /etc/rootnode base directories
ansible.builtin.file:
path: "{{ item }}"
state: directory
owner: root
group: root
mode: "0750"
loop:
- /opt/rootnode
- /etc/rootnode
@@ -0,0 +1,6 @@
---
# Local path (on the machine running ansible-playbook, NOT on either VM) holding the cert
# bundle generated by infra/scripts/docker-tls/generate-docker-tls.sh. The CA private key
# (ca-key.pem) is deliberately never copied to either host by this role — only its public
# ca.pem, plus the one cert/key pair each host actually needs.
docker_tls_local_bundle_dir: "{{ lookup('env', 'ROOTNODE_TLS_BUNDLE_DIR') | default('/does/not/exist', true) }}"
@@ -0,0 +1,22 @@
---
- name: restart docker with tls
ansible.builtin.systemd:
name: docker
state: restarted
daemon_reload: true
# Phase 4.1 — these restart the whole per-host compose wrapper unit (no per-container systemd
# units exist in this project's convention, see the `platform`/`lab-host` roles), which is
# heavier than a surgical single-container restart but consistent with how every other
# config-changed-so-restart-the-stack case is already handled here.
- name: restart lab-router
ansible.builtin.systemd:
name: rootnode-lab-host.service
state: restarted
when: inventory_hostname in groups['lab_host']
- name: restart gateway
ansible.builtin.systemd:
name: rootnode-platform.service
state: restarted
when: inventory_hostname in groups['platform']
@@ -0,0 +1,136 @@
---
# Deploys Docker TLS material generated OFFLINE by infra/scripts/docker-tls/generate-docker-tls.sh
# — this role never generates key material itself, and the CA private key never touches either
# VM (see infra/docker/tls/README.md). Fails loudly if the operator forgot to set
# ROOTNODE_TLS_BUNDLE_DIR / run the generation script first, rather than silently skipping and
# leaving the Provisioner unable to reach rn-lab-host-01.
- name: Verify the local TLS bundle directory exists
ansible.builtin.stat:
path: "{{ docker_tls_local_bundle_dir }}"
register: bundle_dir
delegate_to: localhost
become: false
- name: Fail fast if the TLS bundle wasn't generated
ansible.builtin.fail:
msg: >-
ROOTNODE_TLS_BUNDLE_DIR ({{ docker_tls_local_bundle_dir }}) does not exist. Run
infra/scripts/docker-tls/generate-docker-tls.sh first, then re-run with
ROOTNODE_TLS_BUNDLE_DIR=<output dir> in the environment.
when: not bundle_dir.stat.exists
- name: Create /etc/rootnode/docker-tls
ansible.builtin.file:
path: /etc/rootnode/docker-tls
state: directory
owner: root
group: root
mode: "0700"
- name: Deploy CA cert (public, both hosts)
ansible.builtin.copy:
src: "{{ docker_tls_local_bundle_dir }}/ca.pem"
dest: /etc/rootnode/docker-tls/ca.pem
owner: root
group: root
mode: "0444"
- name: Deploy server cert/key (rn-lab-host-01 only)
ansible.builtin.copy:
src: "{{ docker_tls_local_bundle_dir }}/{{ item.src }}"
dest: "/etc/rootnode/docker-tls/{{ item.dest }}"
owner: root
group: root
mode: "{{ item.mode }}"
loop:
- { src: server-cert.pem, dest: server-cert.pem, mode: "0444" }
- { src: server-key.pem, dest: server-key.pem, mode: "0400" }
when: inventory_hostname in groups['lab_host']
notify: restart docker with tls
- name: Deploy Provisioner client cert/key (rn-platform-01 only)
ansible.builtin.copy:
src: "{{ docker_tls_local_bundle_dir }}/{{ item.src }}"
dest: "/etc/rootnode/docker-tls/{{ item.dest }}"
owner: root
group: root
mode: "{{ item.mode }}"
loop:
- { src: provisioner-client-cert.pem, dest: provisioner-client-cert.pem, mode: "0444" }
- { src: provisioner-client-key.pem, dest: provisioner-client-key.pem, mode: "0400" }
when: inventory_hostname in groups['platform']
- name: Configure dockerd for TLS (rn-lab-host-01 only)
ansible.builtin.template:
src: daemon.json.j2
dest: /etc/docker/daemon.json
owner: root
group: root
mode: "0644"
when: inventory_hostname in groups['lab_host']
notify: restart docker with tls
- name: Override the Docker systemd unit's ExecStart (clears the default -H fd:// flag)
ansible.builtin.file:
path: /etc/systemd/system/docker.service.d
state: directory
mode: "0755"
when: inventory_hostname in groups['lab_host']
- name: Deploy the ExecStart override
ansible.builtin.template:
src: docker-override.conf.j2
dest: /etc/systemd/system/docker.service.d/override.conf
owner: root
group: root
mode: "0644"
when: inventory_hostname in groups['lab_host']
notify: restart docker with tls
# ── Phase 4.1: the DATA-plane mTLS pair (Gateway <-> Lab Host Router). Same CA as the
# CONTROL-plane pair above, separate directory (/etc/rootnode/lab-router-tls) so the two
# concerns stay bind-mount-separable — the `lab-router`/`gateway` containers only ever need
# their own pair mounted in, never the Docker TLS one. See
# docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3.
- name: Create /etc/rootnode/lab-router-tls
ansible.builtin.file:
path: /etc/rootnode/lab-router-tls
state: directory
owner: root
group: root
mode: "0700"
- name: Deploy CA cert for the data-plane pair (public, both hosts)
ansible.builtin.copy:
src: "{{ docker_tls_local_bundle_dir }}/ca.pem"
dest: /etc/rootnode/lab-router-tls/ca.pem
owner: root
group: root
mode: "0444"
- name: Deploy Lab Host Router server cert/key (rn-lab-host-01 only)
ansible.builtin.copy:
src: "{{ docker_tls_local_bundle_dir }}/{{ item.src }}"
dest: "/etc/rootnode/lab-router-tls/{{ item.dest }}"
owner: root
group: root
mode: "{{ item.mode }}"
loop:
- { src: router-server-cert.pem, dest: router-server-cert.pem, mode: "0444" }
- { src: router-server-key.pem, dest: router-server-key.pem, mode: "0400" }
when: inventory_hostname in groups['lab_host']
notify: restart lab-router
- name: Deploy Gateway client cert/key (rn-platform-01 only)
ansible.builtin.copy:
src: "{{ docker_tls_local_bundle_dir }}/{{ item.src }}"
dest: "/etc/rootnode/lab-router-tls/{{ item.dest }}"
owner: root
group: root
mode: "{{ item.mode }}"
loop:
- { src: gateway-client-cert.pem, dest: gateway-client-cert.pem, mode: "0444" }
- { src: gateway-client-key.pem, dest: gateway-client-key.pem, mode: "0400" }
when: inventory_hostname in groups['platform']
notify: restart gateway
@@ -0,0 +1,13 @@
{
"tls": true,
"tlsverify": true,
"tlscacert": "/etc/rootnode/docker-tls/ca.pem",
"tlscert": "/etc/rootnode/docker-tls/server-cert.pem",
"tlskey": "/etc/rootnode/docker-tls/server-key.pem",
"hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2376"],
"storage-driver": "overlay2",
"default-address-pools": [
{ "base": "10.42.0.0/16", "size": 24 }
]
}
@@ -0,0 +1,7 @@
# Docker's systemd unit specifies `-H fd://` on ExecStart by default, which conflicts with
# setting `hosts` in daemon.json ("specified both as a flag and in the configuration file").
# This override clears ExecStart and re-specifies it with no -H flag, letting daemon.json's
# `hosts` list be the only source of truth — the documented fix for this exact conflict.
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd
@@ -0,0 +1,4 @@
---
- name: reload firewall
ansible.builtin.command: netfilter-persistent reload
changed_when: true
@@ -0,0 +1,28 @@
---
# Deploys the matching infra/firewall/*.rules file for this host's group and loads it. Same
# ruleset, same variables as infra/firewall/validate-firewall.sh renders locally — this task
# just uses Ansible's own templating instead of sed, against the real inventory values.
- name: Determine which ruleset applies to this host
ansible.builtin.set_fact:
firewall_ruleset_name: "{{ 'rn-platform-01' if inventory_hostname in groups['platform'] else 'rn-lab-host-01' }}"
- name: Render firewall ruleset for this host
ansible.builtin.template:
src: "{{ role_path }}/../../../firewall/{{ firewall_ruleset_name }}.rules"
dest: /etc/iptables/rules.v4.rootnode
owner: root
group: root
mode: "0600"
notify: reload firewall
- name: Point iptables-persistent at the RootNode ruleset
ansible.builtin.file:
src: /etc/iptables/rules.v4.rootnode
dest: /etc/iptables/rules.v4
state: link
force: true
notify: reload firewall
- name: Flush handlers so the firewall is active before later roles assume it
ansible.builtin.meta: flush_handlers
@@ -0,0 +1,6 @@
---
- name: restart rootnode-lab-host
ansible.builtin.systemd:
name: rootnode-lab-host
state: restarted
daemon_reload: true
@@ -0,0 +1,72 @@
---
# rn-lab-host-01: Docker Engine (TLS config comes from the docker-tls role, run before this
# one), the egress proxy + monitoring agents compose stack, kernel prerequisites for nested
# k3s in privileged Lab containers, and Docker daemon storage/limits configuration.
- name: Install Docker Engine
ansible.builtin.import_tasks: "{{ role_path }}/../common/tasks/docker-engine.yml"
- name: Kernel prerequisites for nested k3s (privileged containers)
ansible.posix.sysctl:
name: "{{ item.name }}"
value: "{{ item.value }}"
sysctl_set: true
state: present
reload: true
loop:
- { name: net.ipv4.ip_forward, value: "1" }
- { name: fs.inotify.max_user_watches, value: "524288" } # k3s/kubelet watch a lot of files
- { name: fs.inotify.max_user_instances, value: "512" }
- { name: vm.max_map_count, value: "262144" } # matches upstream k3s/etcd guidance
- name: Ensure cgroup v2 controllers needed by nested k3s are available
ansible.builtin.command: cat /sys/fs/cgroup/cgroup.controllers
register: cgroup_controllers
changed_when: false
failed_when: false
- name: Warn if cgroup v2 controllers look incomplete (informational only)
ansible.builtin.debug:
msg: >-
cgroup.controllers reports: {{ cgroup_controllers.stdout | default('unreadable') }}.
Expect at least cpu, memory, pids, io for reliable nested k3s — verify manually if any
are missing (see docs/PRODUCTION_VALIDATION_CHECKLIST.md "Lab boot" section).
when: cgroup_controllers.rc == 0
- name: Create data/config directories
ansible.builtin.file:
path: "{{ item }}"
state: directory
owner: root
group: docker
mode: "0750"
loop:
- /opt/rootnode/lab-host
- "{{ rootnode_data_root }}/lab-work" # disposable — see docs/BACKUP_DR.md, never backed up
- name: Render production docker-compose.yml
ansible.builtin.template:
src: "{{ role_path }}/../../../docker/lab-host/docker-compose.production.yml"
dest: /opt/rootnode/lab-host/docker-compose.yml
owner: root
group: root
mode: "0640"
notify: restart rootnode-lab-host
- name: Deploy systemd unit
ansible.builtin.template:
src: rootnode-lab-host.service.j2
dest: /etc/systemd/system/rootnode-lab-host.service
mode: "0644"
notify: restart rootnode-lab-host
- name: Enable and start rootnode-lab-host
ansible.builtin.systemd:
name: rootnode-lab-host
enabled: true
state: started
daemon_reload: true
# Storage-driver/networking daemon options live in the docker-tls role's daemon.json.j2 —
# Docker only ever reads ONE /etc/docker/daemon.json, so that's the single source of truth for
# this host's whole daemon config, not just its TLS settings (see that role's template).
@@ -0,0 +1,17 @@
[Unit]
Description=RootNode Academy Lab Host services (egress proxy + monitoring agents)
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target
[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/rootnode/lab-host
ExecStart=/usr/bin/docker compose -f docker-compose.yml up -d --remove-orphans
ExecStop=/usr/bin/docker compose -f docker-compose.yml down
ExecReload=/usr/bin/docker compose -f docker-compose.yml up -d --remove-orphans
TimeoutStartSec=180
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,16 @@
---
- name: reload prometheus
ansible.builtin.command: docker compose -f /opt/rootnode/platform/docker-compose.yml restart prometheus
changed_when: true
- name: reload alertmanager
ansible.builtin.command: docker compose -f /opt/rootnode/platform/docker-compose.yml restart alertmanager
changed_when: true
- name: restart platform promtail
ansible.builtin.command: docker compose -f /opt/rootnode/platform/docker-compose.yml restart promtail
changed_when: true
- name: restart lab-host promtail
ansible.builtin.command: docker compose -f /opt/rootnode/lab-host/docker-compose.yml restart promtail
changed_when: true
@@ -0,0 +1,75 @@
---
# Deploys infra/monitoring/ configs to whichever host needs them. Named monitoring-agent for
# historical reasons in this repo's role list, but it deploys both the "agent" pieces
# (node-exporter/cadvisor/promtail configs, already referenced directly by the platform/
# lab-host compose files) and the central stack's config (Prometheus/Alertmanager/Loki/Grafana
# on rn-platform-01 only).
- name: Create /opt/rootnode/monitoring (rn-platform-01)
ansible.builtin.file:
path: /opt/rootnode/monitoring/rules
state: directory
mode: "0755"
when: inventory_hostname in groups['platform']
- name: Deploy Prometheus config
ansible.builtin.template:
src: "{{ role_path }}/../../../monitoring/prometheus/prometheus.yml.j2"
dest: /opt/rootnode/monitoring/prometheus.yml
mode: "0644"
when: inventory_hostname in groups['platform']
notify: reload prometheus
- name: Deploy alert rules
ansible.builtin.copy:
src: "{{ role_path }}/../../../monitoring/prometheus/rules/rootnode.yml"
dest: /opt/rootnode/monitoring/rules/rootnode.yml
mode: "0644"
when: inventory_hostname in groups['platform']
notify: reload prometheus
- name: Deploy blackbox_exporter config
ansible.builtin.copy:
src: "{{ role_path }}/../../../monitoring/prometheus/blackbox.yml"
dest: /opt/rootnode/monitoring/blackbox.yml
mode: "0644"
when: inventory_hostname in groups['platform']
- name: Deploy Alertmanager config
ansible.builtin.template:
src: "{{ role_path }}/../../../monitoring/alertmanager/alertmanager.yml.j2"
dest: /opt/rootnode/monitoring/alertmanager.yml
mode: "0644"
when: inventory_hostname in groups['platform']
notify: reload alertmanager
- name: Deploy Loki config
ansible.builtin.copy:
src: "{{ role_path }}/../../../monitoring/loki/loki-config.yml"
dest: /opt/rootnode/monitoring/loki-config.yml
mode: "0644"
when: inventory_hostname in groups['platform']
- name: Deploy Grafana provisioning + dashboards
ansible.builtin.copy:
src: "{{ role_path }}/../../../monitoring/grafana/{{ item }}/"
dest: "/opt/rootnode/monitoring/grafana/{{ item }}/"
mode: "0644"
loop: [provisioning, dashboards]
when: inventory_hostname in groups['platform']
- name: Deploy platform-side promtail config
ansible.builtin.copy:
src: "{{ role_path }}/../../../monitoring/promtail/promtail-config.platform.yml"
dest: /opt/rootnode/monitoring/promtail-config.yml
mode: "0644"
when: inventory_hostname in groups['platform']
notify: restart platform promtail
- name: Deploy lab-host promtail config
ansible.builtin.template:
src: "{{ role_path }}/../../../monitoring/promtail/promtail-config.lab-host.yml.j2"
dest: /opt/rootnode/lab-host/promtail-config.yml
mode: "0644"
when: inventory_hostname in groups['lab_host']
notify: restart lab-host promtail
@@ -0,0 +1,6 @@
---
- name: restart rootnode-platform
ansible.builtin.systemd:
name: rootnode-platform
state: restarted
daemon_reload: true
@@ -0,0 +1,85 @@
---
# rn-platform-01: Docker Engine, directory structure, rendered compose file + env files,
# Postgres tuning overlay, reverse proxy config, and the systemd unit that runs it all.
# Docker TLS material and the host firewall are separate roles (docker-tls, firewall) run
# before this one in site.yml — this role assumes both are already in place.
- name: Install Docker Engine
ansible.builtin.import_tasks: "{{ role_path }}/../common/tasks/docker-engine.yml"
- name: Create data/config directories
ansible.builtin.file:
path: "{{ item }}"
state: directory
owner: root
group: docker
mode: "0750"
loop:
- "{{ rootnode_data_root }}/postgres"
- "{{ rootnode_data_root }}/backups"
- /opt/rootnode/platform
- /opt/rootnode/platform/postgresql.conf.d
- /etc/rootnode/provisioner-config
- name: Deploy Postgres tuning overlay
ansible.builtin.copy:
src: "{{ role_path }}/../../../docker/postgres/postgresql.conf.d/00-rootnode-baseline.conf"
dest: /opt/rootnode/platform/postgresql.conf.d/00-rootnode-baseline.conf
mode: "0644"
notify: restart rootnode-platform
- name: Deploy provisioner-config overlay ({{ env }})
ansible.builtin.copy:
src: "{{ role_path }}/../../../../environments/{{ env }}/provisioner-config/"
dest: /etc/rootnode/provisioner-config/
mode: "0644"
notify: restart rootnode-platform
- name: Deploy per-service env files (from Ansible Vault-backed variables — see group_vars)
ansible.builtin.template:
src: "{{ item }}.env.j2"
dest: "/etc/rootnode/{{ item }}.env"
owner: root
group: root
mode: "0600"
loop:
- platform
- provisioner
- gateway
- postgres
notify: restart rootnode-platform
- name: Render production docker-compose.yml
ansible.builtin.template:
src: "{{ role_path }}/../../../docker/platform/docker-compose.production.yml"
dest: /opt/rootnode/platform/docker-compose.yml
owner: root
group: root
mode: "0640"
notify: restart rootnode-platform
- name: Render Caddyfile
ansible.builtin.template:
src: "{{ role_path }}/../../../docker/platform/Caddyfile.j2"
dest: /opt/rootnode/platform/Caddyfile
owner: root
group: root
mode: "0644"
notify: restart rootnode-platform
- name: Deploy systemd unit
ansible.builtin.template:
src: rootnode-platform.service.j2
dest: /etc/systemd/system/rootnode-platform.service
mode: "0644"
notify: restart rootnode-platform
# The `backup` role (applied after this one in site.yml) deploys the backup script and cron —
# kept separate since it also has to run correctly on a host that ISN'T freshly deployed.
- name: Enable and start rootnode-platform
ansible.builtin.systemd:
name: rootnode-platform
enabled: true
state: started
daemon_reload: true
@@ -0,0 +1,13 @@
# Rendered by Ansible — see platform.env.j2 for the pattern.
PLATFORM_API_URL=http://api:4300
PROVISIONER_URL=http://provisioner:4100
PROVISIONER_SHARED_SECRET={{ vault_provisioner_shared_secret }}
GATEWAY_PORT=4200
# Phase 4.1 — mTLS client identity for the Lab Host Router (data plane). Only used when a
# lab_host's routing_mode is 'router' (see platform/gateway/src/lab-endpoint.js); harmless if
# every configured lab_host is still 'direct'. Paths are inside the container — mounted from
# /etc/rootnode/lab-router-tls on the host by the `docker-tls` role, same pattern as the
# Provisioner's Docker TLS client cert.
LAB_ROUTER_TLS_CA=/tls/ca.pem
LAB_ROUTER_TLS_CERT=/tls/gateway-client-cert.pem
LAB_ROUTER_TLS_KEY=/tls/gateway-client-key.pem
@@ -0,0 +1,24 @@
# Rendered by Ansible from infra/environments/{{ env }}/platform.env.template's shape —
# real values come from group_vars/{{ env }}.yml + the encrypted vault file. Do not hand-edit
# on the host; re-run the platform role instead.
# service names, not localhost — production runs these on a named Docker bridge network
# (rn-platform-internal), not network_mode: host like the dev compose file does.
PLATFORM_DB_URL=postgres://rootnode_platform:{{ vault_postgres_password }}@postgres:5432/rootnode_platform
PROVISIONER_URL=http://provisioner:4100
PROVISIONER_SHARED_SECRET={{ vault_provisioner_shared_secret }}
PLATFORM_API_PORT=4300
STUDENT_SESSION_TTL_HOURS=8
STUDENT_SESSION_IDLE_HOURS=2
INSTRUCTOR_SESSION_TTL_HOURS=12
INSTRUCTOR_SESSION_IDLE_HOURS=2
COOKIE_SECURE=1
TRUST_PROXY=1
# Phase 4.1 §17 — instructor "needs attention" thresholds. Commented out (code defaults apply:
# 3 failures, 7 stale days) — uncomment and override in group_vars/{{ env }}.yml only if a
# cohort's real usage shows the defaults are wrong for it.
# NEEDS_ATTENTION_FAIL_THRESHOLD=3
# NEEDS_ATTENTION_STALE_DAYS=7
@@ -0,0 +1,4 @@
# Rendered by Ansible — see platform.env.j2 for the pattern.
POSTGRES_USER=rootnode_platform
POSTGRES_PASSWORD={{ vault_postgres_password }}
POSTGRES_DB=rootnode_platform
@@ -0,0 +1,11 @@
# Rendered by Ansible — see platform.env.j2 for the pattern.
PLATFORM_DB_URL=postgres://rootnode_platform:{{ vault_postgres_password }}@postgres:5432/rootnode_platform
PROVISIONER_SHARED_SECRET={{ vault_provisioner_shared_secret }}
PROVISIONER_PORT=4100
PROVISIONER_CONFIG_DIR=/app/config
LAB_IMAGE_REF={{ registry }}/rootnode/lab:{{ image_tag }}
EGRESS_PROXY_PORT=3128
DNS_RESOLVER=127.0.0.11
NO_PROXY_CIDRS=10.42.0.0/16,10.43.0.0/16
@@ -0,0 +1,17 @@
[Unit]
Description=RootNode Academy Platform (docker compose)
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target
[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/rootnode/platform
ExecStart=/usr/bin/docker compose -f docker-compose.yml up -d --remove-orphans
ExecStop=/usr/bin/docker compose -f docker-compose.yml down
ExecReload=/usr/bin/docker compose -f docker-compose.yml up -d --remove-orphans
TimeoutStartSec=300
[Install]
WantedBy=multi-user.target
+52
View File
@@ -0,0 +1,52 @@
---
# Top-level playbook — full production/staging deployment. Idempotent: re-running against an
# already-deployed host converges it, doesn't rebuild from scratch (Phase 4 §20 requirement).
#
# Usage:
# ansible-playbook site.yml -i inventory/production.ini --vault-password-file .vault_pass
# ansible-playbook site.yml -i inventory/staging.ini -e env=staging
#
# Role order matters: firewall and docker-tls must land before platform/lab-host bring up
# services that depend on the network policy / TLS material already being in place.
- name: Base hardening — all RootNode hosts
hosts: rootnode
become: true
roles:
- common
- name: Firewall policy — all RootNode hosts
hosts: rootnode
become: true
roles:
- firewall
- name: Docker TLS — both hosts (server cert on lab_host, client cert on platform)
hosts: rootnode
become: true
roles:
- docker-tls
- name: Deploy the Platform VM
hosts: platform
become: true
roles:
- platform
- name: Deploy the Lab Host VM
hosts: lab_host
become: true
roles:
- lab-host
- name: Deploy monitoring configuration — both hosts
hosts: rootnode
become: true
roles:
- monitoring-agent
- name: Deploy backup automation — Platform VM only (that's where Postgres lives)
hosts: platform
become: true
roles:
- backup
+77
View File
@@ -0,0 +1,77 @@
#!/usr/bin/env bash
# Postgres backup — Tier 1 (irreplaceable) data per docs/BACKUP_DR.md. Runs via cron on
# rn-platform-01 (deployed by the `backup` Ansible role), and locally testable against ANY
# reachable Postgres (that's exactly what infra/backup/test-backup-restore.sh does).
#
# Produces one gzip-compressed pg_dump per run, encrypts it, prunes local copies past
# BACKUP_RETENTION_DAYS, and (if BACKUP_OFFSITE_TARGET is set) rsyncs the encrypted copy
# off-host. Writes a Prometheus textfile-collector metric so infra/monitoring/'s BackupFailed/
# BackupAging alerts have real data to fire on.
#
# Required env: PLATFORM_DB_URL, BACKUP_DIR, BACKUP_ENCRYPTION_PASSPHRASE
# Optional env: BACKUP_RETENTION_DAYS (default 30), BACKUP_OFFSITE_TARGET, METRICS_TEXTFILE_DIR
set -euo pipefail
: "${PLATFORM_DB_URL:?PLATFORM_DB_URL is required}"
: "${BACKUP_DIR:?BACKUP_DIR is required}"
: "${BACKUP_ENCRYPTION_PASSPHRASE:?BACKUP_ENCRYPTION_PASSPHRASE is required — see docs/BACKUP_DR.md}"
BACKUP_RETENTION_DAYS="${BACKUP_RETENTION_DAYS:-30}"
METRICS_TEXTFILE_DIR="${METRICS_TEXTFILE_DIR:-}"
TS="$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$BACKUP_DIR"
DUMP_FILE="$BACKUP_DIR/rootnode-platform-$TS.sql.gz"
ENC_FILE="$DUMP_FILE.gpg"
write_metric() {
# $1 = metric line. Written atomically (tmp+rename) so Prometheus's node-exporter
# textfile-collector never scrapes a half-written file.
[ -n "$METRICS_TEXTFILE_DIR" ] || return 0
mkdir -p "$METRICS_TEXTFILE_DIR"
local tmp; tmp="$(mktemp "$METRICS_TEXTFILE_DIR/.rootnode_backup.XXXXXX")"
echo "$1" > "$tmp"
mv -f "$tmp" "$METRICS_TEXTFILE_DIR/rootnode_backup.prom"
}
fail() {
echo "BACKUP FAILED: $1" >&2
write_metric "rootnode_backup_last_run_success 0"
exit 1
}
trap 'fail "unexpected error at line $LINENO"' ERR
echo "==> Dumping $PLATFORM_DB_URL to $DUMP_FILE"
pg_dump --format=plain --no-owner --no-privileges "$PLATFORM_DB_URL" | gzip -9 > "$DUMP_FILE"
# Fail loudly on a suspiciously small dump (empty/truncated) rather than "successfully"
# backing up nothing — a real, if crude, sanity check.
DUMP_SIZE=$(stat -c%s "$DUMP_FILE" 2>/dev/null || stat -f%z "$DUMP_FILE")
[ "$DUMP_SIZE" -gt 1024 ] || fail "dump is suspiciously small ($DUMP_SIZE bytes) — refusing to treat this as a real backup"
echo "==> Encrypting (AES-256, symmetric passphrase — see docs/BACKUP_DR.md for the recovery-key handling policy)"
gpg --batch --yes --passphrase "$BACKUP_ENCRYPTION_PASSPHRASE" --symmetric --cipher-algo AES256 \
--output "$ENC_FILE" "$DUMP_FILE"
rm -f "$DUMP_FILE" # never leave the plaintext dump on disk once the encrypted copy exists
echo "==> Local backup complete: $ENC_FILE ($(stat -c%s "$ENC_FILE" 2>/dev/null || stat -f%z "$ENC_FILE") bytes)"
if [ -n "${BACKUP_OFFSITE_TARGET:-}" ]; then
echo "==> Copying off-host to $BACKUP_OFFSITE_TARGET"
# rsync over SSH works for both a Hetzner Storage Box (SSH/SFTP-native) and any other
# SSH-reachable target — the adapter point the Phase 4 spec asked for is this one variable,
# not a vendor-specific script. An S3-compatible target uses `aws s3 cp` / `rclone` instead;
# see docs/BACKUP_DR.md for both forms.
rsync -az --timeout=120 "$ENC_FILE" "$BACKUP_OFFSITE_TARGET/" \
|| fail "off-host copy to $BACKUP_OFFSITE_TARGET failed"
else
echo "==> BACKUP_OFFSITE_TARGET not set — local-only backup (fine for a local test run, NOT fine for production, see docs/BACKUP_DR.md)"
fi
echo "==> Pruning local backups older than $BACKUP_RETENTION_DAYS days"
find "$BACKUP_DIR" -name 'rootnode-platform-*.sql.gz.gpg' -mtime "+$BACKUP_RETENTION_DAYS" -delete
write_metric "rootnode_backup_last_run_success 1
rootnode_backup_last_success_timestamp_seconds $(date +%s)
rootnode_backup_size_bytes $(stat -c%s "$ENC_FILE" 2>/dev/null || stat -f%z "$ENC_FILE")"
echo "==> Backup succeeded"
+38
View File
@@ -0,0 +1,38 @@
#!/usr/bin/env bash
# Restores an encrypted backup produced by backup-postgres.sh into a target database.
# Deliberately requires an EXPLICIT target — never silently restores over the live production
# database ("what would you do if this ran the wrong way" is a real disaster-recovery failure
# mode, so making that impossible to trigger by accident matters more than convenience here).
#
# Usage: ./restore-postgres.sh <encrypted-backup-file> <target-db-url>
#
# For a REAL production restore, target-db-url should point at a FRESH database created for
# this purpose (docs/RUNBOOK_POSTGRES_RESTORE.md walks through creating it) — restoring into a
# database that already has data merges/conflicts rather than cleanly replacing it.
set -euo pipefail
ENC_FILE="${1:?usage: restore-postgres.sh <encrypted-backup-file> <target-db-url>}"
TARGET_DB_URL="${2:?usage: restore-postgres.sh <encrypted-backup-file> <target-db-url>}"
: "${BACKUP_ENCRYPTION_PASSPHRASE:?BACKUP_ENCRYPTION_PASSPHRASE is required to decrypt}"
[ -f "$ENC_FILE" ] || { echo "No such file: $ENC_FILE" >&2; exit 1; }
TMPDIR="$(mktemp -d)"
trap 'rm -rf "$TMPDIR"' EXIT
DUMP_FILE="$TMPDIR/restore.sql.gz"
echo "==> Decrypting $ENC_FILE"
gpg --batch --yes --passphrase "$BACKUP_ENCRYPTION_PASSPHRASE" --decrypt --output "$DUMP_FILE" "$ENC_FILE"
echo "==> Confirming target is reachable"
psql "$TARGET_DB_URL" -c "SELECT 1" > /dev/null
echo "==> Target database currently has these tables (should be empty/unimportant before you continue):"
psql "$TARGET_DB_URL" -c "\dt" || true
read -r -p "Type RESTORE to proceed against $TARGET_DB_URL: " CONFIRM
[ "$CONFIRM" = "RESTORE" ] || { echo "Aborted."; exit 1; }
echo "==> Restoring"
gunzip -c "$DUMP_FILE" | psql "$TARGET_DB_URL"
echo "==> Restore complete. Verify with infra/backup/verify-restore.sh $TARGET_DB_URL"
+94
View File
@@ -0,0 +1,94 @@
#!/usr/bin/env node
// Seeds realistic academy data for the Phase 4.1 §20 destructive restore drill — an owner, an
// admin instructor, a cohort, 3 students, progress, attempts, an assessment definition +
// session, and an achievement. Uses the REAL Platform API over HTTP for everything that
// doesn't need a live Provisioner/k3s Lab (owner login, instructor/cohort/student/assessment
// creation — genuinely exercises the real RBAC/creation code paths), and direct SQL inserts
// only for progress/attempts/achievements (those need real scenario activity in the full app;
// re-running the whole learning engine here would just re-prove what e2e-phase3b.js already
// proves — this drill is about DATA DURABILITY, not re-testing scenario validation).
//
// Usage: PLATFORM_API_URL=http://localhost:PORT PLATFORM_DB_URL=postgres://... node seed-realistic-data.js
const { Pool } = require('../../platform/api/node_modules/pg');
const API = process.env.PLATFORM_API_URL || 'http://localhost:4390';
const DB_URL = process.env.PLATFORM_DB_URL;
if (!DB_URL) { console.error('Set PLATFORM_DB_URL'); process.exit(1); }
function makeSession() {
const jar = new Map();
function cookieHeader() { return [...jar].map(([k, v]) => `${k}=${v}`).join('; '); }
function absorb(res) {
const cookies = typeof res.headers.getSetCookie === 'function' ? res.headers.getSetCookie() : [];
for (const c of cookies) {
const [pair] = c.split(';');
const eq = pair.indexOf('=');
if (eq === -1) continue;
jar.set(pair.slice(0, eq).trim(), pair.slice(eq + 1).trim());
}
}
return {
async req(method, path, body) {
const headers = { 'Content-Type': 'application/json' };
if (cookieHeader()) headers.Cookie = cookieHeader();
if (!['GET', 'HEAD'].includes(method)) headers['X-CSRF-Token'] = jar.get('rn_csrf_i') || '';
const res = await fetch(`${API}${path}`, { method, headers, body: body ? JSON.stringify(body) : undefined });
absorb(res);
const text = await res.text();
let json; try { json = JSON.parse(text); } catch { json = text; }
if (!res.ok) throw new Error(`${method} ${path} -> ${res.status}: ${JSON.stringify(json)}`);
return json;
},
};
}
async function main() {
const owner = makeSession();
await owner.req('POST', '/api/auth/instructor/login', { email: '[email protected]', password: 'DrillOwnerPass!2026' });
console.log(' owner login ok');
const admin = await owner.req('POST', '/api/admin/instructors', {
email: '[email protected]', display_name: 'Drill Admin', password: 'DrillAdminPass!2026', role: 'admin',
});
console.log(' admin instructor created:', admin.instructor_id);
const cohort = await owner.req('POST', '/api/admin/cohorts', { cohort_id: 'drill-cohort-01', name: 'Restore Drill Cohort' });
console.log(' cohort created');
const students = [];
for (const name of ['Drill Alice', 'Drill Bob', 'Drill Carlos']) {
const s = await owner.req('POST', '/api/admin/students', { display_name: name, cohort_id: 'drill-cohort-01' });
students.push(s.student_id);
}
console.log(' 3 students rostered and enrolled:', students.join(', '));
const assessment = await owner.req('POST', '/api/admin/assessments', {
id: 'drill-exam-01', type: 'exam', title: 'Restore Drill Exam', duration_minutes: 30, hint_policy: 'disabled',
pool_spec: { required: ['pod-basics-mcq'], random: [] },
});
console.log(' assessment definition created:', assessment.assessment_definition_id || 'drill-exam-01');
// Direct SQL for progress/attempts/achievements — see file header for why.
const pool = new Pool({ connectionString: DB_URL });
for (const studentId of students) {
// environment_id is a real FK to the Provisioner's environments table (nullable) — left
// NULL here rather than faking a row that doesn't exist, same as how a genuinely old
// attempt whose environment has long since been cleaned up would look in real data.
await pool.query(
`INSERT INTO scenario_attempts (student_id, scenario_id, status, result, started_at, completed_at, validation_attempt_count, validation_pass_count)
VALUES ($1, 'pod-basics-mcq', 'completed', 'passed', now() - interval '1 hour', now(), 1, 1)`,
[studentId]
);
await pool.query(
`INSERT INTO achievements (student_id, achievement_type, stage_id, achieved_at) VALUES ($1, 'stage_completed', 'foundation', now())`,
[studentId]
);
}
await pool.end();
console.log(' progress/attempts/achievements seeded for all 3 students');
console.log('\nSeed complete.');
}
main().catch((e) => { console.error('SEED FAILED:', e.message); process.exit(1); });
+133
View File
@@ -0,0 +1,133 @@
#!/usr/bin/env bash
# Local test (Phase 4 §29): a REAL end-to-end backup/restore cycle against disposable local
# Postgres containers — not a mock. Applies the actual Platform API migration runner (the same
# code path production uses, platform/api/src/db.js) so the schema under test is genuinely the
# real one, inserts sample data, backs it up, restores into a SECOND empty Postgres, and
# verifies the result. Proves the backup/restore MECHANISM works; does NOT prove anything about
# the real off-host target (Storage Box/S3) — that needs real credentials, marked as
# hardware/credential-dependent below.
set -euo pipefail
cd "$(dirname "$0")/../.." # repo root
SRC_CONTAINER=rootnode-backup-test-src
DST_CONTAINER=rootnode-backup-test-dst
SRC_PORT=55432
DST_PORT=55433
TESTDIR="$(mktemp -d)"
export BACKUP_ENCRYPTION_PASSPHRASE="test-passphrase-not-for-production"
cleanup() {
docker rm -f "$SRC_CONTAINER" "$DST_CONTAINER" >/dev/null 2>&1 || true
rm -rf "$TESTDIR"
}
trap cleanup EXIT
# backup-postgres.sh/restore-postgres.sh assume real postgresql-client + gpg binaries — true on
# a real deployment host (the `backup` Ansible role installs postgresql-client there). This dev
# machine may not have psql/pg_dump installed at all, so for THIS TEST ONLY we shim them as
# thin wrappers around `docker exec` into the two disposable test containers — the scripts
# under test are otherwise completely unmodified from what production actually runs.
if ! command -v psql >/dev/null 2>&1 || ! command -v pg_dump >/dev/null 2>&1; then
echo "==> psql/pg_dump not found locally — shimming both via docker exec for this test run"
SHIM_BIN="$TESTDIR/bin"
mkdir -p "$SHIM_BIN"
# Both wrappers pick a container from the target port (explicit -p, or embedded in a
# postgres://...:PORT/... URL), then re-exec psql/pg_dump INSIDE that container against ITS
# OWN localhost:5432 — the host-side port mapping (55432/55433) doesn't exist inside the
# container's network namespace, so the connection target itself must be replaced, not just
# which container runs the command.
cat > "$SHIM_BIN/psql" <<'EOF'
#!/usr/bin/env bash
args=(); port=""; want_port=0; skip_next=0
for a in "$@"; do
if [ "$skip_next" = "1" ]; then
[ "$want_port" = "1" ] && port="$a"
skip_next=0; want_port=0; continue
fi
if [ "$a" = "-p" ]; then skip_next=1; want_port=1; continue; fi
if [ "$a" = "-h" ]; then skip_next=1; want_port=0; continue; fi
if [[ "$a" =~ ^postgres://[^:]+:[^@]+@[^:]+:([0-9]+)/ ]]; then
port="${BASH_REMATCH[1]}"
continue # drop the URL — reconnect locally inside the container instead
fi
args+=("$a")
done
case "$port" in
55433) c="rootnode-backup-test-dst" ;;
*) c="rootnode-backup-test-src" ;;
esac
# Only forward local stdin (-i) for the one call that actually reads SQL from it (no -c/-tAc
# in its args, i.e. `psql <target>` fed by a pipe) — every -c/-tAc call is self-contained and
# must NOT hold docker exec's stdin open, or it silently steals bytes meant for a later `read`
# elsewhere in the calling script (exactly what broke restore-postgres.sh's confirmation prompt
# the first time this shim was written).
needs_stdin=1
for a in "${args[@]}"; do [ "$a" = "-c" ] || [ "$a" = "-tAc" ] && needs_stdin=0; done
if [ "$needs_stdin" = "1" ]; then
exec docker exec -e PGPASSWORD=postgres -i "$c" psql -h localhost -U postgres "${args[@]}"
else
exec docker exec -e PGPASSWORD=postgres "$c" psql -h localhost -U postgres "${args[@]}"
fi
EOF
cat > "$SHIM_BIN/pg_dump" <<'EOF'
#!/usr/bin/env bash
args=()
for a in "$@"; do
if [[ "$a" =~ ^postgres:// ]]; then continue; fi # source is always the src container here
args+=("$a")
done
exec docker exec -e PGPASSWORD=postgres -i rootnode-backup-test-src pg_dump -h localhost -U postgres "${args[@]}"
EOF
chmod +x "$SHIM_BIN/psql" "$SHIM_BIN/pg_dump"
export PATH="$SHIM_BIN:$PATH"
fi
echo "==> Starting source Postgres (the one we'll back up)"
docker run -d --name "$SRC_CONTAINER" -e POSTGRES_PASSWORD=postgres -p "$SRC_PORT:5432" postgres:16-alpine >/dev/null
echo "==> Starting destination Postgres (the one we'll restore into)"
docker run -d --name "$DST_CONTAINER" -e POSTGRES_PASSWORD=postgres -p "$DST_PORT:5432" postgres:16-alpine >/dev/null
echo "==> Waiting for both to accept connections"
for port in "$SRC_PORT" "$DST_PORT"; do
for i in $(seq 1 30); do
PGPASSWORD=postgres psql -h localhost -p "$port" -U postgres -c "SELECT 1" >/dev/null 2>&1 && break
sleep 1
done
done
SRC_DB_URL="postgres://postgres:postgres@localhost:$SRC_PORT/postgres"
DST_DB_URL="postgres://postgres:postgres@localhost:$DST_PORT/postgres"
echo "==> Applying the REAL Provisioner migrations first (Platform API's scenario_attempts has a real FK to Provisioner's environments table — documented cross-service ordering dependency, see docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md)"
( cd platform/provisioner && [ -d node_modules ] || npm install --no-audit --no-fund >/dev/null
PLATFORM_DB_URL="$SRC_DB_URL" node -e "require('./src/db').runMigrations().then((m)=>console.log('Applied:', m.join(', ')))" )
echo "==> Applying the REAL Platform API migrations against the source"
( cd platform/api && [ -d node_modules ] || npm install --no-audit --no-fund >/dev/null
PLATFORM_DB_URL="$SRC_DB_URL" node -e "require('./src/db').runMigrations().then((m)=>console.log('Applied:', m.join(', ')))" )
echo "==> Inserting sample data (a student, so the backup has something real to carry)"
PGPASSWORD=postgres psql -h localhost -p "$SRC_PORT" -U postgres -c \
"INSERT INTO students (student_id, display_name) VALUES ('student-backup-test', 'Backup Test Student');"
echo "==> Running backup-postgres.sh against the source"
BACKUP_DIR="$TESTDIR/backups" PLATFORM_DB_URL="$SRC_DB_URL" ./infra/backup/backup-postgres.sh
ENC_FILE=$(ls "$TESTDIR"/backups/*.sql.gz.gpg | head -1)
[ -f "$ENC_FILE" ] || { echo "FAIL: no encrypted backup file produced"; exit 1; }
echo "==> Backup produced: $ENC_FILE"
echo "==> Restoring into the destination (non-interactive: piping the confirmation)"
echo "RESTORE" | BACKUP_ENCRYPTION_PASSPHRASE="$BACKUP_ENCRYPTION_PASSPHRASE" ./infra/backup/restore-postgres.sh "$ENC_FILE" "$DST_DB_URL"
echo "==> Verifying the restore"
./infra/backup/verify-restore.sh "$DST_DB_URL"
echo "==> Confirming the sample student actually round-tripped"
RESTORED_NAME=$(PGPASSWORD=postgres psql -h localhost -p "$DST_PORT" -U postgres -tAc \
"SELECT display_name FROM students WHERE student_id = 'student-backup-test'")
[ "$RESTORED_NAME" = "Backup Test Student" ] || { echo "FAIL: restored student data doesn't match"; exit 1; }
echo
echo "OK: full backup -> encrypt -> restore -> verify cycle succeeded against real local Postgres."
echo "NOT TESTED HERE (needs real credentials/hardware): off-host copy to a real Storage Box/S3 target."
+149
View File
@@ -0,0 +1,149 @@
#!/usr/bin/env bash
# Phase 4.1 §20 — the DESTRUCTIVE disaster-recovery drill, one level up from
# test-backup-restore.sh's mechanism-only test: seeds realistic academy data through the REAL
# Platform API (owner, admin instructor, cohort, 3 students, progress, attempts, an assessment
# definition, achievements), backs it up, DESTROYS the disposable Postgres storage entirely
# (docker rm — not just "restore into a second container," the actual source is gone),
# recreates Postgres from nothing, restores, and boots the REAL Platform API again against the
# restored database to prove the whole stack — not just the SQL — comes back.
#
# Never touches the developer's real persistent database — every Postgres here is a disposable
# container on a throwaway port, created and destroyed entirely within this script.
set -euo pipefail
cd "$(dirname "$0")/../.." # repo root
CONTAINER=rootnode-drill-postgres
PORT=55434
API_PORT=4390
TESTDIR="$(mktemp -d)"
# curriculum.js defaults to platform/api/scenarios/{data,bundles} — only present inside the
# Docker image build (COPY scenarios/ ./scenarios/). Running src/server.js directly from
# source (as this drill does, to avoid needing a full container build) needs these pointed at
# the repo's real scenario content instead.
export SCENARIOS_DIR="$PWD/scenarios/data"
export BUNDLES_DIR="$PWD/scenarios/bundles"
export CURRICULUM_PATH_FILE="$PWD/scenarios/rootnode-path.json"
export BACKUP_ENCRYPTION_PASSPHRASE="drill-passphrase-not-for-production"
export PROVISIONER_SHARED_SECRET="drill-shared-secret-not-for-production"
API_PID=""
cleanup() {
[ -n "$API_PID" ] && kill "$API_PID" 2>/dev/null || true
docker rm -f "$CONTAINER" >/dev/null 2>&1 || true
rm -rf "$TESTDIR"
}
trap cleanup EXIT
if ! command -v psql >/dev/null 2>&1 || ! command -v pg_dump >/dev/null 2>&1; then
echo "==> psql/pg_dump not found locally — reusing the same docker-exec shim as test-backup-restore.sh"
SHIM_BIN="$TESTDIR/bin"; mkdir -p "$SHIM_BIN"
# Same shim as test-backup-restore.sh (see that file's comment for the full "why") — the URL
# is DROPPED, not kept, since the host-side port it names doesn't exist inside the
# container's own network namespace; -tAc counts as a self-contained call (needs no stdin)
# the same way -c does, or the admin-verification queries below would hang waiting on stdin.
cat > "$SHIM_BIN/psql" <<'EOF'
#!/usr/bin/env bash
args=(); skip_next=0
for a in "$@"; do
if [ "$skip_next" = "1" ]; then skip_next=0; continue; fi
if [ "$a" = "-p" ] || [ "$a" = "-h" ]; then skip_next=1; continue; fi
if [[ "$a" =~ ^postgres://[^:]+:[^@]+@[^:]+:([0-9]+)/ ]]; then continue; fi # drop the URL — reconnect locally inside the container instead
args+=("$a")
done
container="rootnode-drill-postgres"
needs_stdin=1
for a in "${args[@]}"; do { [ "$a" = "-c" ] || [ "$a" = "-tAc" ]; } && needs_stdin=0; done
if [ "$needs_stdin" = "1" ]; then
exec docker exec -e PGPASSWORD=postgres -i "$container" psql -h localhost -U postgres "${args[@]}"
else
exec docker exec -e PGPASSWORD=postgres "$container" psql -h localhost -U postgres "${args[@]}"
fi
EOF
cat > "$SHIM_BIN/pg_dump" <<'EOF'
#!/usr/bin/env bash
args=()
for a in "$@"; do
if [[ "$a" =~ ^postgres://[^:]+:[^@]+@[^:]+:([0-9]+)/(.*)$ ]]; then args+=("postgres://postgres:postgres@localhost:5432/${BASH_REMATCH[2]}"); continue; fi
args+=("$a")
done
exec docker exec -e PGPASSWORD=postgres rootnode-drill-postgres pg_dump "${args[@]}"
EOF
chmod +x "$SHIM_BIN/psql" "$SHIM_BIN/pg_dump"
export PATH="$SHIM_BIN:$PATH"
fi
DB_URL="postgres://postgres:postgres@localhost:$PORT/postgres"
echo "==> [1/9] Starting disposable Postgres"
docker run -d --name "$CONTAINER" -e POSTGRES_PASSWORD=postgres -p "$PORT:5432" postgres:16-alpine >/dev/null
for i in $(seq 1 30); do PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -c "SELECT 1" >/dev/null 2>&1 && break; sleep 1; done
echo "==> [2/9] Applying real Provisioner + Platform API migrations"
( cd platform/provisioner && [ -d node_modules ] || npm install --no-audit --no-fund >/dev/null
PLATFORM_DB_URL="$DB_URL" node -e "require('./src/db').runMigrations().then((m)=>console.log(' Provisioner:', m.join(', ')))" )
( cd platform/api && [ -d node_modules ] || npm install --no-audit --no-fund >/dev/null
PLATFORM_DB_URL="$DB_URL" node -e "require('./src/db').runMigrations().then((m)=>console.log(' Platform API:', m.join(', ')))" )
echo "==> [3/9] Bootstrapping the drill owner and starting a REAL Platform API"
( cd platform/api && PLATFORM_DB_URL="$DB_URL" node bootstrap-owner.js [email protected] "Drill Owner" 'DrillOwnerPass!2026' )
( cd platform/api && PLATFORM_DB_URL="$DB_URL" PROVISIONER_URL="http://localhost:1" PLATFORM_API_PORT="$API_PORT" node src/server.js > "$TESTDIR/api.log" 2>&1 & echo $! > "$TESTDIR/api.pid" )
API_PID=$(cat "$TESTDIR/api.pid")
for i in $(seq 1 30); do curl -sf "http://localhost:$API_PORT/healthz" >/dev/null 2>&1 && break; sleep 1; done
curl -sf "http://localhost:$API_PORT/healthz" >/dev/null || { echo "FAIL: Platform API never came up"; cat "$TESTDIR/api.log"; exit 1; }
echo " Platform API up on :$API_PORT"
echo "==> [4/9] Seeding realistic academy data through the real API (owner/admin/cohort/3 students/progress/attempts/assessment/achievements)"
( cd infra/backup && PLATFORM_API_URL="http://localhost:$API_PORT" PLATFORM_DB_URL="$DB_URL" node seed-realistic-data.js )
echo "==> [5/9] Backing up (real backup-postgres.sh)"
BACKUP_DIR="$TESTDIR/backups" PLATFORM_DB_URL="$DB_URL" ./infra/backup/backup-postgres.sh
ENC_FILE=$(ls "$TESTDIR"/backups/*.sql.gz.gpg | head -1)
[ -f "$ENC_FILE" ] || { echo "FAIL: no encrypted backup produced"; exit 1; }
echo "==> [6/9] DESTROYING the Postgres storage entirely (this is the destructive part — not restoring into a second container, the original is gone)"
kill "$API_PID" 2>/dev/null || true; API_PID=""
docker rm -f "$CONTAINER" >/dev/null
echo " source container + its storage are gone"
echo "==> [7/9] Recreating Postgres from nothing"
docker run -d --name "$CONTAINER" -e POSTGRES_PASSWORD=postgres -p "$PORT:5432" postgres:16-alpine >/dev/null
for i in $(seq 1 30); do PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -c "SELECT 1" >/dev/null 2>&1 && break; sleep 1; done
echo "==> [8/9] Restoring from the encrypted backup (schema + data both come from the dump — no separate migration step)"
echo "RESTORE" | BACKUP_ENCRYPTION_PASSPHRASE="$BACKUP_ENCRYPTION_PASSPHRASE" ./infra/backup/restore-postgres.sh "$ENC_FILE" "$DB_URL"
./infra/backup/verify-restore.sh "$DB_URL"
echo "==> [9/9] Booting the REAL Platform API again against the RESTORED database"
( cd platform/api && PLATFORM_DB_URL="$DB_URL" PROVISIONER_URL="http://localhost:1" PLATFORM_API_PORT="$API_PORT" node src/server.js > "$TESTDIR/api2.log" 2>&1 & echo $! > "$TESTDIR/api2.pid" )
API_PID=$(cat "$TESTDIR/api2.pid")
for i in $(seq 1 30); do curl -sf "http://localhost:$API_PORT/healthz" >/dev/null 2>&1 && break; sleep 1; done
curl -sf "http://localhost:$API_PORT/healthz" >/dev/null || { echo "FAIL: Platform API did not come back up against the restored DB"; cat "$TESTDIR/api2.log"; exit 1; }
echo " Platform API is back up, against the RESTORED database — no fresh migration run needed"
echo "==> Verifying accounts/RBAC/cohorts/students/progress/attempts/achievements survived"
FAIL=0
check() { local desc="$1" want="$2" got="$3"; if [ "$want" = "$got" ]; then echo " ok — $desc"; else echo " FAIL — $desc (want $want, got $got)"; FAIL=1; fi; }
check "owner account (role) restored" "owner" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT role FROM instructors WHERE email='[email protected]'" | tr -d '[:space:]')"
check "admin instructor restored" "admin" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT role FROM instructors WHERE email='[email protected]'" | tr -d '[:space:]')"
check "cohort restored" "1" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM cohorts WHERE cohort_id='drill-cohort-01'" | tr -d '[:space:]')"
check "3 students restored" "3" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM cohort_enrollments WHERE cohort_id='drill-cohort-01'" | tr -d '[:space:]')"
check "3 completed attempts restored" "3" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM scenario_attempts WHERE result='passed' AND scenario_id='pod-basics-mcq'" | tr -d '[:space:]')"
check "3 achievements restored" "3" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM achievements WHERE stage_id='foundation'" | tr -d '[:space:]')"
check "assessment definition restored" "1" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM assessment_definitions WHERE assessment_definition_id='drill-exam-01'" | tr -d '[:space:]')"
echo "==> Verifying RBAC still works against the restored DB (real login through the real API)"
LOGIN_STATUS=$(curl -s -o /dev/null -w '%{http_code}' -X POST "http://localhost:$API_PORT/api/auth/instructor/login" \
-H 'Content-Type: application/json' -d '{"email":"[email protected]","password":"DrillAdminPass!2026"}')
check "restored admin instructor can log in" "200" "$LOGIN_STATUS"
if [ "$FAIL" = "1" ]; then
echo; echo "DRILL FAILED — see above."; exit 1
fi
echo
echo "OK: full destructive restore drill passed — owner/admin/cohort/3 students/progress/attempts/"
echo "assessment/achievements all survived a real Postgres storage loss + recreation + restore, and"
echo "the real Platform API booted and authenticated against the restored database with no fresh"
echo "migration run required."
echo "NOT TESTED HERE (needs real credentials/hardware): the actual off-host Storage Box/S3 copy."
+26
View File
@@ -0,0 +1,26 @@
#!/usr/bin/env bash
# Sanity-checks a restored database — confirms the expected Platform API schema landed and has
# data, not just that psql exited 0 (a restore that silently imported zero rows would still
# "succeed" by that measure). Used both by restore-postgres.sh's own suggestion and by
# test-backup-restore.sh's automated local test.
set -euo pipefail
DB_URL="${1:?usage: verify-restore.sh <db-url>}"
EXPECTED_TABLES=(students cohorts instructors sessions scenario_attempts assessment_definitions achievements audit_events)
FAIL=0
for t in "${EXPECTED_TABLES[@]}"; do
exists=$(psql "$DB_URL" -tAc "SELECT to_regclass('public.$t') IS NOT NULL")
if [ "$exists" != "t" ]; then
echo "MISSING TABLE: $t" >&2
FAIL=1
fi
done
if [ "$FAIL" = "1" ]; then
echo "FAIL: restored database is missing expected tables" >&2
exit 1
fi
MIGRATIONS_APPLIED=$(psql "$DB_URL" -tAc "SELECT count(*) FROM schema_migrations" 2>/dev/null || echo 0)
echo "OK: all expected tables present. $MIGRATIONS_APPLIED migrations recorded in schema_migrations."
+162
View File
@@ -0,0 +1,162 @@
# Gitea Actions workflow — build/test on every push to main. Gitea Actions uses the same
# workflow syntax as GitHub Actions (a compatible runner), so this is portable if the canonical
# remote ever needs to move CI providers, without being GitHub Actions itself or requiring a
# paid CI provider. Runs against the canonical Gitea repo only — see docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md
# §18 for why this never touches the GitHub kubekosh fork.
#
# Deploy this file to .gitea/workflows/build-and-test.yml at the repo root to activate it (kept
# under infra/ci/ in this PR so it's reviewed alongside the rest of the infra package first,
# per the same "build now, activate deliberately" posture as everything else server-dependent
# in this phase — enabling Gitea Actions on the instance is itself a one-time admin step that
# may not be done yet).
name: build-and-test
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
regression:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_PASSWORD: postgres
ports: ["5432:5432"]
options: >-
--health-cmd "pg_isready -U postgres"
--health-interval 10s --health-timeout 5s --health-retries 10
steps:
- uses: actions/checkout@v4
- name: Set up Node
uses: actions/setup-node@v4
with:
node-version: "20"
- name: Install dependencies (all first-party services)
run: |
for d in backend frontend platform/api platform/gateway platform/provisioner platform/lab-router; do
(cd "$d" && npm ci)
done
- name: Syntax-check every service
run: |
for d in backend frontend/src platform/api platform/gateway platform/provisioner platform/lab-router; do
find "$d" -name '*.js' -not -path '*/node_modules/*' -exec node --check {} \;
done
- name: Lab Host Router unit tests (Phase 4.1 — fast, no Docker daemon needed)
run: (cd platform/lab-router && npm test)
- name: Build frontend
run: (cd frontend && npm run build)
- name: Provisioner migrations against the service Postgres
run: |
cd platform/provisioner
PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5432/postgres \
node -e "require('./src/db').runMigrations().then(m=>console.log('Applied:',m.join(', ')))"
- name: Platform API migrations (depends on Provisioner's, same order as production)
run: |
cd platform/api
PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5432/postgres \
node -e "require('./src/db').runMigrations().then(m=>console.log('Applied:',m.join(', ')))"
- name: Validate infra configs (Phase 4 local tests — no hardware needed)
run: |
./infra/scripts/validate-compose.sh
./infra/firewall/validate-firewall.sh
# The real k3s-boot-dependent e2e suites (platform/test/e2e-phase*.js) need a full Docker
# Compose stack with --privileged Lab containers, which a shared CI runner typically
# can't grant safely — they stay a LOCAL/staging pre-promotion step
# (infra/scripts/staging-smoke-test.sh), not this job. See docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §18.
# Phase 4.1 §24 — free/open-source security scanning, no paid tooling, no signup required for
# any of these. Severity policy (docs/SECURITY_SCANNING.md): CRITICAL/HIGH findings fail the
# build; MEDIUM/LOW are reported but never block a merge — a build that's permanently red over
# an unrelated low-severity transitive dependency finding trains everyone to ignore red builds.
security-scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"
- name: npm audit (dependency vulnerabilities) — fails only on high/critical
run: |
FAIL=0
for d in backend frontend platform/api platform/gateway platform/provisioner platform/lab-router; do
echo "== $d =="
(cd "$d" && npm audit --audit-level=high) || FAIL=1
done
exit $FAIL
- name: Secret scanning (gitleaks — free, no account needed)
run: docker run --rm -v "$PWD:/repo" zricethezav/gitleaks:latest detect --source=/repo --no-git -v
- name: Dockerfile linting (hadolint — free)
run: |
FAIL=0
for f in Dockerfile platform/api/Dockerfile platform/gateway/Dockerfile platform/provisioner/Dockerfile platform/egress-proxy/Dockerfile platform/lab-router/Dockerfile; do
echo "== $f =="
docker run --rm -i hadolint/hadolint < "$f" || FAIL=1
done
exit $FAIL
# -S error, not warning — matches docs/SECURITY_SCANNING.md's severity policy. Real run
# against this tree at -S warning surfaces only style nits (unused loop vars, a missing
# `cd ... || exit` guard) on scripts that are otherwise correct; failing the build on
# those trains everyone to ignore the check.
- name: Shell linting (shellcheck — free, standard)
run: |
sudo apt-get update -qq && sudo apt-get install -y -qq shellcheck
find infra -name '*.sh' -exec shellcheck -S error {} \+
- name: Container image scanning (trivy — free, no account needed)
run: |
docker build -q -t rootnode-scan/platform-api -f platform/api/Dockerfile .
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock aquasec/trivy:latest image \
--severity HIGH,CRITICAL --exit-code 1 rootnode-scan/platform-api
build-images:
needs: [regression, security-scan]
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build and tag images
run: ./infra/scripts/ci/tag-and-push.sh
env:
REGISTRY: ${{ vars.REGISTRY }}
REGISTRY_USER: ${{ secrets.REGISTRY_USER }}
REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
deploy-staging:
needs: build-images
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Deploy to staging (automatic)
run: |
cd infra/ansible
ansible-playbook site.yml -i inventory/staging.ini \
-e "image_tag=staging-${GITHUB_SHA::12}" \
--vault-password-file <(echo "$VAULT_PASSWORD")
env:
VAULT_PASSWORD: ${{ secrets.ANSIBLE_VAULT_PASSWORD }}
- name: Staging smoke test
run: ./infra/scripts/staging-smoke-test.sh
# Production is NEVER automatic — a human runs infra/scripts/deploy-production.sh after
# reviewing the staging smoke test result, per the explicit "manual approval step" preference.
# No job here does that; see docs/RUNBOOK_NORMAL_DEPLOY.md for the actual command.

Some files were not shown because too many files have changed in this diff Show More