infra/backup/test-destructive-restore-drill.sh - realistic academy data
(owner, admin instructor, cohort, 3 students, progress, attempts, an
assessment definition, achievements) seeded through the REAL Platform API,
encrypted-backed-up, the source Postgres container genuinely destroyed
(docker rm, not just left running alongside a second one), a fresh
Postgres created from nothing, restored, and the real Platform API booted
again against the restored database and used to log in as the restored
admin instructor over HTTP. Never touches the developer's real database.
Confirmed passing.
- Real server-side pagination (platform/api/src/pagination.js) on the audit
log (was a flat LIMIT 200), student attempt history (was LIMIT 100), and
assessment session history (was fully unbounded - the one genuine growth
risk found). Frontend updated (AuditView, AssessmentsView, StudentsView).
- Admission control: resource-policy.yaml's new (disabled-by-default)
admission_control.max_concurrent_labs, checked before touching Docker at
all, returns a clean 503 CAPACITY_EXCEEDED instead of failing mid-provision.
Relayed as a clean student-facing message, never a raw 500. New
GET /capacity endpoint feeds the instructor dashboard.
- cohort_instructors.role_in_cohort (existed since Phase 2B, never read) is
now enforced: teacher vs viewer, rbac.js's
requireCohortWriteAccess/requireStudentCohortWriteAccess.
- GET /api/version - app version, deployed Git SHA, latest migration as a
schema-version stand-in.
- Structured assessment authoring UI (AssessmentsView.jsx's PoolSpecEditor)
replaces the raw-JSON pool_spec textarea, backed by a new
GET /api/admin/scenario-catalog endpoint. Owner-only raw-JSON fallback
remains for anything the structured editor doesn't cover.
- NEEDS_ATTENTION_FAIL_THRESHOLD/STALE_DAYS moved to env-var configuration.
Full regression suite (e2e-phase2a/2b/3a/3b, 67/67) re-run clean after
these changes; frontend production build confirmed clean.
Replaces the in-memory sliding-window Map (reset on every restart, not
shareable across instances) with a login_attempts table (migration 016).
No Redis added - login is not a hot path, and this is a single indexed
range-scan query. Restart behavior confirmed for real:
platform/test/e2e-phase3a.js's rate-limiting test passes after a genuine
Platform API restart in the same run.
scenario_attempts.status = 'abandoned' (a valid value since Phase 3B,
never previously set by anything) now gets set on: environment
destroy/reset, logout, session expiry (piggybacked on the existing hourly
session sweeper - no new background job), and switching to a different
scenario while one is still open. Never an academic failure marker -
result stays NULL, history is preserved. Call-site wiring in
server.js/admin-routes.js lands in the next commit alongside other
platform hardening in those same files.
Adds platform/lab-router/ — a narrow service that runs on the Lab Host,
authenticates the Gateway via mTLS, and resolves an opaque routing token to
a local container purely from Docker label state (no Postgres access, no
callback to the Provisioner). Replaces the Gateway's env.internal_ip
assumption (only valid when Gateway and Lab share a Docker daemon) with a
data-driven direct/router decision per lab_host
(lab_hosts.routing_mode/router_endpoint, migration 004), resolved in one
place (platform/gateway/src/lab-endpoint.js's labTarget()) that every
proxy/fetch call site in server.js now goes through.
Extends the existing Docker TLS CA (infra/scripts/docker-tls/generate-docker-tls.sh)
with a second mTLS leaf pair (router-server/gateway-client) for this new
data-plane connection, deployed by the docker-tls Ansible role into a
separate directory from the control-plane pair. Firewall
(rn-lab-host-01.rules) replaces the old speculative wide port-range
allowance with one narrow rule: platform_ip -> the Router's port only.
dev-local stays routing_mode: direct — zero behavior change for local dev,
confirmed by the full existing regression suite passing unmodified.
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs
(PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR,
OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST,
and 12 runbooks (first deployment, normal deploy, rollback, Postgres
restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner
recovery, stuck Lab cleanup, failed migration, backup failure, disk
failure, Lab Host resource pressure).
PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine
architectural gap this phase surfaced: the Gateway's existing
env.internal_ip mechanism for reaching a Lab container assumes Gateway and
Lab share a Docker daemon, which breaks once Labs run on a genuinely
separate rn-lab-host-01 VM. The network-policy half is built (a firewall
allowance for Gateway->Lab-Host on a published port range); the
application half (Provisioner/Gateway address handling) is an explicit,
reviewed, NOT-YET-BUILT follow-up flagged there rather than silently
decided.
Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and
platform/README.md with Phase 4 pointers, consistent with how every prior
phase updated these.
Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
Unauthenticated liveness/readiness probes for uptime monitoring and Ansible
deploy smoke tests (Phase 4). API's checks Postgres reachability via
SELECT 1; Gateway's answers locally without proxying anywhere, so it can
distinguish 'Gateway itself is down' from 'something behind it is down'.
No student/instructor data exposed.
Verified live against the real dev stack: both return 200 with the
expected JSON body. Full regression suite re-run after this change:
e2e-phase2a (10/10), e2e-phase2b (12/12), e2e-phase3a (21/21), e2e-phase3b
(24/24) - all passing, no regressions. Frontend build and the Lab image
(both the multi-service dev compose and the standalone single-container
Dockerfile) also confirmed still building/booting correctly.
boot-labs.js drives the real admin Prepare-Labs endpoint (reusing the
existing session-helper.js from Phase 3A) and records boot time/failure
rate; sample-resources.sh samples real docker stats for running Labs.
capacity-model.js turns measured per-Lab CPU/memory into a recommended/
warning/hard-admission-control concurrency threshold - no concurrency
number is hardcoded as a marketing figure, every input is explicitly
placeholder data until measured against the real host.
Plus deploy/rollback/smoke-test scripts used by the CI/CD and deployment
runbooks. All syntax-checked; validate-compose.sh and capacity-model.js
actually run against real inputs.
Automatic test/build/tag/staging-deploy on push to main; production
promotion stays a manual, human-run step (deploy-production.sh) - never
automated. SHA-tagged images always, semantic tags only on deliberate
release, never :latest alone in staging/production. No paid CI provider or
registry required (uses Gitea's own). Real regression job installs all
services, applies real migrations against a service Postgres, runs the
infra validation scripts.
Validated as real YAML (python3 yaml parse).
pg_dump -> gzip -> gpg AES-256 symmetric encryption, plaintext deleted
immediately, optional off-host rsync-over-SSH copy (Hetzner Storage Box or
any SSH target), retention pruning, Prometheus textfile-collector metrics
for backup success/age/size.
Genuinely tested end-to-end, not just written: real production migrations
(17 total, correct cross-service order) applied to a disposable Postgres,
sample data inserted, backed up, restored into a second disposable
Postgres, verified table-for-table and byte-for-byte. Exit 0.
Named-bridge-network production Compose (not host networking, unlike dev)
for rn-platform-01; egress-proxy + agents for rn-lab-host-01. Caddy reverse
proxy, single public origin, Grafana restricted to management_cidr.
iptables-restore firewall rulesets for all three hosts (Proxmox host,
rn-platform-01, rn-lab-host-01) codifying the Platform/Lab-Host boundary:
no student-to-student traffic, no direct Proxmox exposure, no direct
Postgres/Provisioner exposure from the Lab Host.
Mutual TLS for the Provisioner<->Lab Host Docker connection: certificate
generation/verification scripts, dockerd TLS config, firewall-restricted
port 2376. Application support already existed since Phase 1; this is
purely the operational half.
Validated: docker compose config, caddy validate, firewall structural lint,
and a real openssl s_server/s_client mTLS handshake proving the cert chain
and mutual-auth rejection (legitimate client verified, rogue client
correctly rejected with verify code 19).
common/firewall/docker-tls/platform/lab-host/monitoring-agent/backup roles,
orchestrated by site.yml with the ordering that matters (firewall/TLS land
before the services that depend on them). Ansible Vault for secrets, no new
tool introduced for this project's modest secret surface. Explicit
dev/staging/production environment separation, no secrets committed
(.gitignore covers vault plaintext, TLS keys, Terraform state, *.env).
Validated: ansible-playbook --syntax-check clean.
New docs/PHASE3B_LEARNING_AND_ASSESSMENT.md covers the attempt/learning-record model,
progressive hints (including the Gateway response-rewrite mechanism), curriculum completion
rules, the assessment/exam engine and its timer/draw integrity guarantees, the RootNode
Capstone design, achievement records, the deliberately minimal versioning model, instructor
analytics, the Phase 3B RBAC matrix, security/integrity assumptions, dev/test commands, known
limitations, and the Phase 4 boundary.
Updates ENGINEERING_SOP.md (roadmap items 2/7/8 now done, new section 12, corrects a stale
capstone-placeholder description in section 4), LAB_ARCHITECTURE.md's status line,
README.md (corrects the 'Lab image unmodified' claim now that its frontend bundle carries
Phase 3B UX, plus new platform properties), platform/README.md, and platform/api/README.md.
24 cases: attempt telemetry (start/fail/pass counters, duration), progressive hint sequencing
and exhaustion, hint persistence through a real Lab reset, curriculum stage lock/unlock,
capstone lock-before-prerequisites/unlock-after/random-draw-includes-required-Real-Ops/
cannot-double-start/completion-records-achievement, exam assignment/draw/timer/hint-policy,
cross-student session-read rejection (404, not 403 - consistent with the existing don't-confirm-
existence posture), two-student isolation, timer survives a Lab reset, an assessment session
survives a REAL docker compose restart api, and instructor analytics/needs-attention/RBAC.
Found and fixed two real product bugs in the process: result needed to become nullable (an
in-progress attempt has none yet) and hint_count needed a real DEFAULT 0 (Phase 2B left it
NULL/undefaulted since it wasn't measured yet - it is now, and NULL + 1 stays NULL in SQL).
StudentDashboard is now tabbed: My Learning (the original Phase 2A/2B flow, extracted
unchanged into MyLearningTab and extended with an overall percent-complete bar), Practice,
Assessments (available/active-with-live-countdown/history), Capstone (locked checklist ->
unlocked -> start/resume -> completed), Achievements (a flat dated list, not a leaderboard).
Inside the Lab UI: ProgressiveHints replaces the static hint array render when the Gateway has
stripped hints (detected via scenario.hints being undefined but hint_count present - standalone
dev mode never sees this, so it keeps the original behavior); a new top-level AssessmentBanner
shows a live countdown and completion count whenever a Platform assessment is running,
rendered alongside (not replacing) the pre-existing Lab-local exam-mode UI, which is untouched.
New /api/admin routes: cohort learning summary (completion by stage, active learners),
per-student learning detail (attempts, hint usage, assessment history, capstone state),
scenario analytics (completion rate, avg attempts, failure/hint frequency, avg time), and a
needs-attention view (repeated failures, hints used but still stuck, inactivity, an expired
assessment - deliberately instructional wording, not punitive, with an explicit test asserting
that). Assessment definition CRUD (admin/owner only), cohort assignment and session view/cancel
(any instructor role, cohort-scoped for a plain instructor via the existing Phase 3A RBAC
helpers - no new authorization primitives needed).
New Learning and Assessments tabs in the admin portal, visible to every instructor role;
definition-authoring controls are shown but the server 403s a create/edit from a non-admin/
owner role, same pattern CohortsView already uses for admin-only controls.
New learning-routes.js: scenario start/attempt-state/hint (sequential reveal, 409 once
exhausted, policy-aware under an active assessment), GET /api/curriculum/state (rule-driven,
alongside the unchanged /api/curriculum/path), assessments list/start/active/detail/submit,
GET /api/capstone, GET /api/achievements, GET /api/attempts/recent.
/api/progress/record now routes through attempts.recordValidation instead of a flat insert,
and, on a pass, checks curriculum/capstone achievement consequences and, if the scenario is
part of an active assessment, updates that session (assessment context is derived entirely
server-side from assessment_session_scenarios, never from anything the client asserts).
Gateway changes: two new relay routes (/lab/:token/api/scenarios/:id/start and .../hint,
mirroring the existing validate/answer pattern - pure Platform API operations, the Lab has no
attempt/hint concept) and a response rewrite on GET /lab/:token/api/scenarios/:id that strips
the Lab's own hints array (replacing it with a bare hint_count) so progressive reveal is
actually server-gated instead of just UI-collapsed - the Lab image's bytes stay untouched;
standalone/single-container dev mode never goes through the Gateway and keeps the full array
unaffected.
New achievements table: stage_completed/curriculum_completed/assessment_completed/
capstone_completed, idempotent (WHERE NOT EXISTS guard, not a unique-index dance — simple and
correct at this write volume). Not a certificate — a durable internal record Phase 4/5 can
render one from later. recordAssessmentAchievement only fires on an actual pass, defensively,
even though today's only completion_rule already guarantees that.
New tables: assessment_definitions (reusable template — pool_spec, duration, hint_policy,
completion/unlock rules, a plain incrementing version bumped only on semantic edits),
assessment_cohort_assignments, assessment_sessions (one per student attempt, full snapshot of
what was drawn and under what rules — never affected by a later definition edit),
assessment_session_scenarios (the drawn set, ordered, with a real FK to scenario_attempts).
Deliberately does not touch/reuse exam_sessions (migration 006) — that was already scoped in
Phase 2B as historical-import-only, never the live mechanism; these tables are the real thing.
drawScenarios: every 'required' scenario id included, plus N random picks per pool group,
shuffled together (tiny seeded PRNG, mulberry32 — no new dependency; the seed is recorded for
audit, never shown to the student before the draw). One assessment at a time per student.
Timer is server-authoritative (expires_at fixed at draw time) and lazily expired — checked
whenever a session is touched, no background cron. Ships two seeded defaults: rootnode-capstone
(Real Ops scenarios required, weighted toward operational ability) and rootnode-quick-assessment
(a general example exam matching the spec's category-draw example).
Evolves scenario_attempts (Phase 2B: one row per validation click) into a real attempt
lifecycle — one row per continuous working session, with live counters (validation
attempts/passes/fails, hint reveals, duration) updated in place, not re-inserted. Backward
compatible: every new column is nullable/defaulted, existing rows backfilled under the new
semantics rather than dropped. hint_count (Phase 2B, always NULL — 'not measurable') finally
gets a real DEFAULT 0 and increments, since this is the phase that builds real hint tracking.
curriculum-rules.js computes rule-driven stage lock/complete/percent/next-scenario state from
scenarios/rootnode-path.json's new per-stage completion_rule (all_required/threshold/
prerequisites) against a student's progress. The capstone stage is no longer a hardcoded
locked placeholder — it carries an unlock_rule and a pointer at the assessment definition that
actually defines its scenario draw (see the assessment-engine commit).
Was ambiguous — backend/frontend committed lockfiles, api/gateway/provisioner didn't, with
no stated reason either way. Now explicit: every first-party Node service commits its
package-lock.json, validated with npm ci before each commit that touches it.
backend/ and frontend/ already had committed lockfiles; platform/api/ just gained one.
gateway/ and provisioner/ never had one generated at all — not a deliberate policy, an
inconsistency. Generated via npm install --package-lock-only and validated with a clean
npm ci for both (72 and 153 packages respectively, no resolution surprises).
Was generated locally during Phase 3A work but never committed. Verified it corresponds
to the current package.json via a clean npm ci (118 packages, no resolution changes).
New docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md covers the schema, persistent session
design, instructor accounts, the full RBAC matrix, cohort scoping (including the documented
raw-environment-id simplification), student session hardening (explicitly still
classroom/pilot identification, not production auth), CSRF, rate limiting, audit logging,
the API inventory, a security hardening checklist, test results, known limitations, the
future OIDC path, and production HTTPS/cookie expectations — with an explicit BUILT NOW vs.
FUTURE split.
Updates ENGINEERING_SOP.md (new §9/§10, closing a stale '(§9)' forward-reference that dated
back to Phase 2B), LAB_ARCHITECTURE.md status line, README.md, platform/README.md, and
platform/api/README.md (bootstrap-owner usage, new config vars, no more ADMIN_PASSWORD).
New e2e-phase3a.js (21 cases): no account enumeration, CSRF positive/negative, login rate
limiting, RBAC across all three roles, cohort-scoped instructor access (visibility + route
access), last-owner protection, live role/status re-evaluation (no separate revoke needed),
password change (old password rejected after, other sessions revoked), audit log content
(including a no-secrets-in-metadata assertion), student session RBAC/logout, and session
persistence across a REAL docker compose restart api mid-test.
New session-helper.js — a shared cookie-jar test helper (Node's fetch doesn't persist
cookies across calls) that also auto-attaches X-CSRF-Token from whichever rn_csrf_*
cookie it's holding, replacing the single-cookie-string trackers e2e-phase2a.js and
e2e-phase2b.js each had their own copy of.
e2e-phase2a.js and e2e-phase2b.js updated for the new auth model: admin login is now a real
bootstrapped OWNER_EMAIL/OWNER_PASSWORD account instead of the old shared ADMIN_PASSWORD.
All cases still passing: 10/10 and 12/12 respectively.
No more shared instructor password env var — the api service now needs only
PROVISIONER_SHARED_SECRET. First-run setup is npm run bootstrap-owner instead.
Login form is now email+password (was a bare shared password field). Nav conditionally
shows Instructors (owner only) and Audit (owner/admin) tabs based on the signed-in
instructor's role. New InstructorsView (create/list/edit accounts, role/status controls,
session list + revoke-all, password reset — owner only), AuditView (read-only audit event
table), AccountView (change-password form for the signed-in instructor). CohortsView hides
the create-cohort form and curriculum-assignment button for the 'instructor' role — the
server already enforces this; this just avoids showing controls that would 403.
Non-HttpOnly rn_csrf_s/rn_csrf_i cookies, set on login, echoed back as X-CSRF-Token on every
state-changing request — defense in depth on top of (not instead of) SameSite=Lax. In-memory
sliding-window rate limiting on instructor login (8 attempts / 5 min per ip:email, no new
dependency).
Real bug found while wiring this up: the Gateway's own progress-relay call (triggered when a
student solves a scenario) is a server-to-server POST to /api/progress/record relaying the
browser's session — it needs to forward the CSRF token too, and needs to forward BOTH the
rn_csrf_s cookie and a matching X-CSRF-Token header (forwarding only the header still 403'd,
since the double-submit check needs both on the request Platform API actually receives).
Also tightens the Gateway's session-resolution cache from 5000ms to 2000ms, so a
revoked/expired session stops working at the Gateway within a tighter, tested bound.
Frontend (admin + student API clients) now reads the relevant rn_csrf_* cookie via
document.cookie and attaches it on every POST/PATCH/DELETE.
Three roles (owner > admin > instructor), enforced server-side on every /api/admin/* route
via reusable middleware (requireOwner/requireAdminOrOwner/requireAnyInstructor) — the
frontend hides controls a role can't use, but that's UX, never the actual boundary. A plain
'instructor' is further scoped to cohorts assigned via the new cohort_instructors table
(requireCohortAccess/requireStudentCohortAccess); owner/admin bypass that check entirely.
Owner-only: instructor account lifecycle, audit log. Admin/owner: cohort/student CRUD,
curriculum assignment. Any instructor role (cohort-scoped for 'instructor'): dashboard,
cohort/student views, Prepare/Destroy Labs, provisioning. Deliberately NOT cohort-scoped:
the raw /api/admin/environments/:id/* routes (no cohort context in a bare environment id
without an extra Provisioner round trip on every call) — a plain instructor still has full
Lab control via the cohort/student-scoped routes instead, which already carry the check.
server.js wires the new persistent-session middleware (studentSession/instructorSession —
re-fetches the instructor row from Postgres every request, so a role change or
deactivation takes effect on the very next request with no separate revoke needed) and adds
a last-resort error handler so a DB error in any async route returns 500 instead of
crashing the process or leaking a stack trace.
New audit_events table, distinct from the Provisioner's environment_events (Lab-lifecycle
only). recordAudit() is called from the auth/RBAC/admin layers added in this phase — login
success/failure, logout, password change, instructor CRUD, role changes, session
revocation, cohort/roster changes, curriculum assignment, Prepare/Destroy Labs, and every
403 authorization denial (recorded centrally by the RBAC middleware, not per-route). Never
records plaintext passwords, raw session tokens, Provisioner secrets, or raw routing tokens.
Replaces the single shared ADMIN_PASSWORD (Phase 2B) with real accounts: instructors table
(email/display_name/role/status), scrypt password hashing with a fresh random salt PER
INSTRUCTOR (Phase 2B used one fixed salt because there was exactly one shared password —
that assumption is gone), auth_identities table so a future OIDC login can map to the same
instructor_id without a schema change.
New POST /api/auth/instructor/login, POST /api/auth/logout, GET /api/auth/session,
POST /api/auth/change-password (old password required, revokes other sessions). No account
enumeration: identical response whether the email doesn't exist or the password is wrong.
bootstrap-owner.js is the only supported way to create the first account — no manual
database editing to stand up a fresh deployment. Refuses to run if an active owner already
exists unless the target email is an existing instructor (zero-owners recovery path).
Replaces the in-memory studentSessions/adminSessions Maps (Phase 2B — gone on every
restart) with a sessions table. Browser holds an opaque CSPRNG bearer token; only its
SHA-256 hash is ever stored server-side, same principle as password hashing. Each session
has an absolute expiry (fixed at creation) and a sliding idle expiry (pushed forward on
every authenticated request, capped at the absolute one) — one atomic UPDATE...RETURNING
per resolve, no read-then-write race. Student and instructor sessions stay fully separate
token namespaces, matching Phase 2B's cookie-name separation.
cookies.js centralizes Set-Cookie construction so the Secure attribute (COOKIE_SECURE=1,
for real HTTPS deployments) is applied consistently instead of hand-built per call site.
New docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md covers the Platform API
split, Postgres schema and migration guarantees/rollback, the session
cookie architecture (including the whoami-routing-bug fix), the admin
portal, the API/security boundary, known limitations, and Phase 3+
future work.
Updates ENGINEERING_SOP.md §8, LAB_ARCHITECTURE.md, README.md, and
platform/README.md to reflect Platform API as the sole Provisioner
caller and Postgres as authoritative for identity/progress as of
Phase 2B.
Full acceptance test: admin auth (+ student-can't-call-admin check),
cohort create, 3-student add, curriculum assign, concurrent Prepare Labs
(3/3 ready, instant-reuse timing check), two students completing
different scenarios, dashboard reflecting completions, admin reset with
progress-intact assertion, admin destroy-labs with progress-intact
assertion for everyone, and a regression check that no Lab container
publishes a host port. 12/12 passing.
Removes the old identity service/volume, adds the api service (host
networking, Postgres URL, Provisioner shared secret, ADMIN_PASSWORD,
both required via :?). gateway now points PLATFORM_API_URL at it instead
of the old IDENTITY_BACKEND_URL.
New /admin route (frontend/src/admin/), gated by a separate admin
session (scrypt + timingSafeEqual against ADMIN_PASSWORD, no plaintext
storage, no new dependency). Covers cohort CRUD + roster + curriculum
assignment, and student CRUD + cohort assignment + environment/progress/
attempt views, all under /api/admin/* on the Platform API.
main.jsx now does a three-way path split (lab / admin / platform) so the
existing, unmodified Lab UI, the student dashboard, and the new admin
portal all ship in one Vite bundle without a router dependency.
Replaces the Phase 2A pilot ?sid= query param with an HttpOnly,
SameSite=Lax session cookie. Gateway now derives student identity from
the session (via Platform API's /api/session/whoami) instead of trusting
a client-suppliable id, and checks ownership on every proxied HTTP and
WebSocket Lab request, not just the initial navigation.
Fixes a routing bug found during testing: the fallback route was using
'does this session own a running environment' to decide whether an
unprefixed /api/* call belonged to the Lab UI or the dashboard — true in
both places. Replaced with an explicit LAB_OWNED_PATHS allowlist of the
exact unprefixed paths the existing Lab UI actually calls.
e2e-phase2a.js rewritten to use a cookie jar instead of headers/query
params, with explicit impersonation-rejection tests added. 10/10 passing.
New platform/api/ service owns students, cohorts, curriculum, progress,
scenario attempts, and student/admin sessions. No k3s, no --privileged,
no terminal runtime — starts instantly, unlike the old identity service
which booted an unused k3s cluster just to serve roster/progress APIs.
Provisioner calls (student and admin) are consolidated here too, in
provisioner-client.js, so it's the only service that talks to the
Provisioner directly.
- docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md — the full Phase 2A record: node-pty
root cause and fix, Gateway architecture, student flow (entry through
reset/recreate), API routes added, HTTP/WebSocket routing design (and why
the lab-session cookie exists), security model (what's checked, and the
explicit statement that this is pilot identity, not production auth),
reset/recreate and reaper-recovery semantics, and known limitations found
during implementation (the identity instance's k3s resource cost and its
network_mode: bridge requirement, the Gateway's in-memory cache scope, the
?sid= check's coverage). BUILT NOW vs. FUTURE (Phase 2B) called out
explicitly per the governing requirement.
- docs/LAB_ARCHITECTURE.md, docs/ENGINEERING_SOP.md — status updated to
reflect Phase 2A is implemented and tested, not just Phase 1; roadmap item 5
updated accordingly; ENGINEERING_SOP's Provisioner section corrected (it's
no longer un-fronted — the Gateway is the only caller now).
- README.md — Multi-Student Deployment section updated to describe the
Gateway and student flow, not just the Provisioner; clarifies the
node-pty fix is the only change to the Lab image itself.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
platform/test/e2e-phase2a.js — 15 assert-based tests against a live
platform/docker-compose.dev.yml stack, exercising the full student flow
through the Gateway exactly as a browser would: roster -> login ->
curriculum -> launch (real k3s boot) -> Gateway HTTP routing -> invalid-token
rejection -> cross-student ownership rejection (?sid= mismatch) ->
same-student access -> progress relay (solve in one student's lab, confirm
it shows in their curriculum and NOT the other student's) -> reset (stable
launch_path, progress survives).
All 15 pass. This is slow (each real environment launch is a genuine k3s
cold start) and deliberately not meant for every commit — see
platform/README.md for when to run it.
Does not cover what needs a browser/WebSocket client — real Lab image
terminal (node-pty, kubectl execution, scenario validation) and idle-reaper
timing were verified manually; see
docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md §8 for exactly what and how.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
StudentDashboard.jsx — "Up Next" curriculum view + Launch Lab flow, in one
component since they share state (environment status affects what curriculum
actions make sense).
- Renders GET /api/curriculum/path as-is: the existing RootNode path
(Foundation -> Workloads -> Networking -> Configuration -> Troubleshooting
-> Real Ops -> Capstone), no new structure invented. Capstone stays the
locked placeholder from the earlier slice.
- Launch Lab -> POST /api/environment/launch, polls status every 2.5s while
provisioning/resetting, transitions through none -> provisioning -> running
-> error with a distinct UI state for each (EnvironmentPanel). On running,
navigates to launch_path with the student's own id as ?sid= — a
defense-in-depth ownership check the Gateway verifies on that specific
navigation (see the gateway commit).
- One "Recreate Lab" control, calling POST /api/environment/reset — no raw
destroy-without-recreate exposed to students, matching "don't expose raw
destroy/admin controls beyond what is safe for a student." A visible note
states progress is saved separately and survives.
Platform.module.css reuses the existing design tokens from index.css
(--green, --surface, --text, etc.) so this reads as the same product as the
Lab UI, not a bolted-on separate app.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
New Platform Shell entry point (frontend/src/platform/) alongside the
existing, completely unmodified Lab UI (App.jsx and everything under
frontend/src/components/). main.jsx now splits on the current URL path —
/lab/<token>/ renders the existing <App/> exactly as before; anything else
renders the new <PlatformShell/>. Both live in the same built bundle (it's
served both by the persistent identity instance and by every ephemeral Lab
container), split purely at runtime.
- EntryScreen.jsx — cohort dropdown -> name dropdown (existing
GET /api/roster/cohorts, GET /api/roster?cohort=) -> POST /api/login. No
password. Visibly labeled pilot identity, not production auth, per the
governing requirement that this be stated explicitly, not just true in the
architecture docs.
- api.js — student_id lives in localStorage only, sent as X-Student-Id on
every authenticated request. Every authenticated call throws if no student
is stored rather than silently going out anonymously — this is what
"prevent accidental anonymous progress writes" means in practice, not just
a comment.
- PlatformShell.jsx — entry screen if no stored session, dashboard otherwise.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Real architectural gap found and fixed during Phase 2A testing, not part of
the original plan: scenario validate/answer requests proxied through
/lab/<token>/api/scenarios/:id/{validate,answer} were writing progress into
that ephemeral Lab container's own SQLite — which is destroyed on every
reset/reap. Confirmed by testing: solving a scenario never showed up on the
dashboard's curriculum view, and would have been silently lost on lab
recreate, directly contradicting "academic progress must survive lab
reset/recreate."
Root tension: scenario validation must run against the student's own live
cluster (only exists inside their Lab container) — it can't be re-run
anywhere else. But the canonical progress record needs to live on the
persistent identity instance, which never has the student's actual cluster
state applied to it.
Fix: this endpoint (POST /api/progress/record) records an outcome WITHOUT
re-running validation — it exists purely so the Gateway can relay a
validate/answer result it already obtained from the real Lab. The Gateway
forwards the real request to the Lab, returns that real response to the
browser unchanged, and separately calls this endpoint with the outcome. See
platform/gateway/src/server.js's relayScenarioOutcome().
Verified end-to-end: a scenario solved through a student's Lab shows up in
their curriculum immediately, survives their lab being destroyed and reset,
and is correctly attributed only to that student (not visible to a second
student's own curriculum view).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
New internal component (platform/gateway/) — the only student-facing path to
disposable Lab environments. Browsers never reach a Lab container, the
Provisioner, or Postgres directly.
Routes:
- /api/environment/{status,launch,reset} — student-facing glue (X-Student-Id),
calls the Provisioner internally (shared secret never reaches the browser).
Reuses an existing non-terminal environment on launch rather than always
provisioning a new one; reset destroys+recreates the same environment row.
- /lab/<token>/* — HTTP+WS proxy into that one student's Lab container,
resolved by opaque routing token via a new Provisioner endpoint
(GET /environments/by-token/:token). No raw container IP/port ever reaches
the browser. Rejects invalid tokens (404), unready environments (202/503),
and cross-student access via an optional ?sid= ownership check (403) — see
docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md "Security" for why token-alone,
student-id-alone, and env-id-alone are each individually insufficient and
this is explicitly pilot-grade, not production auth.
- everything else — proxied to the persistent identity/curriculum backend
(roster, login, curriculum, progress, platform-shell frontend), or to the
student's own Lab if a valid lab-session cookie is present.
Routing design note: the existing Lab UI makes root-relative calls
(fetch('/api/scenarios'), the /shell-ws WebSocket) with no idea it might be
served under a /lab/<token>/ prefix — preserving that UI unmodified was a
hard requirement. Solved with an HttpOnly, SameSite=Lax cookie set on the
first /lab/<token>/ response: every later request from that browser carries
it, so the unmodified Lab UI's own unprefixed calls resolve correctly once a
student has navigated into a lab session, with zero changes to its code.
/api/environment/* is explicitly exempted from cookie-based routing so it
never gets hijacked by an active lab session.
Also added platform/docker-compose.dev.yml (the full local stack: Postgres +
egress-proxy + Provisioner + a persistent "identity" instance + Gateway) and
platform/README.md tying the components together. The "identity" instance
required an explicit `network_mode: bridge` — under Compose's own
auto-created bridge network, k3s failed at startup with "no default routes
found in /proc/net/route"; documented as a known limitation, not fully
root-caused.
Found and fixed during testing: express.json() applied globally broke
http-proxy's re-streaming of proxied POST bodies (e.g. POST /api/roster hung
forever, since the body had already been consumed by the parser before
proxy.web() tried to forward it). Removed — none of the Gateway's own routes
actually need a parsed body.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Root cause, confirmed empirically (not guessed):
1. node-gyp generates an incorrect build/config.gypi for this NodeSource-packaged
Node 20 build — it reports node_shared: true (expects a shared libnode.so),
but the actual installed Node is statically linked (no libnode.so.<abi>
exists anywhere on the system; process.config.variables.node_shared on the
real running Node is false). node-gyp trusted the wrong config and linked
node-pty's native addon against -lnode, producing a binary that fails at
require() time with "libnode.so.127: cannot open shared object file" —
confirmed by loading the compiled .node file directly, bypassing node-pty's
own path-search/error-masking logic (which collapses this into a generic
"Cannot find module .../pty.node" across every path it checks).
2. A second, independent bug compounded it: no .dockerignore existed. A
host-side `npm install` (run earlier for backend testing) left
backend/node_modules on disk, built against the host's own environment. The
Dockerfile's `COPY backend/ ./backend/` (placed after the npm-install RUN
layer, for build-cache efficiency) silently overwrote the correctly-built
node_modules with that host copy — including after a first .dockerignore
fix using the pattern `node_modules/`, which only matches a node_modules
directory at the build context ROOT in Docker's matching rules, not at any
depth (unlike .gitignore, where the same-looking pattern does match any
depth — a real, confirmed gotcha, not an assumption).
Fix:
- Dockerfile: `ENV npm_config_node_shared=false` (image-level, not a one-off
RUN prefix, so it can't be silently lost if a native module ever gets
rebuilt again for any reason) + a build-time `node -e "require(...)"` check
that fails the build loudly if this regresses, instead of shipping a silent
runtime failure.
- .dockerignore: `**/node_modules/` (recursive), not `node_modules/`.
No terminal architecture change, no fake shell, no changes to
backend/server.js's PTY handling. RootNode branding and upstream KubeKosh
attribution untouched. The single-container dev workflow
(`docker build -t rootnode-academy:test .`) is the same command as before —
it now just works.
Verified against the real image end-to-end, not a stand-in: build -> container
starts -> k3s reaches Ready -> backend starts, no crash -> nginx starts ->
browser UI 200s -> terminal WebSocket opens showing the RootNode banner ->
`kubectl get nodes` run through the terminal returns real cluster state ->
deploy-nginx scenario solved via the terminal -> all 5 validation checks pass.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
- docs/LAB_ARCHITECTURE.md — the full ephemeral-per-student design: two-VM
production topology (Platform VM / Lab Host VM, correction A), lifecycle,
data model, networking/egress model (correction B), resource model,
routing/auth model, security boundaries, failure/recovery behavior,
migration path from the current single container, and the lab-host
abstraction (correction D). Status updated to reflect Phase 1 is now
implemented, pointing at the two docs below for specifics.
- docs/PHASE1_PLATFORM_FOUNDATION.md — the Phase 1 build plan: exact Postgres
schema, service boundaries, API routes, lifecycle state machine, Docker/
firewall commands, security assumptions, file layout, and a 12-test plan.
Updated with "Implementation notes" at the points where testing surfaced
real deviations from the original design: the per-network (not static)
egress-proxy address resolution, the COALESCE reaper fix, and the
provisioning-failure cleanup fix.
- docs/ENGINEERING_SOP.md — new §8 mapping the platform/provisioner/ and
platform/egress-proxy/ components, what's built vs. not (no Gateway, no
frontend wiring — internal API only), and where identity still lives
(unchanged: Lab image's own SQLite). Roadmap item 5 (per-student isolation)
updated from "not started" to "Phase 1 done, not yet cleared for minors."
- README.md — Multi-Student Deployment section rewritten: the old per-student-
volume persistence model is superseded by the Platform's registry (progress
already lives outside the disposable container as of the prior curriculum/
roster work); points at docs/LAB_ARCHITECTURE.md and platform/provisioner/
for the real architecture instead of describing three delivery modes that
were never actually built out.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
platform/provisioner/test/run.js — 12 assert-based integration tests against a
live docker-compose.dev.yml stack (no framework, matching the project's
existing convention). Covers: health/auth, provision with unique
network/container/token, duplicate-provision rejection, distinct
subnets/IPs for concurrent students, the egress firewall ordering fix
specifically (proxy ACCEPT before the RFC1918 DROP for the same range),
student-to-student isolation, RFC1918 blocking, reset (same identity, fresh
container), destroy (deterministic cleanup), and the audit trail (all
expected events present, routing token never logged).
All 12 pass against the default lightweight test image. Several properties
were validated manually during implementation rather than automated here —
idle reaper timing (needs a real threshold wait), remote TCP+TLS lab-host
(needs a Docker-in-Docker sidecar, too heavy for routine runs),
reconciliation-on-crash (needs an out-of-band container kill + process
restart), and real-Lab-image k3s behavior (environment-specific, see the
Phase 1 final report and platform/provisioner/README.md's known limitations)
— each is noted at the bottom of the file with why it's not automated here.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Background loop (src/reaper.js), config-driven idle/abandoned thresholds and
poll interval (no hard-coded numbers) — without this, "ephemeral by default"
is just a description: a closed browser tab does not stop a running k3s
cluster on its own.
Past the idle threshold: logs a warning event only. Past the abandoned
threshold: destroys via the same path DELETE /environments/:id uses, so
cleanup is identical either way, and logs event_type='reaped_idle'.
Failure-safe — one environment's teardown throwing never stops the reaper
from checking the rest, and a Postgres/Docker hiccup on one tick just means
"try again next tick," never a crashed process.
Uses COALESCE(last_active_at, created_at) rather than bare last_active_at.
Found during testing: an environment that fails during provisioning (before
ever reaching 'running') has a NULL last_active_at, and NULL < anything is
never true in SQL — without the fallback, a failed-and-never-cleaned-up
environment would never have aged out here regardless of how long it sat
there. This is the second, independent safety net on top of
provisionEnvironment's own best-effort cleanup on failure (see the
platform-provisioner commit).
Verified end-to-end: backdated an environment's last_active_at past the
abandoned threshold, waited for a real reaper tick, confirmed full
deterministic cleanup (container, network, firewall chain all gone) with the
correct actor='reaper' attribution in environment_events.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Egress enforcement is a firewall-layer control, not an application-layer
convention (correction B). HTTP_PROXY/HTTPS_PROXY configure containerd for a
clean pull path; they enforce nothing on their own.
- src/egress.js — per-environment dedicated iptables chain
(ROOTNODE-ENV-<short-id>), built with -A (append, order-preserving) rather
than the -I (insert-at-head) pattern an earlier design draft used, which
would have installed rules in reverse order and let the default-deny shadow
the proxy/DNS accepts. Required infrastructure (egress proxy, DNS) is
explicitly permitted first, then RFC1918 ranges are dropped unconditionally,
then a final default-deny catches the public internet. One jump rule per
environment hooks its subnet into its chain from DOCKER-USER (the chain
Docker guarantees it won't silently rewrite). Destroy is deterministic:
unhook, flush, delete — no comment-grep matching.
The ordering fix is empirically verified, not just reasoned about: the
egress proxy's own address is itself an RFC1918 address (it's Platform-side
infrastructure), and testing confirms it's reachable while the broader
RFC1918 range is still blocked — exactly the scenario the original ordering
bug would have broken.
- platform/egress-proxy/ — shared Squid instance (one per deployment, not
per-student), destination-allowlisted (dstdomain ACL against a plain-text
allowlist file), starting set covers docker.io/ghcr.io/gcr.io/quay.io/k8s.io.
Proxy performs domain allowlisting; the firewall chain above is what makes
it unbypassable, not the other way around.
The egress proxy's address is resolved per-student-network (each network's
Docker gateway IP, since the proxy runs with host networking and is present
on every bridge gateway) rather than a fixed hostname — a static address
wouldn't be reachable from more than one isolated student network at a time.
Found while testing with two simultaneous environments.
Verified: student-to-student isolation (A cannot reach B's internal IP),
RFC1918 blocking (A cannot reach a simulated trusted-infra address), and the
proxy allowlist itself (CONNECT tunnel to an allowlisted host succeeds,
non-allowlisted gets a clean 403 from Squid).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Internal-only service (platform/provisioner/) implementing the environment
lifecycle from docs/PHASE1_PLATFORM_FOUNDATION.md: provision -> running ->
(reset -> running | destroy -> destroyed), plus experimental suspend/resume
(decision #3 — never on the guaranteed path, never relied on for correctness).
- src/docker.js — the lab-host abstraction (correction D). Resolves a
lab_hosts row into a connected dockerode client, unix:// socket or tcp://
with TLS client-cert auth. Refuses to connect to a tcp:// endpoint with no
TLS material configured rather than connecting unauthenticated. Validated
against both a local socket and a real remote Docker-in-Docker TLS endpoint
during testing — same code path either way, only the lab_hosts config
differs, which is the entire point.
- src/lifecycle.js — provision/destroy/reset/suspend/resume orchestration,
wired to docker.js + egress.js + the Postgres registry. Every Lab container
is privileged (required for k3s, not avoidable), gets its own Docker network,
no published host ports, deterministic naming (rootnode-lab-<short-id> /
rootnode-lab-net-<short-id>), and a CSPRNG routing token. Includes
reconcileOnStartup(), implementing the "don't trust stale state after a
crash/restart" requirement from docs/LAB_ARCHITECTURE.md §8 — verified by
killing a container out-of-band and confirming a Provisioner restart flags
the mismatch instead of trusting the stale 'running' row.
Found and fixed during testing: a provisioning failure (e.g. readiness
timeout) left an orphaned container/network/firewall chain behind with only
the DB row marked 'error'. provisionEnvironment's catch block now does
best-effort teardown of whatever was already created before marking the row
error — confirmed by deliberately failing a provision and checking no
container/network/chain survived it.
- src/server.js — the internal HTTP API (X-Provisioner-Key shared-secret auth
on every route, matching the instructor-passcode pattern already used for
the roster endpoint). Input validation on student_id/environment_id,
structured JSON errors with a request id, routing tokens redacted from every
response.
- docker-compose.dev.yml — Postgres + egress-proxy + provisioner on one
machine (the documented Phase 1 dev exception). provisioner and egress-proxy
run with network_mode: host + NET_ADMIN/NET_RAW, which is what lets
src/egress.js manipulate the host's DOCKER-USER chain without needing
interactive host sudo.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
New Phase 1 Platform database — PostgreSQL, separate from the Lab image's own
SQLite (which keeps its current students/progress data untouched, decision #2 /
correction C in docs/LAB_ARCHITECTURE.md).
Schema (platform/provisioner/migrations/):
- lab_hosts: the abstraction that keeps Docker connectivity out of code —
endpoint (unix:// or tcp://+TLS), enabled flag. Phase 1 ships one row.
- environments: environment_id (UUID), student_id (opaque external identifier,
no FK — students table isn't in this database yet), lab_host_id (real FK),
status, routing_token (CSPRNG via crypto.randomBytes, opaque), container/
network identity, resource limits, lifecycle timestamps.
- environment_events: append-only audit trail, one row per lifecycle
transition, never logs the routing token in plaintext (see
lifecycle.js's redact()).
File-based migration runner (db.js) matching the check-before-alter idempotency
pattern already used in the Lab image's backend/server.js, applied on every
Provisioner boot.
Resource limits and the lab_hosts seed are config files
(config/resource-policy.yaml, config/lab-hosts.seed.yaml), not database tables —
there's no admin UI to manage them via a database yet, so a DB-backed policy
table would just be a write path nothing uses.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Pre-Phase-1 platform work, approved and tested earlier but not yet committed:
- Terminal banner: "RootNode Academy Lab — node: k8s-lab", upstream KubeKosh
attribution left untouched in footer/README/LICENSE.
- RootNode curriculum mode: scenarios/rootnode-path.json (Foundation -> Workloads
-> Networking -> Configuration -> Troubleshooting -> Real Ops -> Capstone),
built from existing scenario content grouped by category, no new scenarios
invented. New GET /api/curriculum/path endpoint.
- Student roster + per-student progress: instructor-roster, pick-your-name login
(no password), students/environments-adjacent progress table migrated to a
(student_id, scenario_id) composite key so practice progress and exam sessions
are isolated per student instead of one global bucket. New /api/roster,
/api/login endpoints.
- scenarios/data/*.json: added isolation_type metadata (shared_namespace vs
ephemeral_cluster) from an audit of which scenarios execute cluster-scoped or
node-level commands. This was Tier-1-isolation groundwork that got superseded
same-day by the full-ephemeral-per-student architecture (see
docs/LAB_ARCHITECTURE.md) — the field is kept as descriptive metadata only,
per the note now in scenarios/SCHEMA.md. Not wasted: it's an accurate audit of
which scenarios touch cluster-scoped resources, useful independent of isolation
model.
- .gitignore added (node_modules/, frontend/dist/, *.db, /data/, .env) — none
existed before; without it, npm installs during testing would have landed in
a commit.
Known limitation still open: no frontend UI for any of this yet (login screen,
roster-add form, curriculum view) — API-only. No auth on the roster-add endpoint.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
ced4568
