rootnode-academy/docs/PHASE4_1_PREPRODUCTION_HARDENING.md

Phase 4.1 — Pre-Production Hardening

Status: PHASE 4.1 COMPLETE. PRE-PRODUCTION HARDENING COMPLETE. READY FOR HARDWARE DEPLOYMENT.

Phase 4 built the production infrastructure package with the real Hetzner/Proxmox hardware not yet purchased, and flagged one genuine architectural gap: the Gateway's mechanism for reaching a student Lab assumed Gateway and Lab shared a Docker daemon, which is false the moment Labs run on a genuinely separate rn-lab-host-01 VM. Phase 4.1's job was to close that gap and harden everything else discovered along the way, still without the real hardware.

1. The core fix: a Lab Host Router

Before: Gateway → http://${env.internal_ip}:${env.internal_port} directly — a Docker bridge-local address, unreachable across a real VM boundary.

After: Gateway → (mode-aware) → either direct (unchanged, dev/single-host) or router (production): a small new service, platform/lab-router/, runs on the Lab Host itself, authenticates the Gateway via mTLS (the same private CA the Docker TLS control plane already used, extended with two new leaf certs), resolves an opaque routing token to a local container purely from docker ps label state (no Postgres access, no call back to the Provisioner), and reverse-proxies HTTP and WebSocket traffic to it.

The decision of which mode a given lab_host uses is data-driven (lab_hosts.routing_mode/router_endpoint, platform/provisioner/migrations/004_lab_hosts_routing.sql), resolved once per request in platform/gateway/src/lab-endpoint.js's labTarget() — the only place in the Gateway that knows this distinction exists. Every proxy/fetch call site in platform/gateway/src/server.js goes through it.

Exact env.internal_ip replacement: labTarget(env, token) returns { url, headers, agent, mode }. In direct mode, url is exactly the old http://internal_ip:internal_port. In router mode, url is the lab_host's router_endpoint, headers carries X-RootNode-Route-Token: <token> (computed server-side from the Provisioner-verified environment row — never trusted from the browser), and agent is an https.Agent configured with the Gateway's mTLS client identity.

2. No per-student host port

Deliberately not solved by publishing a host port per Lab (the originally-flagged, rejected approach). The Router resolves entirely from Docker's own label state on its own host — no port-per-student anything, no dynamic firewall rule per Lab.

3. Lab Host Router — design

See platform/lab-router/src/server.js's header comment for the full "narrow by design" list: one route (proxy-by-token), requires mTLS (no bearer-token fallback), never exposes Docker API/ container IDs/network info to its caller, never provides student-to-student routing (each request resolves to exactly the one container matching its own token).

4. Multi-Lab-Host ready

Routing is resolved per lab_host_id — adding rn-lab-host-02 later is one new lab_hosts row (its own routing_mode/router_endpoint/TLS material), no Gateway code change. No scheduler was built (explicitly out of scope) — the Provisioner still picks "the one enabled lab_host" (selectLabHost), unchanged from Phase 1.

5. Local development compatibility

dev-local (the Unix-socket lab_host used by docker-compose.dev.yml) stays routing_mode: direct — zero behavior change for the T480/any local dev setup, confirmed by the full existing regression suite passing unmodified (§29 below) and by every new drill in this phase running against the same dev stack. The standalone single-container Lab image build/boot is unaffected (it never touches the Gateway or Provisioner at all).

6-9. Simulation, HTTP/WS validation, security tests, routing token security

infra/scripts/simulate-two-host/ — a real two-Docker-network simulation (not mocks): Gateway attached ONLY to an isolated bridge with no route to any Lab's bridge network, Lab Host Router host-networked (mirrors egress-proxy's existing pattern) so it can reach whatever per-Lab bridge the Provisioner creates. 13/13 real assertions pass, including:

• Gateway genuinely cannot open a TCP connection to a Lab's Docker-internal IP (proven from inside the Gateway container's own network namespace via a raw connect attempt).
• Gateway reaches the Lab correctly through the Router — HTTP AND a real WebSocket terminal session running a genuine kubectl get nodes inside the Lab.
• Alice/Bob isolation (own lab ALLOW, other's DENY — HTTP and WebSocket both), garbage tokens, anonymous requests, and destroyed-environment tokens are all rejected through the real code path.
• The Router refuses any connection with no client certificate — mTLS is the real authentication mechanism.

Routing token security reviewed: already CSPRNG since Phase 1 (generateRoutingToken), never logged in plaintext (the Router's tokenFingerprint truncates), redacted in every API response except the two narrow internal reads that legitimately need it, and — critically — the header the Router trusts is computed server-side from the Provisioner-verified environment row, never read from the browser's own request, so a browser cannot forge routing by supplying its own X-RootNode-Route-Token.

10. Control plane vs data plane

Kept strictly separate — see docs/PRODUCTION_NETWORK_SECURITY.md's dedicated section. Student browser traffic is never proxied through the Docker API at any point.

11. Firewall update

infra/firewall/rn-lab-host-01.rules's old speculative wide port-range allowance replaced with one narrow rule: platform_ip → the Router's port only (8443 default, configurable).

12. Self-paced / remote delivery architecture check

docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md. Short version: student-driven provisioning, cohort-less students, and "idle reaper destroys → progress survives → student returns → fresh Lab → curriculum continues" all already worked before this phase and are now proven end-to-end by two new drills. The one real gap for actual remote/self-paced delivery is student authentication (today's bare student_id login assumes an instructor-controlled roster) — not built here, explicitly Phase 5 scope.

13. Abandoned attempt cleanup

scenario_attempts.status = 'abandoned' (already a valid value since Phase 3B, never previously set) now gets set on: environment destroy/reset (student- or instructor-triggered), logout, session expiry (piggybacked on the existing hourly session sweeper — no new background job), and switching to a different scenario while one is still open. Never a punitive marker — result stays NULL, history is preserved, instructors see an honest "abandoned (reason)" instead of a phantom "still in progress."

14. Gateway session cache

Reviewed — already a tight 2-second TTL since Phase 3A, specifically to bound the revocation-propagation window. Decision: keep it, don't add push-based invalidation — the added coupling (Platform API would need to know how to reach the Gateway) isn't justified by shrinking an already-2-second window further. Documented reasoning, not a silently-accepted gap.

15. Rate limiting

Login rate limiting moved from an in-memory Map (reset on every restart, not shareable across instances) to Postgres (login_attempts table, migrations/016_login_attempts.sql) — platform/api/src/security.js's tooManyAttempts(). No Redis added. Restart behavior confirmed for real: platform/test/e2e-phase3a.js's rate-limiting test passes after a genuine Platform API restart in the same run.

16. Pagination

Real server-side pagination (platform/api/src/pagination.js) on: the audit log (was a flat LIMIT 200), student attempt history (was LIMIT 100), and assessment session history (was fully unbounded — the one genuine growth risk found). Scenario analytics was deliberately left as-is — grouped by scenario id, naturally bounded around ~100 rows, no real growth risk to paginate against. Frontend (AuditView.jsx, AssessmentsView.jsx, StudentsView.jsx) updated to match; frontend build confirmed clean.

17. Needs-attention configuration

NEEDS_ATTENTION_FAIL_THRESHOLD/NEEDS_ATTENTION_STALE_DAYS moved from hardcoded constants to env-var-configurable (matching the existing session-TTL pattern), sensible defaults unchanged.

18. Assessment authoring UX

Raw-JSON pool_spec textarea replaced with a structured editor (frontend/src/admin/AssessmentsView.jsx's PoolSpecEditor): required scenarios and random pools picked from a real scenario catalog (GET /api/admin/scenario-catalog, new), grouped by curriculum stage, via native multi-select — no new UI framework. An owner-only "advanced" raw JSON view remains for anything the structured editor doesn't cover.

19. Cohort role-in-cohort

cohort_instructors.role_in_cohort existed since Phase 2B but was never read. Implemented minimally: teacher (unchanged full capability) vs viewer (read-only — dashboards/analytics still visible, but Prepare/Destroy Labs and provisioning a student's environment are now 403'd server-side, rbac.js's requireCohortWriteAccess/requireStudentCohortWriteAccess).

20. Backup / restore destructive drill

infra/backup/test-destructive-restore-drill.sh — real owner/admin/cohort/3-students/progress/ attempts/assessment/achievements seeded through the real Platform API, encrypted-backed-up, the disposable Postgres container genuinely destroyed (not just restored into a second container), a fresh Postgres created from nothing, restored, and the real Platform API booted again against the restored database and used to log in as the restored admin over HTTP. All data verified present; never touches the developer's real database.

21. Lab Host rebuild drill

infra/scripts/drills/test-lab-host-loss.js — a student's Lab container is destroyed out from under the Provisioner (no clean destroy call, simulating total Lab Host loss). Proves: Platform stays up throughout, the stale environment is never presented as healthy once the Provisioner's real reconcileOnStartup reconciliation runs, academic progress is untouched, and the student can launch a genuinely new Lab and continue.

22. Deployment rollback test

infra/scripts/drills/test-deployment-rollback.sh — builds a known-good and a deliberately broken gateway image, "deploys" each via the real deterministic-tag mechanism, proves the smoke test catches the regression, computes the rollback target with the exact same deploy-history.log logic rollback-production.sh uses, and proves the rollback recovers. Documents explicitly that image rollback never implies a DB migration rollback.

23. CI/CD failure tests

infra/scripts/drills/test-ci-fails-closed.sh — proves all six required failure modes (test/ syntax failure, frontend build failure, image build failure, config validation failure, migration preflight failure, staging smoke test failure) genuinely exit nonzero. Found and fixed two real pre-existing bugs: staging-smoke-test.sh and production-smoke-test.sh both had a classic set -e gotcha (curl ... && echo "OK" as one statement — a failing command that isn't the last in an && chain doesn't trigger -e) that let an unreachable/broken endpoint fall through silently instead of failing the smoke test. Fixed in both files.

24. Security scanning

infra/ci/gitea-workflows/build-and-test.yml's new security-scan job — npm audit, gitleaks (secrets), hadolint (Dockerfiles), shellcheck (shell scripts), trivy (container image CVEs). All free/open-source. Severity policy documented in docs/SECURITY_SCANNING.md: CRITICAL/HIGH blocks the build, MEDIUM/LOW is reported only. security-scan gates build-images the same way regression does.

25. Version / release visibility

GET /api/version (Platform API) — app version (package.json), deployed Git SHA (GIT_SHA env var, set to the same value as the image tag in the production compose file), and the most recently applied migration as a schema-version stand-in. Unauthenticated, minimal, no infrastructure detail.

26. Admission control failure mode

resource-policy.yaml's new (disabled-by-default) admission_control.max_concurrent_labs — when set, the Provisioner checks it BEFORE touching Docker at all, so a full host fails cleanly (503, CAPACITY_EXCEEDED) instead of failing mid-provision. Relayed by the Platform API as a clean "Lab capacity is temporarily full. Try again shortly." — never a raw 500. A new GET /capacity Provisioner endpoint feeds the instructor dashboard's capacity display.

27. Fake cohort dress rehearsal

infra/scripts/drills/test-fake-cohort-rehearsal.js — one continuous narrative: instructor login, cohort + 4 students, curriculum assignment, Prepare Labs, students working different scenarios, hints, a deliberate failed validation, instructor analytics, a Lab reset, a Lab reaped (same consequence as the real idle reaper — see the drill's own comment on why the real multi-minute timer isn't waited out), the student returning and continuing, an exam assigned/taken, and end-of-class Lab destruction — progress verified intact after every destructive step. 14/14 passing.

28. Documentation

This file, docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md, docs/SECURITY_SCANNING.md (new). Updated: docs/LAB_ARCHITECTURE.md, docs/PRODUCTION_NETWORK_SECURITY.md, docs/PRODUCTION_DEPLOYMENT.md, docs/PRODUCTION_VALIDATION_CHECKLIST.md, docs/BACKUP_DR.md, docs/ENGINEERING_SOP.md, README.md, platform/README.md.

29. Regression

Re-ran the full existing suite after every application-code change in this phase: e2e-phase2a.js (10/10), e2e-phase2b.js (12/12), e2e-phase3a.js (21/21 — including a real audit-log pagination-shape fix and confirming rate-limit persistence across a genuine restart), e2e-phase3b.js (24/24). Frontend production build clean. Real Lab image builds/boots (both dev-compose and standalone single-container). tofu validate, ansible-playbook --syntax-check, production Compose config (both platform and lab-host), caddy validate, promtool/amtool, and the backup/restore mechanism all re-validated clean after the infrastructure changes in this phase (new lab-router service, new firewall rule, new Ansible tasks). No knowingly-unexplained regression was left in this codebase at the end of this phase.

What Phase 4.1 deliberately did NOT build

Production deployment itself, Hetzner/Proxmox installation, public course signup, payments, subscriptions, parent portal, marketing site, full LMS, video hosting, email automation, certificates, course marketplace, mobile app, a multi-host scheduler, an AI tutor, invasive proctoring, held scenario content, student email/password authentication (documented as the one real gap for self-paced delivery, not built). All explicit non-goals per this phase's authorization.