rootnode-academy/docs/PHASE4_1_PREPRODUCTION_HARDENING.md

# Phase 4.1 — Pre-Production Hardening

**Status: PHASE 4.1 COMPLETE. PRE-PRODUCTION HARDENING COMPLETE. READY FOR HARDWARE DEPLOYMENT.**

Phase 4 built the production infrastructure package with the real Hetzner/Proxmox hardware not
yet purchased, and flagged one genuine architectural gap: the Gateway's mechanism for reaching
a student Lab assumed Gateway and Lab shared a Docker daemon, which is false the moment Labs run
on a genuinely separate `rn-lab-host-01` VM. Phase 4.1's job was to close that gap and harden
everything else discovered along the way, still without the real hardware.

## 1. The core fix: a Lab Host Router

**Before:** Gateway → `http://${env.internal_ip}:${env.internal_port}` directly — a Docker
bridge-local address, unreachable across a real VM boundary.

**After:** Gateway → (mode-aware) → either `direct` (unchanged, dev/single-host) or `router`
(production): a small new service, `platform/lab-router/`, runs on the Lab Host itself,
authenticates the Gateway via mTLS (the same private CA the Docker TLS control plane already
used, extended with two new leaf certs), resolves an opaque routing token to a local container
purely from `docker ps` label state (no Postgres access, no call back to the Provisioner), and
reverse-proxies HTTP and WebSocket traffic to it.

The decision of which mode a given `lab_host` uses is data-driven
(`lab_hosts.routing_mode`/`router_endpoint`, `platform/provisioner/migrations/004_lab_hosts_routing.sql`),
resolved once per request in `platform/gateway/src/lab-endpoint.js`'s `labTarget()` — the only
place in the Gateway that knows this distinction exists. Every proxy/fetch call site in
`platform/gateway/src/server.js` goes through it.

**Exact `env.internal_ip` replacement:** `labTarget(env, token)` returns `{ url, headers, agent,
mode }`. In `direct` mode, `url` is exactly the old `http://internal_ip:internal_port`. In
`router` mode, `url` is the lab_host's `router_endpoint`, `headers` carries
`X-RootNode-Route-Token: <token>` (computed server-side from the Provisioner-verified
environment row — never trusted from the browser), and `agent` is an `https.Agent` configured
with the Gateway's mTLS client identity.

## 2. No per-student host port

Deliberately not solved by publishing a host port per Lab (the originally-flagged, rejected
approach). The Router resolves entirely from Docker's own label state on its own host — no
port-per-student anything, no dynamic firewall rule per Lab.

## 3. Lab Host Router — design

See `platform/lab-router/src/server.js`'s header comment for the full "narrow by design" list:
one route (proxy-by-token), requires mTLS (no bearer-token fallback), never exposes Docker API/
container IDs/network info to its caller, never provides student-to-student routing (each
request resolves to exactly the one container matching its own token).

## 4. Multi-Lab-Host ready

Routing is resolved per `lab_host_id` — adding `rn-lab-host-02` later is one new `lab_hosts` row
(its own `routing_mode`/`router_endpoint`/TLS material), no Gateway code change. No scheduler
was built (explicitly out of scope) — the Provisioner still picks "the one enabled lab_host"
(`selectLabHost`), unchanged from Phase 1.

## 5. Local development compatibility

`dev-local` (the Unix-socket lab_host used by `docker-compose.dev.yml`) stays `routing_mode:
direct` — zero behavior change for the T480/any local dev setup, confirmed by the full existing
regression suite passing unmodified (§29 below) and by every new drill in this phase running
against the same dev stack. The standalone single-container Lab image build/boot is unaffected
(it never touches the Gateway or Provisioner at all).

## 6-9. Simulation, HTTP/WS validation, security tests, routing token security

`infra/scripts/simulate-two-host/` — a real two-Docker-network simulation (not mocks): Gateway
attached ONLY to an isolated bridge with no route to any Lab's bridge network, Lab Host Router
host-networked (mirrors `egress-proxy`'s existing pattern) so it can reach whatever per-Lab
bridge the Provisioner creates. 13/13 real assertions pass, including:

- Gateway genuinely cannot open a TCP connection to a Lab's Docker-internal IP (proven from
  inside the Gateway container's own network namespace via a raw connect attempt).
- Gateway reaches the Lab correctly through the Router — HTTP AND a real WebSocket terminal
  session running a genuine `kubectl get nodes` inside the Lab.
- Alice/Bob isolation (own lab ALLOW, other's DENY — HTTP and WebSocket both), garbage tokens,
  anonymous requests, and destroyed-environment tokens are all rejected through the real code
  path.
- The Router refuses any connection with no client certificate — mTLS is the real
  authentication mechanism.

Routing token security reviewed: already CSPRNG since Phase 1 (`generateRoutingToken`), never
logged in plaintext (the Router's `tokenFingerprint` truncates), redacted in every API response
except the two narrow internal reads that legitimately need it, and — critically — the header
the Router trusts is computed server-side from the Provisioner-verified environment row, never
read from the browser's own request, so a browser cannot forge routing by supplying its own
`X-RootNode-Route-Token`.

## 10. Control plane vs data plane

Kept strictly separate — see `docs/PRODUCTION_NETWORK_SECURITY.md`'s dedicated section. Student
browser traffic is never proxied through the Docker API at any point.

## 11. Firewall update

`infra/firewall/rn-lab-host-01.rules`'s old speculative wide port-range allowance replaced with
one narrow rule: `platform_ip` → the Router's port only (`8443` default, configurable).

## 12. Self-paced / remote delivery architecture check

`docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`. Short version: student-driven provisioning,
cohort-less students, and "idle reaper destroys → progress survives → student returns → fresh
Lab → curriculum continues" all already worked before this phase and are now proven end-to-end
by two new drills. The one real gap for actual remote/self-paced delivery is student
authentication (today's bare `student_id` login assumes an instructor-controlled roster) — not
built here, explicitly Phase 5 scope.

## 13. Abandoned attempt cleanup

`scenario_attempts.status = 'abandoned'` (already a valid value since Phase 3B, never
previously set) now gets set on: environment destroy/reset (student- or instructor-triggered),
logout, session expiry (piggybacked on the existing hourly session sweeper — no new background
job), and switching to a different scenario while one is still open. Never a punitive marker —
`result` stays NULL, history is preserved, instructors see an honest "abandoned (reason)"
instead of a phantom "still in progress."

## 14. Gateway session cache

Reviewed — already a tight 2-second TTL since Phase 3A, specifically to bound the
revocation-propagation window. Decision: **keep it, don't add push-based invalidation** — the
added coupling (Platform API would need to know how to reach the Gateway) isn't justified by
shrinking an already-2-second window further. Documented reasoning, not a silently-accepted gap.

## 15. Rate limiting

Login rate limiting moved from an in-memory `Map` (reset on every restart, not shareable across
instances) to Postgres (`login_attempts` table, `migrations/016_login_attempts.sql`) —
`platform/api/src/security.js`'s `tooManyAttempts()`. No Redis added. Restart behavior
confirmed for real: `platform/test/e2e-phase3a.js`'s rate-limiting test passes after a genuine
Platform API restart in the same run.

## 16. Pagination

Real server-side pagination (`platform/api/src/pagination.js`) on: the audit log (was a flat
`LIMIT 200`), student attempt history (was `LIMIT 100`), and assessment session history (was
**fully unbounded** — the one genuine growth risk found). Scenario analytics was deliberately
left as-is — grouped by scenario id, naturally bounded around ~100 rows, no real growth risk to
paginate against. Frontend (`AuditView.jsx`, `AssessmentsView.jsx`, `StudentsView.jsx`) updated
to match; `frontend` build confirmed clean.

## 17. Needs-attention configuration

`NEEDS_ATTENTION_FAIL_THRESHOLD`/`NEEDS_ATTENTION_STALE_DAYS` moved from hardcoded constants to
env-var-configurable (matching the existing session-TTL pattern), sensible defaults unchanged.

## 18. Assessment authoring UX

Raw-JSON `pool_spec` textarea replaced with a structured editor
(`frontend/src/admin/AssessmentsView.jsx`'s `PoolSpecEditor`): required scenarios and random
pools picked from a real scenario catalog (`GET /api/admin/scenario-catalog`, new), grouped by
curriculum stage, via native multi-select — no new UI framework. An owner-only "advanced" raw
JSON view remains for anything the structured editor doesn't cover.

## 19. Cohort role-in-cohort

`cohort_instructors.role_in_cohort` existed since Phase 2B but was never read. Implemented
minimally: `teacher` (unchanged full capability) vs `viewer` (read-only — dashboards/analytics
still visible, but Prepare/Destroy Labs and provisioning a student's environment are now 403'd
server-side, `rbac.js`'s `requireCohortWriteAccess`/`requireStudentCohortWriteAccess`).

## 20. Backup / restore destructive drill

`infra/backup/test-destructive-restore-drill.sh` — real owner/admin/cohort/3-students/progress/
attempts/assessment/achievements seeded through the real Platform API, encrypted-backed-up, the
disposable Postgres container **genuinely destroyed** (not just restored into a second
container), a fresh Postgres created from nothing, restored, and the real Platform API booted
again against the restored database and used to log in as the restored admin over HTTP. All
data verified present; never touches the developer's real database.

## 21. Lab Host rebuild drill

`infra/scripts/drills/test-lab-host-loss.js` — a student's Lab container is destroyed out from
under the Provisioner (no clean destroy call, simulating total Lab Host loss). Proves: Platform
stays up throughout, the stale environment is never presented as healthy once the Provisioner's
real `reconcileOnStartup` reconciliation runs, academic progress is untouched, and the student
can launch a genuinely new Lab and continue.

## 22. Deployment rollback test

`infra/scripts/drills/test-deployment-rollback.sh` — builds a known-good and a deliberately
broken gateway image, "deploys" each via the real deterministic-tag mechanism, proves the smoke
test catches the regression, computes the rollback target with the exact same
`deploy-history.log` logic `rollback-production.sh` uses, and proves the rollback recovers.
Documents explicitly that image rollback never implies a DB migration rollback.

## 23. CI/CD failure tests

`infra/scripts/drills/test-ci-fails-closed.sh` — proves all six required failure modes (test/
syntax failure, frontend build failure, image build failure, config validation failure,
migration preflight failure, staging smoke test failure) genuinely exit nonzero. **Found and
fixed two real pre-existing bugs**: `staging-smoke-test.sh` and `production-smoke-test.sh` both
had a classic `set -e` gotcha (`curl ... && echo "OK"` as one statement — a failing command
that isn't the last in an `&&` chain doesn't trigger `-e`) that let an unreachable/broken
endpoint fall through silently instead of failing the smoke test. Fixed in both files.

## 24. Security scanning

`infra/ci/gitea-workflows/build-and-test.yml`'s new `security-scan` job — npm audit, gitleaks
(secrets), hadolint (Dockerfiles), shellcheck (shell scripts), trivy (container image CVEs).
All free/open-source. Severity policy documented in `docs/SECURITY_SCANNING.md`: CRITICAL/HIGH
blocks the build, MEDIUM/LOW is reported only. `security-scan` gates `build-images` the same
way `regression` does.

## 25. Version / release visibility

`GET /api/version` (Platform API) — app version (`package.json`), deployed Git SHA
(`GIT_SHA` env var, set to the same value as the image tag in the production compose file), and
the most recently applied migration as a schema-version stand-in. Unauthenticated, minimal, no
infrastructure detail.

## 26. Admission control failure mode

`resource-policy.yaml`'s new (disabled-by-default) `admission_control.max_concurrent_labs` —
when set, the Provisioner checks it BEFORE touching Docker at all, so a full host fails cleanly
(503, `CAPACITY_EXCEEDED`) instead of failing mid-provision. Relayed by the Platform API as a
clean "Lab capacity is temporarily full. Try again shortly." — never a raw 500. A new
`GET /capacity` Provisioner endpoint feeds the instructor dashboard's capacity display.

## 27. Fake cohort dress rehearsal

`infra/scripts/drills/test-fake-cohort-rehearsal.js` — one continuous narrative: instructor
login, cohort + 4 students, curriculum assignment, Prepare Labs, students working different
scenarios, hints, a deliberate failed validation, instructor analytics, a Lab reset, a Lab
reaped (same consequence as the real idle reaper — see the drill's own comment on why the real
multi-minute timer isn't waited out), the student returning and continuing, an exam
assigned/taken, and end-of-class Lab destruction — progress verified intact after every
destructive step. 14/14 passing.

## 28. Documentation

This file, `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`, `docs/SECURITY_SCANNING.md` (new).
Updated: `docs/LAB_ARCHITECTURE.md`, `docs/PRODUCTION_NETWORK_SECURITY.md`,
`docs/PRODUCTION_DEPLOYMENT.md`, `docs/PRODUCTION_VALIDATION_CHECKLIST.md`, `docs/BACKUP_DR.md`,
`docs/ENGINEERING_SOP.md`, `README.md`, `platform/README.md`.

## 29. Regression

Re-ran the full existing suite after every application-code change in this phase:
`e2e-phase2a.js` (10/10), `e2e-phase2b.js` (12/12), `e2e-phase3a.js` (21/21 — including a real
audit-log pagination-shape fix and confirming rate-limit persistence across a genuine restart),
`e2e-phase3b.js` (24/24). Frontend production build clean. Real Lab image builds/boots (both
dev-compose and standalone single-container). `tofu validate`, `ansible-playbook
--syntax-check`, production Compose config (both platform and lab-host), `caddy validate`,
`promtool`/`amtool`, and the backup/restore mechanism all re-validated clean after the
infrastructure changes in this phase (new `lab-router` service, new firewall rule, new Ansible
tasks). No knowingly-unexplained regression was left in this codebase at the end of this phase.

## What Phase 4.1 deliberately did NOT build

Production deployment itself, Hetzner/Proxmox installation, public course signup, payments,
subscriptions, parent portal, marketing site, full LMS, video hosting, email automation,
certificates, course marketplace, mobile app, a multi-host scheduler, an AI tutor, invasive
proctoring, held scenario content, student email/password authentication (documented as the one
real gap for self-paced delivery, not built). All explicit non-goals per this phase's
authorization.