docs(phase4.1): pre-production hardening documentation

Master overview (PHASE4_1_PREPRODUCTION_HARDENING.md) covering all 27
substantive items, plus REMOTE_AND_SELF_PACED_ARCHITECTURE.md (reviews
what already works for remote/self-paced delivery - student-driven
provisioning, cohort-less students, idle-reaper-destroy-then-continue -
and the one real gap: student authentication, explicitly Phase 5 scope,
not built here).

Updated PRODUCTION_NETWORK_SECURITY.md (the Gateway-to-Lab gap is now
RESOLVED, with the full design writeup and a new control-plane-vs-data-plane
section), PRODUCTION_DEPLOYMENT.md (four-cert TLS bundle),
PRODUCTION_VALIDATION_CHECKLIST.md (Router mTLS check, admission control
checks), PHASE4_PRODUCTION_INFRASTRUCTURE.md and ENGINEERING_SOP.md and
LAB_ARCHITECTURE.md (stale 'known gap' pointers updated to point at the
resolution), BACKUP_DR.md (destructive drill), README.md and
platform/README.md (Phase 4.1 pointers, lab-router component, updated
known-limitations).

Also fixes a real pre-existing bug found while auditing what gets
committed: infra/.gitignore's patterns were written repo-root-relative
(e.g. "infra/terraform/...") but the file lives in infra/, so git resolved
them relative to THAT directory and none of them ever matched anything -
a real .terraform/ provider cache showed up as untracked when it should
have been ignored. Fixed by dropping the redundant leading "infra/" from
every pattern.

Status: PHASE 4.1 COMPLETE / PRE-PRODUCTION HARDENING COMPLETE / READY FOR
HARDWARE DEPLOYMENT.

README.md

@@ -196,8 +196,11 @@ Production topology and what's built vs. not yet: `docs/LAB_ARCHITECTURE.md`,
 
 The production infrastructure package (Proxmox/Ansible/Terraform/firewall/monitoring/backup/CI)
 that runs all of this on real hardware lives in `infra/` — see
-`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`. Built and locally validated now; deployment is
+`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` and `docs/PHASE4_1_PREPRODUCTION_HARDENING.md` (the
-pending purchase of the target Hetzner dedicated server.
+Gateway-to-Lab cross-VM routing fix, plus application hardening — real pagination, abandonment
+handling, a Postgres-backed rate limiter, a structured assessment editor, admission control).
+Built and locally validated now; deployment is pending purchase of the target Hetzner dedicated
+server.
 
 ---
 

docs/BACKUP_DR.md

@@ -37,10 +37,19 @@ requires typing `RESTORE` to confirm, then decrypts and pipes into `psql`.
 
 **Genuinely tested**, not just written: `infra/backup/test-backup-restore.sh` runs the full
 cycle against two disposable local Postgres containers, with the actual production migrations
-applied (all 17 — 3 Provisioner + 14 Platform API, in the correct cross-service order), a
+applied (3 Provisioner + Platform API, in the correct cross-service order), a
 sample row inserted, backed up, restored into the second container, and verified byte-for-byte.
 Confirmed passing, exit 0.
 
+**Phase 4.1's destructive drill goes one step further**
+(`infra/backup/test-destructive-restore-drill.sh`): realistic data (owner, admin instructor,
+cohort, 3 students, progress, attempts, an assessment definition, achievements) seeded through
+the REAL Platform API, backed up, the source Postgres **container genuinely destroyed**
+(`docker rm`, not just left running alongside a second one), a fresh Postgres created from
+nothing, restored, and the real Platform API booted again against the restored database and
+used to log in as the restored admin instructor over HTTP — proving the whole stack recovers,
+not just the SQL. Never touches the developer's real persistent database. Confirmed passing.
+
 ## Disaster recovery scenarios
 
 | Scenario | RPO | RTO (estimate) | What happens |

docs/ENGINEERING_SOP.md

@@ -296,12 +296,40 @@ Compose, firewall, monitoring, backup, CI, load-test tooling).
 - **Where the boundary held:** the Platform VM / Lab Host VM security split from
   `docs/LAB_ARCHITECTURE.md` is codified in firewall rulesets and mutual-TLS Docker config, not
   weakened anywhere.
-- **Known gap surfaced, not silently fixed:** the Gateway's Lab-reachability mechanism doesn't
+- **Known gap surfaced, then resolved in Phase 4.1:** the Gateway's Lab-reachability mechanism
-  survive the move to two real separate VMs — see `docs/PRODUCTION_NETWORK_SECURITY.md`'s
+  didn't survive the move to two real separate VMs — flagged explicitly rather than silently
-  "Known gap" section for what's built (firewall policy) vs. what's an explicit, reviewed,
+  fixed, then actually built and simulation-tested (a Lab Host Router,
-  not-yet-built application follow-up (Provisioner/Gateway address handling).
+  `platform/lab-router/`) in Phase 4.1. See §13 below.
 - **Validated without the hardware:** real `tofu validate`, `ansible-playbook --syntax-check`,
   `docker compose config`, `caddy validate`, `promtool`/`amtool`, and a genuine end-to-end
   backup→encrypt→restore→verify cycle against disposable local Postgres with the real
   production migrations applied. None of this proves the real Hetzner network boundary — that
   needs `docs/PRODUCTION_VALIDATION_CHECKLIST.md` walked against the real server.
+
+## 13. Pre-Production Hardening (Phase 4.1)
+
+Full design: `docs/PHASE4_1_PREPRODUCTION_HARDENING.md`, plus
+`docs/PRODUCTION_NETWORK_SECURITY.md` (routing resolution), `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`,
+`docs/SECURITY_SCANNING.md`.
+
+- **What it is:** closes the Gateway-to-Lab cross-VM gap Phase 4 flagged (a Lab Host Router,
+  `platform/lab-router/`, plus a routing abstraction in the Gateway,
+  `platform/gateway/src/lab-endpoint.js`), then hardens the application layer found along the
+  way: abandoned-attempt handling (`scenario_attempts.status = 'abandoned'`, never a punitive
+  failure), a Postgres-backed login rate limiter (survives a restart, ready to be
+  shared/multi-instance), real pagination on audit/attempt/assessment-session history, a
+  structured assessment authoring UI (owner-only raw JSON remains as a fallback), a minimal
+  `role_in_cohort` enforcement (`teacher`/`viewer`), a `/api/version` endpoint, and admission
+  control (a clean 503 "capacity full" instead of a 500 once a configured cap is reached).
+- **Tested, not just designed:** a real two-Docker-network routing simulation
+  (`infra/scripts/simulate-two-host/`, 13/13), a destructive Postgres restore drill
+  (`infra/backup/test-destructive-restore-drill.sh`), a Lab Host loss/rebuild drill
+  (`infra/scripts/drills/test-lab-host-loss.js`), a deployment rollback drill, a CI-fails-closed
+  drill (which found and fixed a real pre-existing `set -e` bug in both smoke-test scripts —
+  see that drill's own output), and a full fake-cohort dress rehearsal
+  (`infra/scripts/drills/test-fake-cohort-rehearsal.js`) — all passing, all in
+  `infra/scripts/drills/` or `infra/backup/`.
+- **Known limitation carried forward:** real remote/self-paced delivery needs actual student
+  authentication (today's `student_id`-only login relies on instructors controlling who gets a
+  valid id) — see `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md` for the full accounting of what
+  already works vs. what's genuinely Phase 5 scope.

docs/LAB_ARCHITECTURE.md

@@ -491,8 +491,10 @@ networking, egress, resource policy, reaper) against these decisions.
 
 Phase 4 (`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`) codifies the Platform/Lab-Host split this
 document designed as two real, separately-firewalled Proxmox VMs (`rn-platform-01`,
-`rn-lab-host-01`) connected only by mutual-TLS Docker and a tightly scoped firewall allowance —
+`rn-lab-host-01`) connected by two separate mTLS channels (Provisioner↔dockerd CONTROL plane,
-see `docs/PRODUCTION_NETWORK_SECURITY.md`. That document also records a gap Phase 4 surfaced but
+Gateway↔Lab Host Router DATA plane) and a tightly scoped firewall allowance — see
-did not resolve: the Gateway's `env.internal_ip` mechanism for reaching a Lab container assumes
+`docs/PRODUCTION_NETWORK_SECURITY.md`. Phase 4 surfaced a gap in the Gateway's `env.internal_ip`
-Gateway and Lab share a Docker daemon, which breaks once they're genuinely on separate VMs. Read
+mechanism for reaching a Lab container (it assumes Gateway and Lab share a Docker daemon, which
-that document's "Known gap" section before treating two-VM production as fully wired.
+breaks once they're genuinely on separate VMs); Phase 4.1
+(`docs/PHASE4_1_PREPRODUCTION_HARDENING.md`) built and simulation-tested the fix, a Lab Host
+Router (`platform/lab-router/`).

docs/PHASE4_1_PREPRODUCTION_HARDENING.md

@@ -0,0 +1,254 @@
+# Phase 4.1 — Pre-Production Hardening
+
+**Status: PHASE 4.1 COMPLETE. PRE-PRODUCTION HARDENING COMPLETE. READY FOR HARDWARE DEPLOYMENT.**
+
+Phase 4 built the production infrastructure package with the real Hetzner/Proxmox hardware not
+yet purchased, and flagged one genuine architectural gap: the Gateway's mechanism for reaching
+a student Lab assumed Gateway and Lab shared a Docker daemon, which is false the moment Labs run
+on a genuinely separate `rn-lab-host-01` VM. Phase 4.1's job was to close that gap and harden
+everything else discovered along the way, still without the real hardware.
+
+## 1. The core fix: a Lab Host Router
+
+**Before:** Gateway → `http://${env.internal_ip}:${env.internal_port}` directly — a Docker
+bridge-local address, unreachable across a real VM boundary.
+
+**After:** Gateway → (mode-aware) → either `direct` (unchanged, dev/single-host) or `router`
+(production): a small new service, `platform/lab-router/`, runs on the Lab Host itself,
+authenticates the Gateway via mTLS (the same private CA the Docker TLS control plane already
+used, extended with two new leaf certs), resolves an opaque routing token to a local container
+purely from `docker ps` label state (no Postgres access, no call back to the Provisioner), and
+reverse-proxies HTTP and WebSocket traffic to it.
+
+The decision of which mode a given `lab_host` uses is data-driven
+(`lab_hosts.routing_mode`/`router_endpoint`, `platform/provisioner/migrations/004_lab_hosts_routing.sql`),
+resolved once per request in `platform/gateway/src/lab-endpoint.js`'s `labTarget()` — the only
+place in the Gateway that knows this distinction exists. Every proxy/fetch call site in
+`platform/gateway/src/server.js` goes through it.
+
+**Exact `env.internal_ip` replacement:** `labTarget(env, token)` returns `{ url, headers, agent,
+mode }`. In `direct` mode, `url` is exactly the old `http://internal_ip:internal_port`. In
+`router` mode, `url` is the lab_host's `router_endpoint`, `headers` carries
+`X-RootNode-Route-Token: <token>` (computed server-side from the Provisioner-verified
+environment row — never trusted from the browser), and `agent` is an `https.Agent` configured
+with the Gateway's mTLS client identity.
+
+## 2. No per-student host port
+
+Deliberately not solved by publishing a host port per Lab (the originally-flagged, rejected
+approach). The Router resolves entirely from Docker's own label state on its own host — no
+port-per-student anything, no dynamic firewall rule per Lab.
+
+## 3. Lab Host Router — design
+
+See `platform/lab-router/src/server.js`'s header comment for the full "narrow by design" list:
+one route (proxy-by-token), requires mTLS (no bearer-token fallback), never exposes Docker API/
+container IDs/network info to its caller, never provides student-to-student routing (each
+request resolves to exactly the one container matching its own token).
+
+## 4. Multi-Lab-Host ready
+
+Routing is resolved per `lab_host_id` — adding `rn-lab-host-02` later is one new `lab_hosts` row
+(its own `routing_mode`/`router_endpoint`/TLS material), no Gateway code change. No scheduler
+was built (explicitly out of scope) — the Provisioner still picks "the one enabled lab_host"
+(`selectLabHost`), unchanged from Phase 1.
+
+## 5. Local development compatibility
+
+`dev-local` (the Unix-socket lab_host used by `docker-compose.dev.yml`) stays `routing_mode:
+direct` — zero behavior change for the T480/any local dev setup, confirmed by the full existing
+regression suite passing unmodified (§29 below) and by every new drill in this phase running
+against the same dev stack. The standalone single-container Lab image build/boot is unaffected
+(it never touches the Gateway or Provisioner at all).
+
+## 6-9. Simulation, HTTP/WS validation, security tests, routing token security
+
+`infra/scripts/simulate-two-host/` — a real two-Docker-network simulation (not mocks): Gateway
+attached ONLY to an isolated bridge with no route to any Lab's bridge network, Lab Host Router
+host-networked (mirrors `egress-proxy`'s existing pattern) so it can reach whatever per-Lab
+bridge the Provisioner creates. 13/13 real assertions pass, including:
+
+- Gateway genuinely cannot open a TCP connection to a Lab's Docker-internal IP (proven from
+  inside the Gateway container's own network namespace via a raw connect attempt).
+- Gateway reaches the Lab correctly through the Router — HTTP AND a real WebSocket terminal
+  session running a genuine `kubectl get nodes` inside the Lab.
+- Alice/Bob isolation (own lab ALLOW, other's DENY — HTTP and WebSocket both), garbage tokens,
+  anonymous requests, and destroyed-environment tokens are all rejected through the real code
+  path.
+- The Router refuses any connection with no client certificate — mTLS is the real
+  authentication mechanism.
+
+Routing token security reviewed: already CSPRNG since Phase 1 (`generateRoutingToken`), never
+logged in plaintext (the Router's `tokenFingerprint` truncates), redacted in every API response
+except the two narrow internal reads that legitimately need it, and — critically — the header
+the Router trusts is computed server-side from the Provisioner-verified environment row, never
+read from the browser's own request, so a browser cannot forge routing by supplying its own
+`X-RootNode-Route-Token`.
+
+## 10. Control plane vs data plane
+
+Kept strictly separate — see `docs/PRODUCTION_NETWORK_SECURITY.md`'s dedicated section. Student
+browser traffic is never proxied through the Docker API at any point.
+
+## 11. Firewall update
+
+`infra/firewall/rn-lab-host-01.rules`'s old speculative wide port-range allowance replaced with
+one narrow rule: `platform_ip` → the Router's port only (`8443` default, configurable).
+
+## 12. Self-paced / remote delivery architecture check
+
+`docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`. Short version: student-driven provisioning,
+cohort-less students, and "idle reaper destroys → progress survives → student returns → fresh
+Lab → curriculum continues" all already worked before this phase and are now proven end-to-end
+by two new drills. The one real gap for actual remote/self-paced delivery is student
+authentication (today's bare `student_id` login assumes an instructor-controlled roster) — not
+built here, explicitly Phase 5 scope.
+
+## 13. Abandoned attempt cleanup
+
+`scenario_attempts.status = 'abandoned'` (already a valid value since Phase 3B, never
+previously set) now gets set on: environment destroy/reset (student- or instructor-triggered),
+logout, session expiry (piggybacked on the existing hourly session sweeper — no new background
+job), and switching to a different scenario while one is still open. Never a punitive marker —
+`result` stays NULL, history is preserved, instructors see an honest "abandoned (reason)"
+instead of a phantom "still in progress."
+
+## 14. Gateway session cache
+
+Reviewed — already a tight 2-second TTL since Phase 3A, specifically to bound the
+revocation-propagation window. Decision: **keep it, don't add push-based invalidation** — the
+added coupling (Platform API would need to know how to reach the Gateway) isn't justified by
+shrinking an already-2-second window further. Documented reasoning, not a silently-accepted gap.
+
+## 15. Rate limiting
+
+Login rate limiting moved from an in-memory `Map` (reset on every restart, not shareable across
+instances) to Postgres (`login_attempts` table, `migrations/016_login_attempts.sql`) —
+`platform/api/src/security.js`'s `tooManyAttempts()`. No Redis added. Restart behavior
+confirmed for real: `platform/test/e2e-phase3a.js`'s rate-limiting test passes after a genuine
+Platform API restart in the same run.
+
+## 16. Pagination
+
+Real server-side pagination (`platform/api/src/pagination.js`) on: the audit log (was a flat
+`LIMIT 200`), student attempt history (was `LIMIT 100`), and assessment session history (was
+**fully unbounded** — the one genuine growth risk found). Scenario analytics was deliberately
+left as-is — grouped by scenario id, naturally bounded around ~100 rows, no real growth risk to
+paginate against. Frontend (`AuditView.jsx`, `AssessmentsView.jsx`, `StudentsView.jsx`) updated
+to match; `frontend` build confirmed clean.
+
+## 17. Needs-attention configuration
+
+`NEEDS_ATTENTION_FAIL_THRESHOLD`/`NEEDS_ATTENTION_STALE_DAYS` moved from hardcoded constants to
+env-var-configurable (matching the existing session-TTL pattern), sensible defaults unchanged.
+
+## 18. Assessment authoring UX
+
+Raw-JSON `pool_spec` textarea replaced with a structured editor
+(`frontend/src/admin/AssessmentsView.jsx`'s `PoolSpecEditor`): required scenarios and random
+pools picked from a real scenario catalog (`GET /api/admin/scenario-catalog`, new), grouped by
+curriculum stage, via native multi-select — no new UI framework. An owner-only "advanced" raw
+JSON view remains for anything the structured editor doesn't cover.
+
+## 19. Cohort role-in-cohort
+
+`cohort_instructors.role_in_cohort` existed since Phase 2B but was never read. Implemented
+minimally: `teacher` (unchanged full capability) vs `viewer` (read-only — dashboards/analytics
+still visible, but Prepare/Destroy Labs and provisioning a student's environment are now 403'd
+server-side, `rbac.js`'s `requireCohortWriteAccess`/`requireStudentCohortWriteAccess`).
+
+## 20. Backup / restore destructive drill
+
+`infra/backup/test-destructive-restore-drill.sh` — real owner/admin/cohort/3-students/progress/
+attempts/assessment/achievements seeded through the real Platform API, encrypted-backed-up, the
+disposable Postgres container **genuinely destroyed** (not just restored into a second
+container), a fresh Postgres created from nothing, restored, and the real Platform API booted
+again against the restored database and used to log in as the restored admin over HTTP. All
+data verified present; never touches the developer's real database.
+
+## 21. Lab Host rebuild drill
+
+`infra/scripts/drills/test-lab-host-loss.js` — a student's Lab container is destroyed out from
+under the Provisioner (no clean destroy call, simulating total Lab Host loss). Proves: Platform
+stays up throughout, the stale environment is never presented as healthy once the Provisioner's
+real `reconcileOnStartup` reconciliation runs, academic progress is untouched, and the student
+can launch a genuinely new Lab and continue.
+
+## 22. Deployment rollback test
+
+`infra/scripts/drills/test-deployment-rollback.sh` — builds a known-good and a deliberately
+broken gateway image, "deploys" each via the real deterministic-tag mechanism, proves the smoke
+test catches the regression, computes the rollback target with the exact same
+`deploy-history.log` logic `rollback-production.sh` uses, and proves the rollback recovers.
+Documents explicitly that image rollback never implies a DB migration rollback.
+
+## 23. CI/CD failure tests
+
+`infra/scripts/drills/test-ci-fails-closed.sh` — proves all six required failure modes (test/
+syntax failure, frontend build failure, image build failure, config validation failure,
+migration preflight failure, staging smoke test failure) genuinely exit nonzero. **Found and
+fixed two real pre-existing bugs**: `staging-smoke-test.sh` and `production-smoke-test.sh` both
+had a classic `set -e` gotcha (`curl ... && echo "OK"` as one statement — a failing command
+that isn't the last in an `&&` chain doesn't trigger `-e`) that let an unreachable/broken
+endpoint fall through silently instead of failing the smoke test. Fixed in both files.
+
+## 24. Security scanning
+
+`infra/ci/gitea-workflows/build-and-test.yml`'s new `security-scan` job — npm audit, gitleaks
+(secrets), hadolint (Dockerfiles), shellcheck (shell scripts), trivy (container image CVEs).
+All free/open-source. Severity policy documented in `docs/SECURITY_SCANNING.md`: CRITICAL/HIGH
+blocks the build, MEDIUM/LOW is reported only. `security-scan` gates `build-images` the same
+way `regression` does.
+
+## 25. Version / release visibility
+
+`GET /api/version` (Platform API) — app version (`package.json`), deployed Git SHA
+(`GIT_SHA` env var, set to the same value as the image tag in the production compose file), and
+the most recently applied migration as a schema-version stand-in. Unauthenticated, minimal, no
+infrastructure detail.
+
+## 26. Admission control failure mode
+
+`resource-policy.yaml`'s new (disabled-by-default) `admission_control.max_concurrent_labs` —
+when set, the Provisioner checks it BEFORE touching Docker at all, so a full host fails cleanly
+(503, `CAPACITY_EXCEEDED`) instead of failing mid-provision. Relayed by the Platform API as a
+clean "Lab capacity is temporarily full. Try again shortly." — never a raw 500. A new
+`GET /capacity` Provisioner endpoint feeds the instructor dashboard's capacity display.
+
+## 27. Fake cohort dress rehearsal
+
+`infra/scripts/drills/test-fake-cohort-rehearsal.js` — one continuous narrative: instructor
+login, cohort + 4 students, curriculum assignment, Prepare Labs, students working different
+scenarios, hints, a deliberate failed validation, instructor analytics, a Lab reset, a Lab
+reaped (same consequence as the real idle reaper — see the drill's own comment on why the real
+multi-minute timer isn't waited out), the student returning and continuing, an exam
+assigned/taken, and end-of-class Lab destruction — progress verified intact after every
+destructive step. 14/14 passing.
+
+## 28. Documentation
+
+This file, `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`, `docs/SECURITY_SCANNING.md` (new).
+Updated: `docs/LAB_ARCHITECTURE.md`, `docs/PRODUCTION_NETWORK_SECURITY.md`,
+`docs/PRODUCTION_DEPLOYMENT.md`, `docs/PRODUCTION_VALIDATION_CHECKLIST.md`, `docs/BACKUP_DR.md`,
+`docs/ENGINEERING_SOP.md`, `README.md`, `platform/README.md`.
+
+## 29. Regression
+
+Re-ran the full existing suite after every application-code change in this phase:
+`e2e-phase2a.js` (10/10), `e2e-phase2b.js` (12/12), `e2e-phase3a.js` (21/21 — including a real
+audit-log pagination-shape fix and confirming rate-limit persistence across a genuine restart),
+`e2e-phase3b.js` (24/24). Frontend production build clean. Real Lab image builds/boots (both
+dev-compose and standalone single-container). `tofu validate`, `ansible-playbook
+--syntax-check`, production Compose config (both platform and lab-host), `caddy validate`,
+`promtool`/`amtool`, and the backup/restore mechanism all re-validated clean after the
+infrastructure changes in this phase (new `lab-router` service, new firewall rule, new Ansible
+tasks). No knowingly-unexplained regression was left in this codebase at the end of this phase.
+
+## What Phase 4.1 deliberately did NOT build
+
+Production deployment itself, Hetzner/Proxmox installation, public course signup, payments,
+subscriptions, parent portal, marketing site, full LMS, video hosting, email automation,
+certificates, course marketplace, mobile app, a multi-host scheduler, an AI tutor, invasive
+proctoring, held scenario content, student email/password authentication (documented as the one
+real gap for self-paced delivery, not built). All explicit non-goals per this phase's
+authorization.

docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md

@@ -170,15 +170,18 @@ unchanged. No parent portal, billing, held scenarios, or production OIDC provide
 RootNode production infrastructure is dedicated to RootNode Academy — never folded into
 existing Synthos production servers, and canonical source stays Gitea, never GitHub `kubekosh`.
 
-## Known gap surfaced during this phase (not silently papered over)
+## Known gap surfaced during this phase (not silently papered over) — RESOLVED in Phase 4.1
 
 Building the real production Compose/network config surfaced that the Gateway's existing
 mechanism for reaching a Lab container (`env.internal_ip`, a Docker-bridge-local address) only
 works when Gateway and Lab share a Docker daemon — true in every dev/single-host setup so far,
-**not** true once Labs run on a genuinely separate `rn-lab-host-01`. The network-policy half is
+**not** true once Labs run on a genuinely separate `rn-lab-host-01`. At the time this phase
-built (a firewall allowance for Gateway→Lab-Host on a published port range,
+closed, the network-policy half was built but the application half was flagged as an explicit,
-`infra/firewall/rn-lab-host-01.rules`); the application half (the Provisioner recording a
+not-yet-built follow-up rather than guessed at mid-infra-phase.
-reachable external address for each Lab, and the Gateway using it) is an explicit, reviewed,
+
-NOT-YET-BUILT follow-up — see `docs/PRODUCTION_NETWORK_SECURITY.md` for the two candidate
+**Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1) built and proved the application
-designs and why neither was silently chosen here. This must land before real two-VM go-live;
+half** — a Lab Host Router (`platform/lab-router/`) plus a `labTarget()` routing abstraction in
-flagging it clearly was judged better than guessing at an architecture change mid-infra-phase.
+the Gateway, real-tested against a genuine two-network Docker simulation
+(`infra/scripts/simulate-two-host/`), not just designed. See
+`docs/PRODUCTION_NETWORK_SECURITY.md`'s "Gateway-to-Lab reachability" section for the full
+resolution. This item no longer blocks real two-VM go-live.

docs/PRODUCTION_DEPLOYMENT.md

@@ -31,7 +31,7 @@ tofu apply
 Creates `rn-platform-01` and `rn-lab-host-01` from the cloud-init template. Confirm both boot
 and are SSH-reachable from the management network before continuing.
 
-## 2. Generate Docker TLS certificates (offline, once)
+## 2. Generate TLS certificates (offline, once)
 
 ```bash
 export ROOTNODE_TLS_BUNDLE_DIR=/secure/offline/path   # never inside the repo
@@ -39,8 +39,14 @@ export ROOTNODE_TLS_BUNDLE_DIR=/secure/offline/path   # never inside the repo
 ./infra/scripts/docker-tls/verify-docker-tls.sh
 ```
 
-The CA private key stays in `ROOTNODE_TLS_BUNDLE_DIR` and never touches either VM — only the CA
+Generates FOUR leaf certs under one CA (Phase 4.1 extended this from two): the CONTROL-plane
-cert (public) and the relevant leaf cert+key go to each host.
+pair (`server-*`/`provisioner-client-*`, for Provisioner↔dockerd) and the DATA-plane pair
+(`router-server-*`/`gateway-client-*`, for Gateway↔Lab Host Router — see
+`docs/PRODUCTION_NETWORK_SECURITY.md`'s "control plane vs data plane" section). The CA private
+key stays in `ROOTNODE_TLS_BUNDLE_DIR` and never touches either VM — only the CA cert (public)
+and each host's own leaf cert+key go to each host, both pairs deployed by the same `docker-tls`
+Ansible role into two separate directories (`/etc/rootnode/docker-tls/` and
+`/etc/rootnode/lab-router-tls/`).
 
 ## 3. Configure both hosts
 

docs/PRODUCTION_NETWORK_SECURITY.md

@@ -12,7 +12,8 @@ management_cidr ──SSH──▶ rn-platform-01                │
 management_cidr ──SSH──▶ rn-lab-host-01                 │
 proxmox_mgmt_cidr ──22/8006──▶ Proxmox host             │
                                                         │
-rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd)
+rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd)          [CONTROL plane]
+rn-platform-01 (Gateway) ──mTLS:8443──▶ rn-lab-host-01 (Lab Host Router)     [DATA plane]
 rn-lab-host-01 (promtail) ──3100──▶ rn-platform-01 (Loki)
 rn-lab-host-01 (node-exporter/cadvisor) ◀──scrape── rn-platform-01 (Prometheus)
 ```
@@ -37,6 +38,25 @@ Firewall source of truth: `infra/firewall/{proxmox-host,rn-platform-01,rn-lab-ho
   (Grafana is reachable only via Caddy's `/grafana/*` path, itself restricted to
   `management_cidr` by a `caddy` matcher — see `infra/docker/platform/Caddyfile.j2`).
 
+## Control plane vs data plane (Phase 4.1 §10)
+
+Two separate, never-conflated mTLS connections between the two VMs, deliberately kept apart
+both conceptually and operationally:
+
+| | Control plane | Data plane |
+|---|---|---|
+| What | Provisioner manages Lab containers | Gateway serves student browser traffic |
+| Path | Provisioner → dockerd (port 2376) | Gateway → Lab Host Router (port 8443) |
+| Who initiates | Provisioner only | Gateway only |
+| What it can do | Create/start/stop/remove containers, full Docker API | Reverse-proxy one HTTP/WS request to one already-running container by opaque token — nothing else |
+| Certs | `provisioner-client-*` / `server-*` | `gateway-client-*` / `router-server-*` |
+| Exposes Docker API to caller? | Yes (that's its job — Provisioner IS a Docker client) | **Never** — the Router never exposes Docker API, container IDs, or network info to its caller |
+
+The student browser's traffic is proxied at the Gateway/Router layer and never touches the
+Docker API at any point — the Router deliberately has no route that echoes back anything
+Docker-internal (see `platform/lab-router/src/server.js`'s header comment for the full "narrow
+by design" list).
+
 ## Docker TLS (Provisioner ↔ Lab Host)
 
 Mutual TLS on port 2376, restricted at the firewall to `platform_ip` only. Application support
@@ -57,34 +77,51 @@ front this with Cloudflare or expose the origin directly is left as a deployment
 being able to see plaintext unless "Full (strict)" mode is used) — not decided here since it
 doesn't affect anything else in this document.
 
-## Known gap: Gateway-to-Lab reachability across two real VMs
+## Gateway-to-Lab reachability across two real VMs — RESOLVED in Phase 4.1
 
-**This is the most important item in this document.**
+Phase 4 flagged this as the single most important open item; Phase 4.1
+(`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1) closed it. Kept here as the historical record of
+what the gap was and how it was closed, since the reasoning matters for anyone extending this
+later (e.g. adding a second Lab Host).
 
-The Gateway currently reaches a running Lab container via `env.internal_ip` — a Docker
+**The gap, as originally found:** the Gateway used to reach a running Lab container via
-bridge-local address recorded by the Provisioner when it creates the container
+`env.internal_ip` — a Docker bridge-local address recorded by the Provisioner when it creates
-(`platform/gateway/src/server.js`, `platform/provisioner`, built in Phase 2A). This works when
+the container. That only works when Gateway and the Lab share a Docker daemon, true in dev and
-Gateway and the Lab share a Docker daemon, which has been true in every setup so far, including
+any single-host setup, false the moment Labs run on a genuinely separate `rn-lab-host-01`.
-dev and any single-host deployment. **It stops being true once Labs run on a genuinely separate
-`rn-lab-host-01`** — a Docker bridge-internal IP on one host is not routable from a different
-host by default.
 
-What's built now (network-policy half only): a firewall allowance in
+**The fix — a Lab Host Router (`platform/lab-router/`):** a small, narrow service that runs on
-`infra/firewall/rn-lab-host-01.rules` for `platform_ip` → `lab_gateway_port_range`
+rn-lab-host-01 (host-networked, like `egress-proxy`, so it can reach whatever per-Lab bridge
-(`30000:32767` by default, configurable), so that once Lab containers publish ports on the Lab
+network the Provisioner creates) and does exactly one thing: given an opaque routing token over
-Host's own interface, the Gateway VM is permitted to reach them.
+mTLS, find the ONE local container labeled with that token (`docker ps --filter
+label=rootnode.routing-token=<token>` — resolved purely from local Docker state, no Postgres,
+no call back to the Provisioner) and reverse-proxy the request to it, HTTP and WebSocket both.
 
-What's **not yet built** (application half — explicitly flagged, not silently decided here):
+The Gateway now decides `direct` vs `router` per `lab_host` (`lab_hosts.routing_mode` +
-the Provisioner needs to either (a) publish each Lab container's relevant port(s) on the Lab
+`router_endpoint`, `platform/provisioner/migrations/004_lab_hosts_routing.sql`) via
-Host's interface and record a reachable `external_ip`/`external_port` alongside (or instead of)
+`platform/gateway/src/lab-endpoint.js` — a single function, `labTarget(env, token)`, that every
-`internal_ip`/`internal_port`, or (b) route Gateway→Lab traffic through a proxy/tunnel that
+proxy/fetch call site in `server.js` goes through instead of building
-lives on the Lab Host and forwards to the container's internal address. Option (a) is simpler
+`http://${env.internal_ip}:${env.internal_port}` itself. `dev-local` stays `direct` (T480/local
-and consistent with how the egress proxy and cadvisor are already handled in this phase's
+dev unaffected, zero behavior change there); a real `rn-lab-host-01` is configured `router`.
-compose files; option (b) avoids publishing a wide port range but adds a moving part. This is a
+
-small, reviewable, application-level change — not a network-policy change — and should land as
+**Firewall (`infra/firewall/rn-lab-host-01.rules`):** the old speculative
-its own reviewed commit before real two-VM production traffic depends on it. Until it lands,
+`lab_gateway_port_range` allowance (which assumed option (a), publishing per-Lab ports) was
-Gateway↔Lab terminal/scenario traffic will not work correctly across two separate VMs, even
+replaced with a single, narrow rule: `platform_ip` → port `{{ lab_router_port | default('8443')
-though every other production networking, firewall, and TLS boundary is in place and correct.
+}}` only. No wide port range, no per-Lab port publishing at all — the Router is the only thing
+the Gateway ever talks to on rn-lab-host-01 for Lab traffic.
+
+**mTLS:** the Gateway↔Router pair uses the SAME private CA as the Provisioner↔dockerd pair
+(`infra/scripts/docker-tls/generate-docker-tls.sh`, extended in Phase 4.1 to also issue
+`router-server`/`gateway-client` leaf certs), deployed to a separate directory
+(`/etc/rootnode/lab-router-tls/`) by the same `docker-tls` Ansible role.
+
+**Proven, not just designed:** `infra/scripts/simulate-two-host/` builds a real two-network
+Docker simulation (Gateway genuinely isolated on its own bridge, no route to any Lab's bridge)
+and runs 13 real assertions against it — Gateway cannot reach a Lab's Docker-internal IP
+directly, but reaches it correctly through the Router (HTTP and a real WebSocket
+`kubectl get nodes`), Alice/Bob isolation holds, garbage/destroyed tokens are rejected, and the
+Router itself refuses any connection without the Gateway's client certificate. See that
+directory's README for exactly what is and isn't proven by a same-daemon simulation versus real
+separate hardware.
 
 ## Cookie / auth production settings
 

docs/PRODUCTION_VALIDATION_CHECKLIST.md

@@ -19,8 +19,9 @@ can prove in advance.
 
 ## Lab Host VM
 - [ ] Docker daemon TLS verified: `docker --tlsverify --tlscacert=... -H tcp://<lab_host_ip>:2376 info` succeeds from the Platform VM, fails from anywhere else
+- [ ] Lab Host Router mTLS verified: a request to `https://<lab_host_ip>:8443` with no client cert is rejected; with the Gateway's real client cert it isn't (mirrors `infra/scripts/simulate-two-host/test-remote-routing.js`'s "direct public access: NOT AVAILABLE" case, now against the real host)
 - [ ] Provisioner successfully creates a real Lab against the real Lab Host
-- [ ] Gateway can actually reach that Lab's terminal — **this depends on the not-yet-built application change documented in `docs/PRODUCTION_NETWORK_SECURITY.md`'s "Known gap" section; do not mark this item done until that change has landed and been verified for real**
+- [ ] Gateway can actually reach that Lab's terminal (HTTP UI load + WebSocket `kubectl` command) — application mechanism built and simulation-tested in Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1/§6/§7); this line is the real-hardware confirmation that the same result holds across the actual two VMs, not two Docker networks on one daemon
 - [ ] Egress proxy correctly restricts a Lab's outbound traffic (Provisioner's per-Lab `DOCKER-USER` chain)
 
 ## Security
@@ -38,7 +39,7 @@ can prove in advance.
 
 ## Backup
 - [ ] Manual `backup-postgres.sh` run succeeds against real Postgres, real off-host target reachable
-- [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness
+- [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness — Phase 4.1's `infra/backup/test-destructive-restore-drill.sh` proved the full mechanism (realistic data, real storage destruction, restore, real Platform API boot) locally; this line is the real-hardware confirmation
 - [ ] Cron job confirmed scheduled (`crontab -l`)
 
 ## Observability
@@ -46,5 +47,12 @@ can prove in advance.
 - [ ] Trigger one alert deliberately (e.g. stop a service) and confirm it fires and reaches the webhook
 - [ ] Loki has logs flowing from both VMs
 
+## Pre-production hardening (Phase 4.1 — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md)
+- [ ] `resource-policy.yaml`'s `admission_control.max_concurrent_labs` set from a real measured
+      `infra/scripts/capacity-model.js` run against the real Lab Host, not left disabled
+- [ ] A deliberate over-capacity attempt returns the clean 503 "Lab capacity is temporarily
+      full" message, not a 500 — confirms the real host's admission control matches the local test
+- [ ] `docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md`'s steps still make sense once capacity numbers are real
+
 Only once every box above is checked against the real server does the deployment graduate from
 "infrastructure package built" to "actually in production."

docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md

@@ -0,0 +1,85 @@
+# Remote & Self-Paced Delivery Architecture
+
+Phase 4.1 §12. This is an architecture review, not a new product — no self-registration,
+payments, or public enrollment is built here (explicit non-goal). The question this document
+answers: does the infrastructure and application assume "an instructor always provisions the
+Lab," and if so, where, and what would need to change for self-paced delivery to work.
+
+## The short answer: less of that assumption exists than expected
+
+RootNode Academy's Lab lifecycle was built student-driven from Phase 1 — `POST
+/api/environment/launch` is called by the STUDENT'S OWN session, not by an instructor action.
+"Prepare Labs" (`POST /api/admin/cohorts/:id/prepare-labs`) is a classroom **convenience** that
+front-loads several students' launches at once before a synchronous class session, not the only
+path to a running Lab. This was true before Phase 4.1 and confirmed still true by this phase's
+own regression suites (`platform/test/e2e-phase2a.js` roster's two students with `cohort_id:
+null` and takes them through login → curriculum → launch → scenario work with zero cohort
+involvement) and by two new Phase 4.1 drills built specifically to exercise the remote/returning
+-student narrative:
+
+- `infra/scripts/drills/test-lab-host-loss.js` — student launches, makes progress, their Lab is
+  destroyed out from under them (standing in for an idle-reaper destroy), Platform stays up,
+  progress is untouched, they launch again unprompted and continue.
+- `infra/scripts/drills/test-fake-cohort-rehearsal.js` — the same "returns later, fresh Lab,
+  curriculum continues" narrative as part of a fuller classroom rehearsal.
+
+Both pass today, with zero application code changes needed for the self-paced-relevant parts of
+this flow specifically — see the Phase 4.1 report for the full run results.
+
+## What already works, unmodified
+
+- **Student-initiated provisioning** — no instructor step is required between "student wants to
+  work" and "Lab exists." (`platform/api/src/server.js`'s `/api/environment/launch`.)
+- **Cohort-less students** — `cohort_id` is nullable on student creation
+  (`POST /api/admin/students`); a student with no cohort logs in, sees the curriculum, launches
+  Labs, and makes progress identically to a cohorted one. Confirmed by inspection AND by the
+  existing regression suite, not just by schema reading.
+- **Idle reaper → academic state survives → return → continue** — the exact lifecycle this
+  section describes as the target self-paced experience is what Phase 1's reaper +
+  Phase 2B's Postgres-backed academic data (separate from the disposable Lab) were already
+  designed to produce. Phase 4.1 proved it end-to-end, twice (see above).
+- **Curriculum content itself is not cohort-gated** — `GET /api/curriculum/path` reads the one
+  curriculum path file directly (`curriculum.loadCurriculumPath()`); it never checks
+  `curriculum_assignments` or cohort membership at all. A self-paced student sees the full
+  curriculum with no instructor action required to "unlock" it.
+
+## What's genuinely missing for a real self-paced product (Phase 5, not built here)
+
+1. **No self-registration / public enrollment.** A student row is created only by an instructor
+   (`POST /api/admin/students`, admin/owner RBAC-gated). There is no public "sign up" surface.
+   This is almost certainly the right call to leave alone for now — it's explicitly a Phase 5+
+   product/business decision (course commerce, is it free, who's allowed to enroll), not an
+   infrastructure gap.
+
+2. **No real student authentication.** A student session is created from a bare `student_id`
+   (`POST /api/login`, no password) — deliberately, since today every `student_id` is issued and
+   controlled by an instructor via the roster, not chosen or known by the public. A genuinely
+   remote/self-paced product where students access their account without an instructor in the
+   loop needs real credentials (email+password, or a magic link) — reusing the same
+   `password_hash`/`scrypt` machinery `instructors` already has (`src/auth/password.js`) would
+   be the natural extension, not a new auth system. **This is the one item on this list that
+   would actually block self-paced delivery today** — everything else already works or is a
+   product decision, not an infrastructure one.
+
+3. **`curriculum_assignments.student_id` is schema-ready but unwired.** The table has supported
+   a direct per-student curriculum assignment (independent of any cohort) since Phase 2B
+   (`CONSTRAINT curriculum_assignment_target CHECK (cohort_id IS NOT NULL OR student_id IS NOT
+   NULL)`) — but nothing in the application ever reads or writes that column today; there's only
+   one curriculum path, so nothing has needed to differentiate yet. Not a gap that blocks
+   self-paced delivery now (every student already sees the one path with no gate), but the
+   moment a second curriculum path exists, this is where per-individual assignment lands — the
+   schema was designed for exactly this and doesn't need a migration to support it.
+
+4. **No delivery-mode concept anywhere.** In-person, remote, and self-paced sessions aren't
+   distinguished in any data model today, and nothing currently needs them to be — capacity
+   planning, the reaper, and academic records all behave identically regardless of how a student
+   is attending. Worth being aware this is unmodeled, not worth building a taxonomy for before a
+   real product reason exists (YAGNI).
+
+## What Phase 4.1 deliberately did NOT do here
+
+No signup flow, no student password/email auth implementation, no per-student curriculum
+assignment route, no payments, no public course catalog, no held/premium scenario content. All
+explicit non-goals per this phase's authorization. This document exists so Phase 5 planning
+starts from an accurate picture of the gap (one real item: student authentication) instead of a
+guess.

infra/.gitignore

@@ -4,26 +4,31 @@
 
 # Docker TLS material — CA/server/client certs and keys, generated by
 # infra/scripts/docker-tls/generate-ca.sh and friends. Templates/READMEs stay tracked.
-infra/docker/tls/*.pem
+docker/tls/*.pem
-infra/docker/tls/*.key
+docker/tls/*.key
-infra/docker/tls/*.crt
+docker/tls/*.crt
-infra/docker/tls/*.srl
+docker/tls/*.srl
-!infra/docker/tls/**/README.md
+!docker/tls/**/README.md
 
 # Ansible Vault password files and any decrypted vars
-infra/ansible/.vault_pass
+ansible/.vault_pass
-infra/ansible/**/vault_plain.yml
+ansible/**/vault_plain.yml
 
 # Terraform/OpenTofu state and generated vars (may contain resource IDs/IPs)
-infra/terraform/**/*.tfstate
+# Phase 4.1: these patterns were previously written repo-root-relative (e.g.
-infra/terraform/**/*.tfstate.*
+# "infra/terraform/...") inside THIS file, which git resolves relative to the .gitignore's own
-infra/terraform/**/.terraform/
+# directory (infra/) — so they silently matched nothing at all. Found via `git status` showing
-infra/terraform/**/terraform.tfvars
+# a real .terraform/ provider cache as untracked when it should have been ignored. Fixed by
-!infra/terraform/**/terraform.tfvars.example
+# dropping the redundant leading "infra/" from every pattern below.
+terraform/**/*.tfstate
+terraform/**/*.tfstate.*
+terraform/**/.terraform/
+terraform/**/terraform.tfvars
+!terraform/**/terraform.tfvars.example
 
 # Backup output and restore scratch space
-infra/backup/output/
+backup/output/
-infra/backup/restore-test/
+backup/restore-test/
 
 # Load test results (local artifacts, not source)
-infra/scripts/loadtest/results/
+scripts/loadtest/results/

platform/README.md

@@ -23,6 +23,10 @@ entrypoint/Dockerfile unmodified; frontend bundle gained Phase 3B learning UX, s
   `docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md` (routing), `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md`
   (persistent sessions, CSRF), `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` §3 (hint gating).
 - `egress-proxy/` — shared allowlisting proxy every Lab's outbound traffic passes through.
+- `lab-router/` — Phase 4.1 addition. Runs on the Lab Host VM (or `dev-local`'s stand-in in
+  local dev, though `direct` mode there never exercises it); resolves an opaque routing token
+  to a local Lab container purely from Docker state, over mTLS, so the Gateway never needs a
+  Docker-internal address. Design: `docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1-3.
 - `docker-compose.dev.yml` — the full stack (Postgres, egress proxy, Provisioner, Platform API,
   Gateway) on one machine. This is the Phase 1/2A/2B **local dev exception** — production is
   the two-VM Proxmox split described in `docs/LAB_ARCHITECTURE.md` §1.
@@ -77,13 +81,20 @@ those need a browser-equivalent client and aren't automated here.
 See `provisioner/README.md` (Phase 1), `docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md` §9 (Phase
 2A), `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md` (Phase 3A identity limitations), and
 `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` §20 (Phase 3B learning/assessment limitations) for the
-full, honest lists.
+full, honest lists. Phase 3B's "no automatic abandoned-attempt detection" and "no per-cohort
+needs-attention threshold configuration" limitations were both closed in Phase 4.1 (§13/§17 of
+`docs/PHASE4_1_PREPRODUCTION_HARDENING.md`); its raw-JSON pool_spec limitation was replaced with
+a structured editor (§18) with an owner-only raw-JSON fallback still available. Phase 4.1's own
+known limitation: real remote/self-paced delivery needs student email/password authentication,
+not built — see `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`.
 
 ## Running this in production
 
 `../infra/` has the full production infrastructure package (Ansible, Terraform, firewall,
 monitoring, backup, CI/CD) for deploying the Platform API/Gateway/Provisioner/Postgres onto a
-dedicated `rn-platform-01` VM, with student Labs on a separate `rn-lab-host-01` VM. See
+dedicated `rn-platform-01` VM, with student Labs — and a new `lab-router` service
+(`../platform/lab-router/`) — on a separate `rn-lab-host-01` VM. See
 `../docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` for the overview and
-`../docs/PRODUCTION_NETWORK_SECURITY.md` for a known gap in Gateway-to-Lab reachability across
+`../docs/PHASE4_1_PREPRODUCTION_HARDENING.md` for how Gateway-to-Lab reachability across two
-two real VMs that must be closed before production go-live.
+real VMs was closed (a Lab Host Router + mTLS data plane, simulation-tested in
+`../infra/scripts/simulate-two-host/`) and for the rest of Phase 4.1's pre-production hardening.