docs(phase4.1): pre-production hardening documentation

Master overview (PHASE4_1_PREPRODUCTION_HARDENING.md) covering all 27
substantive items, plus REMOTE_AND_SELF_PACED_ARCHITECTURE.md (reviews
what already works for remote/self-paced delivery - student-driven
provisioning, cohort-less students, idle-reaper-destroy-then-continue -
and the one real gap: student authentication, explicitly Phase 5 scope,
not built here).

Updated PRODUCTION_NETWORK_SECURITY.md (the Gateway-to-Lab gap is now
RESOLVED, with the full design writeup and a new control-plane-vs-data-plane
section), PRODUCTION_DEPLOYMENT.md (four-cert TLS bundle),
PRODUCTION_VALIDATION_CHECKLIST.md (Router mTLS check, admission control
checks), PHASE4_PRODUCTION_INFRASTRUCTURE.md and ENGINEERING_SOP.md and
LAB_ARCHITECTURE.md (stale 'known gap' pointers updated to point at the
resolution), BACKUP_DR.md (destructive drill), README.md and
platform/README.md (Phase 4.1 pointers, lab-router component, updated
known-limitations).

Also fixes a real pre-existing bug found while auditing what gets
committed: infra/.gitignore's patterns were written repo-root-relative
(e.g. "infra/terraform/...") but the file lives in infra/, so git resolved
them relative to THAT directory and none of them ever matched anything -
a real .terraform/ provider cache showed up as untracked when it should
have been ignored. Fixed by dropping the redundant leading "infra/" from
every pattern.

Status: PHASE 4.1 COMPLETE / PRE-PRODUCTION HARDENING COMPLETE / READY FOR
HARDWARE DEPLOYMENT.

docs/PRODUCTION_NETWORK_SECURITY.md

@@ -12,7 +12,8 @@ management_cidr ──SSH──▶ rn-platform-01                │
 management_cidr ──SSH──▶ rn-lab-host-01                 │
 proxmox_mgmt_cidr ──22/8006──▶ Proxmox host             │
                                                         │
-rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd)
+rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd)          [CONTROL plane]
+rn-platform-01 (Gateway) ──mTLS:8443──▶ rn-lab-host-01 (Lab Host Router)     [DATA plane]
 rn-lab-host-01 (promtail) ──3100──▶ rn-platform-01 (Loki)
 rn-lab-host-01 (node-exporter/cadvisor) ◀──scrape── rn-platform-01 (Prometheus)
 ```
@@ -37,6 +38,25 @@ Firewall source of truth: `infra/firewall/{proxmox-host,rn-platform-01,rn-lab-ho
   (Grafana is reachable only via Caddy's `/grafana/*` path, itself restricted to
   `management_cidr` by a `caddy` matcher — see `infra/docker/platform/Caddyfile.j2`).
 
+## Control plane vs data plane (Phase 4.1 §10)
+
+Two separate, never-conflated mTLS connections between the two VMs, deliberately kept apart
+both conceptually and operationally:
+
+| | Control plane | Data plane |
+|---|---|---|
+| What | Provisioner manages Lab containers | Gateway serves student browser traffic |
+| Path | Provisioner → dockerd (port 2376) | Gateway → Lab Host Router (port 8443) |
+| Who initiates | Provisioner only | Gateway only |
+| What it can do | Create/start/stop/remove containers, full Docker API | Reverse-proxy one HTTP/WS request to one already-running container by opaque token — nothing else |
+| Certs | `provisioner-client-*` / `server-*` | `gateway-client-*` / `router-server-*` |
+| Exposes Docker API to caller? | Yes (that's its job — Provisioner IS a Docker client) | **Never** — the Router never exposes Docker API, container IDs, or network info to its caller |
+
+The student browser's traffic is proxied at the Gateway/Router layer and never touches the
+Docker API at any point — the Router deliberately has no route that echoes back anything
+Docker-internal (see `platform/lab-router/src/server.js`'s header comment for the full "narrow
+by design" list).
+
 ## Docker TLS (Provisioner ↔ Lab Host)
 
 Mutual TLS on port 2376, restricted at the firewall to `platform_ip` only. Application support
@@ -57,34 +77,51 @@ front this with Cloudflare or expose the origin directly is left as a deployment
 being able to see plaintext unless "Full (strict)" mode is used) — not decided here since it
 doesn't affect anything else in this document.
 
-## Known gap: Gateway-to-Lab reachability across two real VMs
+## Gateway-to-Lab reachability across two real VMs — RESOLVED in Phase 4.1
 
-**This is the most important item in this document.**
+Phase 4 flagged this as the single most important open item; Phase 4.1
+(`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1) closed it. Kept here as the historical record of
+what the gap was and how it was closed, since the reasoning matters for anyone extending this
+later (e.g. adding a second Lab Host).
 
-The Gateway currently reaches a running Lab container via `env.internal_ip` — a Docker
-bridge-local address recorded by the Provisioner when it creates the container
-(`platform/gateway/src/server.js`, `platform/provisioner`, built in Phase 2A). This works when
-Gateway and the Lab share a Docker daemon, which has been true in every setup so far, including
-dev and any single-host deployment. **It stops being true once Labs run on a genuinely separate
-`rn-lab-host-01`** — a Docker bridge-internal IP on one host is not routable from a different
-host by default.
+**The gap, as originally found:** the Gateway used to reach a running Lab container via
+`env.internal_ip` — a Docker bridge-local address recorded by the Provisioner when it creates
+the container. That only works when Gateway and the Lab share a Docker daemon, true in dev and
+any single-host setup, false the moment Labs run on a genuinely separate `rn-lab-host-01`.
 
-What's built now (network-policy half only): a firewall allowance in
-`infra/firewall/rn-lab-host-01.rules` for `platform_ip` → `lab_gateway_port_range`
-(`30000:32767` by default, configurable), so that once Lab containers publish ports on the Lab
-Host's own interface, the Gateway VM is permitted to reach them.
+**The fix — a Lab Host Router (`platform/lab-router/`):** a small, narrow service that runs on
+rn-lab-host-01 (host-networked, like `egress-proxy`, so it can reach whatever per-Lab bridge
+network the Provisioner creates) and does exactly one thing: given an opaque routing token over
+mTLS, find the ONE local container labeled with that token (`docker ps --filter
+label=rootnode.routing-token=<token>` — resolved purely from local Docker state, no Postgres,
+no call back to the Provisioner) and reverse-proxy the request to it, HTTP and WebSocket both.
 
-What's **not yet built** (application half — explicitly flagged, not silently decided here):
-the Provisioner needs to either (a) publish each Lab container's relevant port(s) on the Lab
-Host's interface and record a reachable `external_ip`/`external_port` alongside (or instead of)
-`internal_ip`/`internal_port`, or (b) route Gateway→Lab traffic through a proxy/tunnel that
-lives on the Lab Host and forwards to the container's internal address. Option (a) is simpler
-and consistent with how the egress proxy and cadvisor are already handled in this phase's
-compose files; option (b) avoids publishing a wide port range but adds a moving part. This is a
-small, reviewable, application-level change — not a network-policy change — and should land as
-its own reviewed commit before real two-VM production traffic depends on it. Until it lands,
-Gateway↔Lab terminal/scenario traffic will not work correctly across two separate VMs, even
-though every other production networking, firewall, and TLS boundary is in place and correct.
+The Gateway now decides `direct` vs `router` per `lab_host` (`lab_hosts.routing_mode` +
+`router_endpoint`, `platform/provisioner/migrations/004_lab_hosts_routing.sql`) via
+`platform/gateway/src/lab-endpoint.js` — a single function, `labTarget(env, token)`, that every
+proxy/fetch call site in `server.js` goes through instead of building
+`http://${env.internal_ip}:${env.internal_port}` itself. `dev-local` stays `direct` (T480/local
+dev unaffected, zero behavior change there); a real `rn-lab-host-01` is configured `router`.
+
+**Firewall (`infra/firewall/rn-lab-host-01.rules`):** the old speculative
+`lab_gateway_port_range` allowance (which assumed option (a), publishing per-Lab ports) was
+replaced with a single, narrow rule: `platform_ip` → port `{{ lab_router_port | default('8443')
+}}` only. No wide port range, no per-Lab port publishing at all — the Router is the only thing
+the Gateway ever talks to on rn-lab-host-01 for Lab traffic.
+
+**mTLS:** the Gateway↔Router pair uses the SAME private CA as the Provisioner↔dockerd pair
+(`infra/scripts/docker-tls/generate-docker-tls.sh`, extended in Phase 4.1 to also issue
+`router-server`/`gateway-client` leaf certs), deployed to a separate directory
+(`/etc/rootnode/lab-router-tls/`) by the same `docker-tls` Ansible role.
+
+**Proven, not just designed:** `infra/scripts/simulate-two-host/` builds a real two-network
+Docker simulation (Gateway genuinely isolated on its own bridge, no route to any Lab's bridge)
+and runs 13 real assertions against it — Gateway cannot reach a Lab's Docker-internal IP
+directly, but reaches it correctly through the Router (HTTP and a real WebSocket
+`kubectl get nodes`), Alice/Bob isolation holds, garbage/destroyed tokens are rejected, and the
+Router itself refuses any connection without the Gateway's client certificate. See that
+directory's README for exactly what is and isn't proven by a same-daemon simulation versus real
+separate hardware.
 
 ## Cookie / auth production settings