# Production Network & Security The Platform VM / Lab Host VM boundary is the core production control. This document is where every network decision that touches it gets written down. Nothing here weakens it. ## Topology ``` Internet ──HTTPS(443)──▶ Caddy (rn-platform-01) ──▶ Gateway (internal bridge) │ management_cidr ──SSH──▶ rn-platform-01 │ management_cidr ──SSH──▶ rn-lab-host-01 │ proxmox_mgmt_cidr ──22/8006──▶ Proxmox host │ │ rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd) [CONTROL plane] rn-platform-01 (Gateway) ──mTLS:8443──▶ rn-lab-host-01 (Lab Host Router) [DATA plane] rn-lab-host-01 (promtail) ──3100──▶ rn-platform-01 (Loki) rn-lab-host-01 (node-exporter/cadvisor) ◀──scrape── rn-platform-01 (Prometheus) ``` Firewall source of truth: `infra/firewall/{proxmox-host,rn-platform-01,rn-lab-host-01}.rules` (iptables-restore format, deployed by the `firewall` Ansible role). Validate with `infra/firewall/validate-firewall.sh`. ## What is explicitly denied - Student Lab → another student Lab (no student-to-student traffic, ever). - Student Lab → Proxmox management (both the host firewall and the `lab_subnet_range` DENY rule in `rn-lab-host-01.rules`). - Student Lab → Platform Postgres / Platform API / Gateway internals (logged DENY in `rn-platform-01.rules` for anything from `lab_host_ip` to ports 5432/4100). - rn-lab-host-01 → Postgres (same rule — the Lab Host itself has no legitimate reason to reach Postgres; only the Provisioner on rn-platform-01 does, and it talks to Postgres over the internal Docker bridge, never across the VM boundary). - Any student egress except through the approved egress proxy (enforced by the Provisioner's existing per-Lab `DOCKER-USER` chain manipulation from Phase 1 — unchanged by this phase). - Direct public exposure of the Provisioner, Postgres, Docker TLS port, Proxmox UI, or Grafana (Grafana is reachable only via Caddy's `/grafana/*` path, itself restricted to `management_cidr` by a `caddy` matcher — see `infra/docker/platform/Caddyfile.j2`). ## Control plane vs data plane (Phase 4.1 §10) Two separate, never-conflated mTLS connections between the two VMs, deliberately kept apart both conceptually and operationally: | | Control plane | Data plane | |---|---|---| | What | Provisioner manages Lab containers | Gateway serves student browser traffic | | Path | Provisioner → dockerd (port 2376) | Gateway → Lab Host Router (port 8443) | | Who initiates | Provisioner only | Gateway only | | What it can do | Create/start/stop/remove containers, full Docker API | Reverse-proxy one HTTP/WS request to one already-running container by opaque token — nothing else | | Certs | `provisioner-client-*` / `server-*` | `gateway-client-*` / `router-server-*` | | Exposes Docker API to caller? | Yes (that's its job — Provisioner IS a Docker client) | **Never** — the Router never exposes Docker API, container IDs, or network info to its caller | The student browser's traffic is proxied at the Gateway/Router layer and never touches the Docker API at any point — the Router deliberately has no route that echoes back anything Docker-internal (see `platform/lab-router/src/server.js`'s header comment for the full "narrow by design" list). ## Docker TLS (Provisioner ↔ Lab Host) Mutual TLS on port 2376, restricted at the firewall to `platform_ip` only. Application support already existed since Phase 1 (`platform/provisioner/src/docker.js`'s `buildClient()`); this phase added the operational half: `infra/scripts/docker-tls/generate-docker-tls.sh` (CA + server + client cert generation), the `docker-tls` Ansible role (deploys certs, configures `dockerd`'s `daemon.json`), and `docs/RUNBOOK_DOCKER_TLS_ROTATION.md` for renewal. The CA private key is generated offline and never committed or deployed to either VM — only the CA cert (to verify the peer) and each host's own leaf cert+key. ## Public ingress One origin, Caddy, automatic HTTPS. `reverse_proxy gateway:4200` — Caddy talks only to the Gateway; nothing else is proxied. WebSocket upgrade works automatically (Caddy's default `reverse_proxy` behavior, no special config needed for the terminal WebSocket path). Whether to front this with Cloudflare or expose the origin directly is left as a deployment-time decision (both work with Caddy's ACME flow; Cloudflare adds DDoS absorption at the cost of Cloudflare being able to see plaintext unless "Full (strict)" mode is used) — not decided here since it doesn't affect anything else in this document. ## Gateway-to-Lab reachability across two real VMs — RESOLVED in Phase 4.1 Phase 4 flagged this as the single most important open item; Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1) closed it. Kept here as the historical record of what the gap was and how it was closed, since the reasoning matters for anyone extending this later (e.g. adding a second Lab Host). **The gap, as originally found:** the Gateway used to reach a running Lab container via `env.internal_ip` — a Docker bridge-local address recorded by the Provisioner when it creates the container. That only works when Gateway and the Lab share a Docker daemon, true in dev and any single-host setup, false the moment Labs run on a genuinely separate `rn-lab-host-01`. **The fix — a Lab Host Router (`platform/lab-router/`):** a small, narrow service that runs on rn-lab-host-01 (host-networked, like `egress-proxy`, so it can reach whatever per-Lab bridge network the Provisioner creates) and does exactly one thing: given an opaque routing token over mTLS, find the ONE local container labeled with that token (`docker ps --filter label=rootnode.routing-token=` — resolved purely from local Docker state, no Postgres, no call back to the Provisioner) and reverse-proxy the request to it, HTTP and WebSocket both. The Gateway now decides `direct` vs `router` per `lab_host` (`lab_hosts.routing_mode` + `router_endpoint`, `platform/provisioner/migrations/004_lab_hosts_routing.sql`) via `platform/gateway/src/lab-endpoint.js` — a single function, `labTarget(env, token)`, that every proxy/fetch call site in `server.js` goes through instead of building `http://${env.internal_ip}:${env.internal_port}` itself. `dev-local` stays `direct` (T480/local dev unaffected, zero behavior change there); a real `rn-lab-host-01` is configured `router`. **Firewall (`infra/firewall/rn-lab-host-01.rules`):** the old speculative `lab_gateway_port_range` allowance (which assumed option (a), publishing per-Lab ports) was replaced with a single, narrow rule: `platform_ip` → port `{{ lab_router_port | default('8443') }}` only. No wide port range, no per-Lab port publishing at all — the Router is the only thing the Gateway ever talks to on rn-lab-host-01 for Lab traffic. **mTLS:** the Gateway↔Router pair uses the SAME private CA as the Provisioner↔dockerd pair (`infra/scripts/docker-tls/generate-docker-tls.sh`, extended in Phase 4.1 to also issue `router-server`/`gateway-client` leaf certs), deployed to a separate directory (`/etc/rootnode/lab-router-tls/`) by the same `docker-tls` Ansible role. **Proven, not just designed:** `infra/scripts/simulate-two-host/` builds a real two-network Docker simulation (Gateway genuinely isolated on its own bridge, no route to any Lab's bridge) and runs 13 real assertions against it — Gateway cannot reach a Lab's Docker-internal IP directly, but reaches it correctly through the Router (HTTP and a real WebSocket `kubectl get nodes`), Alice/Bob isolation holds, garbage/destroyed tokens are rejected, and the Router itself refuses any connection without the Gateway's client certificate. See that directory's README for exactly what is and isn't proven by a same-daemon simulation versus real separate hardware. ## Cookie / auth production settings `infra/environments/production/*.env.template`: `COOKIE_SECURE=1`, `TRUST_PROXY=1`. Dev's insecure defaults (`COOKIE_SECURE` unset, no proxy trust) are never inherited — production env files are separate files, not a dev file with overrides layered on.