rootnode-academy/docs/PRODUCTION_NETWORK_SECURITY.md

Production Network & Security

The Platform VM / Lab Host VM boundary is the core production control. This document is where every network decision that touches it gets written down. Nothing here weakens it.

Topology

Internet ──HTTPS(443)──▶ Caddy (rn-platform-01) ──▶ Gateway (internal bridge)
                                                        │
management_cidr ──SSH──▶ rn-platform-01                │
management_cidr ──SSH──▶ rn-lab-host-01                 │
proxmox_mgmt_cidr ──22/8006──▶ Proxmox host             │
                                                        │
rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd)          [CONTROL plane]
rn-platform-01 (Gateway) ──mTLS:8443──▶ rn-lab-host-01 (Lab Host Router)     [DATA plane]
rn-lab-host-01 (promtail) ──3100──▶ rn-platform-01 (Loki)
rn-lab-host-01 (node-exporter/cadvisor) ◀──scrape── rn-platform-01 (Prometheus)

Firewall source of truth: infra/firewall/{proxmox-host,rn-platform-01,rn-lab-host-01}.rules (iptables-restore format, deployed by the firewall Ansible role). Validate with infra/firewall/validate-firewall.sh.

What is explicitly denied

• Student Lab → another student Lab (no student-to-student traffic, ever).
• Student Lab → Proxmox management (both the host firewall and the lab_subnet_range DENY rule in rn-lab-host-01.rules).
• Student Lab → Platform Postgres / Platform API / Gateway internals (logged DENY in rn-platform-01.rules for anything from lab_host_ip to ports 5432/4100).
• rn-lab-host-01 → Postgres (same rule — the Lab Host itself has no legitimate reason to reach Postgres; only the Provisioner on rn-platform-01 does, and it talks to Postgres over the internal Docker bridge, never across the VM boundary).
• Any student egress except through the approved egress proxy (enforced by the Provisioner's existing per-Lab DOCKER-USER chain manipulation from Phase 1 — unchanged by this phase).
• Direct public exposure of the Provisioner, Postgres, Docker TLS port, Proxmox UI, or Grafana (Grafana is reachable only via Caddy's /grafana/* path, itself restricted to management_cidr by a caddy matcher — see infra/docker/platform/Caddyfile.j2).

Control plane vs data plane (Phase 4.1 §10)

Two separate, never-conflated mTLS connections between the two VMs, deliberately kept apart both conceptually and operationally:

	Control plane	Data plane
What	Provisioner manages Lab containers	Gateway serves student browser traffic
Path	Provisioner → dockerd (port 2376)	Gateway → Lab Host Router (port 8443)
Who initiates	Provisioner only	Gateway only
What it can do	Create/start/stop/remove containers, full Docker API	Reverse-proxy one HTTP/WS request to one already-running container by opaque token — nothing else
Certs	provisioner-client-* / server-*	gateway-client-* / router-server-*
Exposes Docker API to caller?	Yes (that's its job — Provisioner IS a Docker client)	Never — the Router never exposes Docker API, container IDs, or network info to its caller

The student browser's traffic is proxied at the Gateway/Router layer and never touches the Docker API at any point — the Router deliberately has no route that echoes back anything Docker-internal (see platform/lab-router/src/server.js's header comment for the full "narrow by design" list).

Docker TLS (Provisioner ↔ Lab Host)

Mutual TLS on port 2376, restricted at the firewall to platform_ip only. Application support already existed since Phase 1 (platform/provisioner/src/docker.js's buildClient()); this phase added the operational half: infra/scripts/docker-tls/generate-docker-tls.sh (CA + server + client cert generation), the docker-tls Ansible role (deploys certs, configures dockerd's daemon.json), and docs/RUNBOOK_DOCKER_TLS_ROTATION.md for renewal. The CA private key is generated offline and never committed or deployed to either VM — only the CA cert (to verify the peer) and each host's own leaf cert+key.

Public ingress

One origin, Caddy, automatic HTTPS. reverse_proxy gateway:4200 — Caddy talks only to the Gateway; nothing else is proxied. WebSocket upgrade works automatically (Caddy's default reverse_proxy behavior, no special config needed for the terminal WebSocket path). Whether to front this with Cloudflare or expose the origin directly is left as a deployment-time decision (both work with Caddy's ACME flow; Cloudflare adds DDoS absorption at the cost of Cloudflare being able to see plaintext unless "Full (strict)" mode is used) — not decided here since it doesn't affect anything else in this document.

Gateway-to-Lab reachability across two real VMs — RESOLVED in Phase 4.1

Phase 4 flagged this as the single most important open item; Phase 4.1 (docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1) closed it. Kept here as the historical record of what the gap was and how it was closed, since the reasoning matters for anyone extending this later (e.g. adding a second Lab Host).

The gap, as originally found: the Gateway used to reach a running Lab container via env.internal_ip — a Docker bridge-local address recorded by the Provisioner when it creates the container. That only works when Gateway and the Lab share a Docker daemon, true in dev and any single-host setup, false the moment Labs run on a genuinely separate rn-lab-host-01.

The fix — a Lab Host Router (platform/lab-router/): a small, narrow service that runs on rn-lab-host-01 (host-networked, like egress-proxy, so it can reach whatever per-Lab bridge network the Provisioner creates) and does exactly one thing: given an opaque routing token over mTLS, find the ONE local container labeled with that token (docker ps --filter label=rootnode.routing-token=<token> — resolved purely from local Docker state, no Postgres, no call back to the Provisioner) and reverse-proxy the request to it, HTTP and WebSocket both.

The Gateway now decides direct vs router per lab_host (lab_hosts.routing_mode + router_endpoint, platform/provisioner/migrations/004_lab_hosts_routing.sql) via platform/gateway/src/lab-endpoint.js — a single function, labTarget(env, token), that every proxy/fetch call site in server.js goes through instead of building http://${env.internal_ip}:${env.internal_port} itself. dev-local stays direct (T480/local dev unaffected, zero behavior change there); a real rn-lab-host-01 is configured router.

Firewall (infra/firewall/rn-lab-host-01.rules): the old speculative lab_gateway_port_range allowance (which assumed option (a), publishing per-Lab ports) was replaced with a single, narrow rule: platform_ip → port {{ lab_router_port | default('8443') }} only. No wide port range, no per-Lab port publishing at all — the Router is the only thing the Gateway ever talks to on rn-lab-host-01 for Lab traffic.

mTLS: the Gateway↔Router pair uses the SAME private CA as the Provisioner↔dockerd pair (infra/scripts/docker-tls/generate-docker-tls.sh, extended in Phase 4.1 to also issue router-server/gateway-client leaf certs), deployed to a separate directory (/etc/rootnode/lab-router-tls/) by the same docker-tls Ansible role.

Proven, not just designed: infra/scripts/simulate-two-host/ builds a real two-network Docker simulation (Gateway genuinely isolated on its own bridge, no route to any Lab's bridge) and runs 13 real assertions against it — Gateway cannot reach a Lab's Docker-internal IP directly, but reaches it correctly through the Router (HTTP and a real WebSocket kubectl get nodes), Alice/Bob isolation holds, garbage/destroyed tokens are rejected, and the Router itself refuses any connection without the Gateway's client certificate. See that directory's README for exactly what is and isn't proven by a same-daemon simulation versus real separate hardware.

Cookie / auth production settings

infra/environments/production/*.env.template: COOKIE_SECURE=1, TRUST_PROXY=1. Dev's insecure defaults (COOKIE_SECURE unset, no proxy trust) are never inherited — production env files are separate files, not a dev file with overrides layered on.