feat(security): production Compose topology, firewall rulesets, Docker mTLS

Named-bridge-network production Compose (not host networking, unlike dev)
for rn-platform-01; egress-proxy + agents for rn-lab-host-01. Caddy reverse
proxy, single public origin, Grafana restricted to management_cidr.
iptables-restore firewall rulesets for all three hosts (Proxmox host,
rn-platform-01, rn-lab-host-01) codifying the Platform/Lab-Host boundary:
no student-to-student traffic, no direct Proxmox exposure, no direct
Postgres/Provisioner exposure from the Lab Host.

Mutual TLS for the Provisioner<->Lab Host Docker connection: certificate
generation/verification scripts, dockerd TLS config, firewall-restricted
port 2376. Application support already existed since Phase 1; this is
purely the operational half.

Validated: docker compose config, caddy validate, firewall structural lint,
and a real openssl s_server/s_client mTLS handshake proving the cert chain
and mutual-auth rejection (legitimate client verified, rogue client
correctly rejected with verify code 19).

infra/firewall/README.md

@@ -0,0 +1,31 @@
+# Production firewall policy
+
+iptables (not nftables) — deliberately, to stay consistent with the Provisioner's own dynamic
+per-Lab egress rules, which already manipulate the `DOCKER-USER` chain via iptables
+(`platform/provisioner/src/firewall.js`, Phase 1). Mixing the nft and legacy iptables backends
+on the same host is a known source of silent rule-ordering bugs; the host-level policy below
+uses the same backend the app already depends on.
+
+Each `.rules` file is `iptables-restore` format — deployed by the matching Ansible role
+(`firewall` for the VMs; the Proxmox host ruleset is applied manually per
+`docs/PRODUCTION_DEPLOYMENT.md` §2, since Proxmox host firewalling is out of Ansible's blast
+radius by design — a mistake there can lock out the only way in).
+
+| File | Applies to | Deployed by |
+|---|---|---|
+| `proxmox-host.rules` | The Proxmox hypervisor itself | manual, `docs/PRODUCTION_DEPLOYMENT.md` §2 |
+| `rn-platform-01.rules` | Platform VM | `infra/ansible/roles/firewall/` |
+| `rn-lab-host-01.rules` | Lab Host VM | `infra/ansible/roles/firewall/` |
+
+All three encode the same policy table from `docs/PRODUCTION_NETWORK_SECURITY.md` §6 — this
+README doesn't restate it, that doc is the source of truth for *why*; these files are the
+*what*, in a form the host can actually load.
+
+**IP addresses/subnets are Ansible template variables** (`{{ platform_ip }}`,
+`{{ lab_host_ip }}`, `{{ management_cidr }}`, etc.) — see `infra/ansible/group_vars/production.yml`.
+Nothing here hardcodes a real production IP, per the explicit Phase 4 instruction not to guess
+those before the real host exists.
+
+`infra/firewall/validate-firewall.sh` renders each ruleset with dummy IPs and runs it through
+`iptables-restore --test` (syntax/semantic validation only — proves the rules are well-formed
+iptables, not that the real network topology is correct, which needs the real hardware).

infra/firewall/proxmox-host.rules

@@ -0,0 +1,42 @@
+# iptables-restore ruleset for the Proxmox VE hypervisor itself. Applied MANUALLY per
+# docs/PRODUCTION_DEPLOYMENT.md §2 — deliberately not Ansible-automated (a mistake in a
+# playbook run against the one host that IS the only way to reach every VM console is a much
+# worse failure mode than a manual, reviewed apply). Prefer configuring this through Proxmox's
+# own Datacenter -> Firewall UI/API where practical instead of hand-editing iptables on the
+# hypervisor — this file documents the equivalent policy either way.
+#
+# Policy summary (docs/PRODUCTION_NETWORK_SECURITY.md §5-6):
+#   - Proxmox web UI (8006) and SSH (22) reachable ONLY from the management CIDR — never from
+#     the internet, never from either VM's network, never from a student Lab subnet
+#   - VM traffic (rn-platform-01 <-> rn-lab-host-01, rn-platform-01 <-> internet via NAT/bridge)
+#     is FORWARDed by the bridge, not filtered by this hypervisor-level ruleset — that's the
+#     VMs' own INPUT chains' job (rn-platform-01.rules / rn-lab-host-01.rules)
+#   - no student Lab subnet, and no rn-lab-host-01 traffic at all, may ever reach Proxmox
+#     management — this is the literal "Proxmox management not reachable from student Lab
+#     networks" requirement, enforced at the hypervisor itself, not just hoped-for via VM
+#     network segmentation
+
+*filter
+:INPUT DROP [0:0]
+:FORWARD ACCEPT [0:0]
+:OUTPUT ACCEPT [0:0]
+
+-A INPUT -i lo -j ACCEPT
+-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
+-A INPUT -m conntrack --ctstate INVALID -j DROP
+-A INPUT -p icmp --icmp-type echo-request -m conntrack --ctstate NEW -j ACCEPT
+
+-A INPUT -p tcp --dport 22 -s {{ management_cidr }} -m conntrack --ctstate NEW -j ACCEPT
+-A INPUT -p tcp --dport 8006 -s {{ management_cidr }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Explicit deny + log for the Lab Host VM's network and the student Lab subnet range reaching
+# Proxmox management directly — the literal "Proxmox management denied from student Lab
+# networks" requirement.
+-A INPUT -s {{ lab_host_ip }} -p tcp -m multiport --dports 22,8006 -j LOG --log-prefix "ROOTNODE-DENY-LABHOST-TO-PROXMOX: " --log-level 4
+-A INPUT -s {{ lab_host_ip }} -p tcp -m multiport --dports 22,8006 -j DROP
+-A INPUT -s {{ lab_subnet_range }} -p tcp -m multiport --dports 22,8006 -j LOG --log-prefix "ROOTNODE-DENY-LAB-TO-PROXMOX: " --log-level 4
+-A INPUT -s {{ lab_subnet_range }} -p tcp -m multiport --dports 22,8006 -j DROP
+
+-A INPUT -j LOG --log-prefix "ROOTNODE-DENY-PROXMOX-DEFAULT: " --log-level 4
+
+COMMIT

infra/firewall/rn-lab-host-01.rules

@@ -0,0 +1,62 @@
+# iptables-restore ruleset for rn-lab-host-01 (Lab Host VM). Deployed by
+# infra/ansible/roles/firewall/. See docs/PRODUCTION_NETWORK_SECURITY.md §6 for full rationale.
+#
+# This is the security-critical host — it runs privileged, disposable, student-controlled
+# containers. Its own INPUT/FORWARD policy is only half the story: the Provisioner's dynamic
+# per-Lab DOCKER-USER chains (platform/provisioner/src/egress.js, unchanged since Phase 1)
+# already enforce egress-proxy-only outbound for every student subnet and implicitly deny
+# Lab-to-Lab and Lab-to-anything-not-allowlisted traffic. This file is the BASE policy under
+# that dynamic layer — it must never be more permissive than what the dynamic chains assume,
+# and it's what protects the host itself (SSH, Docker TLS) rather than inter-container traffic.
+#
+# Policy summary:
+#   - Docker TLS (2376) reachable ONLY from rn-platform-01 — this is the entire point of mTLS
+#     plus a firewall restriction, defense in depth (a leaked client cert alone still isn't
+#     enough without also being on this IP)
+#   - SSH only from the management CIDR
+#   - no port on this host is ever reachable from a student Lab's own Docker subnet — Labs only
+#     ever reach the egress proxy and DNS resolver (already enforced by the Provisioner's
+#     per-Lab chains); this file adds an explicit belt-and-suspenders DROP for the Lab subnet
+#     range as a whole, so even a Lab whose per-environment chain failed to install can't reach
+#     the host's own services
+#   - nothing on this host is ever reachable from the public internet at all
+
+*filter
+:INPUT DROP [0:0]
+:FORWARD DROP [0:0]
+:OUTPUT ACCEPT [0:0]
+
+-A INPUT -i lo -j ACCEPT
+-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
+-A INPUT -m conntrack --ctstate INVALID -j DROP
+-A INPUT -p icmp --icmp-type echo-request -m conntrack --ctstate NEW -j ACCEPT
+
+-A INPUT -p tcp --dport 22 -s {{ management_cidr }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Docker TLS — rn-platform-01 (the Provisioner) only. This is the ONLY control-plane flow
+# allowed into this host at all — see the "Platform Provisioner -> Lab Host Docker TLS
+# endpoint: ALLOW" example in the Phase 4 spec, encoded literally here.
+-A INPUT -p tcp --dport 2376 -s {{ platform_ip }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Gateway -> per-Lab published port range, rn-platform-01 only. REQUIRED so the Gateway can
+# reach each student's Lab UI/terminal over the network once Labs run on a genuinely separate
+# host — see docs/PRODUCTION_NETWORK_SECURITY.md §"Gateway-to-Lab cross-VM reachability" for
+# why this exists and the small Provisioner/Gateway code change it depends on (NOT YET BUILT —
+# this firewall rule is the network-policy half; the port-publishing half is an explicit,
+# reviewed application-code follow-up, tracked there rather than silently assumed here).
+-A INPUT -p tcp --dport {{ lab_gateway_port_range | default('30000:32767') }} -s {{ platform_ip }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Prometheus (rn-platform-01) scraping this host's node-exporter/cadvisor — metrics only,
+# read-only exposition endpoints, still restricted to the one legitimate scraper.
+-A INPUT -p tcp -m multiport --dports 9100,8080 -s {{ platform_ip }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Belt-and-suspenders: explicit deny + log for any student Lab subnet reaching THIS HOST
+# (not another container — the host's own listening ports: SSH, Docker TLS, anything else).
+# The Provisioner's per-Lab DOCKER-USER chains already stop this at the container-egress layer;
+# this is what still protects the host if that layer is ever misconfigured or mid-install.
+-A INPUT -s {{ lab_subnet_range }} -j LOG --log-prefix "ROOTNODE-DENY-LAB-TO-HOST: " --log-level 4
+-A INPUT -s {{ lab_subnet_range }} -j DROP
+
+-A INPUT -j LOG --log-prefix "ROOTNODE-DENY-LABHOST-DEFAULT: " --log-level 4
+
+COMMIT

infra/firewall/rn-platform-01.rules

@@ -0,0 +1,58 @@
+# iptables-restore ruleset for rn-platform-01 (Platform VM). Deployed by
+# infra/ansible/roles/firewall/. Variables ({{ }}) are Ansible Jinja2, rendered before this
+# file reaches the host — see infra/ansible/group_vars/production.yml.
+#
+# Policy summary (full rationale: docs/PRODUCTION_NETWORK_SECURITY.md §6):
+#   - only the reverse proxy's 443/80 are reachable from the internet
+#   - SSH only from the management CIDR (your home/office/VPN, never 0.0.0.0/0)
+#   - Docker TLS (2376) is a rn-lab-host-01 concern, not this host's — nothing to open here
+#   - Postgres, Provisioner's internal API, and the Platform API/Gateway's raw ports are never
+#     exposed on this VM's external interface at all — the reverse proxy is the only public
+#     listener; everything else binds to the docker-compose internal network only, so there is
+#     no host-level rule needed to "hide" them, only to make sure nothing ever publishes them
+#     by accident (this ruleset's default-deny INPUT catches that class of mistake too)
+#   - rn-lab-host-01 may reach exactly one port here: Loki (3100), for its own log shipper —
+#     the Provisioner otherwise calls OUT to rn-lab-host-01, never the other way around — and
+#     may NEVER reach Postgres or the Provisioner's internal API
+
+*filter
+:INPUT DROP [0:0]
+:FORWARD DROP [0:0]
+:OUTPUT ACCEPT [0:0]
+
+# Loopback and established/related — standard baseline
+-A INPUT -i lo -j ACCEPT
+-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
+-A INPUT -m conntrack --ctstate INVALID -j DROP
+
+# ICMP — keep basic reachability diagnostics working
+-A INPUT -p icmp --icmp-type echo-request -m conntrack --ctstate NEW -j ACCEPT
+
+# SSH — management CIDR only, never open to the internet
+-A INPUT -p tcp --dport 22 -s {{ management_cidr }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Public HTTPS/HTTP — the reverse proxy (Caddy), the only public listener on this host
+-A INPUT -p tcp --dport 443 -m conntrack --ctstate NEW -j ACCEPT
+-A INPUT -p tcp --dport 80 -m conntrack --ctstate NEW -j ACCEPT
+
+# Loki log ingestion — rn-lab-host-01's promtail only (that host's own log shipper, pushing its
+# container/journald logs here for centralized viewing in Grafana). Same "one specific source
+# IP, one specific port" pattern as Docker TLS on the other host.
+-A INPUT -p tcp --dport 3100 -s {{ lab_host_ip }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Explicit deny + log for anything from rn-lab-host-01 hitting Postgres or the Provisioner's
+# internal port — defense in depth on top of "these ports were never published," and the one
+# rule in this file that most directly encodes the Lab->Platform boundary the whole platform
+# depends on. Logged (not silently dropped) so an attempted breach attempt is visible in
+# infra/monitoring/ alerts, not just absorbed by the default DROP.
+-A INPUT -s {{ lab_host_ip }} -p tcp --dport 5432 -j LOG --log-prefix "ROOTNODE-DENY-LABHOST-PG: " --log-level 4
+-A INPUT -s {{ lab_host_ip }} -p tcp --dport 5432 -j DROP
+-A INPUT -s {{ lab_host_ip }} -p tcp --dport 4100 -j LOG --log-prefix "ROOTNODE-DENY-LABHOST-PROVISIONER: " --log-level 4
+-A INPUT -s {{ lab_host_ip }} -p tcp --dport 4100 -j DROP
+
+# Default: everything else already falls to the chain policy (DROP), nothing further needed —
+# but an explicit final logged catch-all makes "what got denied and why" visible in logs rather
+# than silent.
+-A INPUT -j LOG --log-prefix "ROOTNODE-DENY-PLATFORM-DEFAULT: " --log-level 4
+
+COMMIT

infra/firewall/validate-firewall.sh

@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+# Local test (Phase 4 §29): renders each .rules file with dummy IPs and validates it.
+#
+# `iptables-restore --test` needs root (it talks to netfilter) — if this shell has passwordless
+# sudo, we use it for a real semantic check. Otherwise we fall back to a structural lint
+# (balanced *table/COMMIT blocks, every rule starts with a real flag, no unrendered {{ }}
+# placeholders left over) — good enough to catch a typo'd rule or a missing COMMIT, NOT a
+# substitute for the real `iptables-restore --test` a human/Ansible with actual root runs
+# before this ever reaches a live host. HARDWARE-DEPENDENT: proving the real IPs/CIDRs are
+# correct needs the actual production network, not this script.
+set -euo pipefail
+cd "$(dirname "$0")"
+
+TMPDIR="$(mktemp -d)"
+trap 'rm -rf "$TMPDIR"' EXIT
+
+render() {
+  sed \
+    -e 's/{{ management_cidr }}/203.0.113.0\/24/g' \
+    -e 's/{{ platform_ip }}/10.10.10.10/g' \
+    -e 's/{{ lab_host_ip }}/10.10.10.20/g' \
+    -e 's/{{ lab_subnet_range }}/10.42.0.0\/16/g' \
+    -e 's/{{ proxmox_mgmt_cidr }}/203.0.113.0\/24/g' \
+    -e "s/{{ lab_gateway_port_range | default('30000:32767') }}/30000:32767/g" \
+    "$1" > "$2"
+}
+
+structural_lint() {
+  local f="$1"
+  grep -q '^\*filter' "$f" || { echo "  missing *filter table declaration"; return 1; }
+  grep -q '^COMMIT' "$f" || { echo "  missing COMMIT"; return 1; }
+  grep -v '^#' "$f" | grep -q '{{' && { echo "  unrendered {{ }} placeholder left in a non-comment line"; return 1; }
+  # every -A/-I rule line should end in a real target (ACCEPT/DROP/REJECT/LOG/a custom chain)
+  while IFS= read -r line; do
+    [[ "$line" =~ ^-[AI]\  ]] || continue
+    [[ "$line" =~ -j\ [A-Za-z0-9_-]+ ]] || { echo "  rule missing a -j target: $line"; return 1; }
+  done < "$f"
+  return 0
+}
+
+HAVE_SUDO=0
+sudo -n true 2>/dev/null && HAVE_SUDO=1
+
+FAIL=0
+for f in proxmox-host.rules rn-platform-01.rules rn-lab-host-01.rules; do
+  render "$f" "$TMPDIR/$f"
+  echo "==> $f"
+  if [ "$HAVE_SUDO" = "1" ] && command -v iptables-restore >/dev/null 2>&1; then
+    if sudo -n iptables-restore --test < "$TMPDIR/$f"; then
+      echo "OK: $f passed a real iptables-restore --test"
+    else
+      echo "FAIL: $f did not pass iptables-restore --test" >&2; FAIL=1
+    fi
+  else
+    if structural_lint "$TMPDIR/$f"; then
+      echo "OK: $f passes structural lint (no root available for a real iptables-restore --test — see script header)"
+    else
+      echo "FAIL: $f failed structural lint" >&2; FAIL=1
+    fi
+  fi
+done
+
+exit $FAIL