feat(security): production Compose topology, firewall rulesets, Docker mTLS

Named-bridge-network production Compose (not host networking, unlike dev)
for rn-platform-01; egress-proxy + agents for rn-lab-host-01. Caddy reverse
proxy, single public origin, Grafana restricted to management_cidr.
iptables-restore firewall rulesets for all three hosts (Proxmox host,
rn-platform-01, rn-lab-host-01) codifying the Platform/Lab-Host boundary:
no student-to-student traffic, no direct Proxmox exposure, no direct
Postgres/Provisioner exposure from the Lab Host.

Mutual TLS for the Provisioner<->Lab Host Docker connection: certificate
generation/verification scripts, dockerd TLS config, firewall-restricted
port 2376. Application support already existed since Phase 1; this is
purely the operational half.

Validated: docker compose config, caddy validate, firewall structural lint,
and a real openssl s_server/s_client mTLS handshake proving the cert chain
and mutual-auth rejection (legitimate client verified, rogue client
correctly rejected with verify code 19).

infra/docker/IMAGE_STRATEGY.md

@@ -0,0 +1,49 @@
+# Container image strategy
+
+## Images
+
+| Image | Built from | Registry path |
+|---|---|---|
+| `rootnode/platform-api` | `platform/api/Dockerfile` | `{{ registry }}/rootnode/platform-api` |
+| `rootnode/gateway` | `platform/gateway/Dockerfile` | `{{ registry }}/rootnode/gateway` |
+| `rootnode/provisioner` | `platform/provisioner/Dockerfile` | `{{ registry }}/rootnode/provisioner` |
+| `rootnode/egress-proxy` | `platform/egress-proxy/Dockerfile` | `{{ registry }}/rootnode/egress-proxy` |
+| `rootnode/lab` | root `Dockerfile` (the student Lab image) | `{{ registry }}/rootnode/lab` |
+
+## Tagging
+
+**Never `:latest` in staging or production.** Every build is tagged with:
+
+- the full git SHA (`sha-<40 hex>`) — always, every build, the ground truth
+- `staging-<short-sha>` on every push to `main` that passes CI (moving pointer, staging only)
+- a semantic tag (`vX.Y.Z`) only on an explicit, human-triggered production release
+
+`infra/scripts/ci/tag-and-push.sh` implements this — see §18 in
+`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`.
+
+Deployment configs (`infra/environments/*/​*.env.template`, Ansible `group_vars`) reference the
+tag as a variable (`image_tag`), never a hardcoded value baked into a compose file — rolling
+back is changing one variable and re-running the `platform`/`lab-host` Ansible role, not editing
+a compose file by hand (see `docs/RUNBOOK_ROLLBACK.md`).
+
+## Registry
+
+No paid registry required. Two supported options, chosen at deploy time via `registry` in
+`infra/ansible/group_vars/production.yml`:
+
+1. **Gitea's own built-in container registry** (`git.synthossystems.com`) — zero new
+   infrastructure, already authenticated the same way the repo itself is. Preferred default.
+2. **A self-hosted registry** (e.g. a `registry:2` container on rn-platform-01 itself, or
+   elsewhere in the homelab) — a documented fallback if the Gitea registry isn't enabled on
+   this instance.
+
+Either way: private registry, credentials pulled the same way any other production secret is
+(`docs/PRODUCTION_DEPLOYMENT.md` §10), never a public Docker Hub push of anything beyond what's
+already public (this project doesn't publish anything to Docker Hub in Phase 4).
+
+## Rollback
+
+Every image is content-addressable by SHA tag, so "roll back" is always available as long as
+the previous tag hasn't been garbage-collected from the registry — see
+`docs/RUNBOOK_ROLLBACK.md` for the exact procedure (re-point `image_tag`, re-run the Ansible
+role, verify).

infra/docker/lab-host/docker-compose.production.yml

@@ -0,0 +1,68 @@
+# Production compose for rn-lab-host-01. Deployed by infra/ansible/roles/lab-host/ to
+# /opt/rootnode/lab-host/docker-compose.yml, run as a systemd unit
+# (rootnode-lab-host.service.j2). Runs exactly one long-lived service: the shared egress proxy
+# every student Lab's outbound traffic passes through (platform/egress-proxy/, unchanged since
+# Phase 1). Student Lab containers themselves are NOT part of this compose file — they're
+# created/destroyed individually by the Provisioner over the remote Docker TLS endpoint
+# (dockerd itself, not compose), same mechanism as dev, just remote.
+
+networks:
+  rn-egress:
+    driver: bridge
+
+services:
+  egress-proxy:
+    image: "{{ registry }}/rootnode/egress-proxy:{{ image_tag }}"
+    restart: unless-stopped
+    ports:
+      - "3128:3128"   # reachable from every per-Lab Docker network on this host — restricted
+                       # to those networks by the Provisioner's per-environment DOCKER-USER
+                       # chains, not by a published-port firewall rule (there is no sensible
+                       # single source IP to restrict this to; the egress proxy's OWN allowlist
+                       # is the actual control, see platform/egress-proxy/allowlist.txt)
+    networks: [rn-egress]
+
+  node-exporter:
+    image: prom/node-exporter:latest
+    restart: unless-stopped
+    network_mode: host
+    pid: host
+    volumes:
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/rootfs:ro
+    command:
+      - "--path.procfs=/host/proc"
+      - "--path.sysfs=/host/sys"
+      - "--path.rootfs=/rootfs"
+      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
+
+  cadvisor:
+    image: gcr.io/cadvisor/cadvisor:latest
+    restart: unless-stopped
+    privileged: true
+    devices:
+      - /dev/kmsg
+    volumes:
+      - /:/rootfs:ro
+      - /var/run:/var/run:ro
+      - /sys:/sys:ro
+      - /var/lib/docker/:/var/lib/docker:ro
+    ports:
+      - "8080:8080"   # scraped remotely by Prometheus on rn-platform-01 — restricted to that
+                       # one source IP by the host firewall (rn-lab-host-01.rules), not by
+                       # binding to localhost, since Prometheus isn't local on this host.
+
+  promtail:
+    image: grafana/promtail:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/lab-host/promtail-config.yml:/etc/promtail/config.yml:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /var/log/journal:/var/log/journal:ro
+      - /etc/machine-id:/etc/machine-id:ro
+    command: ["-config.file=/etc/promtail/config.yml"]
+    # host networking so promtail can reach Loki on rn-platform-01 over the network — this
+    # traffic is OUTBOUND from rn-lab-host-01 (log push, client-initiated), which the host
+    # firewall's default-ACCEPT OUTPUT chain already allows; nothing new needed there.
+    network_mode: host

infra/docker/platform/Caddyfile.j2

@@ -0,0 +1,38 @@
+# Caddy 2 — single stable RootNode Academy origin, automatic HTTPS (Let's Encrypt) unless
+# Cloudflare origin certs are used instead (see docs/PRODUCTION_NETWORK_SECURITY.md §8 for the
+# Cloudflare-vs-direct-LE decision). Rendered by the `platform` Ansible role from
+# infra/ansible/group_vars/production.yml's `academy_domain`.
+#
+# The student/instructor path proxies to the Gateway (platform/gateway) — the sole
+# student-facing entrypoint, unchanged architectural rule from Phase 2A onward. Nothing here
+# reaches the Provisioner, Postgres, or Docker TLS directly; those have no public route at all.
+# Grafana is reachable at /grafana/ but restricted to the management CIDR even though it has
+# its own login — internal observability tooling shouldn't be internet-facing at all, same
+# posture as everything else that isn't the student/instructor product surface.
+
+{{ academy_domain }} {
+	encode gzip zstd
+
+	handle_path /grafana/* {
+		@notmanagement not remote_ip {{ management_cidr }}
+		respond @notmanagement 403
+		reverse_proxy grafana:3000
+	}
+
+	# WebSocket terminal traffic (shell-ws) needs the connection upgraded, not buffered —
+	# Caddy's reverse_proxy handles this automatically for Upgrade: websocket requests, no
+	# special directive needed, but flush_interval -1 avoids buffering SSE/long-poll-shaped
+	# responses too. `gateway` is the Docker Compose service name (rn-platform-internal
+	# network), not localhost — Caddy no longer shares host networking with the Gateway in
+	# production, unlike the dev compose file.
+	reverse_proxy gateway:4200 {
+		flush_interval -1
+	}
+
+	log {
+		output file /var/log/caddy/access.log {
+			roll_size 100mb
+			roll_keep 7
+		}
+	}
+}

infra/docker/platform/docker-compose.production.yml

@@ -0,0 +1,212 @@
+# Production compose for rn-platform-01. Deployed by infra/ansible/roles/platform/ to
+# /opt/rootnode/platform/docker-compose.yml, run as a systemd-managed `docker compose` unit
+# (infra/ansible/roles/platform/templates/rootnode-platform.service.j2).
+#
+# Differs from platform/docker-compose.dev.yml in the ways that actually matter for
+# production, and nothing else:
+#   - images are pulled from a registry by DIGEST-PINNED TAG (never :latest, never built
+#     in-place) — see infra/docker/IMAGE_STRATEGY.md
+#   - Postgres data and backups live on a bind-mounted ZFS dataset, not an anonymous volume
+#   - a reverse proxy (Caddy) terminates TLS and is the only container with a port published to
+#     the internet; a few other services publish loopback-only or single-source-IP-restricted
+#     ports for specific operational needs (Postgres for the host's own backup cron, Loki for
+#     rn-lab-host-01's log shipper) — see each service's own comment
+#   - no `network_mode: host` on gateway/provisioner — production uses a named bridge network,
+#     since rn-platform-01 no longer needs to reach ephemeral container IPs directly (those live
+#     on rn-lab-host-01, reached over the network, not shared Docker networking)
+#   - the api/gateway/provisioner containers still don't reach the internet except through
+#     what's explicitly allowed by the host firewall (infra/firewall/), not Docker's own
+#     default bridge NAT behavior
+#
+# All image tags, ports, and paths below are Ansible Jinja2 variables — this file is a
+# template rendered by the `platform` role, not run directly. See
+# infra/ansible/group_vars/production.yml for the values.
+
+networks:
+  rn-platform-internal:
+    driver: bridge
+    internal: false   # postgres/api/gateway/provisioner need it false to reach rn-lab-host-01
+                       # over the network; the HOST firewall (not Docker) is what actually
+                       # restricts what that traffic can reach — see docs/PRODUCTION_NETWORK_SECURITY.md
+
+volumes:
+  postgres-data:
+    driver: local
+    driver_opts:
+      type: none
+      o: bind
+      device: "{{ rootnode_data_root }}/postgres"
+  caddy-data:
+  caddy-config:
+  prometheus-data:
+  alertmanager-data:
+  grafana-data:
+  loki-data:
+
+services:
+  postgres:
+    image: postgres:16-alpine
+    restart: unless-stopped
+    env_file: /etc/rootnode/postgres.env
+    volumes:
+      - postgres-data:/var/lib/postgresql/data
+      - /opt/rootnode/platform/postgresql.conf.d:/etc/postgresql/conf.d:ro
+    # include_dir appends our tuning files to the image's own auto-generated postgresql.conf —
+    # no need to replace or template the base config file at all.
+    command: ["postgres", "-c", "include_dir=/etc/postgresql/conf.d"]
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-rootnode_platform}"]
+      interval: 10s
+      timeout: 5s
+      retries: 10
+    networks: [rn-platform-internal]
+    ports:
+      - "127.0.0.1:5432:5432"   # loopback-only — reachable from other containers on
+                                 # rn-platform-internal AND from the host's own backup cron
+                                 # (infra/backup/), never from another host on the network, let
+                                 # alone the internet. This is NOT the same as the "Postgres
+                                 # never exposed on this VM's external interface" rule in
+                                 # rn-platform-01.rules — that rule (and the host firewall in
+                                 # general) still applies to this port; 127.0.0.1 binding is a
+                                 # second, independent reason nothing off-host can ever reach it.
+
+  provisioner:
+    image: "{{ registry }}/rootnode/provisioner:{{ image_tag }}"
+    restart: unless-stopped
+    env_file: /etc/rootnode/provisioner.env
+    volumes:
+      - /etc/rootnode/provisioner-config:/app/config:ro
+      - /etc/rootnode/docker-tls:/app/docker-tls:ro
+    depends_on:
+      postgres:
+        condition: service_healthy
+    networks: [rn-platform-internal]
+    cap_add: [NET_ADMIN, NET_RAW]   # only needed while a lab_host row is still a local Unix
+                                    # socket (dev); harmless no-op against a remote TLS endpoint
+
+  api:
+    image: "{{ registry }}/rootnode/platform-api:{{ image_tag }}"
+    restart: unless-stopped
+    env_file: /etc/rootnode/platform.env
+    depends_on:
+      postgres:
+        condition: service_healthy
+      provisioner:
+        condition: service_started
+    networks: [rn-platform-internal]
+
+  gateway:
+    image: "{{ registry }}/rootnode/gateway:{{ image_tag }}"
+    restart: unless-stopped
+    env_file: /etc/rootnode/gateway.env
+    depends_on: [provisioner, api]
+    networks: [rn-platform-internal]
+
+  reverse-proxy:
+    image: caddy:2-alpine
+    restart: unless-stopped
+    ports:
+      - "443:443"
+      - "80:80"   # ACME HTTP-01 challenge / plain-HTTP-to-HTTPS redirect only
+    volumes:
+      - /opt/rootnode/platform/Caddyfile:/etc/caddy/Caddyfile:ro
+      - caddy-data:/data
+      - caddy-config:/config
+    depends_on: [gateway]
+    networks: [rn-platform-internal]
+
+  # ── Observability agents (infra/monitoring/) ────────────────────────────────────────────
+  node-exporter:
+    image: prom/node-exporter:latest
+    restart: unless-stopped
+    network_mode: host   # needs host-level metrics; this is metrics-only, not app traffic
+    pid: host
+    volumes:
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/rootfs:ro
+      - /var/lib/node-exporter/textfile:/textfile:ro   # backup-postgres.sh writes its metric here
+    command:
+      - "--path.procfs=/host/proc"
+      - "--path.sysfs=/host/sys"
+      - "--path.rootfs=/rootfs"
+      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
+      - "--collector.textfile.directory=/textfile"
+
+  postgres-exporter:
+    image: prometheuscommunity/postgres-exporter:latest
+    restart: unless-stopped
+    environment:
+      DATA_SOURCE_NAME: "postgresql://{{ postgres_exporter_user }}:{{ vault_postgres_exporter_password }}@postgres:5432/rootnode_platform?sslmode=disable"
+    depends_on: [postgres]
+    networks: [rn-platform-internal]
+
+  blackbox-exporter:
+    image: prom/blackbox-exporter:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
+    networks: [rn-platform-internal]
+
+  prometheus:
+    image: prom/prometheus:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - /opt/rootnode/monitoring/rules:/etc/prometheus/rules:ro
+      - prometheus-data:/prometheus
+    command:
+      - "--config.file=/etc/prometheus/prometheus.yml"
+      - "--storage.tsdb.retention.time={{ prometheus_retention_days }}d"
+    networks: [rn-platform-internal]
+    ports:
+      - "127.0.0.1:9090:9090"   # bound to loopback only — access via SSH tunnel or Grafana, never public
+
+  alertmanager:
+    image: prom/alertmanager:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
+      - alertmanager-data:/alertmanager
+    networks: [rn-platform-internal]
+
+  grafana:
+    image: grafana/grafana-oss:latest
+    restart: unless-stopped
+    environment:
+      GF_SECURITY_ADMIN_PASSWORD: "{{ vault_grafana_admin_password }}"
+      GF_SERVER_ROOT_URL: "https://{{ academy_domain }}/grafana/"
+      GF_SERVER_SERVE_FROM_SUB_PATH: "true"
+      GF_USERS_ALLOW_SIGN_UP: "false"
+    volumes:
+      - grafana-data:/var/lib/grafana
+      - /opt/rootnode/monitoring/grafana/provisioning:/etc/grafana/provisioning:ro
+      - /opt/rootnode/monitoring/grafana/dashboards:/var/lib/grafana/dashboards:ro
+    networks: [rn-platform-internal]
+    # No published port — reached only via the reverse proxy's /grafana/ path (Caddyfile),
+    # never a direct public listener. See docs/PRODUCTION_NETWORK_SECURITY.md §8.
+
+  loki:
+    image: grafana/loki:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/loki-config.yml:/etc/loki/local-config.yaml:ro
+      - loki-data:/loki
+    command: ["-config.file=/etc/loki/local-config.yaml"]
+    networks: [rn-platform-internal]
+    ports:
+      - "3100:3100"   # reachable from rn-lab-host-01's promtail over the network — restricted
+                       # to that one source IP by the host firewall (rn-platform-01.rules), same
+                       # pattern as Docker TLS's 2376. Not published to the internet.
+
+  promtail:
+    image: grafana/promtail:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/promtail-config.yml:/etc/promtail/config.yml:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /var/log/journal:/var/log/journal:ro
+      - /etc/machine-id:/etc/machine-id:ro
+    command: ["-config.file=/etc/promtail/config.yml"]
+    networks: [rn-platform-internal]
+    depends_on: [loki]

infra/docker/postgres/postgresql.conf.d/00-rootnode-baseline.conf

@@ -0,0 +1,30 @@
+# Conservative production tuning baseline — deliberately NOT hand-tuned to a specific RAM
+# allocation before rn-platform-01's real resources are known (Phase 4 instruction: do not
+# over-tune before actual hardware exists). Sized for a modest ~4GB Postgres allocation, safe
+# defaults everywhere else. Revisit once VM RAM is final — see docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §11.
+
+shared_buffers = 1GB
+effective_cache_size = 3GB
+maintenance_work_mem = 256MB
+work_mem = 16MB
+
+max_connections = 100
+
+# WAL — safe defaults for a single-instance setup with pg_dump/pg_basebackup-style backups
+# (infra/backup/). Not configured for streaming replication (out of scope this phase — see
+# docs/BACKUP_DR.md's DR model for why pg_dump + periodic pg_basebackup is the chosen approach,
+# not a hot standby).
+wal_level = replica
+max_wal_size = 2GB
+min_wal_size = 512MB
+checkpoint_completion_target = 0.9
+
+# Query logging — slow queries only, never full statement logging (would capture bind
+# parameters that can include session tokens' hashes / PII in edge cases). See docs/OBSERVABILITY.md.
+log_min_duration_statement = 500
+log_line_prefix = '%m [%p] %q%u@%d '
+log_checkpoints = on
+log_connections = on
+log_disconnections = on
+
+# Autovacuum stays on defaults — no reason to disable it for this workload size.

infra/docker/tls/README.md

@@ -0,0 +1,19 @@
+# Docker TLS material (generated, not committed)
+
+Generate with `infra/scripts/docker-tls/generate-docker-tls.sh`. Deploy with the `docker-tls`
+Ansible role. Verify with `infra/scripts/docker-tls/verify-docker-tls.sh`.
+
+**Nothing in this directory is ever committed** — see `infra/.gitignore`. This README (and only
+this README) is tracked, so the directory itself exists in the repo as documentation of what
+belongs here operationally.
+
+Files that will exist here on a real deployment host (never in git):
+
+```
+ca.pem                              # public — safe to distribute to both VMs
+ca-key.pem                          # PRIVATE — offline/secrets-vault only, never on either VM
+server-cert.pem / server-key.pem                    # rn-lab-host-01's dockerd
+provisioner-client-cert.pem / provisioner-client-key.pem   # rn-platform-01's Provisioner
+```
+
+Rotation procedure: `docs/RUNBOOK_DOCKER_TLS_ROTATION.md`.