ced4568
5bc0743b08
Master overview (PHASE4_1_PREPRODUCTION_HARDENING.md) covering all 27 substantive items, plus REMOTE_AND_SELF_PACED_ARCHITECTURE.md (reviews what already works for remote/self-paced delivery - student-driven provisioning, cohort-less students, idle-reaper-destroy-then-continue - and the one real gap: student authentication, explicitly Phase 5 scope, not built here). Updated PRODUCTION_NETWORK_SECURITY.md (the Gateway-to-Lab gap is now RESOLVED, with the full design writeup and a new control-plane-vs-data-plane section), PRODUCTION_DEPLOYMENT.md (four-cert TLS bundle), PRODUCTION_VALIDATION_CHECKLIST.md (Router mTLS check, admission control checks), PHASE4_PRODUCTION_INFRASTRUCTURE.md and ENGINEERING_SOP.md and LAB_ARCHITECTURE.md (stale 'known gap' pointers updated to point at the resolution), BACKUP_DR.md (destructive drill), README.md and platform/README.md (Phase 4.1 pointers, lab-router component, updated known-limitations). Also fixes a real pre-existing bug found while auditing what gets committed: infra/.gitignore's patterns were written repo-root-relative (e.g. "infra/terraform/...") but the file lives in infra/, so git resolved them relative to THAT directory and none of them ever matched anything - a real .terraform/ provider cache showed up as untracked when it should have been ignored. Fixed by dropping the redundant leading "infra/" from every pattern. Status: PHASE 4.1 COMPLETE / PRE-PRODUCTION HARDENING COMPLETE / READY FOR HARDWARE DEPLOYMENT.
131 lines
8.2 KiB
Markdown
131 lines
8.2 KiB
Markdown
# Production Network & Security
|
|
|
|
The Platform VM / Lab Host VM boundary is the core production control. This document is where
|
|
every network decision that touches it gets written down. Nothing here weakens it.
|
|
|
|
## Topology
|
|
|
|
```
|
|
Internet ──HTTPS(443)──▶ Caddy (rn-platform-01) ──▶ Gateway (internal bridge)
|
|
│
|
|
management_cidr ──SSH──▶ rn-platform-01 │
|
|
management_cidr ──SSH──▶ rn-lab-host-01 │
|
|
proxmox_mgmt_cidr ──22/8006──▶ Proxmox host │
|
|
│
|
|
rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd) [CONTROL plane]
|
|
rn-platform-01 (Gateway) ──mTLS:8443──▶ rn-lab-host-01 (Lab Host Router) [DATA plane]
|
|
rn-lab-host-01 (promtail) ──3100──▶ rn-platform-01 (Loki)
|
|
rn-lab-host-01 (node-exporter/cadvisor) ◀──scrape── rn-platform-01 (Prometheus)
|
|
```
|
|
|
|
Firewall source of truth: `infra/firewall/{proxmox-host,rn-platform-01,rn-lab-host-01}.rules`
|
|
(iptables-restore format, deployed by the `firewall` Ansible role). Validate with
|
|
`infra/firewall/validate-firewall.sh`.
|
|
|
|
## What is explicitly denied
|
|
|
|
- Student Lab → another student Lab (no student-to-student traffic, ever).
|
|
- Student Lab → Proxmox management (both the host firewall and the `lab_subnet_range` DENY
|
|
rule in `rn-lab-host-01.rules`).
|
|
- Student Lab → Platform Postgres / Platform API / Gateway internals (logged DENY in
|
|
`rn-platform-01.rules` for anything from `lab_host_ip` to ports 5432/4100).
|
|
- rn-lab-host-01 → Postgres (same rule — the Lab Host itself has no legitimate reason to reach
|
|
Postgres; only the Provisioner on rn-platform-01 does, and it talks to Postgres over the
|
|
internal Docker bridge, never across the VM boundary).
|
|
- Any student egress except through the approved egress proxy (enforced by the Provisioner's
|
|
existing per-Lab `DOCKER-USER` chain manipulation from Phase 1 — unchanged by this phase).
|
|
- Direct public exposure of the Provisioner, Postgres, Docker TLS port, Proxmox UI, or Grafana
|
|
(Grafana is reachable only via Caddy's `/grafana/*` path, itself restricted to
|
|
`management_cidr` by a `caddy` matcher — see `infra/docker/platform/Caddyfile.j2`).
|
|
|
|
## Control plane vs data plane (Phase 4.1 §10)
|
|
|
|
Two separate, never-conflated mTLS connections between the two VMs, deliberately kept apart
|
|
both conceptually and operationally:
|
|
|
|
| | Control plane | Data plane |
|
|
|---|---|---|
|
|
| What | Provisioner manages Lab containers | Gateway serves student browser traffic |
|
|
| Path | Provisioner → dockerd (port 2376) | Gateway → Lab Host Router (port 8443) |
|
|
| Who initiates | Provisioner only | Gateway only |
|
|
| What it can do | Create/start/stop/remove containers, full Docker API | Reverse-proxy one HTTP/WS request to one already-running container by opaque token — nothing else |
|
|
| Certs | `provisioner-client-*` / `server-*` | `gateway-client-*` / `router-server-*` |
|
|
| Exposes Docker API to caller? | Yes (that's its job — Provisioner IS a Docker client) | **Never** — the Router never exposes Docker API, container IDs, or network info to its caller |
|
|
|
|
The student browser's traffic is proxied at the Gateway/Router layer and never touches the
|
|
Docker API at any point — the Router deliberately has no route that echoes back anything
|
|
Docker-internal (see `platform/lab-router/src/server.js`'s header comment for the full "narrow
|
|
by design" list).
|
|
|
|
## Docker TLS (Provisioner ↔ Lab Host)
|
|
|
|
Mutual TLS on port 2376, restricted at the firewall to `platform_ip` only. Application support
|
|
already existed since Phase 1 (`platform/provisioner/src/docker.js`'s `buildClient()`); this
|
|
phase added the operational half: `infra/scripts/docker-tls/generate-docker-tls.sh` (CA +
|
|
server + client cert generation), the `docker-tls` Ansible role (deploys certs, configures
|
|
`dockerd`'s `daemon.json`), and `docs/RUNBOOK_DOCKER_TLS_ROTATION.md` for renewal. The CA
|
|
private key is generated offline and never committed or deployed to either VM — only the CA
|
|
cert (to verify the peer) and each host's own leaf cert+key.
|
|
|
|
## Public ingress
|
|
|
|
One origin, Caddy, automatic HTTPS. `reverse_proxy gateway:4200` — Caddy talks only to the
|
|
Gateway; nothing else is proxied. WebSocket upgrade works automatically (Caddy's default
|
|
`reverse_proxy` behavior, no special config needed for the terminal WebSocket path). Whether to
|
|
front this with Cloudflare or expose the origin directly is left as a deployment-time decision
|
|
(both work with Caddy's ACME flow; Cloudflare adds DDoS absorption at the cost of Cloudflare
|
|
being able to see plaintext unless "Full (strict)" mode is used) — not decided here since it
|
|
doesn't affect anything else in this document.
|
|
|
|
## Gateway-to-Lab reachability across two real VMs — RESOLVED in Phase 4.1
|
|
|
|
Phase 4 flagged this as the single most important open item; Phase 4.1
|
|
(`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1) closed it. Kept here as the historical record of
|
|
what the gap was and how it was closed, since the reasoning matters for anyone extending this
|
|
later (e.g. adding a second Lab Host).
|
|
|
|
**The gap, as originally found:** the Gateway used to reach a running Lab container via
|
|
`env.internal_ip` — a Docker bridge-local address recorded by the Provisioner when it creates
|
|
the container. That only works when Gateway and the Lab share a Docker daemon, true in dev and
|
|
any single-host setup, false the moment Labs run on a genuinely separate `rn-lab-host-01`.
|
|
|
|
**The fix — a Lab Host Router (`platform/lab-router/`):** a small, narrow service that runs on
|
|
rn-lab-host-01 (host-networked, like `egress-proxy`, so it can reach whatever per-Lab bridge
|
|
network the Provisioner creates) and does exactly one thing: given an opaque routing token over
|
|
mTLS, find the ONE local container labeled with that token (`docker ps --filter
|
|
label=rootnode.routing-token=<token>` — resolved purely from local Docker state, no Postgres,
|
|
no call back to the Provisioner) and reverse-proxy the request to it, HTTP and WebSocket both.
|
|
|
|
The Gateway now decides `direct` vs `router` per `lab_host` (`lab_hosts.routing_mode` +
|
|
`router_endpoint`, `platform/provisioner/migrations/004_lab_hosts_routing.sql`) via
|
|
`platform/gateway/src/lab-endpoint.js` — a single function, `labTarget(env, token)`, that every
|
|
proxy/fetch call site in `server.js` goes through instead of building
|
|
`http://${env.internal_ip}:${env.internal_port}` itself. `dev-local` stays `direct` (T480/local
|
|
dev unaffected, zero behavior change there); a real `rn-lab-host-01` is configured `router`.
|
|
|
|
**Firewall (`infra/firewall/rn-lab-host-01.rules`):** the old speculative
|
|
`lab_gateway_port_range` allowance (which assumed option (a), publishing per-Lab ports) was
|
|
replaced with a single, narrow rule: `platform_ip` → port `{{ lab_router_port | default('8443')
|
|
}}` only. No wide port range, no per-Lab port publishing at all — the Router is the only thing
|
|
the Gateway ever talks to on rn-lab-host-01 for Lab traffic.
|
|
|
|
**mTLS:** the Gateway↔Router pair uses the SAME private CA as the Provisioner↔dockerd pair
|
|
(`infra/scripts/docker-tls/generate-docker-tls.sh`, extended in Phase 4.1 to also issue
|
|
`router-server`/`gateway-client` leaf certs), deployed to a separate directory
|
|
(`/etc/rootnode/lab-router-tls/`) by the same `docker-tls` Ansible role.
|
|
|
|
**Proven, not just designed:** `infra/scripts/simulate-two-host/` builds a real two-network
|
|
Docker simulation (Gateway genuinely isolated on its own bridge, no route to any Lab's bridge)
|
|
and runs 13 real assertions against it — Gateway cannot reach a Lab's Docker-internal IP
|
|
directly, but reaches it correctly through the Router (HTTP and a real WebSocket
|
|
`kubectl get nodes`), Alice/Bob isolation holds, garbage/destroyed tokens are rejected, and the
|
|
Router itself refuses any connection without the Gateway's client certificate. See that
|
|
directory's README for exactly what is and isn't proven by a same-daemon simulation versus real
|
|
separate hardware.
|
|
|
|
## Cookie / auth production settings
|
|
|
|
`infra/environments/production/*.env.template`: `COOKIE_SECURE=1`, `TRUST_PROXY=1`. Dev's
|
|
insecure defaults (`COOKIE_SECURE` unset, no proxy trust) are never inherited — production env
|
|
files are separate files, not a dev file with overrides layered on.
|