docs(phase4): production infrastructure documentation + runbooks
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs (PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR, OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST, and 12 runbooks (first deployment, normal deploy, rollback, Postgres restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure). PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine architectural gap this phase surfaced: the Gateway's existing env.internal_ip mechanism for reaching a Lab container assumes Gateway and Lab share a Docker daemon, which breaks once Labs run on a genuinely separate rn-lab-host-01 VM. The network-policy half is built (a firewall allowance for Gateway->Lab-Host on a published port range); the application half (Provisioner/Gateway address handling) is an explicit, reviewed, NOT-YET-BUILT follow-up flagged there rather than silently decided. Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and platform/README.md with Phase 4 pointers, consistent with how every prior phase updated these. Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
This commit is contained in:
@@ -0,0 +1,93 @@
|
||||
# Production Network & Security
|
||||
|
||||
The Platform VM / Lab Host VM boundary is the core production control. This document is where
|
||||
every network decision that touches it gets written down. Nothing here weakens it.
|
||||
|
||||
## Topology
|
||||
|
||||
```
|
||||
Internet ──HTTPS(443)──▶ Caddy (rn-platform-01) ──▶ Gateway (internal bridge)
|
||||
│
|
||||
management_cidr ──SSH──▶ rn-platform-01 │
|
||||
management_cidr ──SSH──▶ rn-lab-host-01 │
|
||||
proxmox_mgmt_cidr ──22/8006──▶ Proxmox host │
|
||||
│
|
||||
rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd)
|
||||
rn-lab-host-01 (promtail) ──3100──▶ rn-platform-01 (Loki)
|
||||
rn-lab-host-01 (node-exporter/cadvisor) ◀──scrape── rn-platform-01 (Prometheus)
|
||||
```
|
||||
|
||||
Firewall source of truth: `infra/firewall/{proxmox-host,rn-platform-01,rn-lab-host-01}.rules`
|
||||
(iptables-restore format, deployed by the `firewall` Ansible role). Validate with
|
||||
`infra/firewall/validate-firewall.sh`.
|
||||
|
||||
## What is explicitly denied
|
||||
|
||||
- Student Lab → another student Lab (no student-to-student traffic, ever).
|
||||
- Student Lab → Proxmox management (both the host firewall and the `lab_subnet_range` DENY
|
||||
rule in `rn-lab-host-01.rules`).
|
||||
- Student Lab → Platform Postgres / Platform API / Gateway internals (logged DENY in
|
||||
`rn-platform-01.rules` for anything from `lab_host_ip` to ports 5432/4100).
|
||||
- rn-lab-host-01 → Postgres (same rule — the Lab Host itself has no legitimate reason to reach
|
||||
Postgres; only the Provisioner on rn-platform-01 does, and it talks to Postgres over the
|
||||
internal Docker bridge, never across the VM boundary).
|
||||
- Any student egress except through the approved egress proxy (enforced by the Provisioner's
|
||||
existing per-Lab `DOCKER-USER` chain manipulation from Phase 1 — unchanged by this phase).
|
||||
- Direct public exposure of the Provisioner, Postgres, Docker TLS port, Proxmox UI, or Grafana
|
||||
(Grafana is reachable only via Caddy's `/grafana/*` path, itself restricted to
|
||||
`management_cidr` by a `caddy` matcher — see `infra/docker/platform/Caddyfile.j2`).
|
||||
|
||||
## Docker TLS (Provisioner ↔ Lab Host)
|
||||
|
||||
Mutual TLS on port 2376, restricted at the firewall to `platform_ip` only. Application support
|
||||
already existed since Phase 1 (`platform/provisioner/src/docker.js`'s `buildClient()`); this
|
||||
phase added the operational half: `infra/scripts/docker-tls/generate-docker-tls.sh` (CA +
|
||||
server + client cert generation), the `docker-tls` Ansible role (deploys certs, configures
|
||||
`dockerd`'s `daemon.json`), and `docs/RUNBOOK_DOCKER_TLS_ROTATION.md` for renewal. The CA
|
||||
private key is generated offline and never committed or deployed to either VM — only the CA
|
||||
cert (to verify the peer) and each host's own leaf cert+key.
|
||||
|
||||
## Public ingress
|
||||
|
||||
One origin, Caddy, automatic HTTPS. `reverse_proxy gateway:4200` — Caddy talks only to the
|
||||
Gateway; nothing else is proxied. WebSocket upgrade works automatically (Caddy's default
|
||||
`reverse_proxy` behavior, no special config needed for the terminal WebSocket path). Whether to
|
||||
front this with Cloudflare or expose the origin directly is left as a deployment-time decision
|
||||
(both work with Caddy's ACME flow; Cloudflare adds DDoS absorption at the cost of Cloudflare
|
||||
being able to see plaintext unless "Full (strict)" mode is used) — not decided here since it
|
||||
doesn't affect anything else in this document.
|
||||
|
||||
## Known gap: Gateway-to-Lab reachability across two real VMs
|
||||
|
||||
**This is the most important item in this document.**
|
||||
|
||||
The Gateway currently reaches a running Lab container via `env.internal_ip` — a Docker
|
||||
bridge-local address recorded by the Provisioner when it creates the container
|
||||
(`platform/gateway/src/server.js`, `platform/provisioner`, built in Phase 2A). This works when
|
||||
Gateway and the Lab share a Docker daemon, which has been true in every setup so far, including
|
||||
dev and any single-host deployment. **It stops being true once Labs run on a genuinely separate
|
||||
`rn-lab-host-01`** — a Docker bridge-internal IP on one host is not routable from a different
|
||||
host by default.
|
||||
|
||||
What's built now (network-policy half only): a firewall allowance in
|
||||
`infra/firewall/rn-lab-host-01.rules` for `platform_ip` → `lab_gateway_port_range`
|
||||
(`30000:32767` by default, configurable), so that once Lab containers publish ports on the Lab
|
||||
Host's own interface, the Gateway VM is permitted to reach them.
|
||||
|
||||
What's **not yet built** (application half — explicitly flagged, not silently decided here):
|
||||
the Provisioner needs to either (a) publish each Lab container's relevant port(s) on the Lab
|
||||
Host's interface and record a reachable `external_ip`/`external_port` alongside (or instead of)
|
||||
`internal_ip`/`internal_port`, or (b) route Gateway→Lab traffic through a proxy/tunnel that
|
||||
lives on the Lab Host and forwards to the container's internal address. Option (a) is simpler
|
||||
and consistent with how the egress proxy and cadvisor are already handled in this phase's
|
||||
compose files; option (b) avoids publishing a wide port range but adds a moving part. This is a
|
||||
small, reviewable, application-level change — not a network-policy change — and should land as
|
||||
its own reviewed commit before real two-VM production traffic depends on it. Until it lands,
|
||||
Gateway↔Lab terminal/scenario traffic will not work correctly across two separate VMs, even
|
||||
though every other production networking, firewall, and TLS boundary is in place and correct.
|
||||
|
||||
## Cookie / auth production settings
|
||||
|
||||
`infra/environments/production/*.env.template`: `COOKIE_SECURE=1`, `TRUST_PROXY=1`. Dev's
|
||||
insecure defaults (`COOKIE_SECURE` unset, no proxy trust) are never inherited — production env
|
||||
files are separate files, not a dev file with overrides layered on.
|
||||
Reference in New Issue
Block a user