# Phase 4 — Production Infrastructure Package **Status: PHASE 4 BUILD COMPLETE. PRODUCTION DEPLOYMENT PENDING HARDWARE.** The dedicated Hetzner server this is designed for hasn't been purchased yet. Everything in this document and the `infra/` tree it describes is buildable, testable (against local Docker/Postgres/disposable certs), and reviewable **now** — none of it requires the real hardware to exist. What genuinely can't happen until the hardware exists is called out explicitly throughout, never silently assumed to already work. This document is the map. Detail docs: `docs/PRODUCTION_DEPLOYMENT.md` (step-by-step), `docs/PRODUCTION_NETWORK_SECURITY.md` (network/firewall/TLS design), `docs/BACKUP_DR.md` (backup/restore/disaster recovery), `docs/OBSERVABILITY.md` (metrics/logs/alerts), `docs/CAPACITY_PLANNING.md` (sizing model), and the `docs/RUNBOOK_*.md` files (14 topics, command-oriented). ## 1. What was built ``` infra/ ├── proxmox/ manual bare-metal install docs + cloud-init template definition ├── terraform/ VM creation (rn-platform-01, rn-lab-host-01) — real `tofu validate` pass ├── ansible/ all host configuration — real `ansible-playbook --syntax-check` pass │ ├── roles/ common, firewall, docker-tls, platform, lab-host, monitoring-agent, backup │ ├── group_vars/ production.yml, staging.yml, vault template │ └── site.yml top-level playbook, correct role ordering ├── docker/ production Compose files, Caddyfile, Postgres tuning — real `docker │ compose config` + `caddy validate` pass ├── firewall/ iptables-restore rulesets for all 3 hosts — structural lint pass ├── monitoring/ Prometheus/Alertmanager/Loki/Promtail/Grafana config — real `promtool` │ + `amtool` validation pass ├── backup/ Postgres backup/restore/verify scripts — REAL end-to-end test against │ disposable local Postgres, including the actual production migrations ├── scripts/ deploy/rollback/smoke-test/CI/loadtest/capacity-model tooling ├── ci/ Gitea Actions workflow (build/test/stage automatic, prod manual) └── environments/ dev/staging/production config templates, no secrets committed ``` 18 new docs (this file + 5 detail docs + 12 runbooks), 4 updated (`LAB_ARCHITECTURE.md`, `ENGINEERING_SOP.md`, `README.md`, `platform/README.md`). ## 2. Architecture (unchanged from the design, now codified) ``` Hetzner Dedicated Server └── Proxmox VE ├── rn-platform-01 (frontend, Platform API, Gateway, Provisioner, Postgres, │ Prometheus/Alertmanager/Grafana/Loki, backup cron, Caddy) └── rn-lab-host-01 (Docker Engine w/ TLS, egress proxy, disposable student Labs, node-exporter/cadvisor/promtail) ``` The Platform VM / Lab Host VM security boundary from `docs/LAB_ARCHITECTURE.md` is the one thing this phase was explicitly told never to weaken — it's codified, not just described, in `infra/firewall/` and `infra/ansible/roles/docker-tls/`. ## 3. VM specifications | | rn-platform-01 | rn-lab-host-01 | |---|---|---| | Baseline (default in `infra/terraform/variables.tf`) | 4 cores, 16 GB RAM, 100 GB disk | 24 cores, 96 GB RAM, 500 GB disk | | Recommended (once real host is ~16c/32t/128GB) | same — Postgres/API/Gateway/Provisioner don't need much | majority of remaining host resources | | Max-growth | scale RAM with Postgres `shared_buffers` if instructor/cohort count grows a lot | scale with measured concurrent-Lab capacity (`docs/CAPACITY_PLANNING.md`) | | Configurable via | `terraform.tfvars`' `platform_vm`/`lab_host_vm` objects | same | NUMA/CPU pinning: not configured — not useful at this scale (a handful of VMs on one physical host, not a dozen contending for cache locality). Revisit only if real load testing shows a reason to. ## 4. Storage decision framework Not a final answer — `infra/environments/production/provisioner-config/README.md` and this section exist so the final ZFS layout is a deployment-time decision made against real disks, not guessed now. Candidate layouts (`infra/terraform/variables.tf`'s `vm_disk_datastore` names whichever one gets chosen): - **4× NVMe**: mirrored vdevs (RAID10-equivalent) — two mirror pairs, one for Postgres/platform data (I/O-sensitive, small), one for Lab Host disposable data (throughput-sensitive, large, never needs redundancy since nothing on it is precious). - **2× SSD/NVMe**: one mirrored vdev, logical dataset separation (ZFS datasets, not separate pools) between platform (`{{ rootnode_data_root }}/postgres`, `/backups`) and Lab Host data. - **Mixed SSD/HDD**: only if the purchased server's actual disks require it — HDD for disposable Lab scratch space is acceptable (it's disposable), never for Postgres. **Never single-disk for Postgres data**, in any layout — a mirror is the floor, not a nice-to-have. Pre-deployment disk inspection checklist (model, size, SMART, power-on hours, wear/endurance, sector errors, temperature, firmware): `docs/RUNBOOK_DISK_FAILURE.md`'s inspection section — run once, against the real drives, before committing to a layout. ## 5-8. Network, firewall, Docker TLS, public ingress Full detail: `docs/PRODUCTION_NETWORK_SECURITY.md`. Summary: two private VM networks, Docker TLS (mutual, already application-supported since Phase 1 — this phase only needed the certificate generation/deployment automation, not new app code), a single public origin (Caddy, automatic HTTPS) proxying only to the Gateway, and one genuinely new architectural item this phase surfaced and addressed at the network-policy layer: **Gateway-to-Lab reachability across two real VMs** — see that doc's dedicated section for what's built vs. what still needs a small, reviewed Provisioner/Gateway code change before real go-live. ## 9-10. Production auth/cookie config, secrets `infra/environments/production/*.env.template` set `COOKIE_SECURE=1`/`TRUST_PROXY=1` explicitly (dev's unsafe defaults are never inherited). Secrets: Ansible Vault (already bundled with Ansible — no new tool to install/trust for this project's secret surface), template at `infra/ansible/group_vars/production.vault.yml.template`. Nothing is committed — `infra/.gitignore` covers `*.env`, TLS private key material, vault plaintext, Terraform state. ## 11-13. Postgres, backup, DR Full detail: `docs/BACKUP_DR.md`. Summary: conservative tuning baseline (not hand-tuned before real RAM is known), daily encrypted backups with an off-host adapter point (Storage Box/S3/any SSH target), and a **real, tested** backup→encrypt→restore→verify cycle (`infra/backup/test-backup-restore.sh`) — run against disposable local Postgres with the actual production migrations applied, not a mock. ## 14-17. Observability, logging, alerting, uptime Full detail: `docs/OBSERVABILITY.md`. Prometheus + Grafana + Loki/Promtail (the user's stated Prometheus/Grafana preference, honored) + Alertmanager, all self-hosted, no paid dependency. Deliberately does not duplicate the application's own instructor learning analytics (Phase 3B) — operational observability and learning analytics stay separate concerns with separate audiences, per the explicit instruction. Application-level metrics (active students/Labs, provisioning failures, Gateway error rate) are a documented, **not-yet-built** follow-up — see that doc's honest accounting of what's real today (host/container/Postgres metrics, HTTP healthz probes) vs. what needs app instrumentation later. ## 18-19. CI/CD, image strategy `infra/ci/gitea-workflows/build-and-test.yml` — automatic test/build/tag/staging-deploy, **manual** production promotion (`infra/scripts/deploy-production.sh`), matching the stated preference. SHA-tagged images always, semantic tags on deliberate releases, never `:latest` in staging/production. Full detail: `infra/docker/IMAGE_STRATEGY.md`. ## 20-21. Ansible, Terraform `infra/ansible/site.yml` — idempotent roles, doesn't assume the VMs already exist (Terraform creates them first). Real `ansible-playbook --syntax-check` pass. `infra/terraform/` — real `tofu validate` pass against the actual `bpg/proxmox` provider schema. Neither installs Proxmox itself or builds the base VM template — both stay deliberate manual, documented, one-time steps (`infra/proxmox/README.md`), per the explicit "don't over-automate the fragile one-time step" instruction. ## 22-23. Environment config, staging `infra/environments/{dev,staging,production}/` — explicit separation, production never inherits dev defaults. Staging is designed to run in the homelab or a resource-constrained slice of the real server later — no second dedicated server required. ## 24-25. Load testing, capacity model `infra/scripts/loadtest/` (real Prepare-Labs-driven boot test + `docker stats` sampling) and `infra/scripts/capacity-model.js` (a real, working worksheet — tested against example inputs, produces CPU-vs-memory-bound recommended/warning/hard-admission-control thresholds). No concurrency number is hard-coded as a marketing figure; the example config in `infra/environments/production/capacity.yml.example` is explicitly flagged as placeholder data. ## 26-27. Validation checklist, runbooks `docs/PRODUCTION_VALIDATION_CHECKLIST.md` — one master checklist spanning hardware through academy-workflow verification. 12 runbooks (`docs/RUNBOOK_*.md`) — first deployment, normal deploy, rollback, Postgres restore, VM rebuild (both hosts), Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure. ## What this phase deliberately did NOT touch Learning system design, curriculum content, RBAC model, session architecture — all Phase 3A/3B, unchanged. No parent portal, billing, held scenarios, or production OIDC provider integration. RootNode production infrastructure is dedicated to RootNode Academy — never folded into existing Synthos production servers, and canonical source stays Gitea, never GitHub `kubekosh`. ## Known gap surfaced during this phase (not silently papered over) — RESOLVED in Phase 4.1 Building the real production Compose/network config surfaced that the Gateway's existing mechanism for reaching a Lab container (`env.internal_ip`, a Docker-bridge-local address) only works when Gateway and Lab share a Docker daemon — true in every dev/single-host setup so far, **not** true once Labs run on a genuinely separate `rn-lab-host-01`. At the time this phase closed, the network-policy half was built but the application half was flagged as an explicit, not-yet-built follow-up rather than guessed at mid-infra-phase. **Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1) built and proved the application half** — a Lab Host Router (`platform/lab-router/`) plus a `labTarget()` routing abstraction in the Gateway, real-tested against a genuine two-network Docker simulation (`infra/scripts/simulate-two-host/`), not just designed. See `docs/PRODUCTION_NETWORK_SECURITY.md`'s "Gateway-to-Lab reachability" section for the full resolution. This item no longer blocks real two-VM go-live.