rootnode-academy/docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md

# Phase 4 — Production Infrastructure Package

**Status: PHASE 4 BUILD COMPLETE. PRODUCTION DEPLOYMENT PENDING HARDWARE.**

The dedicated Hetzner server this is designed for hasn't been purchased yet. Everything in this
document and the `infra/` tree it describes is buildable, testable (against local
Docker/Postgres/disposable certs), and reviewable **now** — none of it requires the real
hardware to exist. What genuinely can't happen until the hardware exists is called out
explicitly throughout, never silently assumed to already work.

This document is the map. Detail docs: `docs/PRODUCTION_DEPLOYMENT.md` (step-by-step),
`docs/PRODUCTION_NETWORK_SECURITY.md` (network/firewall/TLS design), `docs/BACKUP_DR.md`
(backup/restore/disaster recovery), `docs/OBSERVABILITY.md` (metrics/logs/alerts),
`docs/CAPACITY_PLANNING.md` (sizing model), and the `docs/RUNBOOK_*.md` files (14 topics,
command-oriented).

## 1. What was built

```
infra/
├── proxmox/          manual bare-metal install docs + cloud-init template definition
├── terraform/         VM creation (rn-platform-01, rn-lab-host-01) — real `tofu validate` pass
├── ansible/            all host configuration — real `ansible-playbook --syntax-check` pass
│   ├── roles/          common, firewall, docker-tls, platform, lab-host, monitoring-agent, backup
│   ├── group_vars/     production.yml, staging.yml, vault template
│   └── site.yml        top-level playbook, correct role ordering
├── docker/            production Compose files, Caddyfile, Postgres tuning — real `docker
│                       compose config` + `caddy validate` pass
├── firewall/           iptables-restore rulesets for all 3 hosts — structural lint pass
├── monitoring/          Prometheus/Alertmanager/Loki/Promtail/Grafana config — real `promtool`
│                        + `amtool` validation pass
├── backup/              Postgres backup/restore/verify scripts — REAL end-to-end test against
│                        disposable local Postgres, including the actual production migrations
├── scripts/             deploy/rollback/smoke-test/CI/loadtest/capacity-model tooling
├── ci/                  Gitea Actions workflow (build/test/stage automatic, prod manual)
└── environments/        dev/staging/production config templates, no secrets committed
```

18 new docs (this file + 5 detail docs + 12 runbooks), 4 updated (`LAB_ARCHITECTURE.md`,
`ENGINEERING_SOP.md`, `README.md`, `platform/README.md`).

## 2. Architecture (unchanged from the design, now codified)

```
Hetzner Dedicated Server
└── Proxmox VE
    ├── rn-platform-01   (frontend, Platform API, Gateway, Provisioner, Postgres,
    │                      Prometheus/Alertmanager/Grafana/Loki, backup cron, Caddy)
    └── rn-lab-host-01   (Docker Engine w/ TLS, egress proxy, disposable student Labs,
                           node-exporter/cadvisor/promtail)
```

The Platform VM / Lab Host VM security boundary from `docs/LAB_ARCHITECTURE.md` is the one
thing this phase was explicitly told never to weaken — it's codified, not just described, in
`infra/firewall/` and `infra/ansible/roles/docker-tls/`.

## 3. VM specifications

| | rn-platform-01 | rn-lab-host-01 |
|---|---|---|
| Baseline (default in `infra/terraform/variables.tf`) | 4 cores, 16 GB RAM, 100 GB disk | 24 cores, 96 GB RAM, 500 GB disk |
| Recommended (once real host is ~16c/32t/128GB) | same — Postgres/API/Gateway/Provisioner don't need much | majority of remaining host resources |
| Max-growth | scale RAM with Postgres `shared_buffers` if instructor/cohort count grows a lot | scale with measured concurrent-Lab capacity (`docs/CAPACITY_PLANNING.md`) |
| Configurable via | `terraform.tfvars`' `platform_vm`/`lab_host_vm` objects | same |

NUMA/CPU pinning: not configured — not useful at this scale (a handful of VMs on one physical
host, not a dozen contending for cache locality). Revisit only if real load testing shows a
reason to.

## 4. Storage decision framework

Not a final answer — `infra/environments/production/provisioner-config/README.md` and this
section exist so the final ZFS layout is a deployment-time decision made against real disks,
not guessed now. Candidate layouts (`infra/terraform/variables.tf`'s `vm_disk_datastore` names
whichever one gets chosen):

- **4× NVMe**: mirrored vdevs (RAID10-equivalent) — two mirror pairs, one for
  Postgres/platform data (I/O-sensitive, small), one for Lab Host disposable data
  (throughput-sensitive, large, never needs redundancy since nothing on it is precious).
- **2× SSD/NVMe**: one mirrored vdev, logical dataset separation (ZFS datasets, not separate
  pools) between platform (`{{ rootnode_data_root }}/postgres`, `/backups`) and Lab Host data.
- **Mixed SSD/HDD**: only if the purchased server's actual disks require it — HDD for
  disposable Lab scratch space is acceptable (it's disposable), never for Postgres.

**Never single-disk for Postgres data**, in any layout — a mirror is the floor, not a nice-to-have.

Pre-deployment disk inspection checklist (model, size, SMART, power-on hours, wear/endurance,
sector errors, temperature, firmware): `docs/RUNBOOK_DISK_FAILURE.md`'s inspection section —
run once, against the real drives, before committing to a layout.

## 5-8. Network, firewall, Docker TLS, public ingress

Full detail: `docs/PRODUCTION_NETWORK_SECURITY.md`. Summary: two private VM networks, Docker TLS
(mutual, already application-supported since Phase 1 — this phase only needed the certificate
generation/deployment automation, not new app code), a single public origin (Caddy, automatic
HTTPS) proxying only to the Gateway, and one genuinely new architectural item this phase
surfaced and addressed at the network-policy layer: **Gateway-to-Lab reachability across two
real VMs** — see that doc's dedicated section for what's built vs. what still needs a small,
reviewed Provisioner/Gateway code change before real go-live.

## 9-10. Production auth/cookie config, secrets

`infra/environments/production/*.env.template` set `COOKIE_SECURE=1`/`TRUST_PROXY=1` explicitly
(dev's unsafe defaults are never inherited). Secrets: Ansible Vault (already bundled with
Ansible — no new tool to install/trust for this project's secret surface), template at
`infra/ansible/group_vars/production.vault.yml.template`. Nothing is committed —
`infra/.gitignore` covers `*.env`, TLS private key material, vault plaintext, Terraform state.

## 11-13. Postgres, backup, DR

Full detail: `docs/BACKUP_DR.md`. Summary: conservative tuning baseline (not hand-tuned before
real RAM is known), daily encrypted backups with an off-host adapter point (Storage Box/S3/any
SSH target), and a **real, tested** backup→encrypt→restore→verify cycle
(`infra/backup/test-backup-restore.sh`) — run against disposable local Postgres with the actual
production migrations applied, not a mock.

## 14-17. Observability, logging, alerting, uptime

Full detail: `docs/OBSERVABILITY.md`. Prometheus + Grafana + Loki/Promtail (the user's stated
Prometheus/Grafana preference, honored) + Alertmanager, all self-hosted, no paid dependency.
Deliberately does not duplicate the application's own instructor learning analytics
(Phase 3B) — operational observability and learning analytics stay separate concerns with
separate audiences, per the explicit instruction. Application-level metrics (active
students/Labs, provisioning failures, Gateway error rate) are a documented, **not-yet-built**
follow-up — see that doc's honest accounting of what's real today (host/container/Postgres
metrics, HTTP healthz probes) vs. what needs app instrumentation later.

## 18-19. CI/CD, image strategy

`infra/ci/gitea-workflows/build-and-test.yml` — automatic test/build/tag/staging-deploy,
**manual** production promotion (`infra/scripts/deploy-production.sh`), matching the stated
preference. SHA-tagged images always, semantic tags on deliberate releases, never `:latest` in
staging/production. Full detail: `infra/docker/IMAGE_STRATEGY.md`.

## 20-21. Ansible, Terraform

`infra/ansible/site.yml` — idempotent roles, doesn't assume the VMs already exist (Terraform
creates them first). Real `ansible-playbook --syntax-check` pass. `infra/terraform/` — real
`tofu validate` pass against the actual `bpg/proxmox` provider schema. Neither installs Proxmox
itself or builds the base VM template — both stay deliberate manual, documented, one-time steps
(`infra/proxmox/README.md`), per the explicit "don't over-automate the fragile one-time step"
instruction.

## 22-23. Environment config, staging

`infra/environments/{dev,staging,production}/` — explicit separation, production never
inherits dev defaults. Staging is designed to run in the homelab or a resource-constrained
slice of the real server later — no second dedicated server required.

## 24-25. Load testing, capacity model

`infra/scripts/loadtest/` (real Prepare-Labs-driven boot test + `docker stats` sampling) and
`infra/scripts/capacity-model.js` (a real, working worksheet — tested against example inputs,
produces CPU-vs-memory-bound recommended/warning/hard-admission-control thresholds). No
concurrency number is hard-coded as a marketing figure; the example config in
`infra/environments/production/capacity.yml.example` is explicitly flagged as placeholder data.

## 26-27. Validation checklist, runbooks

`docs/PRODUCTION_VALIDATION_CHECKLIST.md` — one master checklist spanning hardware through
academy-workflow verification. 12 runbooks (`docs/RUNBOOK_*.md`) — first deployment, normal
deploy, rollback, Postgres restore, VM rebuild (both hosts), Docker TLS rotation, TLS renewal,
owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host
resource pressure.

## What this phase deliberately did NOT touch

Learning system design, curriculum content, RBAC model, session architecture — all Phase 3A/3B,
unchanged. No parent portal, billing, held scenarios, or production OIDC provider integration.
RootNode production infrastructure is dedicated to RootNode Academy — never folded into
existing Synthos production servers, and canonical source stays Gitea, never GitHub `kubekosh`.

## Known gap surfaced during this phase (not silently papered over) — RESOLVED in Phase 4.1

Building the real production Compose/network config surfaced that the Gateway's existing
mechanism for reaching a Lab container (`env.internal_ip`, a Docker-bridge-local address) only
works when Gateway and Lab share a Docker daemon — true in every dev/single-host setup so far,
**not** true once Labs run on a genuinely separate `rn-lab-host-01`. At the time this phase
closed, the network-policy half was built but the application half was flagged as an explicit,
not-yet-built follow-up rather than guessed at mid-infra-phase.

**Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1) built and proved the application
half** — a Lab Host Router (`platform/lab-router/`) plus a `labTarget()` routing abstraction in
the Gateway, real-tested against a genuine two-network Docker simulation
(`infra/scripts/simulate-two-host/`), not just designed. See
`docs/PRODUCTION_NETWORK_SECURITY.md`'s "Gateway-to-Lab reachability" section for the full
resolution. This item no longer blocks real two-VM go-live.