rootnode-academy/docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md

Phase 4 — Production Infrastructure Package

Status: PHASE 4 BUILD COMPLETE. PRODUCTION DEPLOYMENT PENDING HARDWARE.

The dedicated Hetzner server this is designed for hasn't been purchased yet. Everything in this document and the infra/ tree it describes is buildable, testable (against local Docker/Postgres/disposable certs), and reviewable now — none of it requires the real hardware to exist. What genuinely can't happen until the hardware exists is called out explicitly throughout, never silently assumed to already work.

This document is the map. Detail docs: docs/PRODUCTION_DEPLOYMENT.md (step-by-step), docs/PRODUCTION_NETWORK_SECURITY.md (network/firewall/TLS design), docs/BACKUP_DR.md (backup/restore/disaster recovery), docs/OBSERVABILITY.md (metrics/logs/alerts), docs/CAPACITY_PLANNING.md (sizing model), and the docs/RUNBOOK_*.md files (14 topics, command-oriented).

1. What was built

infra/
├── proxmox/          manual bare-metal install docs + cloud-init template definition
├── terraform/         VM creation (rn-platform-01, rn-lab-host-01) — real `tofu validate` pass
├── ansible/            all host configuration — real `ansible-playbook --syntax-check` pass
│   ├── roles/          common, firewall, docker-tls, platform, lab-host, monitoring-agent, backup
│   ├── group_vars/     production.yml, staging.yml, vault template
│   └── site.yml        top-level playbook, correct role ordering
├── docker/            production Compose files, Caddyfile, Postgres tuning — real `docker
│                       compose config` + `caddy validate` pass
├── firewall/           iptables-restore rulesets for all 3 hosts — structural lint pass
├── monitoring/          Prometheus/Alertmanager/Loki/Promtail/Grafana config — real `promtool`
│                        + `amtool` validation pass
├── backup/              Postgres backup/restore/verify scripts — REAL end-to-end test against
│                        disposable local Postgres, including the actual production migrations
├── scripts/             deploy/rollback/smoke-test/CI/loadtest/capacity-model tooling
├── ci/                  Gitea Actions workflow (build/test/stage automatic, prod manual)
└── environments/        dev/staging/production config templates, no secrets committed

18 new docs (this file + 5 detail docs + 12 runbooks), 4 updated (LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, platform/README.md).

2. Architecture (unchanged from the design, now codified)

Hetzner Dedicated Server
└── Proxmox VE
    ├── rn-platform-01   (frontend, Platform API, Gateway, Provisioner, Postgres,
    │                      Prometheus/Alertmanager/Grafana/Loki, backup cron, Caddy)
    └── rn-lab-host-01   (Docker Engine w/ TLS, egress proxy, disposable student Labs,
                           node-exporter/cadvisor/promtail)

The Platform VM / Lab Host VM security boundary from docs/LAB_ARCHITECTURE.md is the one thing this phase was explicitly told never to weaken — it's codified, not just described, in infra/firewall/ and infra/ansible/roles/docker-tls/.

3. VM specifications

	rn-platform-01	rn-lab-host-01
Baseline (default in infra/terraform/variables.tf)	4 cores, 16 GB RAM, 100 GB disk	24 cores, 96 GB RAM, 500 GB disk
Recommended (once real host is ~16c/32t/128GB)	same — Postgres/API/Gateway/Provisioner don't need much	majority of remaining host resources
Max-growth	scale RAM with Postgres shared_buffers if instructor/cohort count grows a lot	scale with measured concurrent-Lab capacity (docs/CAPACITY_PLANNING.md)
Configurable via	terraform.tfvars' platform_vm/lab_host_vm objects	same

NUMA/CPU pinning: not configured — not useful at this scale (a handful of VMs on one physical host, not a dozen contending for cache locality). Revisit only if real load testing shows a reason to.

4. Storage decision framework

Not a final answer — infra/environments/production/provisioner-config/README.md and this section exist so the final ZFS layout is a deployment-time decision made against real disks, not guessed now. Candidate layouts (infra/terraform/variables.tf's vm_disk_datastore names whichever one gets chosen):

• 4× NVMe: mirrored vdevs (RAID10-equivalent) — two mirror pairs, one for Postgres/platform data (I/O-sensitive, small), one for Lab Host disposable data (throughput-sensitive, large, never needs redundancy since nothing on it is precious).
• 2× SSD/NVMe: one mirrored vdev, logical dataset separation (ZFS datasets, not separate pools) between platform ({{ rootnode_data_root }}/postgres, /backups) and Lab Host data.
• Mixed SSD/HDD: only if the purchased server's actual disks require it — HDD for disposable Lab scratch space is acceptable (it's disposable), never for Postgres.

Never single-disk for Postgres data, in any layout — a mirror is the floor, not a nice-to-have.

Pre-deployment disk inspection checklist (model, size, SMART, power-on hours, wear/endurance, sector errors, temperature, firmware): docs/RUNBOOK_DISK_FAILURE.md's inspection section — run once, against the real drives, before committing to a layout.

5-8. Network, firewall, Docker TLS, public ingress

Full detail: docs/PRODUCTION_NETWORK_SECURITY.md. Summary: two private VM networks, Docker TLS (mutual, already application-supported since Phase 1 — this phase only needed the certificate generation/deployment automation, not new app code), a single public origin (Caddy, automatic HTTPS) proxying only to the Gateway, and one genuinely new architectural item this phase surfaced and addressed at the network-policy layer: Gateway-to-Lab reachability across two real VMs — see that doc's dedicated section for what's built vs. what still needs a small, reviewed Provisioner/Gateway code change before real go-live.

9-10. Production auth/cookie config, secrets

infra/environments/production/*.env.template set COOKIE_SECURE=1/TRUST_PROXY=1 explicitly (dev's unsafe defaults are never inherited). Secrets: Ansible Vault (already bundled with Ansible — no new tool to install/trust for this project's secret surface), template at infra/ansible/group_vars/production.vault.yml.template. Nothing is committed — infra/.gitignore covers *.env, TLS private key material, vault plaintext, Terraform state.

11-13. Postgres, backup, DR

Full detail: docs/BACKUP_DR.md. Summary: conservative tuning baseline (not hand-tuned before real RAM is known), daily encrypted backups with an off-host adapter point (Storage Box/S3/any SSH target), and a real, tested backup→encrypt→restore→verify cycle (infra/backup/test-backup-restore.sh) — run against disposable local Postgres with the actual production migrations applied, not a mock.

14-17. Observability, logging, alerting, uptime

Full detail: docs/OBSERVABILITY.md. Prometheus + Grafana + Loki/Promtail (the user's stated Prometheus/Grafana preference, honored) + Alertmanager, all self-hosted, no paid dependency. Deliberately does not duplicate the application's own instructor learning analytics (Phase 3B) — operational observability and learning analytics stay separate concerns with separate audiences, per the explicit instruction. Application-level metrics (active students/Labs, provisioning failures, Gateway error rate) are a documented, not-yet-built follow-up — see that doc's honest accounting of what's real today (host/container/Postgres metrics, HTTP healthz probes) vs. what needs app instrumentation later.

18-19. CI/CD, image strategy

infra/ci/gitea-workflows/build-and-test.yml — automatic test/build/tag/staging-deploy, manual production promotion (infra/scripts/deploy-production.sh), matching the stated preference. SHA-tagged images always, semantic tags on deliberate releases, never :latest in staging/production. Full detail: infra/docker/IMAGE_STRATEGY.md.

20-21. Ansible, Terraform

infra/ansible/site.yml — idempotent roles, doesn't assume the VMs already exist (Terraform creates them first). Real ansible-playbook --syntax-check pass. infra/terraform/ — real tofu validate pass against the actual bpg/proxmox provider schema. Neither installs Proxmox itself or builds the base VM template — both stay deliberate manual, documented, one-time steps (infra/proxmox/README.md), per the explicit "don't over-automate the fragile one-time step" instruction.

22-23. Environment config, staging

infra/environments/{dev,staging,production}/ — explicit separation, production never inherits dev defaults. Staging is designed to run in the homelab or a resource-constrained slice of the real server later — no second dedicated server required.

24-25. Load testing, capacity model

infra/scripts/loadtest/ (real Prepare-Labs-driven boot test + docker stats sampling) and infra/scripts/capacity-model.js (a real, working worksheet — tested against example inputs, produces CPU-vs-memory-bound recommended/warning/hard-admission-control thresholds). No concurrency number is hard-coded as a marketing figure; the example config in infra/environments/production/capacity.yml.example is explicitly flagged as placeholder data.

26-27. Validation checklist, runbooks

docs/PRODUCTION_VALIDATION_CHECKLIST.md — one master checklist spanning hardware through academy-workflow verification. 12 runbooks (docs/RUNBOOK_*.md) — first deployment, normal deploy, rollback, Postgres restore, VM rebuild (both hosts), Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure.

What this phase deliberately did NOT touch

Learning system design, curriculum content, RBAC model, session architecture — all Phase 3A/3B, unchanged. No parent portal, billing, held scenarios, or production OIDC provider integration. RootNode production infrastructure is dedicated to RootNode Academy — never folded into existing Synthos production servers, and canonical source stays Gitea, never GitHub kubekosh.

Known gap surfaced during this phase (not silently papered over)

Building the real production Compose/network config surfaced that the Gateway's existing mechanism for reaching a Lab container (env.internal_ip, a Docker-bridge-local address) only works when Gateway and Lab share a Docker daemon — true in every dev/single-host setup so far, not true once Labs run on a genuinely separate rn-lab-host-01. The network-policy half is built (a firewall allowance for Gateway→Lab-Host on a published port range, infra/firewall/rn-lab-host-01.rules); the application half (the Provisioner recording a reachable external address for each Lab, and the Gateway using it) is an explicit, reviewed, NOT-YET-BUILT follow-up — see docs/PRODUCTION_NETWORK_SECURITY.md for the two candidate designs and why neither was silently chosen here. This must land before real two-VM go-live; flagging it clearly was judged better than guessing at an architecture change mid-infra-phase.