ced4568
bf1e1b5c76
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs (PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR, OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST, and 12 runbooks (first deployment, normal deploy, rollback, Postgres restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure). PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine architectural gap this phase surfaced: the Gateway's existing env.internal_ip mechanism for reaching a Lab container assumes Gateway and Lab share a Docker daemon, which breaks once Labs run on a genuinely separate rn-lab-host-01 VM. The network-policy half is built (a firewall allowance for Gateway->Lab-Host on a published port range); the application half (Provisioner/Gateway address handling) is an explicit, reviewed, NOT-YET-BUILT follow-up flagged there rather than silently decided. Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and platform/README.md with Phase 4 pointers, consistent with how every prior phase updated these. Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
185 lines
11 KiB
Markdown
185 lines
11 KiB
Markdown
# Phase 4 — Production Infrastructure Package
|
||
|
||
**Status: PHASE 4 BUILD COMPLETE. PRODUCTION DEPLOYMENT PENDING HARDWARE.**
|
||
|
||
The dedicated Hetzner server this is designed for hasn't been purchased yet. Everything in this
|
||
document and the `infra/` tree it describes is buildable, testable (against local
|
||
Docker/Postgres/disposable certs), and reviewable **now** — none of it requires the real
|
||
hardware to exist. What genuinely can't happen until the hardware exists is called out
|
||
explicitly throughout, never silently assumed to already work.
|
||
|
||
This document is the map. Detail docs: `docs/PRODUCTION_DEPLOYMENT.md` (step-by-step),
|
||
`docs/PRODUCTION_NETWORK_SECURITY.md` (network/firewall/TLS design), `docs/BACKUP_DR.md`
|
||
(backup/restore/disaster recovery), `docs/OBSERVABILITY.md` (metrics/logs/alerts),
|
||
`docs/CAPACITY_PLANNING.md` (sizing model), and the `docs/RUNBOOK_*.md` files (14 topics,
|
||
command-oriented).
|
||
|
||
## 1. What was built
|
||
|
||
```
|
||
infra/
|
||
├── proxmox/ manual bare-metal install docs + cloud-init template definition
|
||
├── terraform/ VM creation (rn-platform-01, rn-lab-host-01) — real `tofu validate` pass
|
||
├── ansible/ all host configuration — real `ansible-playbook --syntax-check` pass
|
||
│ ├── roles/ common, firewall, docker-tls, platform, lab-host, monitoring-agent, backup
|
||
│ ├── group_vars/ production.yml, staging.yml, vault template
|
||
│ └── site.yml top-level playbook, correct role ordering
|
||
├── docker/ production Compose files, Caddyfile, Postgres tuning — real `docker
|
||
│ compose config` + `caddy validate` pass
|
||
├── firewall/ iptables-restore rulesets for all 3 hosts — structural lint pass
|
||
├── monitoring/ Prometheus/Alertmanager/Loki/Promtail/Grafana config — real `promtool`
|
||
│ + `amtool` validation pass
|
||
├── backup/ Postgres backup/restore/verify scripts — REAL end-to-end test against
|
||
│ disposable local Postgres, including the actual production migrations
|
||
├── scripts/ deploy/rollback/smoke-test/CI/loadtest/capacity-model tooling
|
||
├── ci/ Gitea Actions workflow (build/test/stage automatic, prod manual)
|
||
└── environments/ dev/staging/production config templates, no secrets committed
|
||
```
|
||
|
||
18 new docs (this file + 5 detail docs + 12 runbooks), 4 updated (`LAB_ARCHITECTURE.md`,
|
||
`ENGINEERING_SOP.md`, `README.md`, `platform/README.md`).
|
||
|
||
## 2. Architecture (unchanged from the design, now codified)
|
||
|
||
```
|
||
Hetzner Dedicated Server
|
||
└── Proxmox VE
|
||
├── rn-platform-01 (frontend, Platform API, Gateway, Provisioner, Postgres,
|
||
│ Prometheus/Alertmanager/Grafana/Loki, backup cron, Caddy)
|
||
└── rn-lab-host-01 (Docker Engine w/ TLS, egress proxy, disposable student Labs,
|
||
node-exporter/cadvisor/promtail)
|
||
```
|
||
|
||
The Platform VM / Lab Host VM security boundary from `docs/LAB_ARCHITECTURE.md` is the one
|
||
thing this phase was explicitly told never to weaken — it's codified, not just described, in
|
||
`infra/firewall/` and `infra/ansible/roles/docker-tls/`.
|
||
|
||
## 3. VM specifications
|
||
|
||
| | rn-platform-01 | rn-lab-host-01 |
|
||
|---|---|---|
|
||
| Baseline (default in `infra/terraform/variables.tf`) | 4 cores, 16 GB RAM, 100 GB disk | 24 cores, 96 GB RAM, 500 GB disk |
|
||
| Recommended (once real host is ~16c/32t/128GB) | same — Postgres/API/Gateway/Provisioner don't need much | majority of remaining host resources |
|
||
| Max-growth | scale RAM with Postgres `shared_buffers` if instructor/cohort count grows a lot | scale with measured concurrent-Lab capacity (`docs/CAPACITY_PLANNING.md`) |
|
||
| Configurable via | `terraform.tfvars`' `platform_vm`/`lab_host_vm` objects | same |
|
||
|
||
NUMA/CPU pinning: not configured — not useful at this scale (a handful of VMs on one physical
|
||
host, not a dozen contending for cache locality). Revisit only if real load testing shows a
|
||
reason to.
|
||
|
||
## 4. Storage decision framework
|
||
|
||
Not a final answer — `infra/environments/production/provisioner-config/README.md` and this
|
||
section exist so the final ZFS layout is a deployment-time decision made against real disks,
|
||
not guessed now. Candidate layouts (`infra/terraform/variables.tf`'s `vm_disk_datastore` names
|
||
whichever one gets chosen):
|
||
|
||
- **4× NVMe**: mirrored vdevs (RAID10-equivalent) — two mirror pairs, one for
|
||
Postgres/platform data (I/O-sensitive, small), one for Lab Host disposable data
|
||
(throughput-sensitive, large, never needs redundancy since nothing on it is precious).
|
||
- **2× SSD/NVMe**: one mirrored vdev, logical dataset separation (ZFS datasets, not separate
|
||
pools) between platform (`{{ rootnode_data_root }}/postgres`, `/backups`) and Lab Host data.
|
||
- **Mixed SSD/HDD**: only if the purchased server's actual disks require it — HDD for
|
||
disposable Lab scratch space is acceptable (it's disposable), never for Postgres.
|
||
|
||
**Never single-disk for Postgres data**, in any layout — a mirror is the floor, not a nice-to-have.
|
||
|
||
Pre-deployment disk inspection checklist (model, size, SMART, power-on hours, wear/endurance,
|
||
sector errors, temperature, firmware): `docs/RUNBOOK_DISK_FAILURE.md`'s inspection section —
|
||
run once, against the real drives, before committing to a layout.
|
||
|
||
## 5-8. Network, firewall, Docker TLS, public ingress
|
||
|
||
Full detail: `docs/PRODUCTION_NETWORK_SECURITY.md`. Summary: two private VM networks, Docker TLS
|
||
(mutual, already application-supported since Phase 1 — this phase only needed the certificate
|
||
generation/deployment automation, not new app code), a single public origin (Caddy, automatic
|
||
HTTPS) proxying only to the Gateway, and one genuinely new architectural item this phase
|
||
surfaced and addressed at the network-policy layer: **Gateway-to-Lab reachability across two
|
||
real VMs** — see that doc's dedicated section for what's built vs. what still needs a small,
|
||
reviewed Provisioner/Gateway code change before real go-live.
|
||
|
||
## 9-10. Production auth/cookie config, secrets
|
||
|
||
`infra/environments/production/*.env.template` set `COOKIE_SECURE=1`/`TRUST_PROXY=1` explicitly
|
||
(dev's unsafe defaults are never inherited). Secrets: Ansible Vault (already bundled with
|
||
Ansible — no new tool to install/trust for this project's secret surface), template at
|
||
`infra/ansible/group_vars/production.vault.yml.template`. Nothing is committed —
|
||
`infra/.gitignore` covers `*.env`, TLS private key material, vault plaintext, Terraform state.
|
||
|
||
## 11-13. Postgres, backup, DR
|
||
|
||
Full detail: `docs/BACKUP_DR.md`. Summary: conservative tuning baseline (not hand-tuned before
|
||
real RAM is known), daily encrypted backups with an off-host adapter point (Storage Box/S3/any
|
||
SSH target), and a **real, tested** backup→encrypt→restore→verify cycle
|
||
(`infra/backup/test-backup-restore.sh`) — run against disposable local Postgres with the actual
|
||
production migrations applied, not a mock.
|
||
|
||
## 14-17. Observability, logging, alerting, uptime
|
||
|
||
Full detail: `docs/OBSERVABILITY.md`. Prometheus + Grafana + Loki/Promtail (the user's stated
|
||
Prometheus/Grafana preference, honored) + Alertmanager, all self-hosted, no paid dependency.
|
||
Deliberately does not duplicate the application's own instructor learning analytics
|
||
(Phase 3B) — operational observability and learning analytics stay separate concerns with
|
||
separate audiences, per the explicit instruction. Application-level metrics (active
|
||
students/Labs, provisioning failures, Gateway error rate) are a documented, **not-yet-built**
|
||
follow-up — see that doc's honest accounting of what's real today (host/container/Postgres
|
||
metrics, HTTP healthz probes) vs. what needs app instrumentation later.
|
||
|
||
## 18-19. CI/CD, image strategy
|
||
|
||
`infra/ci/gitea-workflows/build-and-test.yml` — automatic test/build/tag/staging-deploy,
|
||
**manual** production promotion (`infra/scripts/deploy-production.sh`), matching the stated
|
||
preference. SHA-tagged images always, semantic tags on deliberate releases, never `:latest` in
|
||
staging/production. Full detail: `infra/docker/IMAGE_STRATEGY.md`.
|
||
|
||
## 20-21. Ansible, Terraform
|
||
|
||
`infra/ansible/site.yml` — idempotent roles, doesn't assume the VMs already exist (Terraform
|
||
creates them first). Real `ansible-playbook --syntax-check` pass. `infra/terraform/` — real
|
||
`tofu validate` pass against the actual `bpg/proxmox` provider schema. Neither installs Proxmox
|
||
itself or builds the base VM template — both stay deliberate manual, documented, one-time steps
|
||
(`infra/proxmox/README.md`), per the explicit "don't over-automate the fragile one-time step"
|
||
instruction.
|
||
|
||
## 22-23. Environment config, staging
|
||
|
||
`infra/environments/{dev,staging,production}/` — explicit separation, production never
|
||
inherits dev defaults. Staging is designed to run in the homelab or a resource-constrained
|
||
slice of the real server later — no second dedicated server required.
|
||
|
||
## 24-25. Load testing, capacity model
|
||
|
||
`infra/scripts/loadtest/` (real Prepare-Labs-driven boot test + `docker stats` sampling) and
|
||
`infra/scripts/capacity-model.js` (a real, working worksheet — tested against example inputs,
|
||
produces CPU-vs-memory-bound recommended/warning/hard-admission-control thresholds). No
|
||
concurrency number is hard-coded as a marketing figure; the example config in
|
||
`infra/environments/production/capacity.yml.example` is explicitly flagged as placeholder data.
|
||
|
||
## 26-27. Validation checklist, runbooks
|
||
|
||
`docs/PRODUCTION_VALIDATION_CHECKLIST.md` — one master checklist spanning hardware through
|
||
academy-workflow verification. 12 runbooks (`docs/RUNBOOK_*.md`) — first deployment, normal
|
||
deploy, rollback, Postgres restore, VM rebuild (both hosts), Docker TLS rotation, TLS renewal,
|
||
owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host
|
||
resource pressure.
|
||
|
||
## What this phase deliberately did NOT touch
|
||
|
||
Learning system design, curriculum content, RBAC model, session architecture — all Phase 3A/3B,
|
||
unchanged. No parent portal, billing, held scenarios, or production OIDC provider integration.
|
||
RootNode production infrastructure is dedicated to RootNode Academy — never folded into
|
||
existing Synthos production servers, and canonical source stays Gitea, never GitHub `kubekosh`.
|
||
|
||
## Known gap surfaced during this phase (not silently papered over)
|
||
|
||
Building the real production Compose/network config surfaced that the Gateway's existing
|
||
mechanism for reaching a Lab container (`env.internal_ip`, a Docker-bridge-local address) only
|
||
works when Gateway and Lab share a Docker daemon — true in every dev/single-host setup so far,
|
||
**not** true once Labs run on a genuinely separate `rn-lab-host-01`. The network-policy half is
|
||
built (a firewall allowance for Gateway→Lab-Host on a published port range,
|
||
`infra/firewall/rn-lab-host-01.rules`); the application half (the Provisioner recording a
|
||
reachable external address for each Lab, and the Gateway using it) is an explicit, reviewed,
|
||
NOT-YET-BUILT follow-up — see `docs/PRODUCTION_NETWORK_SECURITY.md` for the two candidate
|
||
designs and why neither was silently chosen here. This must land before real two-VM go-live;
|
||
flagging it clearly was judged better than guessing at an architecture change mid-infra-phase.
|