docs(phase4): production infrastructure documentation + runbooks

Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs
(PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR,
OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST,
and 12 runbooks (first deployment, normal deploy, rollback, Postgres
restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner
recovery, stuck Lab cleanup, failed migration, backup failure, disk
failure, Lab Host resource pressure).

PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine
architectural gap this phase surfaced: the Gateway's existing
env.internal_ip mechanism for reaching a Lab container assumes Gateway and
Lab share a Docker daemon, which breaks once Labs run on a genuinely
separate rn-lab-host-01 VM. The network-policy half is built (a firewall
allowance for Gateway->Lab-Host on a published port range); the
application half (Provisioner/Gateway address handling) is an explicit,
reviewed, NOT-YET-BUILT follow-up flagged there rather than silently
decided.

Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and
platform/README.md with Phase 4 pointers, consistent with how every prior
phase updated these.

Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.

.gitignore

@@ -4,3 +4,4 @@ frontend/dist/
 *.db-journal
 /data/
 .env
+.ansible/

README.md

@@ -194,6 +194,11 @@ Production topology and what's built vs. not yet: `docs/LAB_ARCHITECTURE.md`,
 `docs/PHASE1_PLATFORM_FOUNDATION.md`, `docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md`,
 `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md`, `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`.
 
+The production infrastructure package (Proxmox/Ansible/Terraform/firewall/monitoring/backup/CI)
+that runs all of this on real hardware lives in `infra/` — see
+`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`. Built and locally validated now; deployment is
+pending purchase of the target Hetzner dedicated server.
+
 ---
 
 ## Repository Layout

docs/BACKUP_DR.md

@@ -0,0 +1,65 @@
+# Backup & Disaster Recovery
+
+## What's backed up (Tier 1) vs. not
+
+| | Backed up | Why |
+|---|---|---|
+| Postgres data | Yes, daily | The only irreplaceable data — student records, cohort/session state, scenario attempts, assessments, achievements, audit events |
+| `{{ rootnode_data_root }}` config/secrets (Vault files, env templates) | Yes, via the infra repo + Ansible Vault itself | Small, static, versioned |
+| Infra code (`infra/`, Ansible, Terraform state) | Yes — it's in git; Terraform state should be backed up alongside Postgres if not using a remote backend | Needed to rebuild either VM from scratch |
+| Disposable student Labs | **No, intentionally** | Ephemeral by design — a Lab is meant to be destroyed and recreated; backing it up would contradict the whole isolation/disposability model |
+| Prometheus/Loki metrics & log history | No | Operational telemetry, not source-of-truth data; losing 30 days of dashboards on a rebuild is an inconvenience, not data loss |
+
+## Backup mechanism
+
+`infra/backup/backup-postgres.sh` (deployed to `/opt/rootnode/backup/` by the `backup` Ansible
+role, run daily at 03:15 via cron):
+
+1. `pg_dump --format=plain --no-owner --no-privileges | gzip -9`
+2. Sanity check: reject if the compressed dump is under 1KB (almost certainly means Postgres
+   was unreachable or empty, not a real backup).
+3. `gpg --symmetric --cipher-algo AES256` — encrypt, then delete the plaintext dump immediately.
+4. Optional off-host copy via `rsync -az` over SSH — works for Hetzner Storage Box or any SSH
+   target; that's the extension point for S3-compatible alternatives (swap the rsync line for
+   `aws s3 cp`/`rclone` if a different target is chosen later).
+5. Retention pruning (`find ... -mtime +{{ backup_retention_days }} -delete`, default 30 days).
+6. Writes Prometheus textfile-collector metrics
+   (`rootnode_backup_last_run_success`, `..._last_success_timestamp_seconds`, `..._size_bytes`)
+   atomically (tmp file + rename) so a half-written metric file is never scraped.
+
+## Restore
+
+`infra/backup/restore-postgres.sh` — requires an explicit target DB URL (never infers or
+defaults to production, to prevent an accidental overwrite), shows the target's current tables,
+requires typing `RESTORE` to confirm, then decrypts and pipes into `psql`.
+`infra/backup/verify-restore.sh` — confirms all 8 expected tables exist plus the
+`schema_migrations` row count, so a "successful" restore that silently missed a table is caught.
+
+**Genuinely tested**, not just written: `infra/backup/test-backup-restore.sh` runs the full
+cycle against two disposable local Postgres containers, with the actual production migrations
+applied (all 17 — 3 Provisioner + 14 Platform API, in the correct cross-service order), a
+sample row inserted, backed up, restored into the second container, and verified byte-for-byte.
+Confirmed passing, exit 0.
+
+## Disaster recovery scenarios
+
+| Scenario | RPO | RTO (estimate) | What happens |
+|---|---|---|---|
+| Platform VM lost, disks intact | 0 | ~30 min | `docs/RUNBOOK_VM_REBUILD.md` — Ansible re-run against the same disks |
+| Platform VM + its disks lost | Up to 24h (since last backup) | ~1-2h | Rebuild VM (Terraform), re-run Ansible, restore Postgres from the latest off-host backup (`docs/RUNBOOK_POSTGRES_RESTORE.md`) |
+| Lab Host VM lost, any reason | 0 (intentional) | ~30 min | Rebuild VM, re-run Ansible — no data to restore, all Labs were disposable and get recreated on next student login |
+| Single disk failure in a mirror | 0 | 0 (no outage) | `docs/RUNBOOK_DISK_FAILURE.md` — pool stays online DEGRADED, replace and resilver |
+| Whole storage pool lost (both mirror sides) | Up to 24h | ~1-2h | Same as "Platform VM + disks lost" above — this is precisely the scenario the off-host copy exists for; a local-only backup on the same failed pool is useless here |
+| Encryption passphrase lost | N/A | Unrecoverable | The single most important secret to protect and back up separately (e.g., in the same secrets manager as other Vault secrets, not only in `production.vault.yml`) — losing it makes every existing backup file permanently unreadable |
+
+RPO is bounded by backup cadence (daily) — a mid-day incident can lose up to a day of student
+activity. If that's ever unacceptable, the fix is a shorter cron interval or WAL archiving for
+point-in-time recovery, not decided or built here since daily has not yet been proven
+insufficient against real usage.
+
+## What backup does NOT cover
+
+Application code and container images — those live in Gitea and the image registry, both
+already durable and versioned independently of this backup mechanism. TLS certificates — those
+are regenerable (`docs/RUNBOOK_DOCKER_TLS_ROTATION.md`, `docs/RUNBOOK_TLS_RENEWAL.md`), not
+backed up.

docs/CAPACITY_PLANNING.md

@@ -0,0 +1,59 @@
+# Capacity Planning
+
+How many concurrent student Labs `rn-lab-host-01` can actually run — determined from measured
+data after deployment, not guessed or marketed now.
+
+## The model
+
+`infra/scripts/capacity-model.js`:
+
+```
+usable        = hostTotal * (1 - overheadPct/100) - platformReserved
+withMargin    = usable * (1 - marginPct/100)
+perLabLimit   = floor(withMargin / perLabUsage)
+recommended   = min(cpuBoundLimit, memoryBoundLimit)     # whichever runs out first
+warning       = floor(recommended * 0.8)
+hardAdmission = recommended
+```
+
+Computed once for CPU, once for memory; the tighter (smaller) of the two constrains the actual
+recommendation — a host can be memory-starved long before its CPU threads are exhausted, or
+vice versa, and the model doesn't pretend otherwise.
+
+**Inputs** (`infra/environments/production/capacity.yml.example` — explicitly placeholder data
+until real numbers exist):
+- `host_cpu_threads`, `host_memory_mb` — from the real Lab Host VM, once sized
+- `overhead_pct` — Proxmox/hypervisor + OS overhead reserved off the top
+- `platform_reserved_cpu`/`platform_reserved_memory_mb` — headroom for the egress proxy,
+  node-exporter, cadvisor, promtail themselves (they run on the Lab Host too)
+- `per_lab_cpu`/`per_lab_memory_mb` — **measured**, from `infra/scripts/loadtest/`, not assumed
+- `safety_margin_pct` — extra headroom below the raw math, for burst/variance
+
+**Outputs**: recommended concurrent Labs, an 80%-of-recommended warning threshold (for
+alerting — `LabHostHighMemory` in `docs/OBSERVABILITY.md`), and a hard admission-control
+threshold the Provisioner's resource policy should enforce so it refuses to boot a Lab past that
+point rather than degrading every running Lab.
+
+## Load-test tooling that produces the measured inputs
+
+`infra/scripts/loadtest/`:
+- `boot-labs.js` — reuses `platform/test/session-helper.js` (built in Phase 3A) to drive the
+  real admin Prepare-Labs endpoint, booting N Labs and recording boot time/failure rate per Lab
+  into a JSON report.
+- `sample-resources.sh` — samples real `docker stats` for every `rootnode-lab-*` container every
+  5 seconds to CSV (timestamp, name, CPU%, mem usage, mem%, net I/O, block I/O, PIDs).
+
+Together these produce the `per_lab_cpu`/`per_lab_memory_mb` this model needs, plus data on how
+boot time and failure rate change as concurrency rises — the dimensions this phase was told to
+cover: single Lab boot, multiple simultaneous boots, steady-state active Labs, terminal
+WebSocket load, scenario validation load, Gateway throughput, API request volume, Postgres
+connection activity.
+
+## What this document deliberately does not claim
+
+No concurrency number in this repo (`capacity.yml.example`'s numbers included) is a real,
+validated figure — every one is a placeholder until run against the real Lab Host VM with real
+per-Lab measurements. Treating a placeholder as a marketing/sales number would be actively
+misleading; the whole point of the model is that the real number gets filled in once, from
+measurement, after deployment (`docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md` step 4 is the
+trigger to re-measure and update it if the per-Lab footprint later changes).

docs/ENGINEERING_SOP.md

@@ -280,3 +280,28 @@ Full design: `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`.
 - **Known limitations:** see the Phase 3B doc §20 — no automatic abandoned-attempt detection, no
   per-cohort needs-attention threshold configuration, raw-JSON pool_spec authoring in the admin
   UI (correct, not yet a guided form).
+
+## 13. Production Infrastructure (Phase 4)
+
+Full design: `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`, plus detail docs
+`PRODUCTION_DEPLOYMENT.md`, `PRODUCTION_NETWORK_SECURITY.md`, `BACKUP_DR.md`,
+`OBSERVABILITY.md`, `CAPACITY_PLANNING.md`, `PRODUCTION_VALIDATION_CHECKLIST.md`, and 12
+`RUNBOOK_*.md` files, all under `docs/`. Everything lives in `infra/` (Ansible, Terraform,
+Compose, firewall, monitoring, backup, CI, load-test tooling).
+
+- **What it is:** the infrastructure-as-code needed to run rn-platform-01/rn-lab-host-01 on a
+  real Hetzner/Proxmox host — buildable and locally testable now, even though the dedicated
+  server hasn't been purchased. Status is explicitly **PHASE 4 BUILD COMPLETE / PRODUCTION
+  DEPLOYMENT PENDING HARDWARE**, not "deployed."
+- **Where the boundary held:** the Platform VM / Lab Host VM security split from
+  `docs/LAB_ARCHITECTURE.md` is codified in firewall rulesets and mutual-TLS Docker config, not
+  weakened anywhere.
+- **Known gap surfaced, not silently fixed:** the Gateway's Lab-reachability mechanism doesn't
+  survive the move to two real separate VMs — see `docs/PRODUCTION_NETWORK_SECURITY.md`'s
+  "Known gap" section for what's built (firewall policy) vs. what's an explicit, reviewed,
+  not-yet-built application follow-up (Provisioner/Gateway address handling).
+- **Validated without the hardware:** real `tofu validate`, `ansible-playbook --syntax-check`,
+  `docker compose config`, `caddy validate`, `promtool`/`amtool`, and a genuine end-to-end
+  backup→encrypt→restore→verify cycle against disposable local Postgres with the real
+  production migrations applied. None of this proves the real Hetzner network boundary — that
+  needs `docs/PRODUCTION_VALIDATION_CHECKLIST.md` walked against the real server.

docs/LAB_ARCHITECTURE.md

@@ -486,3 +486,13 @@ All four open questions this doc originally ended with are resolved — see the
 at the top and §2–§5 above. Next: `docs/PHASE1_PLATFORM_FOUNDATION.md` — the concrete build
 plan for the platform foundation (Postgres environment registry, provisioner skeleton,
 networking, egress, resource policy, reaper) against these decisions.
+
+## Phase 4: this boundary in production
+
+Phase 4 (`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`) codifies the Platform/Lab-Host split this
+document designed as two real, separately-firewalled Proxmox VMs (`rn-platform-01`,
+`rn-lab-host-01`) connected only by mutual-TLS Docker and a tightly scoped firewall allowance —
+see `docs/PRODUCTION_NETWORK_SECURITY.md`. That document also records a gap Phase 4 surfaced but
+did not resolve: the Gateway's `env.internal_ip` mechanism for reaching a Lab container assumes
+Gateway and Lab share a Docker daemon, which breaks once they're genuinely on separate VMs. Read
+that document's "Known gap" section before treating two-VM production as fully wired.

docs/OBSERVABILITY.md

@@ -0,0 +1,64 @@
+# Observability
+
+Self-hosted Prometheus + Grafana + Alertmanager + Loki/Promtail, deployed via
+`infra/docker/platform/docker-compose.production.yml` on rn-platform-01, with lightweight agents
+(node-exporter, cadvisor, promtail) also on rn-lab-host-01. No paid SaaS.
+
+This is operational observability — host/container/database health, uptime, resource pressure.
+It is deliberately separate from the application's own instructor learning analytics
+(Phase 3B's cohort/scenario/student dashboards) — different audience (ops vs. instructors),
+different data (infrastructure vs. learning outcomes), and mixing them would make both worse.
+
+## What's real today
+
+- **Host metrics** (both VMs): node-exporter — CPU, memory, disk, network.
+- **Container metrics** (Lab Host): cadvisor — per-container CPU/memory/network, including
+  student Lab containers (`rootnode-lab-*`).
+- **Postgres metrics**: postgres-exporter — connections, transaction rate, table sizes.
+- **Uptime/liveness**: blackbox-exporter probing `/healthz` on the Gateway
+  (`platform/gateway/src/server.js`) and the Platform API (`platform/api/src/server.js`), both
+  added this phase specifically for this purpose, plus the Provisioner's pre-existing `/health`.
+- **Logs**: Loki + Promtail, scraping Docker container logs and journald from both VMs, with a
+  redaction pipeline stage that strips `password|token|secret=`-pattern values before they're
+  stored (`infra/monitoring/promtail/`).
+- **Dashboards**: Grafana, reachable only via `https://<origin>/grafana/` and only from
+  `management_cidr` (see `docs/PRODUCTION_NETWORK_SECURITY.md`). One starter dashboard,
+  `infra/monitoring/grafana/provisioning/dashboards/rootnode-overview.json` — 8 panels (Platform
+  + Lab Host CPU/memory/disk, Postgres up/connections, running-Lab-count via cadvisor label
+  match [best-effort, caveated], uptime probes).
+
+## What's NOT built yet (honest accounting, not silently assumed)
+
+Application-level metrics — active student count, Lab provisioning success/failure rate,
+Gateway request latency/error rate, per-endpoint API metrics — would need real instrumentation
+added to the Platform API/Gateway/Provisioner (e.g. a `/metrics` endpoint via `prom-client`).
+`infra/monitoring/prometheus/prometheus.yml.j2` has a commented-out scrape job as the landing
+spot for this, explicitly marked as a follow-up, not wired to anything real yet.
+
+## Alerting
+
+`infra/monitoring/prometheus/rules/rootnode.yml` — two severities:
+
+**Critical** (fires fast, pages): PlatformAPIDown, GatewayDown, PostgresDown, LabHostDown,
+BackupFailed, DiskCriticallyFull.
+
+**Warning** (informational, slower repeat): LabHostHighMemory, HighDiskUsage, BackupAging.
+(RepeatedProvisioningFailures and ExcessiveFailedLogins are present as inert placeholders —
+they need the not-yet-built application metrics above before they can fire on anything real.)
+
+Never alerts on an individual student's scenario failure or a single Lab crash — those are
+expected, routine outcomes of normal usage, not operational incidents.
+
+`infra/monitoring/alertmanager/alertmanager.yml.j2` — critical alerts route separately with a
+30-minute repeat interval, everything else at 4 hours, one webhook receiver
+(`vault_alertmanager_webhook_url` — points at whatever channel gets configured at deploy time,
+e.g. a Slack/Discord/Matrix webhook), and one inhibit rule so a LabHostDown critical suppresses
+its own dependent warnings instead of paging twice for the same root cause.
+
+## Validation performed (local, real tooling — not hardware-dependent)
+
+`infra/monitoring/validate-monitoring.sh` wraps: `promtool check config` (Prometheus config),
+`promtool check rules` (alert rules — 11 rules found, valid), `amtool check-config`
+(Alertmanager — 1 inhibit rule, 2 receivers, valid). These prove syntactic/semantic correctness
+of the configs, not that the real network paths between rn-platform-01 and rn-lab-host-01 work —
+that requires the real hardware and is covered by `docs/PRODUCTION_VALIDATION_CHECKLIST.md`.

docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md

@@ -0,0 +1,184 @@
+# Phase 4 — Production Infrastructure Package
+
+**Status: PHASE 4 BUILD COMPLETE. PRODUCTION DEPLOYMENT PENDING HARDWARE.**
+
+The dedicated Hetzner server this is designed for hasn't been purchased yet. Everything in this
+document and the `infra/` tree it describes is buildable, testable (against local
+Docker/Postgres/disposable certs), and reviewable **now** — none of it requires the real
+hardware to exist. What genuinely can't happen until the hardware exists is called out
+explicitly throughout, never silently assumed to already work.
+
+This document is the map. Detail docs: `docs/PRODUCTION_DEPLOYMENT.md` (step-by-step),
+`docs/PRODUCTION_NETWORK_SECURITY.md` (network/firewall/TLS design), `docs/BACKUP_DR.md`
+(backup/restore/disaster recovery), `docs/OBSERVABILITY.md` (metrics/logs/alerts),
+`docs/CAPACITY_PLANNING.md` (sizing model), and the `docs/RUNBOOK_*.md` files (14 topics,
+command-oriented).
+
+## 1. What was built
+
+```
+infra/
+├── proxmox/          manual bare-metal install docs + cloud-init template definition
+├── terraform/         VM creation (rn-platform-01, rn-lab-host-01) — real `tofu validate` pass
+├── ansible/            all host configuration — real `ansible-playbook --syntax-check` pass
+│   ├── roles/          common, firewall, docker-tls, platform, lab-host, monitoring-agent, backup
+│   ├── group_vars/     production.yml, staging.yml, vault template
+│   └── site.yml        top-level playbook, correct role ordering
+├── docker/            production Compose files, Caddyfile, Postgres tuning — real `docker
+│                       compose config` + `caddy validate` pass
+├── firewall/           iptables-restore rulesets for all 3 hosts — structural lint pass
+├── monitoring/          Prometheus/Alertmanager/Loki/Promtail/Grafana config — real `promtool`
+│                        + `amtool` validation pass
+├── backup/              Postgres backup/restore/verify scripts — REAL end-to-end test against
+│                        disposable local Postgres, including the actual production migrations
+├── scripts/             deploy/rollback/smoke-test/CI/loadtest/capacity-model tooling
+├── ci/                  Gitea Actions workflow (build/test/stage automatic, prod manual)
+└── environments/        dev/staging/production config templates, no secrets committed
+```
+
+18 new docs (this file + 5 detail docs + 12 runbooks), 4 updated (`LAB_ARCHITECTURE.md`,
+`ENGINEERING_SOP.md`, `README.md`, `platform/README.md`).
+
+## 2. Architecture (unchanged from the design, now codified)
+
+```
+Hetzner Dedicated Server
+└── Proxmox VE
+    ├── rn-platform-01   (frontend, Platform API, Gateway, Provisioner, Postgres,
+    │                      Prometheus/Alertmanager/Grafana/Loki, backup cron, Caddy)
+    └── rn-lab-host-01   (Docker Engine w/ TLS, egress proxy, disposable student Labs,
+                           node-exporter/cadvisor/promtail)
+```
+
+The Platform VM / Lab Host VM security boundary from `docs/LAB_ARCHITECTURE.md` is the one
+thing this phase was explicitly told never to weaken — it's codified, not just described, in
+`infra/firewall/` and `infra/ansible/roles/docker-tls/`.
+
+## 3. VM specifications
+
+| | rn-platform-01 | rn-lab-host-01 |
+|---|---|---|
+| Baseline (default in `infra/terraform/variables.tf`) | 4 cores, 16 GB RAM, 100 GB disk | 24 cores, 96 GB RAM, 500 GB disk |
+| Recommended (once real host is ~16c/32t/128GB) | same — Postgres/API/Gateway/Provisioner don't need much | majority of remaining host resources |
+| Max-growth | scale RAM with Postgres `shared_buffers` if instructor/cohort count grows a lot | scale with measured concurrent-Lab capacity (`docs/CAPACITY_PLANNING.md`) |
+| Configurable via | `terraform.tfvars`' `platform_vm`/`lab_host_vm` objects | same |
+
+NUMA/CPU pinning: not configured — not useful at this scale (a handful of VMs on one physical
+host, not a dozen contending for cache locality). Revisit only if real load testing shows a
+reason to.
+
+## 4. Storage decision framework
+
+Not a final answer — `infra/environments/production/provisioner-config/README.md` and this
+section exist so the final ZFS layout is a deployment-time decision made against real disks,
+not guessed now. Candidate layouts (`infra/terraform/variables.tf`'s `vm_disk_datastore` names
+whichever one gets chosen):
+
+- **4× NVMe**: mirrored vdevs (RAID10-equivalent) — two mirror pairs, one for
+  Postgres/platform data (I/O-sensitive, small), one for Lab Host disposable data
+  (throughput-sensitive, large, never needs redundancy since nothing on it is precious).
+- **2× SSD/NVMe**: one mirrored vdev, logical dataset separation (ZFS datasets, not separate
+  pools) between platform (`{{ rootnode_data_root }}/postgres`, `/backups`) and Lab Host data.
+- **Mixed SSD/HDD**: only if the purchased server's actual disks require it — HDD for
+  disposable Lab scratch space is acceptable (it's disposable), never for Postgres.
+
+**Never single-disk for Postgres data**, in any layout — a mirror is the floor, not a nice-to-have.
+
+Pre-deployment disk inspection checklist (model, size, SMART, power-on hours, wear/endurance,
+sector errors, temperature, firmware): `docs/RUNBOOK_DISK_FAILURE.md`'s inspection section —
+run once, against the real drives, before committing to a layout.
+
+## 5-8. Network, firewall, Docker TLS, public ingress
+
+Full detail: `docs/PRODUCTION_NETWORK_SECURITY.md`. Summary: two private VM networks, Docker TLS
+(mutual, already application-supported since Phase 1 — this phase only needed the certificate
+generation/deployment automation, not new app code), a single public origin (Caddy, automatic
+HTTPS) proxying only to the Gateway, and one genuinely new architectural item this phase
+surfaced and addressed at the network-policy layer: **Gateway-to-Lab reachability across two
+real VMs** — see that doc's dedicated section for what's built vs. what still needs a small,
+reviewed Provisioner/Gateway code change before real go-live.
+
+## 9-10. Production auth/cookie config, secrets
+
+`infra/environments/production/*.env.template` set `COOKIE_SECURE=1`/`TRUST_PROXY=1` explicitly
+(dev's unsafe defaults are never inherited). Secrets: Ansible Vault (already bundled with
+Ansible — no new tool to install/trust for this project's secret surface), template at
+`infra/ansible/group_vars/production.vault.yml.template`. Nothing is committed —
+`infra/.gitignore` covers `*.env`, TLS private key material, vault plaintext, Terraform state.
+
+## 11-13. Postgres, backup, DR
+
+Full detail: `docs/BACKUP_DR.md`. Summary: conservative tuning baseline (not hand-tuned before
+real RAM is known), daily encrypted backups with an off-host adapter point (Storage Box/S3/any
+SSH target), and a **real, tested** backup→encrypt→restore→verify cycle
+(`infra/backup/test-backup-restore.sh`) — run against disposable local Postgres with the actual
+production migrations applied, not a mock.
+
+## 14-17. Observability, logging, alerting, uptime
+
+Full detail: `docs/OBSERVABILITY.md`. Prometheus + Grafana + Loki/Promtail (the user's stated
+Prometheus/Grafana preference, honored) + Alertmanager, all self-hosted, no paid dependency.
+Deliberately does not duplicate the application's own instructor learning analytics
+(Phase 3B) — operational observability and learning analytics stay separate concerns with
+separate audiences, per the explicit instruction. Application-level metrics (active
+students/Labs, provisioning failures, Gateway error rate) are a documented, **not-yet-built**
+follow-up — see that doc's honest accounting of what's real today (host/container/Postgres
+metrics, HTTP healthz probes) vs. what needs app instrumentation later.
+
+## 18-19. CI/CD, image strategy
+
+`infra/ci/gitea-workflows/build-and-test.yml` — automatic test/build/tag/staging-deploy,
+**manual** production promotion (`infra/scripts/deploy-production.sh`), matching the stated
+preference. SHA-tagged images always, semantic tags on deliberate releases, never `:latest` in
+staging/production. Full detail: `infra/docker/IMAGE_STRATEGY.md`.
+
+## 20-21. Ansible, Terraform
+
+`infra/ansible/site.yml` — idempotent roles, doesn't assume the VMs already exist (Terraform
+creates them first). Real `ansible-playbook --syntax-check` pass. `infra/terraform/` — real
+`tofu validate` pass against the actual `bpg/proxmox` provider schema. Neither installs Proxmox
+itself or builds the base VM template — both stay deliberate manual, documented, one-time steps
+(`infra/proxmox/README.md`), per the explicit "don't over-automate the fragile one-time step"
+instruction.
+
+## 22-23. Environment config, staging
+
+`infra/environments/{dev,staging,production}/` — explicit separation, production never
+inherits dev defaults. Staging is designed to run in the homelab or a resource-constrained
+slice of the real server later — no second dedicated server required.
+
+## 24-25. Load testing, capacity model
+
+`infra/scripts/loadtest/` (real Prepare-Labs-driven boot test + `docker stats` sampling) and
+`infra/scripts/capacity-model.js` (a real, working worksheet — tested against example inputs,
+produces CPU-vs-memory-bound recommended/warning/hard-admission-control thresholds). No
+concurrency number is hard-coded as a marketing figure; the example config in
+`infra/environments/production/capacity.yml.example` is explicitly flagged as placeholder data.
+
+## 26-27. Validation checklist, runbooks
+
+`docs/PRODUCTION_VALIDATION_CHECKLIST.md` — one master checklist spanning hardware through
+academy-workflow verification. 12 runbooks (`docs/RUNBOOK_*.md`) — first deployment, normal
+deploy, rollback, Postgres restore, VM rebuild (both hosts), Docker TLS rotation, TLS renewal,
+owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host
+resource pressure.
+
+## What this phase deliberately did NOT touch
+
+Learning system design, curriculum content, RBAC model, session architecture — all Phase 3A/3B,
+unchanged. No parent portal, billing, held scenarios, or production OIDC provider integration.
+RootNode production infrastructure is dedicated to RootNode Academy — never folded into
+existing Synthos production servers, and canonical source stays Gitea, never GitHub `kubekosh`.
+
+## Known gap surfaced during this phase (not silently papered over)
+
+Building the real production Compose/network config surfaced that the Gateway's existing
+mechanism for reaching a Lab container (`env.internal_ip`, a Docker-bridge-local address) only
+works when Gateway and Lab share a Docker daemon — true in every dev/single-host setup so far,
+**not** true once Labs run on a genuinely separate `rn-lab-host-01`. The network-policy half is
+built (a firewall allowance for Gateway→Lab-Host on a published port range,
+`infra/firewall/rn-lab-host-01.rules`); the application half (the Provisioner recording a
+reachable external address for each Lab, and the Gateway using it) is an explicit, reviewed,
+NOT-YET-BUILT follow-up — see `docs/PRODUCTION_NETWORK_SECURITY.md` for the two candidate
+designs and why neither was silently chosen here. This must land before real two-VM go-live;
+flagging it clearly was judged better than guessing at an architecture change mid-infra-phase.

docs/PRODUCTION_DEPLOYMENT.md

@@ -0,0 +1,84 @@
+# Production Deployment
+
+Step-by-step, for when the real Hetzner host exists. See
+`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` for the architecture this executes.
+
+## 0. One-time, manual, not automated (deliberately)
+
+1. Install Proxmox VE on the bare-metal host — `infra/proxmox/README.md`.
+2. Inspect the real disks, pick a ZFS layout — `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4,
+   `docs/RUNBOOK_DISK_FAILURE.md`'s inspection checklist.
+3. Build the cloud-init VM template Terraform will clone —
+   `infra/proxmox/cloud-init/user-data.yaml` + `infra/proxmox/README.md`.
+4. Fill in `infra/terraform/terraform.tfvars` (copy from `.tfvars.example`) with the real
+   Proxmox API endpoint, node name, storage/datastore names, and chosen VM sizes.
+5. Fill in `infra/ansible/inventory/production.ini` with the real VM IPs, and
+   `infra/ansible/group_vars/production.yml` with `platform_ip`, `lab_host_ip`,
+   `management_cidr`, `proxmox_mgmt_cidr`.
+6. Copy `infra/ansible/group_vars/production.vault.yml.template` →
+   `production.vault.yml`, fill in every `vault_*` secret with real generated values, then
+   `ansible-vault encrypt group_vars/production.vault.yml`. Never commit the plaintext.
+
+## 1. Create the VMs
+
+```bash
+cd infra/terraform
+tofu init
+tofu plan    # review before applying against real hardware
+tofu apply
+```
+
+Creates `rn-platform-01` and `rn-lab-host-01` from the cloud-init template. Confirm both boot
+and are SSH-reachable from the management network before continuing.
+
+## 2. Generate Docker TLS certificates (offline, once)
+
+```bash
+export ROOTNODE_TLS_BUNDLE_DIR=/secure/offline/path   # never inside the repo
+./infra/scripts/docker-tls/generate-docker-tls.sh
+./infra/scripts/docker-tls/verify-docker-tls.sh
+```
+
+The CA private key stays in `ROOTNODE_TLS_BUNDLE_DIR` and never touches either VM — only the CA
+cert (public) and the relevant leaf cert+key go to each host.
+
+## 3. Configure both hosts
+
+```bash
+cd infra/ansible
+ansible-playbook -i inventory/production.ini site.yml --ask-vault-pass
+```
+
+Role order matters and is fixed in `site.yml`: `common` → `firewall` → `docker-tls` →
+`platform`/`lab-host` → `monitoring-agent` → `backup`. Firewall and TLS land before anything
+that depends on them.
+
+## 4. First application deploy
+
+```bash
+./infra/scripts/deploy-production.sh <image-tag>
+```
+
+Pulls the tagged images, runs the compose stack, applies Postgres migrations (automatic, on
+Platform API/Provisioner startup — see `docs/RUNBOOK_FAILED_MIGRATION.md` if one fails).
+
+## 5. Verify
+
+```bash
+./infra/scripts/production-smoke-test.sh
+```
+
+Curls `/healthz` on the Gateway and Platform API, confirms Postgres is reachable, confirms
+Prometheus/Grafana/Alertmanager are up. Then walk
+`docs/PRODUCTION_VALIDATION_CHECKLIST.md` in full before calling the deployment live.
+
+## Normal (non-first) deploys
+
+`docs/RUNBOOK_NORMAL_DEPLOY.md`. Rollback: `docs/RUNBOOK_ROLLBACK.md`.
+
+## What this document does NOT cover
+
+DNS/Cloudflare setup for the public origin (a one-time manual step outside this repo's scope —
+point the A/AAAA record at the real host and confirm Caddy issues its certificate). Which
+storage layout to pick (decision framework only, §4 of the master doc — the real answer depends
+on the real disks).

docs/PRODUCTION_NETWORK_SECURITY.md

@@ -0,0 +1,93 @@
+# Production Network & Security
+
+The Platform VM / Lab Host VM boundary is the core production control. This document is where
+every network decision that touches it gets written down. Nothing here weakens it.
+
+## Topology
+
+```
+Internet ──HTTPS(443)──▶ Caddy (rn-platform-01) ──▶ Gateway (internal bridge)
+                                                        │
+management_cidr ──SSH──▶ rn-platform-01                │
+management_cidr ──SSH──▶ rn-lab-host-01                 │
+proxmox_mgmt_cidr ──22/8006──▶ Proxmox host             │
+                                                        │
+rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd)
+rn-lab-host-01 (promtail) ──3100──▶ rn-platform-01 (Loki)
+rn-lab-host-01 (node-exporter/cadvisor) ◀──scrape── rn-platform-01 (Prometheus)
+```
+
+Firewall source of truth: `infra/firewall/{proxmox-host,rn-platform-01,rn-lab-host-01}.rules`
+(iptables-restore format, deployed by the `firewall` Ansible role). Validate with
+`infra/firewall/validate-firewall.sh`.
+
+## What is explicitly denied
+
+- Student Lab → another student Lab (no student-to-student traffic, ever).
+- Student Lab → Proxmox management (both the host firewall and the `lab_subnet_range` DENY
+  rule in `rn-lab-host-01.rules`).
+- Student Lab → Platform Postgres / Platform API / Gateway internals (logged DENY in
+  `rn-platform-01.rules` for anything from `lab_host_ip` to ports 5432/4100).
+- rn-lab-host-01 → Postgres (same rule — the Lab Host itself has no legitimate reason to reach
+  Postgres; only the Provisioner on rn-platform-01 does, and it talks to Postgres over the
+  internal Docker bridge, never across the VM boundary).
+- Any student egress except through the approved egress proxy (enforced by the Provisioner's
+  existing per-Lab `DOCKER-USER` chain manipulation from Phase 1 — unchanged by this phase).
+- Direct public exposure of the Provisioner, Postgres, Docker TLS port, Proxmox UI, or Grafana
+  (Grafana is reachable only via Caddy's `/grafana/*` path, itself restricted to
+  `management_cidr` by a `caddy` matcher — see `infra/docker/platform/Caddyfile.j2`).
+
+## Docker TLS (Provisioner ↔ Lab Host)
+
+Mutual TLS on port 2376, restricted at the firewall to `platform_ip` only. Application support
+already existed since Phase 1 (`platform/provisioner/src/docker.js`'s `buildClient()`); this
+phase added the operational half: `infra/scripts/docker-tls/generate-docker-tls.sh` (CA +
+server + client cert generation), the `docker-tls` Ansible role (deploys certs, configures
+`dockerd`'s `daemon.json`), and `docs/RUNBOOK_DOCKER_TLS_ROTATION.md` for renewal. The CA
+private key is generated offline and never committed or deployed to either VM — only the CA
+cert (to verify the peer) and each host's own leaf cert+key.
+
+## Public ingress
+
+One origin, Caddy, automatic HTTPS. `reverse_proxy gateway:4200` — Caddy talks only to the
+Gateway; nothing else is proxied. WebSocket upgrade works automatically (Caddy's default
+`reverse_proxy` behavior, no special config needed for the terminal WebSocket path). Whether to
+front this with Cloudflare or expose the origin directly is left as a deployment-time decision
+(both work with Caddy's ACME flow; Cloudflare adds DDoS absorption at the cost of Cloudflare
+being able to see plaintext unless "Full (strict)" mode is used) — not decided here since it
+doesn't affect anything else in this document.
+
+## Known gap: Gateway-to-Lab reachability across two real VMs
+
+**This is the most important item in this document.**
+
+The Gateway currently reaches a running Lab container via `env.internal_ip` — a Docker
+bridge-local address recorded by the Provisioner when it creates the container
+(`platform/gateway/src/server.js`, `platform/provisioner`, built in Phase 2A). This works when
+Gateway and the Lab share a Docker daemon, which has been true in every setup so far, including
+dev and any single-host deployment. **It stops being true once Labs run on a genuinely separate
+`rn-lab-host-01`** — a Docker bridge-internal IP on one host is not routable from a different
+host by default.
+
+What's built now (network-policy half only): a firewall allowance in
+`infra/firewall/rn-lab-host-01.rules` for `platform_ip` → `lab_gateway_port_range`
+(`30000:32767` by default, configurable), so that once Lab containers publish ports on the Lab
+Host's own interface, the Gateway VM is permitted to reach them.
+
+What's **not yet built** (application half — explicitly flagged, not silently decided here):
+the Provisioner needs to either (a) publish each Lab container's relevant port(s) on the Lab
+Host's interface and record a reachable `external_ip`/`external_port` alongside (or instead of)
+`internal_ip`/`internal_port`, or (b) route Gateway→Lab traffic through a proxy/tunnel that
+lives on the Lab Host and forwards to the container's internal address. Option (a) is simpler
+and consistent with how the egress proxy and cadvisor are already handled in this phase's
+compose files; option (b) avoids publishing a wide port range but adds a moving part. This is a
+small, reviewable, application-level change — not a network-policy change — and should land as
+its own reviewed commit before real two-VM production traffic depends on it. Until it lands,
+Gateway↔Lab terminal/scenario traffic will not work correctly across two separate VMs, even
+though every other production networking, firewall, and TLS boundary is in place and correct.
+
+## Cookie / auth production settings
+
+`infra/environments/production/*.env.template`: `COOKIE_SECURE=1`, `TRUST_PROXY=1`. Dev's
+insecure defaults (`COOKIE_SECURE` unset, no proxy trust) are never inherited — production env
+files are separate files, not a dev file with overrides layered on.

docs/PRODUCTION_VALIDATION_CHECKLIST.md

@@ -0,0 +1,50 @@
+# Production Validation Checklist
+
+Walk this once, top to bottom, before calling a production deployment live. Every item here
+needs the real hardware — this is the "DEPLOY LATER" half of Phase 4, not something this repo
+can prove in advance.
+
+## Hardware / Proxmox
+- [ ] Disk inspection done, layout chosen and matches what's actually in the server (`docs/RUNBOOK_DISK_FAILURE.md` inspection checklist, `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4)
+- [ ] Proxmox VE installed, reachable only from `management_cidr`/`proxmox_mgmt_cidr`
+- [ ] VM template built from real cloud-init (`infra/proxmox/README.md`)
+- [ ] `tofu apply` created both VMs; both boot and are SSH-reachable
+
+## Platform VM
+- [ ] `ansible-playbook site.yml` completed clean, all roles
+- [ ] `docker compose ps` shows every service healthy (postgres, provisioner, api, gateway, reverse-proxy, prometheus, alertmanager, grafana, loki, promtail, exporters)
+- [ ] `curl https://<origin>/healthz` (via Gateway) returns 200
+- [ ] `curl http://localhost:4300/healthz` (Platform API, from the VM) returns `{"ok":true,"db":"ok"}`
+- [ ] TLS certificate issued and valid for the real domain
+
+## Lab Host VM
+- [ ] Docker daemon TLS verified: `docker --tlsverify --tlscacert=... -H tcp://<lab_host_ip>:2376 info` succeeds from the Platform VM, fails from anywhere else
+- [ ] Provisioner successfully creates a real Lab against the real Lab Host
+- [ ] Gateway can actually reach that Lab's terminal — **this depends on the not-yet-built application change documented in `docs/PRODUCTION_NETWORK_SECURITY.md`'s "Known gap" section; do not mark this item done until that change has landed and been verified for real**
+- [ ] Egress proxy correctly restricts a Lab's outbound traffic (Provisioner's per-Lab `DOCKER-USER` chain)
+
+## Security
+- [ ] `infra/firewall/validate-firewall.sh` run against the real hosts with real `iptables-restore --test`, not just the structural lint
+- [ ] Confirm from an unauthorized IP: Postgres, Provisioner, Docker TLS port, Proxmox UI are all unreachable
+- [ ] Confirm Student Lab → another Lab is blocked; confirm Student Lab → Proxmox is blocked
+- [ ] `COOKIE_SECURE`/`TRUST_PROXY` active (check a real Set-Cookie header over HTTPS)
+
+## Academy (application-level smoke test)
+- [ ] Owner bootstrap login works
+- [ ] Instructor can create a cohort, roster a student
+- [ ] Student can log in, boot a Lab, complete a scenario, see progress
+- [ ] An assessment can be assigned, taken, and graded end-to-end
+- [ ] Instructor analytics dashboards populate with real data
+
+## Backup
+- [ ] Manual `backup-postgres.sh` run succeeds against real Postgres, real off-host target reachable
+- [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness
+- [ ] Cron job confirmed scheduled (`crontab -l`)
+
+## Observability
+- [ ] Grafana dashboard shows real data for both VMs
+- [ ] Trigger one alert deliberately (e.g. stop a service) and confirm it fires and reaches the webhook
+- [ ] Loki has logs flowing from both VMs
+
+Only once every box above is checked against the real server does the deployment graduate from
+"infrastructure package built" to "actually in production."

docs/RUNBOOK_BACKUP_FAILURE.md

@@ -0,0 +1,31 @@
+# Runbook: Backup Failure
+
+Triggered by the `BackupFailed`/`BackupAging` alerts (`infra/monitoring/prometheus/rules/rootnode.yml`).
+
+```bash
+# 1. See what actually happened — the cron wrapper logs to syslog
+journalctl -u rootnode-backup --since "24 hours ago" -o cat
+# or, if the cron job never even ran:
+grep rootnode-postgres-backup /var/log/syslog
+
+# 2. Run it manually to see the failure live
+source /etc/rootnode/backup.env
+/opt/rootnode/backup/backup-postgres.sh
+```
+
+**Common causes:**
+- Disk full on `{{ rootnode_data_root }}/backups` — check `df -h`, prune more aggressively
+  (`BACKUP_RETENTION_DAYS`) or the ZFS dataset needs more space (see `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4).
+- Off-host target unreachable (`BACKUP_OFFSITE_TARGET`) — check the SSH key/credentials
+  haven't expired, the target host/bucket is reachable: `ssh <offsite-target-host> echo ok` or
+  the equivalent for whatever adapter is configured (`docs/BACKUP_DR.md`).
+- `BACKUP_ENCRYPTION_PASSPHRASE` missing/wrong in `/etc/rootnode/backup.env` — re-check against
+  the value in Ansible Vault (`vault_backup_encryption_passphrase`); a mismatch here means even
+  a "successful" old backup becomes unrestorable, so treat this specific failure mode as urgent.
+- The dump itself is suspiciously small (script's own sanity check tripped) — Postgres may be
+  unhealthy; check `docker exec rootnode-platform-postgres-1 pg_isready` first.
+
+**After fixing:** run the backup manually once to confirm success (metric updates immediately,
+alert clears within one Prometheus scrape interval), then confirm the cron job is still
+correctly scheduled (`crontab -l` as root, or re-run the `backup` Ansible role if the cron entry
+itself is suspect).

docs/RUNBOOK_DISK_FAILURE.md

@@ -0,0 +1,44 @@
+# Runbook: Disk Failure
+
+Applies once real hardware/ZFS exists — see `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4 for
+the storage decision framework this assumes (mirrored vdevs, never single-disk for Postgres).
+
+```bash
+# 1. Confirm the failure
+zpool status -v
+smartctl -a /dev/sdX   # the suspect drive
+
+# 2. If it's a mirrored vdev (the required layout for anything holding Postgres data — see §4),
+#    the pool stays online in DEGRADED state. No immediate data-loss risk, but no redundancy
+#    either until replaced — treat DEGRADED as urgent, not routine.
+zpool status   # confirm DEGRADED, not FAULTED/UNAVAIL (those mean act immediately, possibly
+                # restore from backup if the redundant mirror side is also compromised)
+
+# 3. Replace the physical drive, then:
+zpool replace <pool> <old-device> <new-device>
+zpool status   # watch the resilver progress
+
+# 4. If a whole pool was lost (both mirror sides failed simultaneously — the scenario mirroring
+#    exists to make vanishingly unlikely, but plan for it anyway):
+#    - rn-lab-host-01's disposable data: nothing to recover, just rebuild the VM
+#      (docs/RUNBOOK_VM_REBUILD.md)
+#    - rn-platform-01's Postgres data: restore from the latest off-host backup
+#      (docs/RUNBOOK_POSTGRES_RESTORE.md) — this is exactly the scenario off-host backup
+#      copies exist for; a local-only backup on the SAME failed pool would be useless here
+```
+
+**Pre-deployment disk inspection checklist** (run once when the real drives are known, before
+they go into service — `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4):
+
+```bash
+for d in /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1; do   # adjust to real device names
+  echo "== $d =="
+  smartctl -i "$d" | grep -E 'Model|Serial|Capacity|Firmware'
+  smartctl -A "$d" | grep -iE 'power_on_hours|reallocated|pending|uncorrectable|wear|percentage.used|temperature'
+  smartctl -H "$d" | grep -i overall
+done
+```
+Record model, size, power-on hours, wear/endurance percentage, reallocated/pending sector
+counts, temperature, and firmware version for every drive before committing to a ZFS layout —
+a drive already showing wear or reallocated sectors shouldn't be trusted with Postgres data
+even in a mirror.

docs/RUNBOOK_DOCKER_TLS_ROTATION.md

@@ -0,0 +1,31 @@
+# Runbook: Docker TLS Rotation
+
+Rotate on a schedule (recommend yearly, well before the ~2.25-year cert lifetime
+`generate-docker-tls.sh` defaults to) or immediately if a client cert may have leaked.
+
+```bash
+# 1. Generate a fresh bundle (new CA too — simplest, avoids any question about whether the old
+#    CA's private key was ever exposed)
+cd infra/scripts/docker-tls
+./generate-docker-tls.sh --out-dir ./out-$(date +%Y%m%d) --lab-host-ip <real-ip> --lab-host-dns rn-lab-host-01.internal
+./verify-docker-tls.sh ./out-$(date +%Y%m%d)
+
+# 2. Redeploy — this restarts dockerd on rn-lab-host-01 (brief interruption to any in-flight
+#    Docker API call from the Provisioner, NOT to already-running student Labs — their
+#    containers keep running through a dockerd restart)
+cd ../../ansible
+ROOTNODE_TLS_BUNDLE_DIR=../scripts/docker-tls/out-$(date +%Y%m%d) \
+  ansible-playbook site.yml -i inventory/production.ini --tags docker-tls --ask-vault-pass
+
+# 3. Verify the Provisioner can still reach rn-lab-host-01
+curl -s http://localhost:4100/health   # from rn-platform-01
+```
+
+**Troubleshooting a failed rotation:**
+- `x509: certificate signed by unknown authority` — the new ca.pem wasn't deployed to both
+  sides, or an old one is cached; re-run the docker-tls role on BOTH hosts.
+- Provisioner logs `ECONNREFUSED` on port 2376 — dockerd didn't actually restart with the new
+  daemon.json; check `systemctl status docker` on rn-lab-host-01 and the ExecStart override
+  (`infra/ansible/roles/docker-tls/templates/docker-override.conf.j2`) actually applied.
+- Firewall — confirm `infra/firewall/rn-lab-host-01.rules`' 2376 rule still matches
+  rn-platform-01's actual IP (didn't change as part of this rotation, but worth a sanity check).

docs/RUNBOOK_FAILED_MIGRATION.md

@@ -0,0 +1,35 @@
+# Runbook: Failed Migration
+
+Both Platform API and Provisioner apply their own migrations automatically on startup
+(`platform/api/src/db.js`, `platform/provisioner/src/db.js`), file-tracked in
+`schema_migrations`, wrapped in a transaction per file — a migration either fully applies or
+fully rolls back, it can't leave a table half-altered.
+
+**If a deploy fails because a migration errored:**
+
+```bash
+# 1. Check what actually happened
+docker logs rootnode-platform-api-1 --tail 50   # the error is right there — migration file + message
+docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
+  "SELECT * FROM schema_migrations ORDER BY applied_at DESC LIMIT 10;"
+
+# 2. Since the failing migration's transaction rolled back, the schema is still in its
+#    PREVIOUS valid state — the app just won't start until the migration succeeds. Fix the
+#    root cause (usually a real data issue the migration didn't anticipate, or the
+#    cross-service ordering dependency — Provisioner's migrations must run before Platform
+#    API's, since scenario_attempts has a real FK to environments; see
+#    docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md).
+
+# 3. Redeploy once fixed — the migration runner is idempotent and will just pick up where it
+#    left off; already-applied migrations are skipped.
+./infra/scripts/deploy-production.sh <same-or-fixed-tag>
+```
+
+**If you need to roll back the CODE but the migration already succeeded** (schema changed,
+code that depended on it is being reverted): rolling back application images
+(`docs/RUNBOOK_ROLLBACK.md`) does NOT undo a Postgres migration — there is no automatic
+down-migration mechanism (deliberate, matches the existing check-before-alter, additive-only
+migration convention documented in `docs/ENGINEERING_SOP.md` §6). If the old code genuinely
+can't run against the new schema, restore from the pre-migration backup instead
+(`docs/RUNBOOK_POSTGRES_RESTORE.md`) rather than trying to hand-write a down-migration under
+pressure.

docs/RUNBOOK_FIRST_DEPLOYMENT.md

@@ -0,0 +1,34 @@
+# Runbook: First Production Deployment
+
+Prerequisites: Proxmox installed (`infra/proxmox/README.md`), VMs created
+(`infra/terraform/`), DNS pointed at rn-platform-01's public IP.
+
+```bash
+# 1. Generate Docker TLS material (offline, on your own machine — never on either VM)
+cd infra/scripts/docker-tls
+./generate-docker-tls.sh --out-dir ./out --lab-host-ip <real-lab-host-ip> --lab-host-dns rn-lab-host-01.internal
+./verify-docker-tls.sh ./out
+
+# 2. Fill in real values
+cd ../../ansible
+cp group_vars/production.vault.yml.template group_vars/production.vault.yml
+$EDITOR group_vars/production.vault.yml   # real passwords/secrets — openssl rand -hex 32 etc.
+ansible-vault encrypt group_vars/production.vault.yml
+$EDITOR group_vars/production.yml         # real IPs, domain, CIDRs
+$EDITOR inventory/production.ini          # real IPs
+
+# 3. Deploy
+ROOTNODE_TLS_BUNDLE_DIR=../scripts/docker-tls/out \
+  ansible-playbook site.yml -i inventory/production.ini --ask-vault-pass
+
+# 4. Bootstrap the owner account (once, by a human)
+ssh rootnode-deploy@rn-platform-01
+docker exec -it rootnode-platform-api-1 node bootstrap-owner.js [email protected] "Your Name"
+# (password comes from BOOTSTRAP_OWNER_PASSWORD in the environment, or the CLI prompt — see
+# platform/api/README.md)
+
+# 5. Verify
+../../infra/scripts/production-smoke-test.sh
+```
+
+Then work through `docs/PRODUCTION_VALIDATION_CHECKLIST.md` in full before calling this done.

docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md

@@ -0,0 +1,29 @@
+# Runbook: High Lab Host Resource Pressure
+
+Triggered by the `LabHostHighMemory`/`HighDiskUsage` warning alerts.
+
+```bash
+# 1. How many Labs are actually running, and against what admission-control threshold?
+curl -s http://localhost:4100/environments | jq '[.[] | select(.status=="running")] | length'
+node infra/scripts/capacity-model.js   # recommended/warning/hard thresholds
+
+# 2. If genuinely over capacity: this is expected pressure from real usage, not a bug.
+#    - Destroy idle/abandoned Labs early via /admin (per-cohort "Destroy Labs" after a class
+#      session ends, rather than waiting for the idle reaper's default window)
+#    - If a specific cohort's class just ended, use /admin -> Cohorts -> [cohort] -> Destroy
+#      Cohort Labs immediately rather than waiting
+
+# 3. If pressure is high with FEWER Labs running than capacity-model.js's threshold: something
+#    is consuming more per-Lab resources than the model assumes (a scenario doing something
+#    unusually heavy, a memory leak in a long-running Lab, zombie processes from a crashed
+#    reset). Check per-container usage:
+docker stats --no-stream | grep rootnode-lab-
+
+# 4. Update infra/environments/production/capacity.yml with fresh measured numbers
+#    (infra/scripts/loadtest/) if the per-Lab footprint has genuinely changed, so the
+#    admission-control threshold reflects reality going forward.
+```
+
+Do not silently raise resource limits in `provisioner-config/resource-policy.yaml` to make an
+alert go away without understanding why usage is higher than modeled — that just moves the
+same pressure onto host RAM/CPU directly instead of onto individual Lab limits.

docs/RUNBOOK_NORMAL_DEPLOY.md

@@ -0,0 +1,17 @@
+# Runbook: Normal Deploy
+
+For a routine code change already merged to `main` on the canonical Gitea repo.
+
+1. CI (`infra/ci/gitea-workflows/build-and-test.yml`) runs automatically: tests, image build/tag/push, staging deploy, staging smoke test.
+2. Review the staging smoke test result and, ideally, poke around `https://staging.<domain>` yourself.
+3. Promote to production (manual, deliberate):
+
+```bash
+cd infra/scripts
+./deploy-production.sh sha-<12-char-sha>    # or a release tag, e.g. v1.3.0
+./production-smoke-test.sh
+```
+
+4. Watch Grafana (`/grafana/`) for 10-15 minutes — error rates, Postgres connections, Lab Host
+   memory — before considering the deploy "settled."
+5. If anything looks wrong: `docs/RUNBOOK_ROLLBACK.md`.

docs/RUNBOOK_OWNER_RECOVERY.md

@@ -0,0 +1,27 @@
+# Runbook: Owner Account Recovery
+
+**Locked out with zero working owner accounts** (all deactivated, forgot the only owner's
+password, etc.):
+
+```bash
+ssh rootnode-deploy@rn-platform-01
+docker exec -it rootnode-platform-api-1 sh
+node bootstrap-owner.js [email protected] "Your Name"
+# bootstrap-owner.js refuses to run if an active owner already exists UNLESS the target email
+# belongs to an EXISTING instructor row — in that case it promotes them to owner. Point it at
+# any known instructor email to recover through them, or a brand-new email to create a fresh
+# owner outright if the instructors table itself is in an unknown state.
+```
+
+**Forgot password, account still active and you have SSH access:**
+
+```bash
+docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
+  "SELECT instructor_id, email, role, status FROM instructors WHERE role='owner';"
+# Then use another owner's account to reset via /admin -> Instructors -> [account] -> Reset
+# Password, OR re-run bootstrap-owner.js targeting that email (also resets the password).
+```
+
+**Never** hand-edit `password_hash` in the database — it's `scrypt(salt):hash` in a specific
+format (`platform/api/src/auth/password.js`); a malformed value just breaks login rather than
+setting a usable password. Always go through `bootstrap-owner.js` or the admin UI's reset flow.

docs/RUNBOOK_POSTGRES_RESTORE.md

@@ -0,0 +1,45 @@
+# Runbook: Postgres Restore
+
+**When to use:** Postgres corruption/data loss on rn-platform-01, or verifying a backup is
+actually good (do this periodically even without an incident — an untested backup is a
+hypothesis, not a backup).
+
+```bash
+# 1. Find the backup to restore
+ls -la /srv/rootnode-data/backups/    # or wherever backup_offsite_target points, if local copies are gone
+
+# 2. Create a FRESH target database — never restore over the live one directly
+docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
+  "CREATE DATABASE rootnode_platform_restore_test;"
+
+# 3. Restore into it
+source /etc/rootnode/backup.env   # for BACKUP_ENCRYPTION_PASSPHRASE
+/opt/rootnode/backup/restore-postgres.sh \
+  /srv/rootnode-data/backups/rootnode-platform-<timestamp>.sql.gz.gpg \
+  "postgres://rootnode_platform:$PGPASS@localhost:5432/rootnode_platform_restore_test"
+# type RESTORE when prompted
+
+# 4. Verify
+/opt/rootnode/backup/verify-restore.sh \
+  "postgres://rootnode_platform:$PGPASS@localhost:5432/rootnode_platform_restore_test"
+
+# 5a. If this was a drill: drop the test database
+docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
+  "DROP DATABASE rootnode_platform_restore_test;"
+
+# 5b. If this was a REAL incident: stop the app stack, rename the broken DB, rename the
+#     restored one into its place, restart.
+systemctl stop rootnode-platform
+docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
+  "ALTER DATABASE rootnode_platform RENAME TO rootnode_platform_broken_$(date +%s);"
+docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
+  "ALTER DATABASE rootnode_platform_restore_test RENAME TO rootnode_platform;"
+systemctl start rootnode-platform
+```
+
+Local, non-production dry run of this exact flow: `infra/backup/test-backup-restore.sh` (uses
+disposable containers, safe to run anytime, no real data involved).
+
+**Recovery point objective:** up to 24 hours of academic data (daily backup cadence) — see
+`docs/BACKUP_DR.md` for the full RPO/RTO discussion and why that's the honest number, not an
+aspirational one.

docs/RUNBOOK_ROLLBACK.md

@@ -0,0 +1,18 @@
+# Runbook: Rollback
+
+```bash
+cd infra/scripts
+./rollback-production.sh              # rolls back to the previous deployed tag automatically
+# or, to go back further than one step:
+./rollback-production.sh sha-<older-sha>
+
+./production-smoke-test.sh
+```
+
+This rolls back **application images only**. If the bad deploy included a database migration
+that needs undoing too, schema changes aren't automatically reversible — see
+`docs/RUNBOOK_FAILED_MIGRATION.md` instead (or in addition).
+
+After rolling back: file an issue / note in the deploy log (`/opt/rootnode/deploy-history.log`
+already has the rollback recorded) about what went wrong before re-attempting the forward
+deploy.

docs/RUNBOOK_STUCK_LAB_CLEANUP.md

@@ -0,0 +1,40 @@
+# Runbook: Stuck Lab Cleanup
+
+A Lab that's stuck `provisioning`/`resetting` past its expected boot window, or a container
+that outlived its Postgres `environments` row (crash during teardown, etc.).
+
+**Via the admin UI (preferred, no host access needed):** `/admin` -> Students -> [student] ->
+Environment -> Destroy, then Provision again if they need a fresh one immediately.
+
+**Via the API directly** (same effect, useful for scripting a batch cleanup):
+```bash
+curl -s -b <owner-session-cookie> -X DELETE https://academy.example.com/api/admin/environments/<environment_id>
+```
+
+**A container the Provisioner itself has lost track of** (rare — e.g. a crash during
+provisioning left an orphaned container/network/firewall chain with no matching `environments`
+row, or vice versa):
+
+```bash
+# On rn-lab-host-01 — list what's actually running
+docker ps -a --format '{{.Names}}\t{{.Status}}' | grep '^rootnode-lab-'
+
+# Cross-reference against the Provisioner's own view
+curl -s http://localhost:4100/environments | jq '.[] | {environment_id, status, container_name}'
+
+# Provisioner already does this automatically on its own restart (reconcileOnStartup,
+# platform/provisioner/src/lifecycle.js) — a simple `systemctl restart rootnode-platform`
+# (which restarts the provisioner container as part of the stack) is often the fastest fix for
+# a batch of orphans rather than hand-cleaning each one.
+
+# Manual last resort — remove the container/network/firewall chain directly:
+docker rm -f rootnode-lab-<id>
+docker network rm rootnode-lab-net-<id>
+iptables -D DOCKER-USER -s <subnet> -j ROOTNODE-ENV-<id>   # remove the jump rule FIRST
+iptables -F ROOTNODE-ENV-<id> && iptables -X ROOTNODE-ENV-<id>
+```
+
+Never manually delete an `environments` row from Postgres directly unless you've confirmed the
+container/network/firewall chain are already gone — the Provisioner treats that table as
+authoritative and will behave unpredictably against real Docker/iptables state it doesn't know
+matches.

docs/RUNBOOK_TLS_RENEWAL.md

@@ -0,0 +1,19 @@
+# Runbook: Public TLS Certificate Renewal
+
+If using Caddy's built-in Let's Encrypt (the default in `infra/docker/platform/Caddyfile.j2`):
+**nothing to do** — Caddy renews automatically well before expiry, as long as port 80 stays
+reachable for the ACME HTTP-01 challenge (`infra/firewall/rn-platform-01.rules` already allows
+it). `infra/scripts/production-smoke-test.sh` checks the live expiry date on every run as an
+early warning if auto-renewal ever silently breaks.
+
+If using Cloudflare origin certs instead (see `docs/PRODUCTION_NETWORK_SECURITY.md` §8 for
+that decision): Cloudflare-issued origin certs are typically 15-year validity — this is a
+once-per-network-topology-change concern, not a recurring one. Re-issue from the Cloudflare
+dashboard (SSL/TLS -> Origin Server) and redeploy the cert/key via the `platform` Ansible role
+if it ever needs replacing (domain change, suspected compromise).
+
+**Manual check:**
+```bash
+echo | openssl s_client -connect academy.example.com:443 -servername academy.example.com 2>/dev/null \
+  | openssl x509 -noout -enddate
+```

docs/RUNBOOK_VM_REBUILD.md

@@ -0,0 +1,31 @@
+# Runbook: Platform VM / Lab Host VM Rebuild
+
+Both VMs are designed to be disposable and rebuildable from code — nothing irreplaceable
+should ever live only on either VM's disk outside of what's backed up (`docs/BACKUP_DR.md`).
+
+## rn-platform-01 lost
+
+Academic data survives (it's in Postgres backups, off-host). Instructor/student accounts and
+progress are recoverable; anything since the last backup is not (`docs/BACKUP_DR.md`'s RPO).
+
+```bash
+cd infra/terraform && terraform apply   # recreates the VM from the template
+cd ../ansible && ansible-playbook site.yml -i inventory/production.ini --limit platform --ask-vault-pass
+# Then restore Postgres — docs/RUNBOOK_POSTGRES_RESTORE.md, step 5b variant (fresh DB this time,
+# no need to rename since it's a fresh instance)
+./infra/scripts/production-smoke-test.sh
+```
+
+## rn-lab-host-01 lost
+
+**No academic data is lost** — this host only ever holds disposable Lab containers by design
+(`docs/BACKUP_DR.md`'s Tier distinction). Every enrolled student's Lab needs re-provisioning
+after this, which is a normal "Prepare Labs" click, not a recovery operation.
+
+```bash
+cd infra/terraform && terraform apply
+cd ../ansible && ansible-playbook site.yml -i inventory/production.ini --limit lab_host --ask-vault-pass
+# Re-generate Docker TLS material for the new host if its identity (IP/DNS) changed —
+# docs/RUNBOOK_DOCKER_TLS_ROTATION.md
+# From /admin: Prepare Labs again for any cohort that needs it.
+```

infra/.gitignore

@@ -0,0 +1,29 @@
+# Never commit filled secrets — only *.env.template files are tracked.
+*.env
+!*.env.template
+
+# Docker TLS material — CA/server/client certs and keys, generated by
+# infra/scripts/docker-tls/generate-ca.sh and friends. Templates/READMEs stay tracked.
+infra/docker/tls/*.pem
+infra/docker/tls/*.key
+infra/docker/tls/*.crt
+infra/docker/tls/*.srl
+!infra/docker/tls/**/README.md
+
+# Ansible Vault password files and any decrypted vars
+infra/ansible/.vault_pass
+infra/ansible/**/vault_plain.yml
+
+# Terraform/OpenTofu state and generated vars (may contain resource IDs/IPs)
+infra/terraform/**/*.tfstate
+infra/terraform/**/*.tfstate.*
+infra/terraform/**/.terraform/
+infra/terraform/**/terraform.tfvars
+!infra/terraform/**/terraform.tfvars.example
+
+# Backup output and restore scratch space
+infra/backup/output/
+infra/backup/restore-test/
+
+# Load test results (local artifacts, not source)
+infra/scripts/loadtest/results/

infra/ansible/ansible.cfg

@@ -0,0 +1,10 @@
+[defaults]
+inventory = inventory/production.ini
+roles_path = roles
+host_key_checking = True
+retry_files_enabled = False
+stdout_callback = yaml
+
+[privilege_escalation]
+become = True
+become_method = sudo

infra/ansible/group_vars/production.vault.yml.template

@@ -0,0 +1,22 @@
+# Ansible Vault template — copy to production.vault.yml and encrypt it, NEVER commit the
+# plaintext or the encrypted file to git (infra/.gitignore already excludes
+# **/vault_plain.yml; the encrypted production.vault.yml itself is small/safe to commit IF
+# encrypted — see docs/PRODUCTION_DEPLOYMENT.md §10 for the actual decision made). Ansible
+# Vault chosen over SOPS/age here because it's already bundled with Ansible (no new tool to
+# install/trust) and this project's secret surface is small enough that Vault's simplicity
+# outweighs SOPS's per-key-value granularity — revisit if that changes.
+#
+#   cp production.vault.yml.template production.vault.yml
+#   # fill in real values below
+#   ansible-vault encrypt production.vault.yml
+#
+# Reference a vaulted value from group_vars/production.yml or a role like:
+#   postgres_password: "{{ vault_postgres_password }}"
+
+vault_postgres_password: __REPLACE__
+vault_provisioner_shared_secret: __REPLACE_openssl_rand_hex_32__
+vault_owner_bootstrap_password: __REPLACE__          # used once, by a human, per docs/RUNBOOK_FIRST_DEPLOYMENT.md
+vault_postgres_exporter_password: __REPLACE__
+vault_backup_encryption_passphrase: __REPLACE_openssl_rand_base64_32__
+vault_backup_offsite_credentials: __REPLACE_e.g._Hetzner_Storage_Box_ssh_key_passphrase_or_S3_secret_key__
+vault_grafana_admin_password: __REPLACE__

infra/ansible/group_vars/production.yml

@@ -0,0 +1,33 @@
+# Production variables — referenced by every {{ }} placeholder in infra/docker/, infra/firewall/,
+# and the roles below. Fill in the __PLACEHOLDER__ values once the real server exists
+# (docs/PRODUCTION_DEPLOYMENT.md §1). Nothing here is a secret — real secrets live in
+# infra/environments/production/*.env (gitignored) and Ansible Vault (see
+# infra/ansible/group_vars/production.vault.yml.template), never in this plain-vars file.
+
+env: production
+academy_domain: __REPLACE_e.g._academy.rootnode.example__
+
+# ── Network topology (Phase 4 §5 — placeholders until real hardware exists) ────────────────
+platform_ip: __PLATFORM_VM_IP__
+lab_host_ip: __LAB_HOST_VM_IP__
+management_cidr: __YOUR_ADMIN_ACCESS_CIDR__       # e.g. a VPN/WireGuard subnet, never 0.0.0.0/0
+proxmox_mgmt_cidr: __PROXMOX_HOST_MGMT_CIDR__
+lab_subnet_range: "10.42.0.0/16"                    # matches NO_PROXY_CIDRS already used by the app
+lab_gateway_port_range: "30000:32767"                # see docs/PRODUCTION_NETWORK_SECURITY.md — Gateway-to-Lab reachability follow-up
+
+# ── Storage (Phase 4 §4 — a decision framework, not a final layout; see docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §4)
+rootnode_data_root: /srv/rootnode-data   # expected to be a ZFS dataset mountpoint on the real host
+
+# ── Registry / images (infra/docker/IMAGE_STRATEGY.md) ─────────────────────────────────────
+registry: git.synthossystems.com
+image_tag: __GIT_SHA_OR_RELEASE_TAG__
+
+postgres_exporter_user: rootnode_exporter
+
+# ── Observability (infra/monitoring/) ───────────────────────────────────────────────────────
+prometheus_retention_days: 30
+alertmanager_webhook_url: __REPLACE_e.g._uptime_kuma_or_ntfy_webhook__
+
+# ── Backup (infra/backup/) ───────────────────────────────────────────────────────────────────
+backup_retention_days: 30
+backup_offsite_target: __REPLACE_e.g._Hetzner_Storage_Box_path_or_S3_bucket__

infra/ansible/group_vars/staging.yml

@@ -0,0 +1,22 @@
+# Staging variables — same shape as production.yml, lower stakes. See
+# infra/environments/staging/README.md.
+
+env: staging
+academy_domain: __REPLACE_e.g._staging.academy.rootnode.example_or_unset_for_no_public_TLS__
+
+platform_ip: __STAGING_PLATFORM_IP__
+lab_host_ip: __STAGING_LAB_HOST_IP__
+management_cidr: __YOUR_ADMIN_ACCESS_CIDR__
+proxmox_mgmt_cidr: __PROXMOX_HOST_MGMT_CIDR__
+lab_subnet_range: "10.42.0.0/16"
+
+rootnode_data_root: /srv/rootnode-data-staging
+
+registry: git.synthossystems.com
+image_tag: __STAGING_TAG_e.g._staging-<sha>__
+
+prometheus_retention_days: 7
+alertmanager_webhook_url: __OPTIONAL_staging_can_skip_alerting__
+
+backup_retention_days: 7
+backup_offsite_target: __OPTIONAL_staging_backups_are_lower_priority__

infra/ansible/inventory/production.ini

@@ -0,0 +1,16 @@
+# Production inventory. IPs/hostnames are placeholders — fill in once the real VMs exist
+# (docs/PRODUCTION_DEPLOYMENT.md §1). This file is safe to commit (no secrets, just topology).
+
+[platform]
+rn-platform-01 ansible_host=__PLATFORM_VM_IP__
+
+[lab_host]
+rn-lab-host-01 ansible_host=__LAB_HOST_VM_IP__
+
+[rootnode:children]
+platform
+lab_host
+
+[rootnode:vars]
+ansible_user=rootnode-deploy
+ansible_ssh_common_args='-o StrictHostKeyChecking=yes'

infra/ansible/inventory/staging.ini

@@ -0,0 +1,16 @@
+# Staging inventory — same group shape as production, pointed at homelab VMs/containers or a
+# resource-constrained slice of the dedicated server once it exists (infra/environments/staging/README.md).
+
+[platform]
+rn-platform-staging ansible_host=__STAGING_PLATFORM_IP__
+
+[lab_host]
+rn-lab-host-staging ansible_host=__STAGING_LAB_HOST_IP__
+
+[rootnode:children]
+platform
+lab_host
+
+[rootnode:vars]
+ansible_user=rootnode-deploy
+env=staging

infra/ansible/roles/backup/tasks/main.yml

@@ -0,0 +1,63 @@
+---
+# Deploys the backup script + a daily cron job on rn-platform-01. The script itself
+# (infra/backup/backup-postgres.sh) is identical to what infra/backup/test-backup-restore.sh
+# exercises locally — same code path, just pointed at real production values here.
+
+- name: Install postgresql-client + gpg (backup script dependencies)
+  ansible.builtin.apt:
+    name: [postgresql-client, gnupg]
+    state: present
+
+- name: Create backup directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0700"
+  loop:
+    - /opt/rootnode/backup
+    - "{{ rootnode_data_root }}/backups"
+
+- name: Deploy backup script
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../backup/backup-postgres.sh"
+    dest: /opt/rootnode/backup/backup-postgres.sh
+    mode: "0750"
+
+- name: Deploy restore + verify scripts (not run by cron, kept alongside for the runbook)
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../backup/{{ item }}"
+    dest: "/opt/rootnode/backup/{{ item }}"
+    mode: "0750"
+  loop:
+    - restore-postgres.sh
+    - verify-restore.sh
+
+- name: Deploy the backup env file (PLATFORM_DB_URL, encryption passphrase, offsite target)
+  ansible.builtin.template:
+    src: backup.env.j2
+    dest: /etc/rootnode/backup.env
+    owner: root
+    group: root
+    mode: "0600"
+
+- name: Deploy cron wrapper (sources the env file, writes to syslog too)
+  ansible.builtin.template:
+    src: run-backup.sh.j2
+    dest: /opt/rootnode/backup/run-backup.sh
+    mode: "0750"
+
+- name: Install daily backup cron job
+  ansible.builtin.cron:
+    name: rootnode-postgres-backup
+    user: root
+    minute: "15"
+    hour: "3"
+    job: /opt/rootnode/backup/run-backup.sh
+
+- name: Ensure node-exporter's textfile collector directory exists (backup metrics land here)
+  ansible.builtin.file:
+    path: /var/lib/node-exporter/textfile
+    state: directory
+    mode: "0755"

infra/ansible/roles/backup/templates/backup.env.j2

@@ -0,0 +1,6 @@
+PLATFORM_DB_URL=postgres://rootnode_platform:{{ vault_postgres_password }}@localhost:5432/rootnode_platform
+BACKUP_DIR={{ rootnode_data_root }}/backups
+BACKUP_ENCRYPTION_PASSPHRASE={{ vault_backup_encryption_passphrase }}
+BACKUP_RETENTION_DAYS={{ backup_retention_days }}
+BACKUP_OFFSITE_TARGET={{ backup_offsite_target }}
+METRICS_TEXTFILE_DIR=/var/lib/node-exporter/textfile

infra/ansible/roles/backup/templates/run-backup.sh.j2

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+# Cron wrapper — sources the deployed env file and runs the real backup script, logging to
+# syslog (visible via journalctl and shipped to Loki by promtail like every other unit) in
+# addition to whatever cron itself captures.
+set -euo pipefail
+set -a
+source /etc/rootnode/backup.env
+set +a
+exec /opt/rootnode/backup/backup-postgres.sh 2>&1 | systemd-cat -t rootnode-backup

infra/ansible/roles/common/defaults/main.yml

@@ -0,0 +1,3 @@
+---
+common_timezone: UTC
+common_unattended_upgrades: true

infra/ansible/roles/common/tasks/docker-engine.yml

@@ -0,0 +1,35 @@
+---
+# Shared Docker Engine install — used by both platform and lab-host roles (import_tasks from
+# common, not duplicated). Follows Docker's own official apt-repo installation method.
+
+- name: Add Docker's GPG key
+  ansible.builtin.get_url:
+    url: https://download.docker.com/linux/ubuntu/gpg
+    dest: /etc/apt/keyrings/docker.asc
+    mode: "0644"
+  register: docker_gpg
+  retries: 3
+  delay: 5
+  until: docker_gpg is succeeded
+
+- name: Add Docker apt repository
+  ansible.builtin.apt_repository:
+    repo: "deb [arch={{ ansible_architecture | replace('x86_64','amd64') }} signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable"
+    state: present
+    filename: docker
+
+- name: Install Docker Engine + Compose plugin
+  ansible.builtin.apt:
+    name:
+      - docker-ce
+      - docker-ce-cli
+      - containerd.io
+      - docker-compose-plugin
+    state: present
+    update_cache: true
+
+- name: Enable and start Docker
+  ansible.builtin.systemd:
+    name: docker
+    enabled: true
+    state: started

infra/ansible/roles/common/tasks/main.yml

@@ -0,0 +1,58 @@
+---
+# Baseline hardening applied to both rn-platform-01 and rn-lab-host-01 before anything
+# RootNode-specific — idempotent, safe to re-run (Phase 4 §20 requirement).
+
+- name: Set timezone
+  community.general.timezone:
+    name: "{{ common_timezone }}"
+
+- name: Install base packages
+  ansible.builtin.apt:
+    name:
+      - curl
+      - ca-certificates
+      - gnupg
+      - iptables
+      - iptables-persistent
+      - unattended-upgrades
+      - fail2ban
+      - jq
+    state: present
+    update_cache: true
+    cache_valid_time: 3600
+
+- name: Enable unattended security upgrades
+  ansible.builtin.copy:
+    dest: /etc/apt/apt.conf.d/20auto-upgrades
+    content: |
+      APT::Periodic::Update-Package-Lists "1";
+      APT::Periodic::Unattended-Upgrade "1";
+    mode: "0644"
+  when: common_unattended_upgrades
+
+- name: Create the rootnode-deploy service account (if not the connecting user already)
+  ansible.builtin.user:
+    name: rootnode-deploy
+    shell: /bin/bash
+    groups: docker
+    append: true
+    create_home: true
+  register: deploy_user_result
+  failed_when: false   # non-fatal — some hosts may already use this as ansible_user
+
+- name: Ensure fail2ban is enabled for SSH
+  ansible.builtin.service:
+    name: fail2ban
+    state: started
+    enabled: true
+
+- name: Create /opt/rootnode and /etc/rootnode base directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0750"
+  loop:
+    - /opt/rootnode
+    - /etc/rootnode

infra/ansible/roles/docker-tls/defaults/main.yml

@@ -0,0 +1,6 @@
+---
+# Local path (on the machine running ansible-playbook, NOT on either VM) holding the cert
+# bundle generated by infra/scripts/docker-tls/generate-docker-tls.sh. The CA private key
+# (ca-key.pem) is deliberately never copied to either host by this role — only its public
+# ca.pem, plus the one cert/key pair each host actually needs.
+docker_tls_local_bundle_dir: "{{ lookup('env', 'ROOTNODE_TLS_BUNDLE_DIR') | default('/does/not/exist', true) }}"

infra/ansible/roles/docker-tls/handlers/main.yml

@@ -0,0 +1,6 @@
+---
+- name: restart docker with tls
+  ansible.builtin.systemd:
+    name: docker
+    state: restarted
+    daemon_reload: true

infra/ansible/roles/docker-tls/tasks/main.yml

@@ -0,0 +1,89 @@
+---
+# Deploys Docker TLS material generated OFFLINE by infra/scripts/docker-tls/generate-docker-tls.sh
+# — this role never generates key material itself, and the CA private key never touches either
+# VM (see infra/docker/tls/README.md). Fails loudly if the operator forgot to set
+# ROOTNODE_TLS_BUNDLE_DIR / run the generation script first, rather than silently skipping and
+# leaving the Provisioner unable to reach rn-lab-host-01.
+
+- name: Verify the local TLS bundle directory exists
+  ansible.builtin.stat:
+    path: "{{ docker_tls_local_bundle_dir }}"
+  register: bundle_dir
+  delegate_to: localhost
+  become: false
+
+- name: Fail fast if the TLS bundle wasn't generated
+  ansible.builtin.fail:
+    msg: >-
+      ROOTNODE_TLS_BUNDLE_DIR ({{ docker_tls_local_bundle_dir }}) does not exist. Run
+      infra/scripts/docker-tls/generate-docker-tls.sh first, then re-run with
+      ROOTNODE_TLS_BUNDLE_DIR=<output dir> in the environment.
+  when: not bundle_dir.stat.exists
+
+- name: Create /etc/rootnode/docker-tls
+  ansible.builtin.file:
+    path: /etc/rootnode/docker-tls
+    state: directory
+    owner: root
+    group: root
+    mode: "0700"
+
+- name: Deploy CA cert (public, both hosts)
+  ansible.builtin.copy:
+    src: "{{ docker_tls_local_bundle_dir }}/ca.pem"
+    dest: /etc/rootnode/docker-tls/ca.pem
+    owner: root
+    group: root
+    mode: "0444"
+
+- name: Deploy server cert/key (rn-lab-host-01 only)
+  ansible.builtin.copy:
+    src: "{{ docker_tls_local_bundle_dir }}/{{ item.src }}"
+    dest: "/etc/rootnode/docker-tls/{{ item.dest }}"
+    owner: root
+    group: root
+    mode: "{{ item.mode }}"
+  loop:
+    - { src: server-cert.pem, dest: server-cert.pem, mode: "0444" }
+    - { src: server-key.pem, dest: server-key.pem, mode: "0400" }
+  when: inventory_hostname in groups['lab_host']
+  notify: restart docker with tls
+
+- name: Deploy Provisioner client cert/key (rn-platform-01 only)
+  ansible.builtin.copy:
+    src: "{{ docker_tls_local_bundle_dir }}/{{ item.src }}"
+    dest: "/etc/rootnode/docker-tls/{{ item.dest }}"
+    owner: root
+    group: root
+    mode: "{{ item.mode }}"
+  loop:
+    - { src: provisioner-client-cert.pem, dest: provisioner-client-cert.pem, mode: "0444" }
+    - { src: provisioner-client-key.pem, dest: provisioner-client-key.pem, mode: "0400" }
+  when: inventory_hostname in groups['platform']
+
+- name: Configure dockerd for TLS (rn-lab-host-01 only)
+  ansible.builtin.template:
+    src: daemon.json.j2
+    dest: /etc/docker/daemon.json
+    owner: root
+    group: root
+    mode: "0644"
+  when: inventory_hostname in groups['lab_host']
+  notify: restart docker with tls
+
+- name: Override the Docker systemd unit's ExecStart (clears the default -H fd:// flag)
+  ansible.builtin.file:
+    path: /etc/systemd/system/docker.service.d
+    state: directory
+    mode: "0755"
+  when: inventory_hostname in groups['lab_host']
+
+- name: Deploy the ExecStart override
+  ansible.builtin.template:
+    src: docker-override.conf.j2
+    dest: /etc/systemd/system/docker.service.d/override.conf
+    owner: root
+    group: root
+    mode: "0644"
+  when: inventory_hostname in groups['lab_host']
+  notify: restart docker with tls

infra/ansible/roles/docker-tls/templates/daemon.json.j2

@@ -0,0 +1,13 @@
+{
+  "tls": true,
+  "tlsverify": true,
+  "tlscacert": "/etc/rootnode/docker-tls/ca.pem",
+  "tlscert": "/etc/rootnode/docker-tls/server-cert.pem",
+  "tlskey": "/etc/rootnode/docker-tls/server-key.pem",
+  "hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2376"],
+
+  "storage-driver": "overlay2",
+  "default-address-pools": [
+    { "base": "10.42.0.0/16", "size": 24 }
+  ]
+}

infra/ansible/roles/docker-tls/templates/docker-override.conf.j2

@@ -0,0 +1,7 @@
+# Docker's systemd unit specifies `-H fd://` on ExecStart by default, which conflicts with
+# setting `hosts` in daemon.json ("specified both as a flag and in the configuration file").
+# This override clears ExecStart and re-specifies it with no -H flag, letting daemon.json's
+# `hosts` list be the only source of truth — the documented fix for this exact conflict.
+[Service]
+ExecStart=
+ExecStart=/usr/bin/dockerd

infra/ansible/roles/firewall/handlers/main.yml

@@ -0,0 +1,4 @@
+---
+- name: reload firewall
+  ansible.builtin.command: netfilter-persistent reload
+  changed_when: true

infra/ansible/roles/firewall/tasks/main.yml

@@ -0,0 +1,28 @@
+---
+# Deploys the matching infra/firewall/*.rules file for this host's group and loads it. Same
+# ruleset, same variables as infra/firewall/validate-firewall.sh renders locally — this task
+# just uses Ansible's own templating instead of sed, against the real inventory values.
+
+- name: Determine which ruleset applies to this host
+  ansible.builtin.set_fact:
+    firewall_ruleset_name: "{{ 'rn-platform-01' if inventory_hostname in groups['platform'] else 'rn-lab-host-01' }}"
+
+- name: Render firewall ruleset for this host
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../firewall/{{ firewall_ruleset_name }}.rules"
+    dest: /etc/iptables/rules.v4.rootnode
+    owner: root
+    group: root
+    mode: "0600"
+  notify: reload firewall
+
+- name: Point iptables-persistent at the RootNode ruleset
+  ansible.builtin.file:
+    src: /etc/iptables/rules.v4.rootnode
+    dest: /etc/iptables/rules.v4
+    state: link
+    force: true
+  notify: reload firewall
+
+- name: Flush handlers so the firewall is active before later roles assume it
+  ansible.builtin.meta: flush_handlers

infra/ansible/roles/lab-host/handlers/main.yml

@@ -0,0 +1,6 @@
+---
+- name: restart rootnode-lab-host
+  ansible.builtin.systemd:
+    name: rootnode-lab-host
+    state: restarted
+    daemon_reload: true

infra/ansible/roles/lab-host/tasks/main.yml

@@ -0,0 +1,72 @@
+---
+# rn-lab-host-01: Docker Engine (TLS config comes from the docker-tls role, run before this
+# one), the egress proxy + monitoring agents compose stack, kernel prerequisites for nested
+# k3s in privileged Lab containers, and Docker daemon storage/limits configuration.
+
+- name: Install Docker Engine
+  ansible.builtin.import_tasks: "{{ role_path }}/../common/tasks/docker-engine.yml"
+
+- name: Kernel prerequisites for nested k3s (privileged containers)
+  ansible.posix.sysctl:
+    name: "{{ item.name }}"
+    value: "{{ item.value }}"
+    sysctl_set: true
+    state: present
+    reload: true
+  loop:
+    - { name: net.ipv4.ip_forward, value: "1" }
+    - { name: fs.inotify.max_user_watches, value: "524288" }   # k3s/kubelet watch a lot of files
+    - { name: fs.inotify.max_user_instances, value: "512" }
+    - { name: vm.max_map_count, value: "262144" }               # matches upstream k3s/etcd guidance
+
+- name: Ensure cgroup v2 controllers needed by nested k3s are available
+  ansible.builtin.command: cat /sys/fs/cgroup/cgroup.controllers
+  register: cgroup_controllers
+  changed_when: false
+  failed_when: false
+
+- name: Warn if cgroup v2 controllers look incomplete (informational only)
+  ansible.builtin.debug:
+    msg: >-
+      cgroup.controllers reports: {{ cgroup_controllers.stdout | default('unreadable') }}.
+      Expect at least cpu, memory, pids, io for reliable nested k3s — verify manually if any
+      are missing (see docs/PRODUCTION_VALIDATION_CHECKLIST.md "Lab boot" section).
+  when: cgroup_controllers.rc == 0
+
+- name: Create data/config directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: docker
+    mode: "0750"
+  loop:
+    - /opt/rootnode/lab-host
+    - "{{ rootnode_data_root }}/lab-work"   # disposable — see docs/BACKUP_DR.md, never backed up
+
+- name: Render production docker-compose.yml
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../docker/lab-host/docker-compose.production.yml"
+    dest: /opt/rootnode/lab-host/docker-compose.yml
+    owner: root
+    group: root
+    mode: "0640"
+  notify: restart rootnode-lab-host
+
+- name: Deploy systemd unit
+  ansible.builtin.template:
+    src: rootnode-lab-host.service.j2
+    dest: /etc/systemd/system/rootnode-lab-host.service
+    mode: "0644"
+  notify: restart rootnode-lab-host
+
+- name: Enable and start rootnode-lab-host
+  ansible.builtin.systemd:
+    name: rootnode-lab-host
+    enabled: true
+    state: started
+    daemon_reload: true
+
+# Storage-driver/networking daemon options live in the docker-tls role's daemon.json.j2 —
+# Docker only ever reads ONE /etc/docker/daemon.json, so that's the single source of truth for
+# this host's whole daemon config, not just its TLS settings (see that role's template).

infra/ansible/roles/lab-host/templates/rootnode-lab-host.service.j2

@@ -0,0 +1,17 @@
+[Unit]
+Description=RootNode Academy Lab Host services (egress proxy + monitoring agents)
+Requires=docker.service
+After=docker.service network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+WorkingDirectory=/opt/rootnode/lab-host
+ExecStart=/usr/bin/docker compose -f docker-compose.yml up -d --remove-orphans
+ExecStop=/usr/bin/docker compose -f docker-compose.yml down
+ExecReload=/usr/bin/docker compose -f docker-compose.yml up -d --remove-orphans
+TimeoutStartSec=180
+
+[Install]
+WantedBy=multi-user.target

infra/ansible/roles/monitoring-agent/handlers/main.yml

@@ -0,0 +1,16 @@
+---
+- name: reload prometheus
+  ansible.builtin.command: docker compose -f /opt/rootnode/platform/docker-compose.yml restart prometheus
+  changed_when: true
+
+- name: reload alertmanager
+  ansible.builtin.command: docker compose -f /opt/rootnode/platform/docker-compose.yml restart alertmanager
+  changed_when: true
+
+- name: restart platform promtail
+  ansible.builtin.command: docker compose -f /opt/rootnode/platform/docker-compose.yml restart promtail
+  changed_when: true
+
+- name: restart lab-host promtail
+  ansible.builtin.command: docker compose -f /opt/rootnode/lab-host/docker-compose.yml restart promtail
+  changed_when: true

infra/ansible/roles/monitoring-agent/tasks/main.yml

@@ -0,0 +1,75 @@
+---
+# Deploys infra/monitoring/ configs to whichever host needs them. Named monitoring-agent for
+# historical reasons in this repo's role list, but it deploys both the "agent" pieces
+# (node-exporter/cadvisor/promtail configs, already referenced directly by the platform/
+# lab-host compose files) and the central stack's config (Prometheus/Alertmanager/Loki/Grafana
+# on rn-platform-01 only).
+
+- name: Create /opt/rootnode/monitoring (rn-platform-01)
+  ansible.builtin.file:
+    path: /opt/rootnode/monitoring/rules
+    state: directory
+    mode: "0755"
+  when: inventory_hostname in groups['platform']
+
+- name: Deploy Prometheus config
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../monitoring/prometheus/prometheus.yml.j2"
+    dest: /opt/rootnode/monitoring/prometheus.yml
+    mode: "0644"
+  when: inventory_hostname in groups['platform']
+  notify: reload prometheus
+
+- name: Deploy alert rules
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../monitoring/prometheus/rules/rootnode.yml"
+    dest: /opt/rootnode/monitoring/rules/rootnode.yml
+    mode: "0644"
+  when: inventory_hostname in groups['platform']
+  notify: reload prometheus
+
+- name: Deploy blackbox_exporter config
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../monitoring/prometheus/blackbox.yml"
+    dest: /opt/rootnode/monitoring/blackbox.yml
+    mode: "0644"
+  when: inventory_hostname in groups['platform']
+
+- name: Deploy Alertmanager config
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../monitoring/alertmanager/alertmanager.yml.j2"
+    dest: /opt/rootnode/monitoring/alertmanager.yml
+    mode: "0644"
+  when: inventory_hostname in groups['platform']
+  notify: reload alertmanager
+
+- name: Deploy Loki config
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../monitoring/loki/loki-config.yml"
+    dest: /opt/rootnode/monitoring/loki-config.yml
+    mode: "0644"
+  when: inventory_hostname in groups['platform']
+
+- name: Deploy Grafana provisioning + dashboards
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../monitoring/grafana/{{ item }}/"
+    dest: "/opt/rootnode/monitoring/grafana/{{ item }}/"
+    mode: "0644"
+  loop: [provisioning, dashboards]
+  when: inventory_hostname in groups['platform']
+
+- name: Deploy platform-side promtail config
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../monitoring/promtail/promtail-config.platform.yml"
+    dest: /opt/rootnode/monitoring/promtail-config.yml
+    mode: "0644"
+  when: inventory_hostname in groups['platform']
+  notify: restart platform promtail
+
+- name: Deploy lab-host promtail config
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../monitoring/promtail/promtail-config.lab-host.yml.j2"
+    dest: /opt/rootnode/lab-host/promtail-config.yml
+    mode: "0644"
+  when: inventory_hostname in groups['lab_host']
+  notify: restart lab-host promtail

infra/ansible/roles/platform/handlers/main.yml

@@ -0,0 +1,6 @@
+---
+- name: restart rootnode-platform
+  ansible.builtin.systemd:
+    name: rootnode-platform
+    state: restarted
+    daemon_reload: true

infra/ansible/roles/platform/tasks/main.yml

@@ -0,0 +1,85 @@
+---
+# rn-platform-01: Docker Engine, directory structure, rendered compose file + env files,
+# Postgres tuning overlay, reverse proxy config, and the systemd unit that runs it all.
+# Docker TLS material and the host firewall are separate roles (docker-tls, firewall) run
+# before this one in site.yml — this role assumes both are already in place.
+
+- name: Install Docker Engine
+  ansible.builtin.import_tasks: "{{ role_path }}/../common/tasks/docker-engine.yml"
+
+- name: Create data/config directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: docker
+    mode: "0750"
+  loop:
+    - "{{ rootnode_data_root }}/postgres"
+    - "{{ rootnode_data_root }}/backups"
+    - /opt/rootnode/platform
+    - /opt/rootnode/platform/postgresql.conf.d
+    - /etc/rootnode/provisioner-config
+
+- name: Deploy Postgres tuning overlay
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../docker/postgres/postgresql.conf.d/00-rootnode-baseline.conf"
+    dest: /opt/rootnode/platform/postgresql.conf.d/00-rootnode-baseline.conf
+    mode: "0644"
+  notify: restart rootnode-platform
+
+- name: Deploy provisioner-config overlay ({{ env }})
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../../environments/{{ env }}/provisioner-config/"
+    dest: /etc/rootnode/provisioner-config/
+    mode: "0644"
+  notify: restart rootnode-platform
+
+- name: Deploy per-service env files (from Ansible Vault-backed variables — see group_vars)
+  ansible.builtin.template:
+    src: "{{ item }}.env.j2"
+    dest: "/etc/rootnode/{{ item }}.env"
+    owner: root
+    group: root
+    mode: "0600"
+  loop:
+    - platform
+    - provisioner
+    - gateway
+    - postgres
+  notify: restart rootnode-platform
+
+- name: Render production docker-compose.yml
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../docker/platform/docker-compose.production.yml"
+    dest: /opt/rootnode/platform/docker-compose.yml
+    owner: root
+    group: root
+    mode: "0640"
+  notify: restart rootnode-platform
+
+- name: Render Caddyfile
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../docker/platform/Caddyfile.j2"
+    dest: /opt/rootnode/platform/Caddyfile
+    owner: root
+    group: root
+    mode: "0644"
+  notify: restart rootnode-platform
+
+- name: Deploy systemd unit
+  ansible.builtin.template:
+    src: rootnode-platform.service.j2
+    dest: /etc/systemd/system/rootnode-platform.service
+    mode: "0644"
+  notify: restart rootnode-platform
+
+# The `backup` role (applied after this one in site.yml) deploys the backup script and cron —
+# kept separate since it also has to run correctly on a host that ISN'T freshly deployed.
+
+- name: Enable and start rootnode-platform
+  ansible.builtin.systemd:
+    name: rootnode-platform
+    enabled: true
+    state: started
+    daemon_reload: true

infra/ansible/roles/platform/templates/gateway.env.j2

@@ -0,0 +1,5 @@
+# Rendered by Ansible — see platform.env.j2 for the pattern.
+PLATFORM_API_URL=http://api:4300
+PROVISIONER_URL=http://provisioner:4100
+PROVISIONER_SHARED_SECRET={{ vault_provisioner_shared_secret }}
+GATEWAY_PORT=4200

infra/ansible/roles/platform/templates/platform.env.j2

@@ -0,0 +1,18 @@
+# Rendered by Ansible from infra/environments/{{ env }}/platform.env.template's shape —
+# real values come from group_vars/{{ env }}.yml + the encrypted vault file. Do not hand-edit
+# on the host; re-run the platform role instead.
+
+# service names, not localhost — production runs these on a named Docker bridge network
+# (rn-platform-internal), not network_mode: host like the dev compose file does.
+PLATFORM_DB_URL=postgres://rootnode_platform:{{ vault_postgres_password }}@postgres:5432/rootnode_platform
+PROVISIONER_URL=http://provisioner:4100
+PROVISIONER_SHARED_SECRET={{ vault_provisioner_shared_secret }}
+PLATFORM_API_PORT=4300
+
+STUDENT_SESSION_TTL_HOURS=8
+STUDENT_SESSION_IDLE_HOURS=2
+INSTRUCTOR_SESSION_TTL_HOURS=12
+INSTRUCTOR_SESSION_IDLE_HOURS=2
+
+COOKIE_SECURE=1
+TRUST_PROXY=1

infra/ansible/roles/platform/templates/postgres.env.j2

@@ -0,0 +1,4 @@
+# Rendered by Ansible — see platform.env.j2 for the pattern.
+POSTGRES_USER=rootnode_platform
+POSTGRES_PASSWORD={{ vault_postgres_password }}
+POSTGRES_DB=rootnode_platform

infra/ansible/roles/platform/templates/provisioner.env.j2

@@ -0,0 +1,11 @@
+# Rendered by Ansible — see platform.env.j2 for the pattern.
+PLATFORM_DB_URL=postgres://rootnode_platform:{{ vault_postgres_password }}@postgres:5432/rootnode_platform
+PROVISIONER_SHARED_SECRET={{ vault_provisioner_shared_secret }}
+PROVISIONER_PORT=4100
+PROVISIONER_CONFIG_DIR=/app/config
+
+LAB_IMAGE_REF={{ registry }}/rootnode/lab:{{ image_tag }}
+
+EGRESS_PROXY_PORT=3128
+DNS_RESOLVER=127.0.0.11
+NO_PROXY_CIDRS=10.42.0.0/16,10.43.0.0/16

infra/ansible/roles/platform/templates/rootnode-platform.service.j2

@@ -0,0 +1,17 @@
+[Unit]
+Description=RootNode Academy Platform (docker compose)
+Requires=docker.service
+After=docker.service network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+WorkingDirectory=/opt/rootnode/platform
+ExecStart=/usr/bin/docker compose -f docker-compose.yml up -d --remove-orphans
+ExecStop=/usr/bin/docker compose -f docker-compose.yml down
+ExecReload=/usr/bin/docker compose -f docker-compose.yml up -d --remove-orphans
+TimeoutStartSec=300
+
+[Install]
+WantedBy=multi-user.target

infra/ansible/site.yml

@@ -0,0 +1,52 @@
+---
+# Top-level playbook — full production/staging deployment. Idempotent: re-running against an
+# already-deployed host converges it, doesn't rebuild from scratch (Phase 4 §20 requirement).
+#
+# Usage:
+#   ansible-playbook site.yml -i inventory/production.ini --vault-password-file .vault_pass
+#   ansible-playbook site.yml -i inventory/staging.ini -e env=staging
+#
+# Role order matters: firewall and docker-tls must land before platform/lab-host bring up
+# services that depend on the network policy / TLS material already being in place.
+
+- name: Base hardening — all RootNode hosts
+  hosts: rootnode
+  become: true
+  roles:
+    - common
+
+- name: Firewall policy — all RootNode hosts
+  hosts: rootnode
+  become: true
+  roles:
+    - firewall
+
+- name: Docker TLS — both hosts (server cert on lab_host, client cert on platform)
+  hosts: rootnode
+  become: true
+  roles:
+    - docker-tls
+
+- name: Deploy the Platform VM
+  hosts: platform
+  become: true
+  roles:
+    - platform
+
+- name: Deploy the Lab Host VM
+  hosts: lab_host
+  become: true
+  roles:
+    - lab-host
+
+- name: Deploy monitoring configuration — both hosts
+  hosts: rootnode
+  become: true
+  roles:
+    - monitoring-agent
+
+- name: Deploy backup automation — Platform VM only (that's where Postgres lives)
+  hosts: platform
+  become: true
+  roles:
+    - backup

infra/backup/backup-postgres.sh

@@ -0,0 +1,77 @@
+#!/usr/bin/env bash
+# Postgres backup — Tier 1 (irreplaceable) data per docs/BACKUP_DR.md. Runs via cron on
+# rn-platform-01 (deployed by the `backup` Ansible role), and locally testable against ANY
+# reachable Postgres (that's exactly what infra/backup/test-backup-restore.sh does).
+#
+# Produces one gzip-compressed pg_dump per run, encrypts it, prunes local copies past
+# BACKUP_RETENTION_DAYS, and (if BACKUP_OFFSITE_TARGET is set) rsyncs the encrypted copy
+# off-host. Writes a Prometheus textfile-collector metric so infra/monitoring/'s BackupFailed/
+# BackupAging alerts have real data to fire on.
+#
+# Required env: PLATFORM_DB_URL, BACKUP_DIR, BACKUP_ENCRYPTION_PASSPHRASE
+# Optional env: BACKUP_RETENTION_DAYS (default 30), BACKUP_OFFSITE_TARGET, METRICS_TEXTFILE_DIR
+set -euo pipefail
+
+: "${PLATFORM_DB_URL:?PLATFORM_DB_URL is required}"
+: "${BACKUP_DIR:?BACKUP_DIR is required}"
+: "${BACKUP_ENCRYPTION_PASSPHRASE:?BACKUP_ENCRYPTION_PASSPHRASE is required — see docs/BACKUP_DR.md}"
+BACKUP_RETENTION_DAYS="${BACKUP_RETENTION_DAYS:-30}"
+METRICS_TEXTFILE_DIR="${METRICS_TEXTFILE_DIR:-}"
+
+TS="$(date -u +%Y%m%dT%H%M%SZ)"
+mkdir -p "$BACKUP_DIR"
+DUMP_FILE="$BACKUP_DIR/rootnode-platform-$TS.sql.gz"
+ENC_FILE="$DUMP_FILE.gpg"
+
+write_metric() {
+  # $1 = metric line. Written atomically (tmp+rename) so Prometheus's node-exporter
+  # textfile-collector never scrapes a half-written file.
+  [ -n "$METRICS_TEXTFILE_DIR" ] || return 0
+  mkdir -p "$METRICS_TEXTFILE_DIR"
+  local tmp; tmp="$(mktemp "$METRICS_TEXTFILE_DIR/.rootnode_backup.XXXXXX")"
+  echo "$1" > "$tmp"
+  mv -f "$tmp" "$METRICS_TEXTFILE_DIR/rootnode_backup.prom"
+}
+
+fail() {
+  echo "BACKUP FAILED: $1" >&2
+  write_metric "rootnode_backup_last_run_success 0"
+  exit 1
+}
+trap 'fail "unexpected error at line $LINENO"' ERR
+
+echo "==> Dumping $PLATFORM_DB_URL to $DUMP_FILE"
+pg_dump --format=plain --no-owner --no-privileges "$PLATFORM_DB_URL" | gzip -9 > "$DUMP_FILE"
+
+# Fail loudly on a suspiciously small dump (empty/truncated) rather than "successfully"
+# backing up nothing — a real, if crude, sanity check.
+DUMP_SIZE=$(stat -c%s "$DUMP_FILE" 2>/dev/null || stat -f%z "$DUMP_FILE")
+[ "$DUMP_SIZE" -gt 1024 ] || fail "dump is suspiciously small ($DUMP_SIZE bytes) — refusing to treat this as a real backup"
+
+echo "==> Encrypting (AES-256, symmetric passphrase — see docs/BACKUP_DR.md for the recovery-key handling policy)"
+gpg --batch --yes --passphrase "$BACKUP_ENCRYPTION_PASSPHRASE" --symmetric --cipher-algo AES256 \
+  --output "$ENC_FILE" "$DUMP_FILE"
+rm -f "$DUMP_FILE"   # never leave the plaintext dump on disk once the encrypted copy exists
+
+echo "==> Local backup complete: $ENC_FILE ($(stat -c%s "$ENC_FILE" 2>/dev/null || stat -f%z "$ENC_FILE") bytes)"
+
+if [ -n "${BACKUP_OFFSITE_TARGET:-}" ]; then
+  echo "==> Copying off-host to $BACKUP_OFFSITE_TARGET"
+  # rsync over SSH works for both a Hetzner Storage Box (SSH/SFTP-native) and any other
+  # SSH-reachable target — the adapter point the Phase 4 spec asked for is this one variable,
+  # not a vendor-specific script. An S3-compatible target uses `aws s3 cp` / `rclone` instead;
+  # see docs/BACKUP_DR.md for both forms.
+  rsync -az --timeout=120 "$ENC_FILE" "$BACKUP_OFFSITE_TARGET/" \
+    || fail "off-host copy to $BACKUP_OFFSITE_TARGET failed"
+else
+  echo "==> BACKUP_OFFSITE_TARGET not set — local-only backup (fine for a local test run, NOT fine for production, see docs/BACKUP_DR.md)"
+fi
+
+echo "==> Pruning local backups older than $BACKUP_RETENTION_DAYS days"
+find "$BACKUP_DIR" -name 'rootnode-platform-*.sql.gz.gpg' -mtime "+$BACKUP_RETENTION_DAYS" -delete
+
+write_metric "rootnode_backup_last_run_success 1
+rootnode_backup_last_success_timestamp_seconds $(date +%s)
+rootnode_backup_size_bytes $(stat -c%s "$ENC_FILE" 2>/dev/null || stat -f%z "$ENC_FILE")"
+
+echo "==> Backup succeeded"

infra/backup/restore-postgres.sh

@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+# Restores an encrypted backup produced by backup-postgres.sh into a target database.
+# Deliberately requires an EXPLICIT target — never silently restores over the live production
+# database ("what would you do if this ran the wrong way" is a real disaster-recovery failure
+# mode, so making that impossible to trigger by accident matters more than convenience here).
+#
+# Usage: ./restore-postgres.sh <encrypted-backup-file> <target-db-url>
+#
+# For a REAL production restore, target-db-url should point at a FRESH database created for
+# this purpose (docs/RUNBOOK_POSTGRES_RESTORE.md walks through creating it) — restoring into a
+# database that already has data merges/conflicts rather than cleanly replacing it.
+set -euo pipefail
+
+ENC_FILE="${1:?usage: restore-postgres.sh <encrypted-backup-file> <target-db-url>}"
+TARGET_DB_URL="${2:?usage: restore-postgres.sh <encrypted-backup-file> <target-db-url>}"
+: "${BACKUP_ENCRYPTION_PASSPHRASE:?BACKUP_ENCRYPTION_PASSPHRASE is required to decrypt}"
+
+[ -f "$ENC_FILE" ] || { echo "No such file: $ENC_FILE" >&2; exit 1; }
+
+TMPDIR="$(mktemp -d)"
+trap 'rm -rf "$TMPDIR"' EXIT
+DUMP_FILE="$TMPDIR/restore.sql.gz"
+
+echo "==> Decrypting $ENC_FILE"
+gpg --batch --yes --passphrase "$BACKUP_ENCRYPTION_PASSPHRASE" --decrypt --output "$DUMP_FILE" "$ENC_FILE"
+
+echo "==> Confirming target is reachable"
+psql "$TARGET_DB_URL" -c "SELECT 1" > /dev/null
+
+echo "==> Target database currently has these tables (should be empty/unimportant before you continue):"
+psql "$TARGET_DB_URL" -c "\dt" || true
+read -r -p "Type RESTORE to proceed against $TARGET_DB_URL: " CONFIRM
+[ "$CONFIRM" = "RESTORE" ] || { echo "Aborted."; exit 1; }
+
+echo "==> Restoring"
+gunzip -c "$DUMP_FILE" | psql "$TARGET_DB_URL"
+
+echo "==> Restore complete. Verify with infra/backup/verify-restore.sh $TARGET_DB_URL"

infra/backup/test-backup-restore.sh

@@ -0,0 +1,133 @@
+#!/usr/bin/env bash
+# Local test (Phase 4 §29): a REAL end-to-end backup/restore cycle against disposable local
+# Postgres containers — not a mock. Applies the actual Platform API migration runner (the same
+# code path production uses, platform/api/src/db.js) so the schema under test is genuinely the
+# real one, inserts sample data, backs it up, restores into a SECOND empty Postgres, and
+# verifies the result. Proves the backup/restore MECHANISM works; does NOT prove anything about
+# the real off-host target (Storage Box/S3) — that needs real credentials, marked as
+# hardware/credential-dependent below.
+set -euo pipefail
+cd "$(dirname "$0")/../.."   # repo root
+
+SRC_CONTAINER=rootnode-backup-test-src
+DST_CONTAINER=rootnode-backup-test-dst
+SRC_PORT=55432
+DST_PORT=55433
+TESTDIR="$(mktemp -d)"
+export BACKUP_ENCRYPTION_PASSPHRASE="test-passphrase-not-for-production"
+
+cleanup() {
+  docker rm -f "$SRC_CONTAINER" "$DST_CONTAINER" >/dev/null 2>&1 || true
+  rm -rf "$TESTDIR"
+}
+trap cleanup EXIT
+
+# backup-postgres.sh/restore-postgres.sh assume real postgresql-client + gpg binaries — true on
+# a real deployment host (the `backup` Ansible role installs postgresql-client there). This dev
+# machine may not have psql/pg_dump installed at all, so for THIS TEST ONLY we shim them as
+# thin wrappers around `docker exec` into the two disposable test containers — the scripts
+# under test are otherwise completely unmodified from what production actually runs.
+if ! command -v psql >/dev/null 2>&1 || ! command -v pg_dump >/dev/null 2>&1; then
+  echo "==> psql/pg_dump not found locally — shimming both via docker exec for this test run"
+  SHIM_BIN="$TESTDIR/bin"
+  mkdir -p "$SHIM_BIN"
+  # Both wrappers pick a container from the target port (explicit -p, or embedded in a
+  # postgres://...:PORT/... URL), then re-exec psql/pg_dump INSIDE that container against ITS
+  # OWN localhost:5432 — the host-side port mapping (55432/55433) doesn't exist inside the
+  # container's network namespace, so the connection target itself must be replaced, not just
+  # which container runs the command.
+  cat > "$SHIM_BIN/psql" <<'EOF'
+#!/usr/bin/env bash
+args=(); port=""; want_port=0; skip_next=0
+for a in "$@"; do
+  if [ "$skip_next" = "1" ]; then
+    [ "$want_port" = "1" ] && port="$a"
+    skip_next=0; want_port=0; continue
+  fi
+  if [ "$a" = "-p" ]; then skip_next=1; want_port=1; continue; fi
+  if [ "$a" = "-h" ]; then skip_next=1; want_port=0; continue; fi
+  if [[ "$a" =~ ^postgres://[^:]+:[^@]+@[^:]+:([0-9]+)/ ]]; then
+    port="${BASH_REMATCH[1]}"
+    continue   # drop the URL — reconnect locally inside the container instead
+  fi
+  args+=("$a")
+done
+case "$port" in
+  55433) c="rootnode-backup-test-dst" ;;
+  *) c="rootnode-backup-test-src" ;;
+esac
+# Only forward local stdin (-i) for the one call that actually reads SQL from it (no -c/-tAc
+# in its args, i.e. `psql <target>` fed by a pipe) — every -c/-tAc call is self-contained and
+# must NOT hold docker exec's stdin open, or it silently steals bytes meant for a later `read`
+# elsewhere in the calling script (exactly what broke restore-postgres.sh's confirmation prompt
+# the first time this shim was written).
+needs_stdin=1
+for a in "${args[@]}"; do [ "$a" = "-c" ] || [ "$a" = "-tAc" ] && needs_stdin=0; done
+if [ "$needs_stdin" = "1" ]; then
+  exec docker exec -e PGPASSWORD=postgres -i "$c" psql -h localhost -U postgres "${args[@]}"
+else
+  exec docker exec -e PGPASSWORD=postgres "$c" psql -h localhost -U postgres "${args[@]}"
+fi
+EOF
+  cat > "$SHIM_BIN/pg_dump" <<'EOF'
+#!/usr/bin/env bash
+args=()
+for a in "$@"; do
+  if [[ "$a" =~ ^postgres:// ]]; then continue; fi   # source is always the src container here
+  args+=("$a")
+done
+exec docker exec -e PGPASSWORD=postgres -i rootnode-backup-test-src pg_dump -h localhost -U postgres "${args[@]}"
+EOF
+  chmod +x "$SHIM_BIN/psql" "$SHIM_BIN/pg_dump"
+  export PATH="$SHIM_BIN:$PATH"
+fi
+
+echo "==> Starting source Postgres (the one we'll back up)"
+docker run -d --name "$SRC_CONTAINER" -e POSTGRES_PASSWORD=postgres -p "$SRC_PORT:5432" postgres:16-alpine >/dev/null
+echo "==> Starting destination Postgres (the one we'll restore into)"
+docker run -d --name "$DST_CONTAINER" -e POSTGRES_PASSWORD=postgres -p "$DST_PORT:5432" postgres:16-alpine >/dev/null
+
+echo "==> Waiting for both to accept connections"
+for port in "$SRC_PORT" "$DST_PORT"; do
+  for i in $(seq 1 30); do
+    PGPASSWORD=postgres psql -h localhost -p "$port" -U postgres -c "SELECT 1" >/dev/null 2>&1 && break
+    sleep 1
+  done
+done
+
+SRC_DB_URL="postgres://postgres:postgres@localhost:$SRC_PORT/postgres"
+DST_DB_URL="postgres://postgres:postgres@localhost:$DST_PORT/postgres"
+
+echo "==> Applying the REAL Provisioner migrations first (Platform API's scenario_attempts has a real FK to Provisioner's environments table — documented cross-service ordering dependency, see docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md)"
+( cd platform/provisioner && [ -d node_modules ] || npm install --no-audit --no-fund >/dev/null
+  PLATFORM_DB_URL="$SRC_DB_URL" node -e "require('./src/db').runMigrations().then((m)=>console.log('Applied:', m.join(', ')))" )
+
+echo "==> Applying the REAL Platform API migrations against the source"
+( cd platform/api && [ -d node_modules ] || npm install --no-audit --no-fund >/dev/null
+  PLATFORM_DB_URL="$SRC_DB_URL" node -e "require('./src/db').runMigrations().then((m)=>console.log('Applied:', m.join(', ')))" )
+
+echo "==> Inserting sample data (a student, so the backup has something real to carry)"
+PGPASSWORD=postgres psql -h localhost -p "$SRC_PORT" -U postgres -c \
+  "INSERT INTO students (student_id, display_name) VALUES ('student-backup-test', 'Backup Test Student');"
+
+echo "==> Running backup-postgres.sh against the source"
+BACKUP_DIR="$TESTDIR/backups" PLATFORM_DB_URL="$SRC_DB_URL" ./infra/backup/backup-postgres.sh
+
+ENC_FILE=$(ls "$TESTDIR"/backups/*.sql.gz.gpg | head -1)
+[ -f "$ENC_FILE" ] || { echo "FAIL: no encrypted backup file produced"; exit 1; }
+echo "==> Backup produced: $ENC_FILE"
+
+echo "==> Restoring into the destination (non-interactive: piping the confirmation)"
+echo "RESTORE" | BACKUP_ENCRYPTION_PASSPHRASE="$BACKUP_ENCRYPTION_PASSPHRASE" ./infra/backup/restore-postgres.sh "$ENC_FILE" "$DST_DB_URL"
+
+echo "==> Verifying the restore"
+./infra/backup/verify-restore.sh "$DST_DB_URL"
+
+echo "==> Confirming the sample student actually round-tripped"
+RESTORED_NAME=$(PGPASSWORD=postgres psql -h localhost -p "$DST_PORT" -U postgres -tAc \
+  "SELECT display_name FROM students WHERE student_id = 'student-backup-test'")
+[ "$RESTORED_NAME" = "Backup Test Student" ] || { echo "FAIL: restored student data doesn't match"; exit 1; }
+
+echo
+echo "OK: full backup -> encrypt -> restore -> verify cycle succeeded against real local Postgres."
+echo "NOT TESTED HERE (needs real credentials/hardware): off-host copy to a real Storage Box/S3 target."

infra/backup/verify-restore.sh

@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+# Sanity-checks a restored database — confirms the expected Platform API schema landed and has
+# data, not just that psql exited 0 (a restore that silently imported zero rows would still
+# "succeed" by that measure). Used both by restore-postgres.sh's own suggestion and by
+# test-backup-restore.sh's automated local test.
+set -euo pipefail
+DB_URL="${1:?usage: verify-restore.sh <db-url>}"
+
+EXPECTED_TABLES=(students cohorts instructors sessions scenario_attempts assessment_definitions achievements audit_events)
+FAIL=0
+
+for t in "${EXPECTED_TABLES[@]}"; do
+  exists=$(psql "$DB_URL" -tAc "SELECT to_regclass('public.$t') IS NOT NULL")
+  if [ "$exists" != "t" ]; then
+    echo "MISSING TABLE: $t" >&2
+    FAIL=1
+  fi
+done
+
+if [ "$FAIL" = "1" ]; then
+  echo "FAIL: restored database is missing expected tables" >&2
+  exit 1
+fi
+
+MIGRATIONS_APPLIED=$(psql "$DB_URL" -tAc "SELECT count(*) FROM schema_migrations" 2>/dev/null || echo 0)
+echo "OK: all expected tables present. $MIGRATIONS_APPLIED migrations recorded in schema_migrations."

infra/ci/gitea-workflows/build-and-test.yml

@@ -0,0 +1,111 @@
+# Gitea Actions workflow — build/test on every push to main. Gitea Actions uses the same
+# workflow syntax as GitHub Actions (a compatible runner), so this is portable if the canonical
+# remote ever needs to move CI providers, without being GitHub Actions itself or requiring a
+# paid CI provider. Runs against the canonical Gitea repo only — see docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md
+# §18 for why this never touches the GitHub kubekosh fork.
+#
+# Deploy this file to .gitea/workflows/build-and-test.yml at the repo root to activate it (kept
+# under infra/ci/ in this PR so it's reviewed alongside the rest of the infra package first,
+# per the same "build now, activate deliberately" posture as everything else server-dependent
+# in this phase — enabling Gitea Actions on the instance is itself a one-time admin step that
+# may not be done yet).
+
+name: build-and-test
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  regression:
+    runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: postgres:16-alpine
+        env:
+          POSTGRES_PASSWORD: postgres
+        ports: ["5432:5432"]
+        options: >-
+          --health-cmd "pg_isready -U postgres"
+          --health-interval 10s --health-timeout 5s --health-retries 10
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Node
+        uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+
+      - name: Install dependencies (all first-party services)
+        run: |
+          for d in backend frontend platform/api platform/gateway platform/provisioner; do
+            (cd "$d" && npm ci)
+          done
+
+      - name: Syntax-check every service
+        run: |
+          for d in backend frontend/src platform/api platform/gateway platform/provisioner; do
+            find "$d" -name '*.js' -not -path '*/node_modules/*' -exec node --check {} \;
+          done
+
+      - name: Build frontend
+        run: (cd frontend && npm run build)
+
+      - name: Provisioner migrations against the service Postgres
+        run: |
+          cd platform/provisioner
+          PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5432/postgres \
+            node -e "require('./src/db').runMigrations().then(m=>console.log('Applied:',m.join(', ')))"
+
+      - name: Platform API migrations (depends on Provisioner's, same order as production)
+        run: |
+          cd platform/api
+          PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5432/postgres \
+            node -e "require('./src/db').runMigrations().then(m=>console.log('Applied:',m.join(', ')))"
+
+      - name: Validate infra configs (Phase 4 local tests — no hardware needed)
+        run: |
+          ./infra/scripts/validate-compose.sh
+          ./infra/firewall/validate-firewall.sh
+
+      # The real k3s-boot-dependent e2e suites (platform/test/e2e-phase*.js) need a full Docker
+      # Compose stack with --privileged Lab containers, which a shared CI runner typically
+      # can't grant safely — they stay a LOCAL/staging pre-promotion step
+      # (infra/scripts/staging-smoke-test.sh), not this job. See docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §18.
+
+  build-images:
+    needs: regression
+    if: github.ref == 'refs/heads/main'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Build and tag images
+        run: ./infra/scripts/ci/tag-and-push.sh
+        env:
+          REGISTRY: ${{ vars.REGISTRY }}
+          REGISTRY_USER: ${{ secrets.REGISTRY_USER }}
+          REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
+
+  deploy-staging:
+    needs: build-images
+    if: github.ref == 'refs/heads/main'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Deploy to staging (automatic)
+        run: |
+          cd infra/ansible
+          ansible-playbook site.yml -i inventory/staging.ini \
+            -e "image_tag=staging-${GITHUB_SHA::12}" \
+            --vault-password-file <(echo "$VAULT_PASSWORD")
+        env:
+          VAULT_PASSWORD: ${{ secrets.ANSIBLE_VAULT_PASSWORD }}
+
+      - name: Staging smoke test
+        run: ./infra/scripts/staging-smoke-test.sh
+
+  # Production is NEVER automatic — a human runs infra/scripts/deploy-production.sh after
+  # reviewing the staging smoke test result, per the explicit "manual approval step" preference.
+  # No job here does that; see docs/RUNBOOK_NORMAL_DEPLOY.md for the actual command.

infra/docker/IMAGE_STRATEGY.md

@@ -0,0 +1,49 @@
+# Container image strategy
+
+## Images
+
+| Image | Built from | Registry path |
+|---|---|---|
+| `rootnode/platform-api` | `platform/api/Dockerfile` | `{{ registry }}/rootnode/platform-api` |
+| `rootnode/gateway` | `platform/gateway/Dockerfile` | `{{ registry }}/rootnode/gateway` |
+| `rootnode/provisioner` | `platform/provisioner/Dockerfile` | `{{ registry }}/rootnode/provisioner` |
+| `rootnode/egress-proxy` | `platform/egress-proxy/Dockerfile` | `{{ registry }}/rootnode/egress-proxy` |
+| `rootnode/lab` | root `Dockerfile` (the student Lab image) | `{{ registry }}/rootnode/lab` |
+
+## Tagging
+
+**Never `:latest` in staging or production.** Every build is tagged with:
+
+- the full git SHA (`sha-<40 hex>`) — always, every build, the ground truth
+- `staging-<short-sha>` on every push to `main` that passes CI (moving pointer, staging only)
+- a semantic tag (`vX.Y.Z`) only on an explicit, human-triggered production release
+
+`infra/scripts/ci/tag-and-push.sh` implements this — see §18 in
+`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`.
+
+Deployment configs (`infra/environments/*/​*.env.template`, Ansible `group_vars`) reference the
+tag as a variable (`image_tag`), never a hardcoded value baked into a compose file — rolling
+back is changing one variable and re-running the `platform`/`lab-host` Ansible role, not editing
+a compose file by hand (see `docs/RUNBOOK_ROLLBACK.md`).
+
+## Registry
+
+No paid registry required. Two supported options, chosen at deploy time via `registry` in
+`infra/ansible/group_vars/production.yml`:
+
+1. **Gitea's own built-in container registry** (`git.synthossystems.com`) — zero new
+   infrastructure, already authenticated the same way the repo itself is. Preferred default.
+2. **A self-hosted registry** (e.g. a `registry:2` container on rn-platform-01 itself, or
+   elsewhere in the homelab) — a documented fallback if the Gitea registry isn't enabled on
+   this instance.
+
+Either way: private registry, credentials pulled the same way any other production secret is
+(`docs/PRODUCTION_DEPLOYMENT.md` §10), never a public Docker Hub push of anything beyond what's
+already public (this project doesn't publish anything to Docker Hub in Phase 4).
+
+## Rollback
+
+Every image is content-addressable by SHA tag, so "roll back" is always available as long as
+the previous tag hasn't been garbage-collected from the registry — see
+`docs/RUNBOOK_ROLLBACK.md` for the exact procedure (re-point `image_tag`, re-run the Ansible
+role, verify).

infra/docker/lab-host/docker-compose.production.yml

@@ -0,0 +1,68 @@
+# Production compose for rn-lab-host-01. Deployed by infra/ansible/roles/lab-host/ to
+# /opt/rootnode/lab-host/docker-compose.yml, run as a systemd unit
+# (rootnode-lab-host.service.j2). Runs exactly one long-lived service: the shared egress proxy
+# every student Lab's outbound traffic passes through (platform/egress-proxy/, unchanged since
+# Phase 1). Student Lab containers themselves are NOT part of this compose file — they're
+# created/destroyed individually by the Provisioner over the remote Docker TLS endpoint
+# (dockerd itself, not compose), same mechanism as dev, just remote.
+
+networks:
+  rn-egress:
+    driver: bridge
+
+services:
+  egress-proxy:
+    image: "{{ registry }}/rootnode/egress-proxy:{{ image_tag }}"
+    restart: unless-stopped
+    ports:
+      - "3128:3128"   # reachable from every per-Lab Docker network on this host — restricted
+                       # to those networks by the Provisioner's per-environment DOCKER-USER
+                       # chains, not by a published-port firewall rule (there is no sensible
+                       # single source IP to restrict this to; the egress proxy's OWN allowlist
+                       # is the actual control, see platform/egress-proxy/allowlist.txt)
+    networks: [rn-egress]
+
+  node-exporter:
+    image: prom/node-exporter:latest
+    restart: unless-stopped
+    network_mode: host
+    pid: host
+    volumes:
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/rootfs:ro
+    command:
+      - "--path.procfs=/host/proc"
+      - "--path.sysfs=/host/sys"
+      - "--path.rootfs=/rootfs"
+      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
+
+  cadvisor:
+    image: gcr.io/cadvisor/cadvisor:latest
+    restart: unless-stopped
+    privileged: true
+    devices:
+      - /dev/kmsg
+    volumes:
+      - /:/rootfs:ro
+      - /var/run:/var/run:ro
+      - /sys:/sys:ro
+      - /var/lib/docker/:/var/lib/docker:ro
+    ports:
+      - "8080:8080"   # scraped remotely by Prometheus on rn-platform-01 — restricted to that
+                       # one source IP by the host firewall (rn-lab-host-01.rules), not by
+                       # binding to localhost, since Prometheus isn't local on this host.
+
+  promtail:
+    image: grafana/promtail:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/lab-host/promtail-config.yml:/etc/promtail/config.yml:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /var/log/journal:/var/log/journal:ro
+      - /etc/machine-id:/etc/machine-id:ro
+    command: ["-config.file=/etc/promtail/config.yml"]
+    # host networking so promtail can reach Loki on rn-platform-01 over the network — this
+    # traffic is OUTBOUND from rn-lab-host-01 (log push, client-initiated), which the host
+    # firewall's default-ACCEPT OUTPUT chain already allows; nothing new needed there.
+    network_mode: host

infra/docker/platform/Caddyfile.j2

@@ -0,0 +1,38 @@
+# Caddy 2 — single stable RootNode Academy origin, automatic HTTPS (Let's Encrypt) unless
+# Cloudflare origin certs are used instead (see docs/PRODUCTION_NETWORK_SECURITY.md §8 for the
+# Cloudflare-vs-direct-LE decision). Rendered by the `platform` Ansible role from
+# infra/ansible/group_vars/production.yml's `academy_domain`.
+#
+# The student/instructor path proxies to the Gateway (platform/gateway) — the sole
+# student-facing entrypoint, unchanged architectural rule from Phase 2A onward. Nothing here
+# reaches the Provisioner, Postgres, or Docker TLS directly; those have no public route at all.
+# Grafana is reachable at /grafana/ but restricted to the management CIDR even though it has
+# its own login — internal observability tooling shouldn't be internet-facing at all, same
+# posture as everything else that isn't the student/instructor product surface.
+
+{{ academy_domain }} {
+	encode gzip zstd
+
+	handle_path /grafana/* {
+		@notmanagement not remote_ip {{ management_cidr }}
+		respond @notmanagement 403
+		reverse_proxy grafana:3000
+	}
+
+	# WebSocket terminal traffic (shell-ws) needs the connection upgraded, not buffered —
+	# Caddy's reverse_proxy handles this automatically for Upgrade: websocket requests, no
+	# special directive needed, but flush_interval -1 avoids buffering SSE/long-poll-shaped
+	# responses too. `gateway` is the Docker Compose service name (rn-platform-internal
+	# network), not localhost — Caddy no longer shares host networking with the Gateway in
+	# production, unlike the dev compose file.
+	reverse_proxy gateway:4200 {
+		flush_interval -1
+	}
+
+	log {
+		output file /var/log/caddy/access.log {
+			roll_size 100mb
+			roll_keep 7
+		}
+	}
+}

infra/docker/platform/docker-compose.production.yml

@@ -0,0 +1,212 @@
+# Production compose for rn-platform-01. Deployed by infra/ansible/roles/platform/ to
+# /opt/rootnode/platform/docker-compose.yml, run as a systemd-managed `docker compose` unit
+# (infra/ansible/roles/platform/templates/rootnode-platform.service.j2).
+#
+# Differs from platform/docker-compose.dev.yml in the ways that actually matter for
+# production, and nothing else:
+#   - images are pulled from a registry by DIGEST-PINNED TAG (never :latest, never built
+#     in-place) — see infra/docker/IMAGE_STRATEGY.md
+#   - Postgres data and backups live on a bind-mounted ZFS dataset, not an anonymous volume
+#   - a reverse proxy (Caddy) terminates TLS and is the only container with a port published to
+#     the internet; a few other services publish loopback-only or single-source-IP-restricted
+#     ports for specific operational needs (Postgres for the host's own backup cron, Loki for
+#     rn-lab-host-01's log shipper) — see each service's own comment
+#   - no `network_mode: host` on gateway/provisioner — production uses a named bridge network,
+#     since rn-platform-01 no longer needs to reach ephemeral container IPs directly (those live
+#     on rn-lab-host-01, reached over the network, not shared Docker networking)
+#   - the api/gateway/provisioner containers still don't reach the internet except through
+#     what's explicitly allowed by the host firewall (infra/firewall/), not Docker's own
+#     default bridge NAT behavior
+#
+# All image tags, ports, and paths below are Ansible Jinja2 variables — this file is a
+# template rendered by the `platform` role, not run directly. See
+# infra/ansible/group_vars/production.yml for the values.
+
+networks:
+  rn-platform-internal:
+    driver: bridge
+    internal: false   # postgres/api/gateway/provisioner need it false to reach rn-lab-host-01
+                       # over the network; the HOST firewall (not Docker) is what actually
+                       # restricts what that traffic can reach — see docs/PRODUCTION_NETWORK_SECURITY.md
+
+volumes:
+  postgres-data:
+    driver: local
+    driver_opts:
+      type: none
+      o: bind
+      device: "{{ rootnode_data_root }}/postgres"
+  caddy-data:
+  caddy-config:
+  prometheus-data:
+  alertmanager-data:
+  grafana-data:
+  loki-data:
+
+services:
+  postgres:
+    image: postgres:16-alpine
+    restart: unless-stopped
+    env_file: /etc/rootnode/postgres.env
+    volumes:
+      - postgres-data:/var/lib/postgresql/data
+      - /opt/rootnode/platform/postgresql.conf.d:/etc/postgresql/conf.d:ro
+    # include_dir appends our tuning files to the image's own auto-generated postgresql.conf —
+    # no need to replace or template the base config file at all.
+    command: ["postgres", "-c", "include_dir=/etc/postgresql/conf.d"]
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-rootnode_platform}"]
+      interval: 10s
+      timeout: 5s
+      retries: 10
+    networks: [rn-platform-internal]
+    ports:
+      - "127.0.0.1:5432:5432"   # loopback-only — reachable from other containers on
+                                 # rn-platform-internal AND from the host's own backup cron
+                                 # (infra/backup/), never from another host on the network, let
+                                 # alone the internet. This is NOT the same as the "Postgres
+                                 # never exposed on this VM's external interface" rule in
+                                 # rn-platform-01.rules — that rule (and the host firewall in
+                                 # general) still applies to this port; 127.0.0.1 binding is a
+                                 # second, independent reason nothing off-host can ever reach it.
+
+  provisioner:
+    image: "{{ registry }}/rootnode/provisioner:{{ image_tag }}"
+    restart: unless-stopped
+    env_file: /etc/rootnode/provisioner.env
+    volumes:
+      - /etc/rootnode/provisioner-config:/app/config:ro
+      - /etc/rootnode/docker-tls:/app/docker-tls:ro
+    depends_on:
+      postgres:
+        condition: service_healthy
+    networks: [rn-platform-internal]
+    cap_add: [NET_ADMIN, NET_RAW]   # only needed while a lab_host row is still a local Unix
+                                    # socket (dev); harmless no-op against a remote TLS endpoint
+
+  api:
+    image: "{{ registry }}/rootnode/platform-api:{{ image_tag }}"
+    restart: unless-stopped
+    env_file: /etc/rootnode/platform.env
+    depends_on:
+      postgres:
+        condition: service_healthy
+      provisioner:
+        condition: service_started
+    networks: [rn-platform-internal]
+
+  gateway:
+    image: "{{ registry }}/rootnode/gateway:{{ image_tag }}"
+    restart: unless-stopped
+    env_file: /etc/rootnode/gateway.env
+    depends_on: [provisioner, api]
+    networks: [rn-platform-internal]
+
+  reverse-proxy:
+    image: caddy:2-alpine
+    restart: unless-stopped
+    ports:
+      - "443:443"
+      - "80:80"   # ACME HTTP-01 challenge / plain-HTTP-to-HTTPS redirect only
+    volumes:
+      - /opt/rootnode/platform/Caddyfile:/etc/caddy/Caddyfile:ro
+      - caddy-data:/data
+      - caddy-config:/config
+    depends_on: [gateway]
+    networks: [rn-platform-internal]
+
+  # ── Observability agents (infra/monitoring/) ────────────────────────────────────────────
+  node-exporter:
+    image: prom/node-exporter:latest
+    restart: unless-stopped
+    network_mode: host   # needs host-level metrics; this is metrics-only, not app traffic
+    pid: host
+    volumes:
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/rootfs:ro
+      - /var/lib/node-exporter/textfile:/textfile:ro   # backup-postgres.sh writes its metric here
+    command:
+      - "--path.procfs=/host/proc"
+      - "--path.sysfs=/host/sys"
+      - "--path.rootfs=/rootfs"
+      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
+      - "--collector.textfile.directory=/textfile"
+
+  postgres-exporter:
+    image: prometheuscommunity/postgres-exporter:latest
+    restart: unless-stopped
+    environment:
+      DATA_SOURCE_NAME: "postgresql://{{ postgres_exporter_user }}:{{ vault_postgres_exporter_password }}@postgres:5432/rootnode_platform?sslmode=disable"
+    depends_on: [postgres]
+    networks: [rn-platform-internal]
+
+  blackbox-exporter:
+    image: prom/blackbox-exporter:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
+    networks: [rn-platform-internal]
+
+  prometheus:
+    image: prom/prometheus:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - /opt/rootnode/monitoring/rules:/etc/prometheus/rules:ro
+      - prometheus-data:/prometheus
+    command:
+      - "--config.file=/etc/prometheus/prometheus.yml"
+      - "--storage.tsdb.retention.time={{ prometheus_retention_days }}d"
+    networks: [rn-platform-internal]
+    ports:
+      - "127.0.0.1:9090:9090"   # bound to loopback only — access via SSH tunnel or Grafana, never public
+
+  alertmanager:
+    image: prom/alertmanager:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
+      - alertmanager-data:/alertmanager
+    networks: [rn-platform-internal]
+
+  grafana:
+    image: grafana/grafana-oss:latest
+    restart: unless-stopped
+    environment:
+      GF_SECURITY_ADMIN_PASSWORD: "{{ vault_grafana_admin_password }}"
+      GF_SERVER_ROOT_URL: "https://{{ academy_domain }}/grafana/"
+      GF_SERVER_SERVE_FROM_SUB_PATH: "true"
+      GF_USERS_ALLOW_SIGN_UP: "false"
+    volumes:
+      - grafana-data:/var/lib/grafana
+      - /opt/rootnode/monitoring/grafana/provisioning:/etc/grafana/provisioning:ro
+      - /opt/rootnode/monitoring/grafana/dashboards:/var/lib/grafana/dashboards:ro
+    networks: [rn-platform-internal]
+    # No published port — reached only via the reverse proxy's /grafana/ path (Caddyfile),
+    # never a direct public listener. See docs/PRODUCTION_NETWORK_SECURITY.md §8.
+
+  loki:
+    image: grafana/loki:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/loki-config.yml:/etc/loki/local-config.yaml:ro
+      - loki-data:/loki
+    command: ["-config.file=/etc/loki/local-config.yaml"]
+    networks: [rn-platform-internal]
+    ports:
+      - "3100:3100"   # reachable from rn-lab-host-01's promtail over the network — restricted
+                       # to that one source IP by the host firewall (rn-platform-01.rules), same
+                       # pattern as Docker TLS's 2376. Not published to the internet.
+
+  promtail:
+    image: grafana/promtail:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/promtail-config.yml:/etc/promtail/config.yml:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /var/log/journal:/var/log/journal:ro
+      - /etc/machine-id:/etc/machine-id:ro
+    command: ["-config.file=/etc/promtail/config.yml"]
+    networks: [rn-platform-internal]
+    depends_on: [loki]

infra/docker/postgres/postgresql.conf.d/00-rootnode-baseline.conf

@@ -0,0 +1,30 @@
+# Conservative production tuning baseline — deliberately NOT hand-tuned to a specific RAM
+# allocation before rn-platform-01's real resources are known (Phase 4 instruction: do not
+# over-tune before actual hardware exists). Sized for a modest ~4GB Postgres allocation, safe
+# defaults everywhere else. Revisit once VM RAM is final — see docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §11.
+
+shared_buffers = 1GB
+effective_cache_size = 3GB
+maintenance_work_mem = 256MB
+work_mem = 16MB
+
+max_connections = 100
+
+# WAL — safe defaults for a single-instance setup with pg_dump/pg_basebackup-style backups
+# (infra/backup/). Not configured for streaming replication (out of scope this phase — see
+# docs/BACKUP_DR.md's DR model for why pg_dump + periodic pg_basebackup is the chosen approach,
+# not a hot standby).
+wal_level = replica
+max_wal_size = 2GB
+min_wal_size = 512MB
+checkpoint_completion_target = 0.9
+
+# Query logging — slow queries only, never full statement logging (would capture bind
+# parameters that can include session tokens' hashes / PII in edge cases). See docs/OBSERVABILITY.md.
+log_min_duration_statement = 500
+log_line_prefix = '%m [%p] %q%u@%d '
+log_checkpoints = on
+log_connections = on
+log_disconnections = on
+
+# Autovacuum stays on defaults — no reason to disable it for this workload size.

infra/docker/tls/README.md

@@ -0,0 +1,19 @@
+# Docker TLS material (generated, not committed)
+
+Generate with `infra/scripts/docker-tls/generate-docker-tls.sh`. Deploy with the `docker-tls`
+Ansible role. Verify with `infra/scripts/docker-tls/verify-docker-tls.sh`.
+
+**Nothing in this directory is ever committed** — see `infra/.gitignore`. This README (and only
+this README) is tracked, so the directory itself exists in the repo as documentation of what
+belongs here operationally.
+
+Files that will exist here on a real deployment host (never in git):
+
+```
+ca.pem                              # public — safe to distribute to both VMs
+ca-key.pem                          # PRIVATE — offline/secrets-vault only, never on either VM
+server-cert.pem / server-key.pem                    # rn-lab-host-01's dockerd
+provisioner-client-cert.pem / provisioner-client-key.pem   # rn-platform-01's Provisioner
+```
+
+Rotation procedure: `docs/RUNBOOK_DOCKER_TLS_ROTATION.md`.

infra/environments/README.md

@@ -0,0 +1,21 @@
+# Environment configuration
+
+Three environments, explicitly separated so production never silently inherits a dev default:
+
+| | dev | staging | production |
+|---|---|---|---|
+| Where it runs | any dev machine, `platform/docker-compose.dev.yml` | homelab or a resource-limited slice of the dedicated server once it exists | rn-platform-01 / rn-lab-host-01 on Proxmox |
+| `COOKIE_SECURE` | unset (plain HTTP) | `1` if served over HTTPS, else unset | `1` — required |
+| `TRUST_PROXY` | unset | `1` if fronted by a proxy | `1` — required, real reverse proxy in front |
+| `ADMIN`/owner bootstrap | `platform/api/bootstrap-owner.js` manually | same, against staging DB | same, against production DB, run once by a human, credential handled per `docs/PRODUCTION_DEPLOYMENT.md` §10 |
+| Secrets | `platform/.env` (gitignored, throwaway) | `infra/environments/staging/*.env` (gitignored, real but low-stakes) | `infra/environments/production/*.env` (gitignored, real, backed up per `docs/BACKUP_DR.md`) |
+| Lab image | built locally, `:test` tag | tagged build from CI, `:staging-<sha>` | tagged release, `:<sha>` or `:vX.Y.Z`, never `:latest` |
+
+Each environment directory holds `*.env.template` files (tracked) — copy to `*.env` (gitignored
+via `infra/.gitignore`) and fill in real values. **Never commit a filled `.env` file.** See
+`docs/PRODUCTION_DEPLOYMENT.md` §10 for how production secrets are actually generated, stored,
+and rotated (not just "don't commit them").
+
+`dev/` is intentionally thin — it just points back at the existing
+`platform/docker-compose.dev.yml` workflow (Phase 1–3B, unchanged); there was nothing new to
+build there for Phase 4.

infra/environments/dev/README.md

@@ -0,0 +1,7 @@
+# Dev environment
+
+Nothing new here for Phase 4 — dev still means `platform/docker-compose.dev.yml` +
+`platform/.env` (gitignored), exactly as built in Phases 1–3B. See `platform/README.md`.
+
+This directory exists only so `infra/environments/` visibly has all three tiers, not because
+dev needed new infrastructure code.

infra/environments/production/capacity.yml.example

@@ -0,0 +1,18 @@
+# Capacity planning inputs (Phase 4 §25). Copy to capacity.yml and fill in REAL measured values
+# once the actual Hetzner host and a real load test (infra/scripts/loadtest/) exist. The
+# numbers below are illustrative placeholders for a ~16c/32t/128GB host — do not treat them as
+# a real recommendation; run infra/scripts/capacity-model.js against real ones once measured.
+_placeholder: true
+
+host_cpu_cores: 32          # threads, if the host does hyperthreading — matches how Docker/cgroups count them
+host_mem_mb: 131072          # 128 GB
+
+platform_reserved_cpu_cores: 4    # rn-platform-01's VM allocation, reserved off the Lab Host's own pool
+platform_reserved_mem_mb: 16384
+
+proxmox_overhead_pct: 5      # hypervisor + host OS overhead — a conservative starting estimate
+
+measured_lab_cpu_cores: 2    # from infra/scripts/loadtest/sample-resources.sh peak CPU per Lab
+measured_lab_mem_mb: 2048    # from the same — peak memory per Lab
+
+safety_margin_pct: 20        # headroom kept unallocated even at the "recommended" threshold

infra/environments/production/gateway.env.template

@@ -0,0 +1,8 @@
+# Production environment template for rn-platform-01's `gateway` service.
+# `api`/`provisioner` are Docker Compose service names, not localhost — production runs these
+# on a named bridge network (rn-platform-internal), not network_mode: host like dev.
+
+PLATFORM_API_URL=http://api:4300
+PROVISIONER_URL=http://provisioner:4100
+PROVISIONER_SHARED_SECRET=__REPLACE_same_value_as_platform.env_and_provisioner.env__
+GATEWAY_PORT=4200

infra/environments/production/platform.env.template

@@ -0,0 +1,38 @@
+# Production environment template for rn-platform-01's `api` service.
+# Copy to platform.env OUTSIDE git (see infra/backup/README.md "secrets" and
+# docs/PRODUCTION_DEPLOYMENT.md §10), fill in real values, never commit the filled file.
+# infra/.gitignore already excludes *.env (only *.env.template is tracked).
+
+# ── Postgres ────────────────────────────────────────────────────────────────────────────
+# `postgres`/`provisioner` are Docker Compose service names, not localhost — production runs
+# these on a named bridge network (rn-platform-internal), not network_mode: host like dev.
+PLATFORM_DB_URL=postgres://rootnode_platform:__REPLACE_WITH_STRONG_PASSWORD__@postgres:5432/rootnode_platform
+
+# ── Provisioner ─────────────────────────────────────────────────────────────────────────
+PROVISIONER_URL=http://provisioner:4100
+PROVISIONER_SHARED_SECRET=__REPLACE_openssl_rand_hex_32__
+
+# ── Listen ──────────────────────────────────────────────────────────────────────────────
+PLATFORM_API_PORT=4300
+
+# ── Sessions (production values — do NOT inherit dev defaults) ────────────────────────────
+STUDENT_SESSION_TTL_HOURS=8
+STUDENT_SESSION_IDLE_HOURS=2
+INSTRUCTOR_SESSION_TTL_HOURS=12
+INSTRUCTOR_SESSION_IDLE_HOURS=2
+
+# ── Cookies / proxy — MUST be set for production, unlike dev ──────────────────────────────
+# Secure requires the origin to actually be HTTPS end-to-end (see docs/PRODUCTION_NETWORK_SECURITY.md §8).
+COOKIE_SECURE=1
+# Only set to 1 once a trusted reverse proxy (Cloudflare/nginx/Caddy, see infra/docker/reverse-proxy/)
+# is actually in front of the Gateway — otherwise req.ip becomes spoofable via X-Forwarded-For.
+TRUST_PROXY=1
+
+# ── Frontend (baked into the platform/api image at build time; listed here for reference) ─
+# FRONTEND_DIST_DIR=/frontend-dist
+
+# ── Future OIDC (Phase 3A adapter — not wired yet, see docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md §16) ─
+# OIDC_ISSUER_URL=
+# OIDC_CLIENT_ID=
+# OIDC_CLIENT_SECRET=__do_not_commit__
+# OIDC_REDIRECT_URL=https://academy.example.com/api/auth/oidc/callback

infra/environments/production/postgres.env.template

@@ -0,0 +1,14 @@
+# Production environment template for rn-platform-01's `postgres` service.
+# Postgres itself binds only to localhost/the platform-internal Docker network — never exposed
+# beyond rn-platform-01, see docs/PRODUCTION_NETWORK_SECURITY.md §6.
+
+POSTGRES_USER=rootnode_platform
+POSTGRES_PASSWORD=__REPLACE_WITH_STRONG_PASSWORD__
+POSTGRES_DB=rootnode_platform
+
+# Tuning baseline — conservative, meant to be revisited once rn-platform-01's real allocated
+# RAM is known (see docs/production-postgres section in docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md
+# §11). Applied via infra/docker/postgres/postgresql.conf.d/ — do not hand-edit the container.
+# POSTGRES_SHARED_BUFFERS=1GB
+# POSTGRES_EFFECTIVE_CACHE_SIZE=3GB
+# POSTGRES_MAX_CONNECTIONS=100

infra/environments/production/provisioner-config/README.md

@@ -0,0 +1,15 @@
+# Production Provisioner config overlay
+
+Deployed by the `platform` Ansible role to `/etc/rootnode/provisioner-config/` on
+rn-platform-01 (`PROVISIONER_CONFIG_DIR`), replacing the dev-only files shipped in
+`platform/provisioner/config/`.
+
+- `lab-hosts.yaml` — points at the real rn-lab-host-01 VM over TLS instead of the dev Unix
+  socket. TLS paths reference material generated by `infra/scripts/docker-tls/` and deployed by
+  the `docker-tls` Ansible role — **never checked into this repo** (see
+  `infra/docker/tls/.gitignore`).
+- `resource-policy.yaml` — deliberately still the Phase 1 dev defaults, unchanged, with a
+  loud comment explaining why. Per the explicit Phase 4 instruction: do not hard-code final
+  per-Lab CPU/RAM numbers before the real host is benchmarked (`docs/CAPACITY_PLANNING.md`).
+  Whoever runs the first production load test (`infra/scripts/loadtest/`) updates this file
+  with real measured numbers, not this deployment.

infra/environments/production/provisioner-config/lab-hosts.yaml

@@ -0,0 +1,16 @@
+# Production lab_hosts seed — points at the real rn-lab-host-01 VM instead of Phase 1's dev
+# Unix socket. Same mechanism the Provisioner has had since Phase 1
+# (platform/provisioner/config/lab-hosts.seed.yaml), just pointed at a real remote endpoint —
+# no application code changes required for this, see docs/PRODUCTION_NETWORK_SECURITY.md §7.
+#
+# {{ lab_host_ip }} is an Ansible template variable — see infra/ansible/group_vars/production.yml.
+# Rendered by the `platform` role; the raw file here keeps the placeholder for readability.
+
+lab_hosts:
+  - lab_host_id: labhost-01
+    display_name: "RootNode Lab Host VM 01 (production)"
+    endpoint: "tcp://{{ lab_host_ip }}:2376"
+    tls_ca_path: /etc/rootnode/docker-tls/ca.pem
+    tls_cert_path: /etc/rootnode/docker-tls/provisioner-client-cert.pem
+    tls_key_path: /etc/rootnode/docker-tls/provisioner-client-key.pem
+    enabled: true

infra/environments/production/provisioner-config/resource-policy.yaml

@@ -0,0 +1,25 @@
+# PRODUCTION OVERLAY — placeholder, deliberately unchanged from the Phase 1 dev defaults.
+# Do not hand-tune these before the real host exists (docs/CAPACITY_PLANNING.md, Phase 4 §25).
+# After the first real load test on rn-lab-host-01 (infra/scripts/loadtest/), replace this
+# comment block and the numbers below with actual measured values, then redeploy via the
+# `platform` Ansible role (idempotent — re-running just re-renders this file).
+#
+# Phase 1 resource policy — config-driven, not hard-coded (decision #4).
+# Target host class for planning is ~16 cores/32 threads, 128GB RAM, SSD/NVMe.
+
+default:
+  cpu_limit: 2.0          # vCPUs
+  mem_limit_mb: 4096
+  pids_limit: 512
+  storage_limit_mb: 8192  # only enforced if the host's Docker storage driver supports
+                           # --storage-opt size= (see known-limitations in the SOP)
+
+idle_threshold_minutes: 30       # past this: log a warning event only
+abandoned_threshold_minutes: 120 # past this: reaper destroys for real
+
+reaper_poll_interval_seconds: 60
+
+health_check:
+  path: /api/health
+  timeout_ms: 240000   # k3s cold start in a nested container can be slow — matches the
+                        # ~60-attempt wait already in scripts/entrypoint.sh

infra/environments/production/provisioner.env.template

@@ -0,0 +1,20 @@
+# Production environment template for rn-platform-01's `provisioner` service.
+# See platform.env.template for the secrets-handling convention.
+#
+# Docker-connectivity config (lab_hosts) and resource limits are NOT env vars — they're YAML
+# files the Provisioner already loads from PROVISIONER_CONFIG_DIR (Phase 1 design, unchanged).
+# Production overlays for both live at infra/environments/production/provisioner-config/ — see
+# that directory's README before touching resource-policy.yaml numbers.
+
+PLATFORM_DB_URL=postgres://rootnode_platform:__REPLACE_WITH_STRONG_PASSWORD__@localhost:5432/rootnode_platform
+PROVISIONER_SHARED_SECRET=__REPLACE_same_value_as_platform.env__
+PROVISIONER_PORT=4100
+PROVISIONER_CONFIG_DIR=/etc/rootnode/provisioner-config
+
+# Real Lab image, tagged (never :latest in production — see infra/docker/IMAGE_STRATEGY.md)
+LAB_IMAGE_REF=registry.example.com/rootnode/lab:__GIT_SHA_OR_TAG__
+
+# ── Egress proxy on rn-lab-host-01 ─────────────────────────────────────────────────────────
+EGRESS_PROXY_PORT=3128
+DNS_RESOLVER=127.0.0.11
+NO_PROXY_CIDRS=10.42.0.0/16,10.43.0.0/16

infra/environments/staging/README.md

@@ -0,0 +1,16 @@
+# Staging environment
+
+Same shape as production (`docs/PRODUCTION_DEPLOYMENT.md` applies to both — pass `-e env=staging`
+to the Ansible playbooks), lower stakes: separate database name/port, staging-tagged images, and
+`COOKIE_SECURE` only if staging actually sits behind HTTPS. See `docs/CI_CD` section in
+`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §18 for how a Gitea Actions build lands here
+automatically before a human promotes it to production.
+
+Staging is designed to run either in the homelab (a couple of low-resource VMs/containers) or,
+once the dedicated server exists, as a resource-constrained slice of it — see
+`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §23 for the sizing note. It does not require a
+second dedicated server.
+
+Staging acceptance = the same four things a person actually needs before promoting a build:
+app deploys, migrations run clean, one or two disposable Labs boot through the Gateway, and an
+assessment can be started and completed. See `infra/scripts/staging-smoke-test.sh`.

infra/environments/staging/gateway.env.template

@@ -0,0 +1,6 @@
+# Production environment template for rn-platform-01's `gateway` service.
+
+PLATFORM_API_URL=http://localhost:4300
+PROVISIONER_URL=http://localhost:4100
+PROVISIONER_SHARED_SECRET=__REPLACE_same_value_as_platform.env_and_provisioner.env__
+GATEWAY_PORT=4200

infra/environments/staging/platform.env.template

@@ -0,0 +1,36 @@
+# Production environment template for rn-platform-01's `api` service.
+# Copy to platform.env OUTSIDE git (see infra/backup/README.md "secrets" and
+# docs/PRODUCTION_DEPLOYMENT.md §10), fill in real values, never commit the filled file.
+# infra/.gitignore already excludes *.env (only *.env.template is tracked).
+
+# ── Postgres ────────────────────────────────────────────────────────────────────────────
+PLATFORM_DB_URL=postgres://rootnode_platform_staging:__REPLACE_WITH_STRONG_PASSWORD__@localhost:5433/rootnode_platform_staging
+
+# ── Provisioner ─────────────────────────────────────────────────────────────────────────
+PROVISIONER_URL=http://localhost:4100
+PROVISIONER_SHARED_SECRET=__REPLACE_openssl_rand_hex_32__
+
+# ── Listen ──────────────────────────────────────────────────────────────────────────────
+PLATFORM_API_PORT=4300
+
+# ── Sessions (production values — do NOT inherit dev defaults) ────────────────────────────
+STUDENT_SESSION_TTL_HOURS=8
+STUDENT_SESSION_IDLE_HOURS=2
+INSTRUCTOR_SESSION_TTL_HOURS=12
+INSTRUCTOR_SESSION_IDLE_HOURS=2
+
+# ── Cookies / proxy — MUST be set for production, unlike dev ──────────────────────────────
+# Secure requires the origin to actually be HTTPS end-to-end (see docs/PRODUCTION_NETWORK_SECURITY.md §8).
+COOKIE_SECURE=1  # unset if staging is not served over HTTPS
+# Only set to 1 once a trusted reverse proxy (Cloudflare/nginx/Caddy, see infra/docker/reverse-proxy/)
+# is actually in front of the Gateway — otherwise req.ip becomes spoofable via X-Forwarded-For.
+TRUST_PROXY=1
+
+# ── Frontend (baked into the platform/api image at build time; listed here for reference) ─
+# FRONTEND_DIST_DIR=/frontend-dist
+
+# ── Future OIDC (Phase 3A adapter — not wired yet, see docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md §16) ─
+# OIDC_ISSUER_URL=
+# OIDC_CLIENT_ID=
+# OIDC_CLIENT_SECRET=__do_not_commit__
+# OIDC_REDIRECT_URL=https://academy.example.com/api/auth/oidc/callback

infra/environments/staging/postgres.env.template

@@ -0,0 +1,14 @@
+# Production environment template for rn-platform-01's `postgres` service.
+# Postgres itself binds only to localhost/the platform-internal Docker network — never exposed
+# beyond rn-platform-01, see docs/PRODUCTION_NETWORK_SECURITY.md §6.
+
+POSTGRES_USER=rootnode_platform
+POSTGRES_PASSWORD=__REPLACE_WITH_STRONG_PASSWORD__
+POSTGRES_DB=rootnode_platform
+
+# Tuning baseline — conservative, meant to be revisited once rn-platform-01's real allocated
+# RAM is known (see docs/production-postgres section in docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md
+# §11). Applied via infra/docker/postgres/postgresql.conf.d/ — do not hand-edit the container.
+# POSTGRES_SHARED_BUFFERS=1GB
+# POSTGRES_EFFECTIVE_CACHE_SIZE=3GB
+# POSTGRES_MAX_CONNECTIONS=100

infra/environments/staging/provisioner-config/lab-hosts.yaml

@@ -0,0 +1,16 @@
+# Production lab_hosts seed — points at the real rn-lab-host-01 VM instead of Phase 1's dev
+# Unix socket. Same mechanism the Provisioner has had since Phase 1
+# (platform/provisioner/config/lab-hosts.seed.yaml), just pointed at a real remote endpoint —
+# no application code changes required for this, see docs/PRODUCTION_NETWORK_SECURITY.md §7.
+#
+# {{ lab_host_ip }} is an Ansible template variable — see infra/ansible/group_vars/production.yml.
+# Rendered by the `platform` role; the raw file here keeps the placeholder for readability.
+
+lab_hosts:
+  - lab_host_id: labhost-01
+    display_name: "RootNode Lab Host VM 01 (production)"
+    endpoint: "tcp://{{ lab_host_ip }}:2376"
+    tls_ca_path: /etc/rootnode/docker-tls/ca.pem
+    tls_cert_path: /etc/rootnode/docker-tls/provisioner-client-cert.pem
+    tls_key_path: /etc/rootnode/docker-tls/provisioner-client-key.pem
+    enabled: true

infra/environments/staging/provisioner-config/resource-policy.yaml

@@ -0,0 +1,25 @@
+# PRODUCTION OVERLAY — placeholder, deliberately unchanged from the Phase 1 dev defaults.
+# Do not hand-tune these before the real host exists (docs/CAPACITY_PLANNING.md, Phase 4 §25).
+# After the first real load test on rn-lab-host-01 (infra/scripts/loadtest/), replace this
+# comment block and the numbers below with actual measured values, then redeploy via the
+# `platform` Ansible role (idempotent — re-running just re-renders this file).
+#
+# Phase 1 resource policy — config-driven, not hard-coded (decision #4).
+# Target host class for planning is ~16 cores/32 threads, 128GB RAM, SSD/NVMe.
+
+default:
+  cpu_limit: 2.0          # vCPUs
+  mem_limit_mb: 4096
+  pids_limit: 512
+  storage_limit_mb: 8192  # only enforced if the host's Docker storage driver supports
+                           # --storage-opt size= (see known-limitations in the SOP)
+
+idle_threshold_minutes: 30       # past this: log a warning event only
+abandoned_threshold_minutes: 120 # past this: reaper destroys for real
+
+reaper_poll_interval_seconds: 60
+
+health_check:
+  path: /api/health
+  timeout_ms: 240000   # k3s cold start in a nested container can be slow — matches the
+                        # ~60-attempt wait already in scripts/entrypoint.sh

infra/environments/staging/provisioner.env.template

@@ -0,0 +1,20 @@
+# Production environment template for rn-platform-01's `provisioner` service.
+# See platform.env.template for the secrets-handling convention.
+#
+# Docker-connectivity config (lab_hosts) and resource limits are NOT env vars — they're YAML
+# files the Provisioner already loads from PROVISIONER_CONFIG_DIR (Phase 1 design, unchanged).
+# Production overlays for both live at infra/environments/production/provisioner-config/ — see
+# that directory's README before touching resource-policy.yaml numbers.
+
+PLATFORM_DB_URL=postgres://rootnode_platform:__REPLACE_WITH_STRONG_PASSWORD__@localhost:5433/rootnode_platform
+PROVISIONER_SHARED_SECRET=__REPLACE_same_value_as_platform.env__
+PROVISIONER_PORT=4100
+PROVISIONER_CONFIG_DIR=/etc/rootnode/provisioner-config
+
+# Real Lab image, tagged (never :latest in production — see infra/docker/IMAGE_STRATEGY.md)
+LAB_IMAGE_REF=registry.example.com/rootnode/lab:staging-__GIT_SHA__
+
+# ── Egress proxy on rn-lab-host-01 ─────────────────────────────────────────────────────────
+EGRESS_PROXY_PORT=3128
+DNS_RESOLVER=127.0.0.11
+NO_PROXY_CIDRS=10.42.0.0/16,10.43.0.0/16

infra/firewall/README.md

@@ -0,0 +1,31 @@
+# Production firewall policy
+
+iptables (not nftables) — deliberately, to stay consistent with the Provisioner's own dynamic
+per-Lab egress rules, which already manipulate the `DOCKER-USER` chain via iptables
+(`platform/provisioner/src/firewall.js`, Phase 1). Mixing the nft and legacy iptables backends
+on the same host is a known source of silent rule-ordering bugs; the host-level policy below
+uses the same backend the app already depends on.
+
+Each `.rules` file is `iptables-restore` format — deployed by the matching Ansible role
+(`firewall` for the VMs; the Proxmox host ruleset is applied manually per
+`docs/PRODUCTION_DEPLOYMENT.md` §2, since Proxmox host firewalling is out of Ansible's blast
+radius by design — a mistake there can lock out the only way in).
+
+| File | Applies to | Deployed by |
+|---|---|---|
+| `proxmox-host.rules` | The Proxmox hypervisor itself | manual, `docs/PRODUCTION_DEPLOYMENT.md` §2 |
+| `rn-platform-01.rules` | Platform VM | `infra/ansible/roles/firewall/` |
+| `rn-lab-host-01.rules` | Lab Host VM | `infra/ansible/roles/firewall/` |
+
+All three encode the same policy table from `docs/PRODUCTION_NETWORK_SECURITY.md` §6 — this
+README doesn't restate it, that doc is the source of truth for *why*; these files are the
+*what*, in a form the host can actually load.
+
+**IP addresses/subnets are Ansible template variables** (`{{ platform_ip }}`,
+`{{ lab_host_ip }}`, `{{ management_cidr }}`, etc.) — see `infra/ansible/group_vars/production.yml`.
+Nothing here hardcodes a real production IP, per the explicit Phase 4 instruction not to guess
+those before the real host exists.
+
+`infra/firewall/validate-firewall.sh` renders each ruleset with dummy IPs and runs it through
+`iptables-restore --test` (syntax/semantic validation only — proves the rules are well-formed
+iptables, not that the real network topology is correct, which needs the real hardware).

infra/firewall/proxmox-host.rules

@@ -0,0 +1,42 @@
+# iptables-restore ruleset for the Proxmox VE hypervisor itself. Applied MANUALLY per
+# docs/PRODUCTION_DEPLOYMENT.md §2 — deliberately not Ansible-automated (a mistake in a
+# playbook run against the one host that IS the only way to reach every VM console is a much
+# worse failure mode than a manual, reviewed apply). Prefer configuring this through Proxmox's
+# own Datacenter -> Firewall UI/API where practical instead of hand-editing iptables on the
+# hypervisor — this file documents the equivalent policy either way.
+#
+# Policy summary (docs/PRODUCTION_NETWORK_SECURITY.md §5-6):
+#   - Proxmox web UI (8006) and SSH (22) reachable ONLY from the management CIDR — never from
+#     the internet, never from either VM's network, never from a student Lab subnet
+#   - VM traffic (rn-platform-01 <-> rn-lab-host-01, rn-platform-01 <-> internet via NAT/bridge)
+#     is FORWARDed by the bridge, not filtered by this hypervisor-level ruleset — that's the
+#     VMs' own INPUT chains' job (rn-platform-01.rules / rn-lab-host-01.rules)
+#   - no student Lab subnet, and no rn-lab-host-01 traffic at all, may ever reach Proxmox
+#     management — this is the literal "Proxmox management not reachable from student Lab
+#     networks" requirement, enforced at the hypervisor itself, not just hoped-for via VM
+#     network segmentation
+
+*filter
+:INPUT DROP [0:0]
+:FORWARD ACCEPT [0:0]
+:OUTPUT ACCEPT [0:0]
+
+-A INPUT -i lo -j ACCEPT
+-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
+-A INPUT -m conntrack --ctstate INVALID -j DROP
+-A INPUT -p icmp --icmp-type echo-request -m conntrack --ctstate NEW -j ACCEPT
+
+-A INPUT -p tcp --dport 22 -s {{ management_cidr }} -m conntrack --ctstate NEW -j ACCEPT
+-A INPUT -p tcp --dport 8006 -s {{ management_cidr }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Explicit deny + log for the Lab Host VM's network and the student Lab subnet range reaching
+# Proxmox management directly — the literal "Proxmox management denied from student Lab
+# networks" requirement.
+-A INPUT -s {{ lab_host_ip }} -p tcp -m multiport --dports 22,8006 -j LOG --log-prefix "ROOTNODE-DENY-LABHOST-TO-PROXMOX: " --log-level 4
+-A INPUT -s {{ lab_host_ip }} -p tcp -m multiport --dports 22,8006 -j DROP
+-A INPUT -s {{ lab_subnet_range }} -p tcp -m multiport --dports 22,8006 -j LOG --log-prefix "ROOTNODE-DENY-LAB-TO-PROXMOX: " --log-level 4
+-A INPUT -s {{ lab_subnet_range }} -p tcp -m multiport --dports 22,8006 -j DROP
+
+-A INPUT -j LOG --log-prefix "ROOTNODE-DENY-PROXMOX-DEFAULT: " --log-level 4
+
+COMMIT

infra/firewall/rn-lab-host-01.rules

@@ -0,0 +1,62 @@
+# iptables-restore ruleset for rn-lab-host-01 (Lab Host VM). Deployed by
+# infra/ansible/roles/firewall/. See docs/PRODUCTION_NETWORK_SECURITY.md §6 for full rationale.
+#
+# This is the security-critical host — it runs privileged, disposable, student-controlled
+# containers. Its own INPUT/FORWARD policy is only half the story: the Provisioner's dynamic
+# per-Lab DOCKER-USER chains (platform/provisioner/src/egress.js, unchanged since Phase 1)
+# already enforce egress-proxy-only outbound for every student subnet and implicitly deny
+# Lab-to-Lab and Lab-to-anything-not-allowlisted traffic. This file is the BASE policy under
+# that dynamic layer — it must never be more permissive than what the dynamic chains assume,
+# and it's what protects the host itself (SSH, Docker TLS) rather than inter-container traffic.
+#
+# Policy summary:
+#   - Docker TLS (2376) reachable ONLY from rn-platform-01 — this is the entire point of mTLS
+#     plus a firewall restriction, defense in depth (a leaked client cert alone still isn't
+#     enough without also being on this IP)
+#   - SSH only from the management CIDR
+#   - no port on this host is ever reachable from a student Lab's own Docker subnet — Labs only
+#     ever reach the egress proxy and DNS resolver (already enforced by the Provisioner's
+#     per-Lab chains); this file adds an explicit belt-and-suspenders DROP for the Lab subnet
+#     range as a whole, so even a Lab whose per-environment chain failed to install can't reach
+#     the host's own services
+#   - nothing on this host is ever reachable from the public internet at all
+
+*filter
+:INPUT DROP [0:0]
+:FORWARD DROP [0:0]
+:OUTPUT ACCEPT [0:0]
+
+-A INPUT -i lo -j ACCEPT
+-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
+-A INPUT -m conntrack --ctstate INVALID -j DROP
+-A INPUT -p icmp --icmp-type echo-request -m conntrack --ctstate NEW -j ACCEPT
+
+-A INPUT -p tcp --dport 22 -s {{ management_cidr }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Docker TLS — rn-platform-01 (the Provisioner) only. This is the ONLY control-plane flow
+# allowed into this host at all — see the "Platform Provisioner -> Lab Host Docker TLS
+# endpoint: ALLOW" example in the Phase 4 spec, encoded literally here.
+-A INPUT -p tcp --dport 2376 -s {{ platform_ip }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Gateway -> per-Lab published port range, rn-platform-01 only. REQUIRED so the Gateway can
+# reach each student's Lab UI/terminal over the network once Labs run on a genuinely separate
+# host — see docs/PRODUCTION_NETWORK_SECURITY.md §"Gateway-to-Lab cross-VM reachability" for
+# why this exists and the small Provisioner/Gateway code change it depends on (NOT YET BUILT —
+# this firewall rule is the network-policy half; the port-publishing half is an explicit,
+# reviewed application-code follow-up, tracked there rather than silently assumed here).
+-A INPUT -p tcp --dport {{ lab_gateway_port_range | default('30000:32767') }} -s {{ platform_ip }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Prometheus (rn-platform-01) scraping this host's node-exporter/cadvisor — metrics only,
+# read-only exposition endpoints, still restricted to the one legitimate scraper.
+-A INPUT -p tcp -m multiport --dports 9100,8080 -s {{ platform_ip }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Belt-and-suspenders: explicit deny + log for any student Lab subnet reaching THIS HOST
+# (not another container — the host's own listening ports: SSH, Docker TLS, anything else).
+# The Provisioner's per-Lab DOCKER-USER chains already stop this at the container-egress layer;
+# this is what still protects the host if that layer is ever misconfigured or mid-install.
+-A INPUT -s {{ lab_subnet_range }} -j LOG --log-prefix "ROOTNODE-DENY-LAB-TO-HOST: " --log-level 4
+-A INPUT -s {{ lab_subnet_range }} -j DROP
+
+-A INPUT -j LOG --log-prefix "ROOTNODE-DENY-LABHOST-DEFAULT: " --log-level 4
+
+COMMIT

infra/firewall/rn-platform-01.rules

@@ -0,0 +1,58 @@
+# iptables-restore ruleset for rn-platform-01 (Platform VM). Deployed by
+# infra/ansible/roles/firewall/. Variables ({{ }}) are Ansible Jinja2, rendered before this
+# file reaches the host — see infra/ansible/group_vars/production.yml.
+#
+# Policy summary (full rationale: docs/PRODUCTION_NETWORK_SECURITY.md §6):
+#   - only the reverse proxy's 443/80 are reachable from the internet
+#   - SSH only from the management CIDR (your home/office/VPN, never 0.0.0.0/0)
+#   - Docker TLS (2376) is a rn-lab-host-01 concern, not this host's — nothing to open here
+#   - Postgres, Provisioner's internal API, and the Platform API/Gateway's raw ports are never
+#     exposed on this VM's external interface at all — the reverse proxy is the only public
+#     listener; everything else binds to the docker-compose internal network only, so there is
+#     no host-level rule needed to "hide" them, only to make sure nothing ever publishes them
+#     by accident (this ruleset's default-deny INPUT catches that class of mistake too)
+#   - rn-lab-host-01 may reach exactly one port here: Loki (3100), for its own log shipper —
+#     the Provisioner otherwise calls OUT to rn-lab-host-01, never the other way around — and
+#     may NEVER reach Postgres or the Provisioner's internal API
+
+*filter
+:INPUT DROP [0:0]
+:FORWARD DROP [0:0]
+:OUTPUT ACCEPT [0:0]
+
+# Loopback and established/related — standard baseline
+-A INPUT -i lo -j ACCEPT
+-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
+-A INPUT -m conntrack --ctstate INVALID -j DROP
+
+# ICMP — keep basic reachability diagnostics working
+-A INPUT -p icmp --icmp-type echo-request -m conntrack --ctstate NEW -j ACCEPT
+
+# SSH — management CIDR only, never open to the internet
+-A INPUT -p tcp --dport 22 -s {{ management_cidr }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Public HTTPS/HTTP — the reverse proxy (Caddy), the only public listener on this host
+-A INPUT -p tcp --dport 443 -m conntrack --ctstate NEW -j ACCEPT
+-A INPUT -p tcp --dport 80 -m conntrack --ctstate NEW -j ACCEPT
+
+# Loki log ingestion — rn-lab-host-01's promtail only (that host's own log shipper, pushing its
+# container/journald logs here for centralized viewing in Grafana). Same "one specific source
+# IP, one specific port" pattern as Docker TLS on the other host.
+-A INPUT -p tcp --dport 3100 -s {{ lab_host_ip }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Explicit deny + log for anything from rn-lab-host-01 hitting Postgres or the Provisioner's
+# internal port — defense in depth on top of "these ports were never published," and the one
+# rule in this file that most directly encodes the Lab->Platform boundary the whole platform
+# depends on. Logged (not silently dropped) so an attempted breach attempt is visible in
+# infra/monitoring/ alerts, not just absorbed by the default DROP.
+-A INPUT -s {{ lab_host_ip }} -p tcp --dport 5432 -j LOG --log-prefix "ROOTNODE-DENY-LABHOST-PG: " --log-level 4
+-A INPUT -s {{ lab_host_ip }} -p tcp --dport 5432 -j DROP
+-A INPUT -s {{ lab_host_ip }} -p tcp --dport 4100 -j LOG --log-prefix "ROOTNODE-DENY-LABHOST-PROVISIONER: " --log-level 4
+-A INPUT -s {{ lab_host_ip }} -p tcp --dport 4100 -j DROP
+
+# Default: everything else already falls to the chain policy (DROP), nothing further needed —
+# but an explicit final logged catch-all makes "what got denied and why" visible in logs rather
+# than silent.
+-A INPUT -j LOG --log-prefix "ROOTNODE-DENY-PLATFORM-DEFAULT: " --log-level 4
+
+COMMIT

infra/firewall/validate-firewall.sh

@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+# Local test (Phase 4 §29): renders each .rules file with dummy IPs and validates it.
+#
+# `iptables-restore --test` needs root (it talks to netfilter) — if this shell has passwordless
+# sudo, we use it for a real semantic check. Otherwise we fall back to a structural lint
+# (balanced *table/COMMIT blocks, every rule starts with a real flag, no unrendered {{ }}
+# placeholders left over) — good enough to catch a typo'd rule or a missing COMMIT, NOT a
+# substitute for the real `iptables-restore --test` a human/Ansible with actual root runs
+# before this ever reaches a live host. HARDWARE-DEPENDENT: proving the real IPs/CIDRs are
+# correct needs the actual production network, not this script.
+set -euo pipefail
+cd "$(dirname "$0")"
+
+TMPDIR="$(mktemp -d)"
+trap 'rm -rf "$TMPDIR"' EXIT
+
+render() {
+  sed \
+    -e 's/{{ management_cidr }}/203.0.113.0\/24/g' \
+    -e 's/{{ platform_ip }}/10.10.10.10/g' \
+    -e 's/{{ lab_host_ip }}/10.10.10.20/g' \
+    -e 's/{{ lab_subnet_range }}/10.42.0.0\/16/g' \
+    -e 's/{{ proxmox_mgmt_cidr }}/203.0.113.0\/24/g' \
+    -e "s/{{ lab_gateway_port_range | default('30000:32767') }}/30000:32767/g" \
+    "$1" > "$2"
+}
+
+structural_lint() {
+  local f="$1"
+  grep -q '^\*filter' "$f" || { echo "  missing *filter table declaration"; return 1; }
+  grep -q '^COMMIT' "$f" || { echo "  missing COMMIT"; return 1; }
+  grep -v '^#' "$f" | grep -q '{{' && { echo "  unrendered {{ }} placeholder left in a non-comment line"; return 1; }
+  # every -A/-I rule line should end in a real target (ACCEPT/DROP/REJECT/LOG/a custom chain)
+  while IFS= read -r line; do
+    [[ "$line" =~ ^-[AI]\  ]] || continue
+    [[ "$line" =~ -j\ [A-Za-z0-9_-]+ ]] || { echo "  rule missing a -j target: $line"; return 1; }
+  done < "$f"
+  return 0
+}
+
+HAVE_SUDO=0
+sudo -n true 2>/dev/null && HAVE_SUDO=1
+
+FAIL=0
+for f in proxmox-host.rules rn-platform-01.rules rn-lab-host-01.rules; do
+  render "$f" "$TMPDIR/$f"
+  echo "==> $f"
+  if [ "$HAVE_SUDO" = "1" ] && command -v iptables-restore >/dev/null 2>&1; then
+    if sudo -n iptables-restore --test < "$TMPDIR/$f"; then
+      echo "OK: $f passed a real iptables-restore --test"
+    else
+      echo "FAIL: $f did not pass iptables-restore --test" >&2; FAIL=1
+    fi
+  else
+    if structural_lint "$TMPDIR/$f"; then
+      echo "OK: $f passes structural lint (no root available for a real iptables-restore --test — see script header)"
+    else
+      echo "FAIL: $f failed structural lint" >&2; FAIL=1
+    fi
+  fi
+done
+
+exit $FAIL

infra/monitoring/alertmanager/alertmanager.yml.j2

@@ -0,0 +1,39 @@
+# Alertmanager config — routes CRITICAL alerts and WARNING alerts to the same webhook by
+# default (Uptime Kuma's push-monitor webhook, or ntfy, or another already-available target —
+# not a new paid service; the user's stated preference for Uptime Kuma is honored here without
+# forcing it if a cleaner existing target turns out to be available at deploy time, per the
+# Phase 4 instruction). Swap the receiver's url for whatever's actually wired up.
+
+global:
+  resolve_timeout: 5m
+
+route:
+  receiver: rootnode-default
+  group_by: ["alertname", "host"]
+  group_wait: 30s
+  group_interval: 5m
+  repeat_interval: 4h
+  routes:
+    - match:
+        severity: critical
+      receiver: rootnode-critical
+      repeat_interval: 30m
+
+receivers:
+  - name: rootnode-default
+    webhook_configs:
+      - url: "{{ vault_alertmanager_webhook_url | default('http://example.invalid/webhook') }}"
+        send_resolved: true
+
+  - name: rootnode-critical
+    webhook_configs:
+      - url: "{{ vault_alertmanager_webhook_url | default('http://example.invalid/webhook') }}"
+        send_resolved: true
+
+inhibit_rules:
+  # If the whole Lab Host is down, don't also page separately for every dependent warning.
+  - source_match:
+      alertname: LabHostDown
+    target_match:
+      severity: warning
+    equal: ["host"]

infra/monitoring/grafana/dashboards/rootnode-overview.json

@@ -0,0 +1,54 @@
+{
+  "title": "RootNode Academy — Infrastructure Overview",
+  "uid": "rootnode-overview",
+  "schemaVersion": 39,
+  "version": 1,
+  "timezone": "utc",
+  "refresh": "30s",
+  "time": { "from": "now-6h", "to": "now" },
+  "panels": [
+    {
+      "id": 1, "title": "Platform VM CPU", "type": "timeseries",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
+      "targets": [{ "expr": "100 - (avg(rate(node_cpu_seconds_total{job=\"platform-node\",mode=\"idle\"}[5m])) * 100)", "legendFormat": "CPU %" }]
+    },
+    {
+      "id": 2, "title": "Lab Host VM CPU", "type": "timeseries",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
+      "targets": [{ "expr": "100 - (avg(rate(node_cpu_seconds_total{job=\"lab-host-node\",mode=\"idle\"}[5m])) * 100)", "legendFormat": "CPU %" }]
+    },
+    {
+      "id": 3, "title": "Platform VM Memory Available", "type": "timeseries",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
+      "targets": [{ "expr": "node_memory_MemAvailable_bytes{job=\"platform-node\"}", "legendFormat": "Available" }]
+    },
+    {
+      "id": 4, "title": "Lab Host VM Memory Available", "type": "timeseries",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
+      "targets": [{ "expr": "node_memory_MemAvailable_bytes{job=\"lab-host-node\"}", "legendFormat": "Available" }]
+    },
+    {
+      "id": 5, "title": "Disk Usage (root filesystem, both hosts)", "type": "timeseries",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 },
+      "targets": [{ "expr": "1 - (node_filesystem_avail_bytes{mountpoint=\"/\"} / node_filesystem_size_bytes{mountpoint=\"/\"})", "legendFormat": "{{host}}" }]
+    },
+    {
+      "id": 6, "title": "Postgres — Up / Connections", "type": "timeseries",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 16 },
+      "targets": [
+        { "expr": "pg_up", "legendFormat": "up" },
+        { "expr": "pg_stat_database_numbackends{datname=\"rootnode_platform\"}", "legendFormat": "connections" }
+      ]
+    },
+    {
+      "id": 7, "title": "Running Docker Containers — Lab Host", "type": "timeseries",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 24 },
+      "targets": [{ "expr": "count(container_last_seen{name=~\"rootnode-lab-.*\"})", "legendFormat": "active labs (best-effort via cadvisor label match)" }]
+    },
+    {
+      "id": 8, "title": "Uptime Probes (Gateway / Platform API / Provisioner)", "type": "timeseries",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 24 },
+      "targets": [{ "expr": "probe_success", "legendFormat": "{{job}}" }]
+    }
+  ]
+}

infra/monitoring/grafana/provisioning/dashboards/dashboards.yml

@@ -0,0 +1,7 @@
+apiVersion: 1
+providers:
+  - name: RootNode
+    folder: RootNode
+    type: file
+    options:
+      path: /var/lib/grafana/dashboards

infra/monitoring/grafana/provisioning/datasources/datasources.yml

@@ -0,0 +1,11 @@
+apiVersion: 1
+datasources:
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: true
+  - name: Loki
+    type: loki
+    access: proxy
+    url: http://loki:3100

infra/monitoring/loki/loki-config.yml

@@ -0,0 +1,36 @@
+# Minimal single-binary Loki config — local filesystem storage, no external dependency, sized
+# for this deployment (a handful of services' logs, not internet-scale ingestion).
+# Retention is set here, not left to grow forever — see docs/OBSERVABILITY.md §"retention."
+
+auth_enabled: false
+
+server:
+  http_listen_port: 3100
+
+common:
+  path_prefix: /loki
+  storage:
+    filesystem:
+      chunks_directory: /loki/chunks
+      rules_directory: /loki/rules
+  replication_factor: 1
+  ring:
+    kvstore:
+      store: inmemory
+
+schema_config:
+  configs:
+    - from: 2024-01-01
+      store: tsdb
+      object_store: filesystem
+      schema: v13
+      index:
+        prefix: index_
+        period: 24h
+
+limits_config:
+  retention_period: 720h   # 30 days — matches Prometheus's default retention (production.yml var)
+
+compactor:
+  working_directory: /loki/compactor
+  retention_enabled: true

infra/monitoring/prometheus/blackbox.yml

@@ -0,0 +1,13 @@
+# blackbox_exporter config — HTTP probes for uptime monitoring (Phase 4 §17). Probes the
+# /healthz endpoints added to Platform API and Gateway this phase, plus the Provisioner's
+# existing /health (Phase 1). Prometheus scrapes blackbox_exporter with these probe targets
+# (see prometheus.yml.j2's blackbox-* jobs — add them there once blackbox_exporter is deployed;
+# left as a documented follow-up wiring step rather than duplicated here).
+
+modules:
+  http_2xx:
+    prober: http
+    timeout: 5s
+    http:
+      valid_status_codes: [200]
+      method: GET

infra/monitoring/prometheus/prometheus.yml.j2

@@ -0,0 +1,93 @@
+# Prometheus config — deployed to rn-platform-01 (runs alongside the app stack; see
+# infra/ansible/roles/monitoring-agent/). Scrapes both hosts remotely — node-exporter/
+# postgres-exporter here on rn-platform-01 (same-host, local network), node-exporter/cadvisor
+# on rn-lab-host-01 over the network (allowed by the firewall rule added for exactly this,
+# infra/firewall/rn-lab-host-01.rules).
+#
+# Deliberately does NOT scrape or duplicate the application's own instructor learning
+# analytics (docs/PHASE3B_LEARNING_AND_ASSESSMENT.md §12) — operational observability
+# (uptime/CPU/errors) and learning analytics (completion rates/hint usage) are separate
+# concerns with separate audiences, per the explicit Phase 4 instruction.
+
+global:
+  scrape_interval: 15s
+  evaluation_interval: 15s
+
+alerting:
+  alertmanagers:
+    - static_configs:
+        - targets: ["localhost:9093"]
+
+rule_files:
+  - /etc/prometheus/rules/*.yml
+
+scrape_configs:
+  - job_name: platform-node
+    static_configs:
+      - targets: ["localhost:9100"]
+        labels: { host: rn-platform-01 }
+
+  - job_name: platform-postgres
+    static_configs:
+      - targets: ["postgres-exporter:9187"]
+        labels: { host: rn-platform-01 }
+
+  - job_name: lab-host-node
+    static_configs:
+      - targets: ["{{ lab_host_ip }}:9100"]
+        labels: { host: rn-lab-host-01 }
+
+  - job_name: lab-host-cadvisor
+    static_configs:
+      - targets: ["{{ lab_host_ip }}:8080"]
+        labels: { host: rn-lab-host-01 }
+
+  # Uptime checks (Phase 4 §17) — blackbox_exporter probing the healthz endpoints added this
+  # phase (Platform API, Gateway) plus the Provisioner's existing /health (Phase 1).
+  - job_name: blackbox-gateway
+    metrics_path: /probe
+    params: { module: [http_2xx] }
+    static_configs:
+      - targets: ["https://{{ academy_domain }}/healthz"]
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: __param_target
+      - source_labels: [__param_target]
+        target_label: instance
+      - target_label: __address__
+        replacement: localhost:9115
+
+  - job_name: blackbox-platform-api
+    metrics_path: /probe
+    params: { module: [http_2xx] }
+    static_configs:
+      - targets: ["http://api:4300/healthz"]
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: __param_target
+      - source_labels: [__param_target]
+        target_label: instance
+      - target_label: __address__
+        replacement: localhost:9115
+
+  - job_name: blackbox-provisioner
+    metrics_path: /probe
+    params: { module: [http_2xx] }
+    static_configs:
+      - targets: ["http://provisioner:4100/health"]
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: __param_target
+      - source_labels: [__param_target]
+        target_label: instance
+      - target_label: __address__
+        replacement: localhost:9115
+
+  # RootNode application metrics — see docs/OBSERVABILITY.md §"exposing /metrics" for the
+  # documented, NOT-YET-BUILT follow-up (Platform API/Gateway/Provisioner don't currently
+  # expose a Prometheus /metrics endpoint; this job is ready for when that lands, disabled
+  # until then rather than scraping a 404 forever).
+  # - job_name: rootnode-platform-api
+  #   static_configs:
+  #     - targets: ["api:4300"]
+  #   metrics_path: /metrics

infra/monitoring/prometheus/rules/rootnode.yml

@@ -0,0 +1,97 @@
+# Alert rules (Phase 4 §16). CRITICAL pages immediately; WARNING is for daytime attention.
+# Deliberately does NOT alert on individual student scenario failures or validation
+# failures — those are instructional signal (docs/PHASE3B_LEARNING_AND_ASSESSMENT.md's Needs
+# Attention view), not operational incidents.
+
+groups:
+  - name: rootnode-critical
+    rules:
+      - alert: PlatformAPIDown
+        expr: probe_success{job="blackbox-platform-api"} == 0
+        for: 2m
+        labels: { severity: critical }
+        annotations:
+          summary: "Platform API is unreachable"
+          description: "/healthz on rn-platform-01's Platform API has failed for 2+ minutes."
+
+      - alert: GatewayDown
+        expr: probe_success{job="blackbox-gateway"} == 0
+        for: 2m
+        labels: { severity: critical }
+        annotations:
+          summary: "Gateway is unreachable"
+          description: "The Gateway's /healthz has failed for 2+ minutes — students cannot reach any Lab."
+
+      - alert: PostgresDown
+        expr: pg_up == 0
+        for: 1m
+        labels: { severity: critical }
+        annotations:
+          summary: "Postgres is down"
+          description: "postgres-exporter cannot reach rootnode_platform Postgres."
+
+      - alert: LabHostDown
+        expr: up{job="lab-host-node"} == 0
+        for: 2m
+        labels: { severity: critical }
+        annotations:
+          summary: "rn-lab-host-01 is unreachable"
+          description: "node-exporter on the Lab Host VM has stopped responding — no student can launch or use a Lab."
+
+      - alert: BackupFailed
+        expr: rootnode_backup_last_success_timestamp_seconds < (time() - 90000)
+        labels: { severity: critical }
+        annotations:
+          summary: "Postgres backup hasn't succeeded in 25+ hours"
+          description: "infra/backup/backup-postgres.sh's last successful run is stale — see docs/RUNBOOK_BACKUP_FAILURE.md."
+
+      - alert: DiskCriticallyFull
+        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) < 0.10
+        for: 5m
+        labels: { severity: critical }
+        annotations:
+          summary: "Root filesystem below 10% free on {{ $labels.host }}"
+
+  - name: rootnode-warning
+    rules:
+      - alert: LabHostHighMemory
+        expr: (1 - (node_memory_MemAvailable_bytes{job="lab-host-node"} / node_memory_MemTotal_bytes{job="lab-host-node"})) > 0.85
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "rn-lab-host-01 memory usage above 85% for 10+ minutes"
+          description: "Check active Lab count against docs/CAPACITY_PLANNING.md's admission thresholds."
+
+      - alert: HighDiskUsage
+        expr: (1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)) > 0.80
+        for: 15m
+        labels: { severity: warning }
+        annotations:
+          summary: "Disk usage above 80% on {{ $labels.host }}"
+
+      - alert: RepeatedProvisioningFailures
+        expr: increase(rootnode_provisioning_failures_total[15m]) >= 3
+        labels: { severity: warning }
+        annotations:
+          summary: "3+ Lab provisioning failures in 15 minutes"
+          description: >-
+            Depends on the Provisioner exposing this counter — see
+            docs/OBSERVABILITY.md's "application metrics — not yet built" note. Placeholder rule,
+            inert until that counter exists.
+
+      - alert: BackupAging
+        expr: rootnode_backup_last_success_timestamp_seconds < (time() - 43200)
+        labels: { severity: warning }
+        annotations:
+          summary: "Postgres backup hasn't succeeded in 12+ hours"
+
+      - alert: ExcessiveFailedLogins
+        expr: increase(rootnode_audit_login_failures_total[5m]) > 20
+        labels: { severity: warning }
+        annotations:
+          summary: "20+ failed instructor logins in 5 minutes"
+          description: >-
+            Depends on exporting audit_events counts — see docs/OBSERVABILITY.md's
+            "application metrics — not yet built" note. Placeholder rule, inert until built;
+            the audit_events table itself already exists (Phase 3A) and can be queried
+            manually via /api/admin/audit-events in the meantime.

infra/monitoring/promtail/promtail-config.lab-host.yml.j2

@@ -0,0 +1,38 @@
+# Promtail on rn-lab-host-01 — same shape as promtail-config.platform.yml, but pushes to Loki
+# over the network (Loki lives on rn-platform-01, not locally) since these two hosts don't
+# share a Docker network. {{ platform_ip }} is an Ansible variable, rendered by the lab-host
+# role — see infra/firewall/rn-platform-01.rules for the matching inbound rule (3100, this
+# host's IP only).
+
+server:
+  http_listen_port: 9080
+
+positions:
+  filename: /tmp/positions.yaml
+
+clients:
+  - url: "http://{{ platform_ip }}:3100/loki/api/v1/push"
+
+scrape_configs:
+  - job_name: docker
+    docker_sd_configs:
+      - host: unix:///var/run/docker.sock
+        refresh_interval: 15s
+    relabel_configs:
+      - source_labels: [__meta_docker_container_name]
+        target_label: container
+      - source_labels: [__meta_docker_container_label_com_docker_compose_service]
+        target_label: service
+    pipeline_stages:
+      - replace:
+          expression: '(?i)(password|token|secret)=[^&\s]+'
+          replace: '$1=***REDACTED***'
+
+  - job_name: journal
+    journal:
+      max_age: 24h
+      labels:
+        job: systemd-journal
+    relabel_configs:
+      - source_labels: [__journal__systemd_unit]
+        target_label: unit

infra/monitoring/promtail/promtail-config.platform.yml

@@ -0,0 +1,42 @@
+# Promtail — ships container logs (via the Docker socket, all containers on this host) and
+# journald (systemd units: rootnode-platform.service, rootnode-lab-host.service, docker,
+# sshd, fail2ban) to Loki. Deliberately does NOT centralize raw student terminal/command
+# output — the Lab image's own stdout is Lab-lifecycle noise (k3s boot logs, entrypoint
+# output), not the actual PTY session content, which never leaves the container's terminal
+# WebSocket in the first place. See docs/OBSERVABILITY.md §"what's excluded and why."
+
+server:
+  http_listen_port: 9080
+
+positions:
+  filename: /tmp/positions.yaml
+
+clients:
+  - url: http://loki:3100/loki/api/v1/push
+
+scrape_configs:
+  - job_name: docker
+    docker_sd_configs:
+      - host: unix:///var/run/docker.sock
+        refresh_interval: 15s
+    relabel_configs:
+      - source_labels: [__meta_docker_container_name]
+        target_label: container
+      - source_labels: [__meta_docker_container_label_com_docker_compose_service]
+        target_label: service
+    pipeline_stages:
+      # Never ship secrets — a defense-in-depth scrub in case a service ever accidentally logs
+      # one (the app itself is expected to never log tokens/passwords per its own conventions,
+      # e.g. docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md's "never log raw session tokens").
+      - replace:
+          expression: '(?i)(password|token|secret)=[^&\s]+'
+          replace: '$1=***REDACTED***'
+
+  - job_name: journal
+    journal:
+      max_age: 24h
+      labels:
+        job: systemd-journal
+    relabel_configs:
+      - source_labels: [__journal__systemd_unit]
+        target_label: unit