docs(phase4.1): pre-production hardening documentation

Master overview (PHASE4_1_PREPRODUCTION_HARDENING.md) covering all 27
substantive items, plus REMOTE_AND_SELF_PACED_ARCHITECTURE.md (reviews
what already works for remote/self-paced delivery - student-driven
provisioning, cohort-less students, idle-reaper-destroy-then-continue -
and the one real gap: student authentication, explicitly Phase 5 scope,
not built here).

Updated PRODUCTION_NETWORK_SECURITY.md (the Gateway-to-Lab gap is now
RESOLVED, with the full design writeup and a new control-plane-vs-data-plane
section), PRODUCTION_DEPLOYMENT.md (four-cert TLS bundle),
PRODUCTION_VALIDATION_CHECKLIST.md (Router mTLS check, admission control
checks), PHASE4_PRODUCTION_INFRASTRUCTURE.md and ENGINEERING_SOP.md and
LAB_ARCHITECTURE.md (stale 'known gap' pointers updated to point at the
resolution), BACKUP_DR.md (destructive drill), README.md and
platform/README.md (Phase 4.1 pointers, lab-router component, updated
known-limitations).

Also fixes a real pre-existing bug found while auditing what gets
committed: infra/.gitignore's patterns were written repo-root-relative
(e.g. "infra/terraform/...") but the file lives in infra/, so git resolved
them relative to THAT directory and none of them ever matched anything -
a real .terraform/ provider cache showed up as untracked when it should
have been ignored. Fixed by dropping the redundant leading "infra/" from
every pattern.

Status: PHASE 4.1 COMPLETE / PRE-PRODUCTION HARDENING COMPLETE / READY FOR
HARDWARE DEPLOYMENT.

.gitignore

@@ -4,3 +4,4 @@ frontend/dist/
 *.db-journal
 /data/
 .env
+.ansible/

README.md

@@ -194,6 +194,14 @@ Production topology and what's built vs. not yet: `docs/LAB_ARCHITECTURE.md`,
 `docs/PHASE1_PLATFORM_FOUNDATION.md`, `docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md`,
 `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md`, `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`.
 
+The production infrastructure package (Proxmox/Ansible/Terraform/firewall/monitoring/backup/CI)
+that runs all of this on real hardware lives in `infra/` — see
+`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` and `docs/PHASE4_1_PREPRODUCTION_HARDENING.md` (the
+Gateway-to-Lab cross-VM routing fix, plus application hardening — real pagination, abandonment
+handling, a Postgres-backed rate limiter, a structured assessment editor, admission control).
+Built and locally validated now; deployment is pending purchase of the target Hetzner dedicated
+server.
+
 ---
 
 ## Repository Layout

docs/BACKUP_DR.md

@@ -0,0 +1,74 @@
+# Backup & Disaster Recovery
+
+## What's backed up (Tier 1) vs. not
+
+| | Backed up | Why |
+|---|---|---|
+| Postgres data | Yes, daily | The only irreplaceable data — student records, cohort/session state, scenario attempts, assessments, achievements, audit events |
+| `{{ rootnode_data_root }}` config/secrets (Vault files, env templates) | Yes, via the infra repo + Ansible Vault itself | Small, static, versioned |
+| Infra code (`infra/`, Ansible, Terraform state) | Yes — it's in git; Terraform state should be backed up alongside Postgres if not using a remote backend | Needed to rebuild either VM from scratch |
+| Disposable student Labs | **No, intentionally** | Ephemeral by design — a Lab is meant to be destroyed and recreated; backing it up would contradict the whole isolation/disposability model |
+| Prometheus/Loki metrics & log history | No | Operational telemetry, not source-of-truth data; losing 30 days of dashboards on a rebuild is an inconvenience, not data loss |
+
+## Backup mechanism
+
+`infra/backup/backup-postgres.sh` (deployed to `/opt/rootnode/backup/` by the `backup` Ansible
+role, run daily at 03:15 via cron):
+
+1. `pg_dump --format=plain --no-owner --no-privileges | gzip -9`
+2. Sanity check: reject if the compressed dump is under 1KB (almost certainly means Postgres
+   was unreachable or empty, not a real backup).
+3. `gpg --symmetric --cipher-algo AES256` — encrypt, then delete the plaintext dump immediately.
+4. Optional off-host copy via `rsync -az` over SSH — works for Hetzner Storage Box or any SSH
+   target; that's the extension point for S3-compatible alternatives (swap the rsync line for
+   `aws s3 cp`/`rclone` if a different target is chosen later).
+5. Retention pruning (`find ... -mtime +{{ backup_retention_days }} -delete`, default 30 days).
+6. Writes Prometheus textfile-collector metrics
+   (`rootnode_backup_last_run_success`, `..._last_success_timestamp_seconds`, `..._size_bytes`)
+   atomically (tmp file + rename) so a half-written metric file is never scraped.
+
+## Restore
+
+`infra/backup/restore-postgres.sh` — requires an explicit target DB URL (never infers or
+defaults to production, to prevent an accidental overwrite), shows the target's current tables,
+requires typing `RESTORE` to confirm, then decrypts and pipes into `psql`.
+`infra/backup/verify-restore.sh` — confirms all 8 expected tables exist plus the
+`schema_migrations` row count, so a "successful" restore that silently missed a table is caught.
+
+**Genuinely tested**, not just written: `infra/backup/test-backup-restore.sh` runs the full
+cycle against two disposable local Postgres containers, with the actual production migrations
+applied (3 Provisioner + Platform API, in the correct cross-service order), a
+sample row inserted, backed up, restored into the second container, and verified byte-for-byte.
+Confirmed passing, exit 0.
+
+**Phase 4.1's destructive drill goes one step further**
+(`infra/backup/test-destructive-restore-drill.sh`): realistic data (owner, admin instructor,
+cohort, 3 students, progress, attempts, an assessment definition, achievements) seeded through
+the REAL Platform API, backed up, the source Postgres **container genuinely destroyed**
+(`docker rm`, not just left running alongside a second one), a fresh Postgres created from
+nothing, restored, and the real Platform API booted again against the restored database and
+used to log in as the restored admin instructor over HTTP — proving the whole stack recovers,
+not just the SQL. Never touches the developer's real persistent database. Confirmed passing.
+
+## Disaster recovery scenarios
+
+| Scenario | RPO | RTO (estimate) | What happens |
+|---|---|---|---|
+| Platform VM lost, disks intact | 0 | ~30 min | `docs/RUNBOOK_VM_REBUILD.md` — Ansible re-run against the same disks |
+| Platform VM + its disks lost | Up to 24h (since last backup) | ~1-2h | Rebuild VM (Terraform), re-run Ansible, restore Postgres from the latest off-host backup (`docs/RUNBOOK_POSTGRES_RESTORE.md`) |
+| Lab Host VM lost, any reason | 0 (intentional) | ~30 min | Rebuild VM, re-run Ansible — no data to restore, all Labs were disposable and get recreated on next student login |
+| Single disk failure in a mirror | 0 | 0 (no outage) | `docs/RUNBOOK_DISK_FAILURE.md` — pool stays online DEGRADED, replace and resilver |
+| Whole storage pool lost (both mirror sides) | Up to 24h | ~1-2h | Same as "Platform VM + disks lost" above — this is precisely the scenario the off-host copy exists for; a local-only backup on the same failed pool is useless here |
+| Encryption passphrase lost | N/A | Unrecoverable | The single most important secret to protect and back up separately (e.g., in the same secrets manager as other Vault secrets, not only in `production.vault.yml`) — losing it makes every existing backup file permanently unreadable |
+
+RPO is bounded by backup cadence (daily) — a mid-day incident can lose up to a day of student
+activity. If that's ever unacceptable, the fix is a shorter cron interval or WAL archiving for
+point-in-time recovery, not decided or built here since daily has not yet been proven
+insufficient against real usage.
+
+## What backup does NOT cover
+
+Application code and container images — those live in Gitea and the image registry, both
+already durable and versioned independently of this backup mechanism. TLS certificates — those
+are regenerable (`docs/RUNBOOK_DOCKER_TLS_ROTATION.md`, `docs/RUNBOOK_TLS_RENEWAL.md`), not
+backed up.

docs/CAPACITY_PLANNING.md

@@ -0,0 +1,59 @@
+# Capacity Planning
+
+How many concurrent student Labs `rn-lab-host-01` can actually run — determined from measured
+data after deployment, not guessed or marketed now.
+
+## The model
+
+`infra/scripts/capacity-model.js`:
+
+```
+usable        = hostTotal * (1 - overheadPct/100) - platformReserved
+withMargin    = usable * (1 - marginPct/100)
+perLabLimit   = floor(withMargin / perLabUsage)
+recommended   = min(cpuBoundLimit, memoryBoundLimit)     # whichever runs out first
+warning       = floor(recommended * 0.8)
+hardAdmission = recommended
+```
+
+Computed once for CPU, once for memory; the tighter (smaller) of the two constrains the actual
+recommendation — a host can be memory-starved long before its CPU threads are exhausted, or
+vice versa, and the model doesn't pretend otherwise.
+
+**Inputs** (`infra/environments/production/capacity.yml.example` — explicitly placeholder data
+until real numbers exist):
+- `host_cpu_threads`, `host_memory_mb` — from the real Lab Host VM, once sized
+- `overhead_pct` — Proxmox/hypervisor + OS overhead reserved off the top
+- `platform_reserved_cpu`/`platform_reserved_memory_mb` — headroom for the egress proxy,
+  node-exporter, cadvisor, promtail themselves (they run on the Lab Host too)
+- `per_lab_cpu`/`per_lab_memory_mb` — **measured**, from `infra/scripts/loadtest/`, not assumed
+- `safety_margin_pct` — extra headroom below the raw math, for burst/variance
+
+**Outputs**: recommended concurrent Labs, an 80%-of-recommended warning threshold (for
+alerting — `LabHostHighMemory` in `docs/OBSERVABILITY.md`), and a hard admission-control
+threshold the Provisioner's resource policy should enforce so it refuses to boot a Lab past that
+point rather than degrading every running Lab.
+
+## Load-test tooling that produces the measured inputs
+
+`infra/scripts/loadtest/`:
+- `boot-labs.js` — reuses `platform/test/session-helper.js` (built in Phase 3A) to drive the
+  real admin Prepare-Labs endpoint, booting N Labs and recording boot time/failure rate per Lab
+  into a JSON report.
+- `sample-resources.sh` — samples real `docker stats` for every `rootnode-lab-*` container every
+  5 seconds to CSV (timestamp, name, CPU%, mem usage, mem%, net I/O, block I/O, PIDs).
+
+Together these produce the `per_lab_cpu`/`per_lab_memory_mb` this model needs, plus data on how
+boot time and failure rate change as concurrency rises — the dimensions this phase was told to
+cover: single Lab boot, multiple simultaneous boots, steady-state active Labs, terminal
+WebSocket load, scenario validation load, Gateway throughput, API request volume, Postgres
+connection activity.
+
+## What this document deliberately does not claim
+
+No concurrency number in this repo (`capacity.yml.example`'s numbers included) is a real,
+validated figure — every one is a placeholder until run against the real Lab Host VM with real
+per-Lab measurements. Treating a placeholder as a marketing/sales number would be actively
+misleading; the whole point of the model is that the real number gets filled in once, from
+measurement, after deployment (`docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md` step 4 is the
+trigger to re-measure and update it if the per-Lab footprint later changes).

docs/ENGINEERING_SOP.md

@@ -280,3 +280,56 @@ Full design: `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`.
 - **Known limitations:** see the Phase 3B doc §20 — no automatic abandoned-attempt detection, no
   per-cohort needs-attention threshold configuration, raw-JSON pool_spec authoring in the admin
   UI (correct, not yet a guided form).
+
+## 13. Production Infrastructure (Phase 4)
+
+Full design: `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`, plus detail docs
+`PRODUCTION_DEPLOYMENT.md`, `PRODUCTION_NETWORK_SECURITY.md`, `BACKUP_DR.md`,
+`OBSERVABILITY.md`, `CAPACITY_PLANNING.md`, `PRODUCTION_VALIDATION_CHECKLIST.md`, and 12
+`RUNBOOK_*.md` files, all under `docs/`. Everything lives in `infra/` (Ansible, Terraform,
+Compose, firewall, monitoring, backup, CI, load-test tooling).
+
+- **What it is:** the infrastructure-as-code needed to run rn-platform-01/rn-lab-host-01 on a
+  real Hetzner/Proxmox host — buildable and locally testable now, even though the dedicated
+  server hasn't been purchased. Status is explicitly **PHASE 4 BUILD COMPLETE / PRODUCTION
+  DEPLOYMENT PENDING HARDWARE**, not "deployed."
+- **Where the boundary held:** the Platform VM / Lab Host VM security split from
+  `docs/LAB_ARCHITECTURE.md` is codified in firewall rulesets and mutual-TLS Docker config, not
+  weakened anywhere.
+- **Known gap surfaced, then resolved in Phase 4.1:** the Gateway's Lab-reachability mechanism
+  didn't survive the move to two real separate VMs — flagged explicitly rather than silently
+  fixed, then actually built and simulation-tested (a Lab Host Router,
+  `platform/lab-router/`) in Phase 4.1. See §13 below.
+- **Validated without the hardware:** real `tofu validate`, `ansible-playbook --syntax-check`,
+  `docker compose config`, `caddy validate`, `promtool`/`amtool`, and a genuine end-to-end
+  backup→encrypt→restore→verify cycle against disposable local Postgres with the real
+  production migrations applied. None of this proves the real Hetzner network boundary — that
+  needs `docs/PRODUCTION_VALIDATION_CHECKLIST.md` walked against the real server.
+
+## 13. Pre-Production Hardening (Phase 4.1)
+
+Full design: `docs/PHASE4_1_PREPRODUCTION_HARDENING.md`, plus
+`docs/PRODUCTION_NETWORK_SECURITY.md` (routing resolution), `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`,
+`docs/SECURITY_SCANNING.md`.
+
+- **What it is:** closes the Gateway-to-Lab cross-VM gap Phase 4 flagged (a Lab Host Router,
+  `platform/lab-router/`, plus a routing abstraction in the Gateway,
+  `platform/gateway/src/lab-endpoint.js`), then hardens the application layer found along the
+  way: abandoned-attempt handling (`scenario_attempts.status = 'abandoned'`, never a punitive
+  failure), a Postgres-backed login rate limiter (survives a restart, ready to be
+  shared/multi-instance), real pagination on audit/attempt/assessment-session history, a
+  structured assessment authoring UI (owner-only raw JSON remains as a fallback), a minimal
+  `role_in_cohort` enforcement (`teacher`/`viewer`), a `/api/version` endpoint, and admission
+  control (a clean 503 "capacity full" instead of a 500 once a configured cap is reached).
+- **Tested, not just designed:** a real two-Docker-network routing simulation
+  (`infra/scripts/simulate-two-host/`, 13/13), a destructive Postgres restore drill
+  (`infra/backup/test-destructive-restore-drill.sh`), a Lab Host loss/rebuild drill
+  (`infra/scripts/drills/test-lab-host-loss.js`), a deployment rollback drill, a CI-fails-closed
+  drill (which found and fixed a real pre-existing `set -e` bug in both smoke-test scripts —
+  see that drill's own output), and a full fake-cohort dress rehearsal
+  (`infra/scripts/drills/test-fake-cohort-rehearsal.js`) — all passing, all in
+  `infra/scripts/drills/` or `infra/backup/`.
+- **Known limitation carried forward:** real remote/self-paced delivery needs actual student
+  authentication (today's `student_id`-only login relies on instructors controlling who gets a
+  valid id) — see `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md` for the full accounting of what
+  already works vs. what's genuinely Phase 5 scope.

docs/LAB_ARCHITECTURE.md

@@ -486,3 +486,15 @@ All four open questions this doc originally ended with are resolved — see the
 at the top and §2–§5 above. Next: `docs/PHASE1_PLATFORM_FOUNDATION.md` — the concrete build
 plan for the platform foundation (Postgres environment registry, provisioner skeleton,
 networking, egress, resource policy, reaper) against these decisions.
+
+## Phase 4: this boundary in production
+
+Phase 4 (`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`) codifies the Platform/Lab-Host split this
+document designed as two real, separately-firewalled Proxmox VMs (`rn-platform-01`,
+`rn-lab-host-01`) connected by two separate mTLS channels (Provisioner↔dockerd CONTROL plane,
+Gateway↔Lab Host Router DATA plane) and a tightly scoped firewall allowance — see
+`docs/PRODUCTION_NETWORK_SECURITY.md`. Phase 4 surfaced a gap in the Gateway's `env.internal_ip`
+mechanism for reaching a Lab container (it assumes Gateway and Lab share a Docker daemon, which
+breaks once they're genuinely on separate VMs); Phase 4.1
+(`docs/PHASE4_1_PREPRODUCTION_HARDENING.md`) built and simulation-tested the fix, a Lab Host
+Router (`platform/lab-router/`).

docs/OBSERVABILITY.md

@@ -0,0 +1,64 @@
+# Observability
+
+Self-hosted Prometheus + Grafana + Alertmanager + Loki/Promtail, deployed via
+`infra/docker/platform/docker-compose.production.yml` on rn-platform-01, with lightweight agents
+(node-exporter, cadvisor, promtail) also on rn-lab-host-01. No paid SaaS.
+
+This is operational observability — host/container/database health, uptime, resource pressure.
+It is deliberately separate from the application's own instructor learning analytics
+(Phase 3B's cohort/scenario/student dashboards) — different audience (ops vs. instructors),
+different data (infrastructure vs. learning outcomes), and mixing them would make both worse.
+
+## What's real today
+
+- **Host metrics** (both VMs): node-exporter — CPU, memory, disk, network.
+- **Container metrics** (Lab Host): cadvisor — per-container CPU/memory/network, including
+  student Lab containers (`rootnode-lab-*`).
+- **Postgres metrics**: postgres-exporter — connections, transaction rate, table sizes.
+- **Uptime/liveness**: blackbox-exporter probing `/healthz` on the Gateway
+  (`platform/gateway/src/server.js`) and the Platform API (`platform/api/src/server.js`), both
+  added this phase specifically for this purpose, plus the Provisioner's pre-existing `/health`.
+- **Logs**: Loki + Promtail, scraping Docker container logs and journald from both VMs, with a
+  redaction pipeline stage that strips `password|token|secret=`-pattern values before they're
+  stored (`infra/monitoring/promtail/`).
+- **Dashboards**: Grafana, reachable only via `https://<origin>/grafana/` and only from
+  `management_cidr` (see `docs/PRODUCTION_NETWORK_SECURITY.md`). One starter dashboard,
+  `infra/monitoring/grafana/provisioning/dashboards/rootnode-overview.json` — 8 panels (Platform
+  + Lab Host CPU/memory/disk, Postgres up/connections, running-Lab-count via cadvisor label
+  match [best-effort, caveated], uptime probes).
+
+## What's NOT built yet (honest accounting, not silently assumed)
+
+Application-level metrics — active student count, Lab provisioning success/failure rate,
+Gateway request latency/error rate, per-endpoint API metrics — would need real instrumentation
+added to the Platform API/Gateway/Provisioner (e.g. a `/metrics` endpoint via `prom-client`).
+`infra/monitoring/prometheus/prometheus.yml.j2` has a commented-out scrape job as the landing
+spot for this, explicitly marked as a follow-up, not wired to anything real yet.
+
+## Alerting
+
+`infra/monitoring/prometheus/rules/rootnode.yml` — two severities:
+
+**Critical** (fires fast, pages): PlatformAPIDown, GatewayDown, PostgresDown, LabHostDown,
+BackupFailed, DiskCriticallyFull.
+
+**Warning** (informational, slower repeat): LabHostHighMemory, HighDiskUsage, BackupAging.
+(RepeatedProvisioningFailures and ExcessiveFailedLogins are present as inert placeholders —
+they need the not-yet-built application metrics above before they can fire on anything real.)
+
+Never alerts on an individual student's scenario failure or a single Lab crash — those are
+expected, routine outcomes of normal usage, not operational incidents.
+
+`infra/monitoring/alertmanager/alertmanager.yml.j2` — critical alerts route separately with a
+30-minute repeat interval, everything else at 4 hours, one webhook receiver
+(`vault_alertmanager_webhook_url` — points at whatever channel gets configured at deploy time,
+e.g. a Slack/Discord/Matrix webhook), and one inhibit rule so a LabHostDown critical suppresses
+its own dependent warnings instead of paging twice for the same root cause.
+
+## Validation performed (local, real tooling — not hardware-dependent)
+
+`infra/monitoring/validate-monitoring.sh` wraps: `promtool check config` (Prometheus config),
+`promtool check rules` (alert rules — 11 rules found, valid), `amtool check-config`
+(Alertmanager — 1 inhibit rule, 2 receivers, valid). These prove syntactic/semantic correctness
+of the configs, not that the real network paths between rn-platform-01 and rn-lab-host-01 work —
+that requires the real hardware and is covered by `docs/PRODUCTION_VALIDATION_CHECKLIST.md`.

docs/PHASE4_1_PREPRODUCTION_HARDENING.md

@@ -0,0 +1,254 @@
+# Phase 4.1 — Pre-Production Hardening
+
+**Status: PHASE 4.1 COMPLETE. PRE-PRODUCTION HARDENING COMPLETE. READY FOR HARDWARE DEPLOYMENT.**
+
+Phase 4 built the production infrastructure package with the real Hetzner/Proxmox hardware not
+yet purchased, and flagged one genuine architectural gap: the Gateway's mechanism for reaching
+a student Lab assumed Gateway and Lab shared a Docker daemon, which is false the moment Labs run
+on a genuinely separate `rn-lab-host-01` VM. Phase 4.1's job was to close that gap and harden
+everything else discovered along the way, still without the real hardware.
+
+## 1. The core fix: a Lab Host Router
+
+**Before:** Gateway → `http://${env.internal_ip}:${env.internal_port}` directly — a Docker
+bridge-local address, unreachable across a real VM boundary.
+
+**After:** Gateway → (mode-aware) → either `direct` (unchanged, dev/single-host) or `router`
+(production): a small new service, `platform/lab-router/`, runs on the Lab Host itself,
+authenticates the Gateway via mTLS (the same private CA the Docker TLS control plane already
+used, extended with two new leaf certs), resolves an opaque routing token to a local container
+purely from `docker ps` label state (no Postgres access, no call back to the Provisioner), and
+reverse-proxies HTTP and WebSocket traffic to it.
+
+The decision of which mode a given `lab_host` uses is data-driven
+(`lab_hosts.routing_mode`/`router_endpoint`, `platform/provisioner/migrations/004_lab_hosts_routing.sql`),
+resolved once per request in `platform/gateway/src/lab-endpoint.js`'s `labTarget()` — the only
+place in the Gateway that knows this distinction exists. Every proxy/fetch call site in
+`platform/gateway/src/server.js` goes through it.
+
+**Exact `env.internal_ip` replacement:** `labTarget(env, token)` returns `{ url, headers, agent,
+mode }`. In `direct` mode, `url` is exactly the old `http://internal_ip:internal_port`. In
+`router` mode, `url` is the lab_host's `router_endpoint`, `headers` carries
+`X-RootNode-Route-Token: <token>` (computed server-side from the Provisioner-verified
+environment row — never trusted from the browser), and `agent` is an `https.Agent` configured
+with the Gateway's mTLS client identity.
+
+## 2. No per-student host port
+
+Deliberately not solved by publishing a host port per Lab (the originally-flagged, rejected
+approach). The Router resolves entirely from Docker's own label state on its own host — no
+port-per-student anything, no dynamic firewall rule per Lab.
+
+## 3. Lab Host Router — design
+
+See `platform/lab-router/src/server.js`'s header comment for the full "narrow by design" list:
+one route (proxy-by-token), requires mTLS (no bearer-token fallback), never exposes Docker API/
+container IDs/network info to its caller, never provides student-to-student routing (each
+request resolves to exactly the one container matching its own token).
+
+## 4. Multi-Lab-Host ready
+
+Routing is resolved per `lab_host_id` — adding `rn-lab-host-02` later is one new `lab_hosts` row
+(its own `routing_mode`/`router_endpoint`/TLS material), no Gateway code change. No scheduler
+was built (explicitly out of scope) — the Provisioner still picks "the one enabled lab_host"
+(`selectLabHost`), unchanged from Phase 1.
+
+## 5. Local development compatibility
+
+`dev-local` (the Unix-socket lab_host used by `docker-compose.dev.yml`) stays `routing_mode:
+direct` — zero behavior change for the T480/any local dev setup, confirmed by the full existing
+regression suite passing unmodified (§29 below) and by every new drill in this phase running
+against the same dev stack. The standalone single-container Lab image build/boot is unaffected
+(it never touches the Gateway or Provisioner at all).
+
+## 6-9. Simulation, HTTP/WS validation, security tests, routing token security
+
+`infra/scripts/simulate-two-host/` — a real two-Docker-network simulation (not mocks): Gateway
+attached ONLY to an isolated bridge with no route to any Lab's bridge network, Lab Host Router
+host-networked (mirrors `egress-proxy`'s existing pattern) so it can reach whatever per-Lab
+bridge the Provisioner creates. 13/13 real assertions pass, including:
+
+- Gateway genuinely cannot open a TCP connection to a Lab's Docker-internal IP (proven from
+  inside the Gateway container's own network namespace via a raw connect attempt).
+- Gateway reaches the Lab correctly through the Router — HTTP AND a real WebSocket terminal
+  session running a genuine `kubectl get nodes` inside the Lab.
+- Alice/Bob isolation (own lab ALLOW, other's DENY — HTTP and WebSocket both), garbage tokens,
+  anonymous requests, and destroyed-environment tokens are all rejected through the real code
+  path.
+- The Router refuses any connection with no client certificate — mTLS is the real
+  authentication mechanism.
+
+Routing token security reviewed: already CSPRNG since Phase 1 (`generateRoutingToken`), never
+logged in plaintext (the Router's `tokenFingerprint` truncates), redacted in every API response
+except the two narrow internal reads that legitimately need it, and — critically — the header
+the Router trusts is computed server-side from the Provisioner-verified environment row, never
+read from the browser's own request, so a browser cannot forge routing by supplying its own
+`X-RootNode-Route-Token`.
+
+## 10. Control plane vs data plane
+
+Kept strictly separate — see `docs/PRODUCTION_NETWORK_SECURITY.md`'s dedicated section. Student
+browser traffic is never proxied through the Docker API at any point.
+
+## 11. Firewall update
+
+`infra/firewall/rn-lab-host-01.rules`'s old speculative wide port-range allowance replaced with
+one narrow rule: `platform_ip` → the Router's port only (`8443` default, configurable).
+
+## 12. Self-paced / remote delivery architecture check
+
+`docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`. Short version: student-driven provisioning,
+cohort-less students, and "idle reaper destroys → progress survives → student returns → fresh
+Lab → curriculum continues" all already worked before this phase and are now proven end-to-end
+by two new drills. The one real gap for actual remote/self-paced delivery is student
+authentication (today's bare `student_id` login assumes an instructor-controlled roster) — not
+built here, explicitly Phase 5 scope.
+
+## 13. Abandoned attempt cleanup
+
+`scenario_attempts.status = 'abandoned'` (already a valid value since Phase 3B, never
+previously set) now gets set on: environment destroy/reset (student- or instructor-triggered),
+logout, session expiry (piggybacked on the existing hourly session sweeper — no new background
+job), and switching to a different scenario while one is still open. Never a punitive marker —
+`result` stays NULL, history is preserved, instructors see an honest "abandoned (reason)"
+instead of a phantom "still in progress."
+
+## 14. Gateway session cache
+
+Reviewed — already a tight 2-second TTL since Phase 3A, specifically to bound the
+revocation-propagation window. Decision: **keep it, don't add push-based invalidation** — the
+added coupling (Platform API would need to know how to reach the Gateway) isn't justified by
+shrinking an already-2-second window further. Documented reasoning, not a silently-accepted gap.
+
+## 15. Rate limiting
+
+Login rate limiting moved from an in-memory `Map` (reset on every restart, not shareable across
+instances) to Postgres (`login_attempts` table, `migrations/016_login_attempts.sql`) —
+`platform/api/src/security.js`'s `tooManyAttempts()`. No Redis added. Restart behavior
+confirmed for real: `platform/test/e2e-phase3a.js`'s rate-limiting test passes after a genuine
+Platform API restart in the same run.
+
+## 16. Pagination
+
+Real server-side pagination (`platform/api/src/pagination.js`) on: the audit log (was a flat
+`LIMIT 200`), student attempt history (was `LIMIT 100`), and assessment session history (was
+**fully unbounded** — the one genuine growth risk found). Scenario analytics was deliberately
+left as-is — grouped by scenario id, naturally bounded around ~100 rows, no real growth risk to
+paginate against. Frontend (`AuditView.jsx`, `AssessmentsView.jsx`, `StudentsView.jsx`) updated
+to match; `frontend` build confirmed clean.
+
+## 17. Needs-attention configuration
+
+`NEEDS_ATTENTION_FAIL_THRESHOLD`/`NEEDS_ATTENTION_STALE_DAYS` moved from hardcoded constants to
+env-var-configurable (matching the existing session-TTL pattern), sensible defaults unchanged.
+
+## 18. Assessment authoring UX
+
+Raw-JSON `pool_spec` textarea replaced with a structured editor
+(`frontend/src/admin/AssessmentsView.jsx`'s `PoolSpecEditor`): required scenarios and random
+pools picked from a real scenario catalog (`GET /api/admin/scenario-catalog`, new), grouped by
+curriculum stage, via native multi-select — no new UI framework. An owner-only "advanced" raw
+JSON view remains for anything the structured editor doesn't cover.
+
+## 19. Cohort role-in-cohort
+
+`cohort_instructors.role_in_cohort` existed since Phase 2B but was never read. Implemented
+minimally: `teacher` (unchanged full capability) vs `viewer` (read-only — dashboards/analytics
+still visible, but Prepare/Destroy Labs and provisioning a student's environment are now 403'd
+server-side, `rbac.js`'s `requireCohortWriteAccess`/`requireStudentCohortWriteAccess`).
+
+## 20. Backup / restore destructive drill
+
+`infra/backup/test-destructive-restore-drill.sh` — real owner/admin/cohort/3-students/progress/
+attempts/assessment/achievements seeded through the real Platform API, encrypted-backed-up, the
+disposable Postgres container **genuinely destroyed** (not just restored into a second
+container), a fresh Postgres created from nothing, restored, and the real Platform API booted
+again against the restored database and used to log in as the restored admin over HTTP. All
+data verified present; never touches the developer's real database.
+
+## 21. Lab Host rebuild drill
+
+`infra/scripts/drills/test-lab-host-loss.js` — a student's Lab container is destroyed out from
+under the Provisioner (no clean destroy call, simulating total Lab Host loss). Proves: Platform
+stays up throughout, the stale environment is never presented as healthy once the Provisioner's
+real `reconcileOnStartup` reconciliation runs, academic progress is untouched, and the student
+can launch a genuinely new Lab and continue.
+
+## 22. Deployment rollback test
+
+`infra/scripts/drills/test-deployment-rollback.sh` — builds a known-good and a deliberately
+broken gateway image, "deploys" each via the real deterministic-tag mechanism, proves the smoke
+test catches the regression, computes the rollback target with the exact same
+`deploy-history.log` logic `rollback-production.sh` uses, and proves the rollback recovers.
+Documents explicitly that image rollback never implies a DB migration rollback.
+
+## 23. CI/CD failure tests
+
+`infra/scripts/drills/test-ci-fails-closed.sh` — proves all six required failure modes (test/
+syntax failure, frontend build failure, image build failure, config validation failure,
+migration preflight failure, staging smoke test failure) genuinely exit nonzero. **Found and
+fixed two real pre-existing bugs**: `staging-smoke-test.sh` and `production-smoke-test.sh` both
+had a classic `set -e` gotcha (`curl ... && echo "OK"` as one statement — a failing command
+that isn't the last in an `&&` chain doesn't trigger `-e`) that let an unreachable/broken
+endpoint fall through silently instead of failing the smoke test. Fixed in both files.
+
+## 24. Security scanning
+
+`infra/ci/gitea-workflows/build-and-test.yml`'s new `security-scan` job — npm audit, gitleaks
+(secrets), hadolint (Dockerfiles), shellcheck (shell scripts), trivy (container image CVEs).
+All free/open-source. Severity policy documented in `docs/SECURITY_SCANNING.md`: CRITICAL/HIGH
+blocks the build, MEDIUM/LOW is reported only. `security-scan` gates `build-images` the same
+way `regression` does.
+
+## 25. Version / release visibility
+
+`GET /api/version` (Platform API) — app version (`package.json`), deployed Git SHA
+(`GIT_SHA` env var, set to the same value as the image tag in the production compose file), and
+the most recently applied migration as a schema-version stand-in. Unauthenticated, minimal, no
+infrastructure detail.
+
+## 26. Admission control failure mode
+
+`resource-policy.yaml`'s new (disabled-by-default) `admission_control.max_concurrent_labs` —
+when set, the Provisioner checks it BEFORE touching Docker at all, so a full host fails cleanly
+(503, `CAPACITY_EXCEEDED`) instead of failing mid-provision. Relayed by the Platform API as a
+clean "Lab capacity is temporarily full. Try again shortly." — never a raw 500. A new
+`GET /capacity` Provisioner endpoint feeds the instructor dashboard's capacity display.
+
+## 27. Fake cohort dress rehearsal
+
+`infra/scripts/drills/test-fake-cohort-rehearsal.js` — one continuous narrative: instructor
+login, cohort + 4 students, curriculum assignment, Prepare Labs, students working different
+scenarios, hints, a deliberate failed validation, instructor analytics, a Lab reset, a Lab
+reaped (same consequence as the real idle reaper — see the drill's own comment on why the real
+multi-minute timer isn't waited out), the student returning and continuing, an exam
+assigned/taken, and end-of-class Lab destruction — progress verified intact after every
+destructive step. 14/14 passing.
+
+## 28. Documentation
+
+This file, `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`, `docs/SECURITY_SCANNING.md` (new).
+Updated: `docs/LAB_ARCHITECTURE.md`, `docs/PRODUCTION_NETWORK_SECURITY.md`,
+`docs/PRODUCTION_DEPLOYMENT.md`, `docs/PRODUCTION_VALIDATION_CHECKLIST.md`, `docs/BACKUP_DR.md`,
+`docs/ENGINEERING_SOP.md`, `README.md`, `platform/README.md`.
+
+## 29. Regression
+
+Re-ran the full existing suite after every application-code change in this phase:
+`e2e-phase2a.js` (10/10), `e2e-phase2b.js` (12/12), `e2e-phase3a.js` (21/21 — including a real
+audit-log pagination-shape fix and confirming rate-limit persistence across a genuine restart),
+`e2e-phase3b.js` (24/24). Frontend production build clean. Real Lab image builds/boots (both
+dev-compose and standalone single-container). `tofu validate`, `ansible-playbook
+--syntax-check`, production Compose config (both platform and lab-host), `caddy validate`,
+`promtool`/`amtool`, and the backup/restore mechanism all re-validated clean after the
+infrastructure changes in this phase (new `lab-router` service, new firewall rule, new Ansible
+tasks). No knowingly-unexplained regression was left in this codebase at the end of this phase.
+
+## What Phase 4.1 deliberately did NOT build
+
+Production deployment itself, Hetzner/Proxmox installation, public course signup, payments,
+subscriptions, parent portal, marketing site, full LMS, video hosting, email automation,
+certificates, course marketplace, mobile app, a multi-host scheduler, an AI tutor, invasive
+proctoring, held scenario content, student email/password authentication (documented as the one
+real gap for self-paced delivery, not built). All explicit non-goals per this phase's
+authorization.

docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md

@@ -0,0 +1,187 @@
+# Phase 4 — Production Infrastructure Package
+
+**Status: PHASE 4 BUILD COMPLETE. PRODUCTION DEPLOYMENT PENDING HARDWARE.**
+
+The dedicated Hetzner server this is designed for hasn't been purchased yet. Everything in this
+document and the `infra/` tree it describes is buildable, testable (against local
+Docker/Postgres/disposable certs), and reviewable **now** — none of it requires the real
+hardware to exist. What genuinely can't happen until the hardware exists is called out
+explicitly throughout, never silently assumed to already work.
+
+This document is the map. Detail docs: `docs/PRODUCTION_DEPLOYMENT.md` (step-by-step),
+`docs/PRODUCTION_NETWORK_SECURITY.md` (network/firewall/TLS design), `docs/BACKUP_DR.md`
+(backup/restore/disaster recovery), `docs/OBSERVABILITY.md` (metrics/logs/alerts),
+`docs/CAPACITY_PLANNING.md` (sizing model), and the `docs/RUNBOOK_*.md` files (14 topics,
+command-oriented).
+
+## 1. What was built
+
+```
+infra/
+├── proxmox/          manual bare-metal install docs + cloud-init template definition
+├── terraform/         VM creation (rn-platform-01, rn-lab-host-01) — real `tofu validate` pass
+├── ansible/            all host configuration — real `ansible-playbook --syntax-check` pass
+│   ├── roles/          common, firewall, docker-tls, platform, lab-host, monitoring-agent, backup
+│   ├── group_vars/     production.yml, staging.yml, vault template
+│   └── site.yml        top-level playbook, correct role ordering
+├── docker/            production Compose files, Caddyfile, Postgres tuning — real `docker
+│                       compose config` + `caddy validate` pass
+├── firewall/           iptables-restore rulesets for all 3 hosts — structural lint pass
+├── monitoring/          Prometheus/Alertmanager/Loki/Promtail/Grafana config — real `promtool`
+│                        + `amtool` validation pass
+├── backup/              Postgres backup/restore/verify scripts — REAL end-to-end test against
+│                        disposable local Postgres, including the actual production migrations
+├── scripts/             deploy/rollback/smoke-test/CI/loadtest/capacity-model tooling
+├── ci/                  Gitea Actions workflow (build/test/stage automatic, prod manual)
+└── environments/        dev/staging/production config templates, no secrets committed
+```
+
+18 new docs (this file + 5 detail docs + 12 runbooks), 4 updated (`LAB_ARCHITECTURE.md`,
+`ENGINEERING_SOP.md`, `README.md`, `platform/README.md`).
+
+## 2. Architecture (unchanged from the design, now codified)
+
+```
+Hetzner Dedicated Server
+└── Proxmox VE
+    ├── rn-platform-01   (frontend, Platform API, Gateway, Provisioner, Postgres,
+    │                      Prometheus/Alertmanager/Grafana/Loki, backup cron, Caddy)
+    └── rn-lab-host-01   (Docker Engine w/ TLS, egress proxy, disposable student Labs,
+                           node-exporter/cadvisor/promtail)
+```
+
+The Platform VM / Lab Host VM security boundary from `docs/LAB_ARCHITECTURE.md` is the one
+thing this phase was explicitly told never to weaken — it's codified, not just described, in
+`infra/firewall/` and `infra/ansible/roles/docker-tls/`.
+
+## 3. VM specifications
+
+| | rn-platform-01 | rn-lab-host-01 |
+|---|---|---|
+| Baseline (default in `infra/terraform/variables.tf`) | 4 cores, 16 GB RAM, 100 GB disk | 24 cores, 96 GB RAM, 500 GB disk |
+| Recommended (once real host is ~16c/32t/128GB) | same — Postgres/API/Gateway/Provisioner don't need much | majority of remaining host resources |
+| Max-growth | scale RAM with Postgres `shared_buffers` if instructor/cohort count grows a lot | scale with measured concurrent-Lab capacity (`docs/CAPACITY_PLANNING.md`) |
+| Configurable via | `terraform.tfvars`' `platform_vm`/`lab_host_vm` objects | same |
+
+NUMA/CPU pinning: not configured — not useful at this scale (a handful of VMs on one physical
+host, not a dozen contending for cache locality). Revisit only if real load testing shows a
+reason to.
+
+## 4. Storage decision framework
+
+Not a final answer — `infra/environments/production/provisioner-config/README.md` and this
+section exist so the final ZFS layout is a deployment-time decision made against real disks,
+not guessed now. Candidate layouts (`infra/terraform/variables.tf`'s `vm_disk_datastore` names
+whichever one gets chosen):
+
+- **4× NVMe**: mirrored vdevs (RAID10-equivalent) — two mirror pairs, one for
+  Postgres/platform data (I/O-sensitive, small), one for Lab Host disposable data
+  (throughput-sensitive, large, never needs redundancy since nothing on it is precious).
+- **2× SSD/NVMe**: one mirrored vdev, logical dataset separation (ZFS datasets, not separate
+  pools) between platform (`{{ rootnode_data_root }}/postgres`, `/backups`) and Lab Host data.
+- **Mixed SSD/HDD**: only if the purchased server's actual disks require it — HDD for
+  disposable Lab scratch space is acceptable (it's disposable), never for Postgres.
+
+**Never single-disk for Postgres data**, in any layout — a mirror is the floor, not a nice-to-have.
+
+Pre-deployment disk inspection checklist (model, size, SMART, power-on hours, wear/endurance,
+sector errors, temperature, firmware): `docs/RUNBOOK_DISK_FAILURE.md`'s inspection section —
+run once, against the real drives, before committing to a layout.
+
+## 5-8. Network, firewall, Docker TLS, public ingress
+
+Full detail: `docs/PRODUCTION_NETWORK_SECURITY.md`. Summary: two private VM networks, Docker TLS
+(mutual, already application-supported since Phase 1 — this phase only needed the certificate
+generation/deployment automation, not new app code), a single public origin (Caddy, automatic
+HTTPS) proxying only to the Gateway, and one genuinely new architectural item this phase
+surfaced and addressed at the network-policy layer: **Gateway-to-Lab reachability across two
+real VMs** — see that doc's dedicated section for what's built vs. what still needs a small,
+reviewed Provisioner/Gateway code change before real go-live.
+
+## 9-10. Production auth/cookie config, secrets
+
+`infra/environments/production/*.env.template` set `COOKIE_SECURE=1`/`TRUST_PROXY=1` explicitly
+(dev's unsafe defaults are never inherited). Secrets: Ansible Vault (already bundled with
+Ansible — no new tool to install/trust for this project's secret surface), template at
+`infra/ansible/group_vars/production.vault.yml.template`. Nothing is committed —
+`infra/.gitignore` covers `*.env`, TLS private key material, vault plaintext, Terraform state.
+
+## 11-13. Postgres, backup, DR
+
+Full detail: `docs/BACKUP_DR.md`. Summary: conservative tuning baseline (not hand-tuned before
+real RAM is known), daily encrypted backups with an off-host adapter point (Storage Box/S3/any
+SSH target), and a **real, tested** backup→encrypt→restore→verify cycle
+(`infra/backup/test-backup-restore.sh`) — run against disposable local Postgres with the actual
+production migrations applied, not a mock.
+
+## 14-17. Observability, logging, alerting, uptime
+
+Full detail: `docs/OBSERVABILITY.md`. Prometheus + Grafana + Loki/Promtail (the user's stated
+Prometheus/Grafana preference, honored) + Alertmanager, all self-hosted, no paid dependency.
+Deliberately does not duplicate the application's own instructor learning analytics
+(Phase 3B) — operational observability and learning analytics stay separate concerns with
+separate audiences, per the explicit instruction. Application-level metrics (active
+students/Labs, provisioning failures, Gateway error rate) are a documented, **not-yet-built**
+follow-up — see that doc's honest accounting of what's real today (host/container/Postgres
+metrics, HTTP healthz probes) vs. what needs app instrumentation later.
+
+## 18-19. CI/CD, image strategy
+
+`infra/ci/gitea-workflows/build-and-test.yml` — automatic test/build/tag/staging-deploy,
+**manual** production promotion (`infra/scripts/deploy-production.sh`), matching the stated
+preference. SHA-tagged images always, semantic tags on deliberate releases, never `:latest` in
+staging/production. Full detail: `infra/docker/IMAGE_STRATEGY.md`.
+
+## 20-21. Ansible, Terraform
+
+`infra/ansible/site.yml` — idempotent roles, doesn't assume the VMs already exist (Terraform
+creates them first). Real `ansible-playbook --syntax-check` pass. `infra/terraform/` — real
+`tofu validate` pass against the actual `bpg/proxmox` provider schema. Neither installs Proxmox
+itself or builds the base VM template — both stay deliberate manual, documented, one-time steps
+(`infra/proxmox/README.md`), per the explicit "don't over-automate the fragile one-time step"
+instruction.
+
+## 22-23. Environment config, staging
+
+`infra/environments/{dev,staging,production}/` — explicit separation, production never
+inherits dev defaults. Staging is designed to run in the homelab or a resource-constrained
+slice of the real server later — no second dedicated server required.
+
+## 24-25. Load testing, capacity model
+
+`infra/scripts/loadtest/` (real Prepare-Labs-driven boot test + `docker stats` sampling) and
+`infra/scripts/capacity-model.js` (a real, working worksheet — tested against example inputs,
+produces CPU-vs-memory-bound recommended/warning/hard-admission-control thresholds). No
+concurrency number is hard-coded as a marketing figure; the example config in
+`infra/environments/production/capacity.yml.example` is explicitly flagged as placeholder data.
+
+## 26-27. Validation checklist, runbooks
+
+`docs/PRODUCTION_VALIDATION_CHECKLIST.md` — one master checklist spanning hardware through
+academy-workflow verification. 12 runbooks (`docs/RUNBOOK_*.md`) — first deployment, normal
+deploy, rollback, Postgres restore, VM rebuild (both hosts), Docker TLS rotation, TLS renewal,
+owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host
+resource pressure.
+
+## What this phase deliberately did NOT touch
+
+Learning system design, curriculum content, RBAC model, session architecture — all Phase 3A/3B,
+unchanged. No parent portal, billing, held scenarios, or production OIDC provider integration.
+RootNode production infrastructure is dedicated to RootNode Academy — never folded into
+existing Synthos production servers, and canonical source stays Gitea, never GitHub `kubekosh`.
+
+## Known gap surfaced during this phase (not silently papered over) — RESOLVED in Phase 4.1
+
+Building the real production Compose/network config surfaced that the Gateway's existing
+mechanism for reaching a Lab container (`env.internal_ip`, a Docker-bridge-local address) only
+works when Gateway and Lab share a Docker daemon — true in every dev/single-host setup so far,
+**not** true once Labs run on a genuinely separate `rn-lab-host-01`. At the time this phase
+closed, the network-policy half was built but the application half was flagged as an explicit,
+not-yet-built follow-up rather than guessed at mid-infra-phase.
+
+**Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1) built and proved the application
+half** — a Lab Host Router (`platform/lab-router/`) plus a `labTarget()` routing abstraction in
+the Gateway, real-tested against a genuine two-network Docker simulation
+(`infra/scripts/simulate-two-host/`), not just designed. See
+`docs/PRODUCTION_NETWORK_SECURITY.md`'s "Gateway-to-Lab reachability" section for the full
+resolution. This item no longer blocks real two-VM go-live.

docs/PRODUCTION_DEPLOYMENT.md

@@ -0,0 +1,90 @@
+# Production Deployment
+
+Step-by-step, for when the real Hetzner host exists. See
+`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` for the architecture this executes.
+
+## 0. One-time, manual, not automated (deliberately)
+
+1. Install Proxmox VE on the bare-metal host — `infra/proxmox/README.md`.
+2. Inspect the real disks, pick a ZFS layout — `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4,
+   `docs/RUNBOOK_DISK_FAILURE.md`'s inspection checklist.
+3. Build the cloud-init VM template Terraform will clone —
+   `infra/proxmox/cloud-init/user-data.yaml` + `infra/proxmox/README.md`.
+4. Fill in `infra/terraform/terraform.tfvars` (copy from `.tfvars.example`) with the real
+   Proxmox API endpoint, node name, storage/datastore names, and chosen VM sizes.
+5. Fill in `infra/ansible/inventory/production.ini` with the real VM IPs, and
+   `infra/ansible/group_vars/production.yml` with `platform_ip`, `lab_host_ip`,
+   `management_cidr`, `proxmox_mgmt_cidr`.
+6. Copy `infra/ansible/group_vars/production.vault.yml.template` →
+   `production.vault.yml`, fill in every `vault_*` secret with real generated values, then
+   `ansible-vault encrypt group_vars/production.vault.yml`. Never commit the plaintext.
+
+## 1. Create the VMs
+
+```bash
+cd infra/terraform
+tofu init
+tofu plan    # review before applying against real hardware
+tofu apply
+```
+
+Creates `rn-platform-01` and `rn-lab-host-01` from the cloud-init template. Confirm both boot
+and are SSH-reachable from the management network before continuing.
+
+## 2. Generate TLS certificates (offline, once)
+
+```bash
+export ROOTNODE_TLS_BUNDLE_DIR=/secure/offline/path   # never inside the repo
+./infra/scripts/docker-tls/generate-docker-tls.sh
+./infra/scripts/docker-tls/verify-docker-tls.sh
+```
+
+Generates FOUR leaf certs under one CA (Phase 4.1 extended this from two): the CONTROL-plane
+pair (`server-*`/`provisioner-client-*`, for Provisioner↔dockerd) and the DATA-plane pair
+(`router-server-*`/`gateway-client-*`, for Gateway↔Lab Host Router — see
+`docs/PRODUCTION_NETWORK_SECURITY.md`'s "control plane vs data plane" section). The CA private
+key stays in `ROOTNODE_TLS_BUNDLE_DIR` and never touches either VM — only the CA cert (public)
+and each host's own leaf cert+key go to each host, both pairs deployed by the same `docker-tls`
+Ansible role into two separate directories (`/etc/rootnode/docker-tls/` and
+`/etc/rootnode/lab-router-tls/`).
+
+## 3. Configure both hosts
+
+```bash
+cd infra/ansible
+ansible-playbook -i inventory/production.ini site.yml --ask-vault-pass
+```
+
+Role order matters and is fixed in `site.yml`: `common` → `firewall` → `docker-tls` →
+`platform`/`lab-host` → `monitoring-agent` → `backup`. Firewall and TLS land before anything
+that depends on them.
+
+## 4. First application deploy
+
+```bash
+./infra/scripts/deploy-production.sh <image-tag>
+```
+
+Pulls the tagged images, runs the compose stack, applies Postgres migrations (automatic, on
+Platform API/Provisioner startup — see `docs/RUNBOOK_FAILED_MIGRATION.md` if one fails).
+
+## 5. Verify
+
+```bash
+./infra/scripts/production-smoke-test.sh
+```
+
+Curls `/healthz` on the Gateway and Platform API, confirms Postgres is reachable, confirms
+Prometheus/Grafana/Alertmanager are up. Then walk
+`docs/PRODUCTION_VALIDATION_CHECKLIST.md` in full before calling the deployment live.
+
+## Normal (non-first) deploys
+
+`docs/RUNBOOK_NORMAL_DEPLOY.md`. Rollback: `docs/RUNBOOK_ROLLBACK.md`.
+
+## What this document does NOT cover
+
+DNS/Cloudflare setup for the public origin (a one-time manual step outside this repo's scope —
+point the A/AAAA record at the real host and confirm Caddy issues its certificate). Which
+storage layout to pick (decision framework only, §4 of the master doc — the real answer depends
+on the real disks).

docs/PRODUCTION_NETWORK_SECURITY.md

@@ -0,0 +1,130 @@
+# Production Network & Security
+
+The Platform VM / Lab Host VM boundary is the core production control. This document is where
+every network decision that touches it gets written down. Nothing here weakens it.
+
+## Topology
+
+```
+Internet ──HTTPS(443)──▶ Caddy (rn-platform-01) ──▶ Gateway (internal bridge)
+                                                        │
+management_cidr ──SSH──▶ rn-platform-01                │
+management_cidr ──SSH──▶ rn-lab-host-01                 │
+proxmox_mgmt_cidr ──22/8006──▶ Proxmox host             │
+                                                        │
+rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd)          [CONTROL plane]
+rn-platform-01 (Gateway) ──mTLS:8443──▶ rn-lab-host-01 (Lab Host Router)     [DATA plane]
+rn-lab-host-01 (promtail) ──3100──▶ rn-platform-01 (Loki)
+rn-lab-host-01 (node-exporter/cadvisor) ◀──scrape── rn-platform-01 (Prometheus)
+```
+
+Firewall source of truth: `infra/firewall/{proxmox-host,rn-platform-01,rn-lab-host-01}.rules`
+(iptables-restore format, deployed by the `firewall` Ansible role). Validate with
+`infra/firewall/validate-firewall.sh`.
+
+## What is explicitly denied
+
+- Student Lab → another student Lab (no student-to-student traffic, ever).
+- Student Lab → Proxmox management (both the host firewall and the `lab_subnet_range` DENY
+  rule in `rn-lab-host-01.rules`).
+- Student Lab → Platform Postgres / Platform API / Gateway internals (logged DENY in
+  `rn-platform-01.rules` for anything from `lab_host_ip` to ports 5432/4100).
+- rn-lab-host-01 → Postgres (same rule — the Lab Host itself has no legitimate reason to reach
+  Postgres; only the Provisioner on rn-platform-01 does, and it talks to Postgres over the
+  internal Docker bridge, never across the VM boundary).
+- Any student egress except through the approved egress proxy (enforced by the Provisioner's
+  existing per-Lab `DOCKER-USER` chain manipulation from Phase 1 — unchanged by this phase).
+- Direct public exposure of the Provisioner, Postgres, Docker TLS port, Proxmox UI, or Grafana
+  (Grafana is reachable only via Caddy's `/grafana/*` path, itself restricted to
+  `management_cidr` by a `caddy` matcher — see `infra/docker/platform/Caddyfile.j2`).
+
+## Control plane vs data plane (Phase 4.1 §10)
+
+Two separate, never-conflated mTLS connections between the two VMs, deliberately kept apart
+both conceptually and operationally:
+
+| | Control plane | Data plane |
+|---|---|---|
+| What | Provisioner manages Lab containers | Gateway serves student browser traffic |
+| Path | Provisioner → dockerd (port 2376) | Gateway → Lab Host Router (port 8443) |
+| Who initiates | Provisioner only | Gateway only |
+| What it can do | Create/start/stop/remove containers, full Docker API | Reverse-proxy one HTTP/WS request to one already-running container by opaque token — nothing else |
+| Certs | `provisioner-client-*` / `server-*` | `gateway-client-*` / `router-server-*` |
+| Exposes Docker API to caller? | Yes (that's its job — Provisioner IS a Docker client) | **Never** — the Router never exposes Docker API, container IDs, or network info to its caller |
+
+The student browser's traffic is proxied at the Gateway/Router layer and never touches the
+Docker API at any point — the Router deliberately has no route that echoes back anything
+Docker-internal (see `platform/lab-router/src/server.js`'s header comment for the full "narrow
+by design" list).
+
+## Docker TLS (Provisioner ↔ Lab Host)
+
+Mutual TLS on port 2376, restricted at the firewall to `platform_ip` only. Application support
+already existed since Phase 1 (`platform/provisioner/src/docker.js`'s `buildClient()`); this
+phase added the operational half: `infra/scripts/docker-tls/generate-docker-tls.sh` (CA +
+server + client cert generation), the `docker-tls` Ansible role (deploys certs, configures
+`dockerd`'s `daemon.json`), and `docs/RUNBOOK_DOCKER_TLS_ROTATION.md` for renewal. The CA
+private key is generated offline and never committed or deployed to either VM — only the CA
+cert (to verify the peer) and each host's own leaf cert+key.
+
+## Public ingress
+
+One origin, Caddy, automatic HTTPS. `reverse_proxy gateway:4200` — Caddy talks only to the
+Gateway; nothing else is proxied. WebSocket upgrade works automatically (Caddy's default
+`reverse_proxy` behavior, no special config needed for the terminal WebSocket path). Whether to
+front this with Cloudflare or expose the origin directly is left as a deployment-time decision
+(both work with Caddy's ACME flow; Cloudflare adds DDoS absorption at the cost of Cloudflare
+being able to see plaintext unless "Full (strict)" mode is used) — not decided here since it
+doesn't affect anything else in this document.
+
+## Gateway-to-Lab reachability across two real VMs — RESOLVED in Phase 4.1
+
+Phase 4 flagged this as the single most important open item; Phase 4.1
+(`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1) closed it. Kept here as the historical record of
+what the gap was and how it was closed, since the reasoning matters for anyone extending this
+later (e.g. adding a second Lab Host).
+
+**The gap, as originally found:** the Gateway used to reach a running Lab container via
+`env.internal_ip` — a Docker bridge-local address recorded by the Provisioner when it creates
+the container. That only works when Gateway and the Lab share a Docker daemon, true in dev and
+any single-host setup, false the moment Labs run on a genuinely separate `rn-lab-host-01`.
+
+**The fix — a Lab Host Router (`platform/lab-router/`):** a small, narrow service that runs on
+rn-lab-host-01 (host-networked, like `egress-proxy`, so it can reach whatever per-Lab bridge
+network the Provisioner creates) and does exactly one thing: given an opaque routing token over
+mTLS, find the ONE local container labeled with that token (`docker ps --filter
+label=rootnode.routing-token=<token>` — resolved purely from local Docker state, no Postgres,
+no call back to the Provisioner) and reverse-proxy the request to it, HTTP and WebSocket both.
+
+The Gateway now decides `direct` vs `router` per `lab_host` (`lab_hosts.routing_mode` +
+`router_endpoint`, `platform/provisioner/migrations/004_lab_hosts_routing.sql`) via
+`platform/gateway/src/lab-endpoint.js` — a single function, `labTarget(env, token)`, that every
+proxy/fetch call site in `server.js` goes through instead of building
+`http://${env.internal_ip}:${env.internal_port}` itself. `dev-local` stays `direct` (T480/local
+dev unaffected, zero behavior change there); a real `rn-lab-host-01` is configured `router`.
+
+**Firewall (`infra/firewall/rn-lab-host-01.rules`):** the old speculative
+`lab_gateway_port_range` allowance (which assumed option (a), publishing per-Lab ports) was
+replaced with a single, narrow rule: `platform_ip` → port `{{ lab_router_port | default('8443')
+}}` only. No wide port range, no per-Lab port publishing at all — the Router is the only thing
+the Gateway ever talks to on rn-lab-host-01 for Lab traffic.
+
+**mTLS:** the Gateway↔Router pair uses the SAME private CA as the Provisioner↔dockerd pair
+(`infra/scripts/docker-tls/generate-docker-tls.sh`, extended in Phase 4.1 to also issue
+`router-server`/`gateway-client` leaf certs), deployed to a separate directory
+(`/etc/rootnode/lab-router-tls/`) by the same `docker-tls` Ansible role.
+
+**Proven, not just designed:** `infra/scripts/simulate-two-host/` builds a real two-network
+Docker simulation (Gateway genuinely isolated on its own bridge, no route to any Lab's bridge)
+and runs 13 real assertions against it — Gateway cannot reach a Lab's Docker-internal IP
+directly, but reaches it correctly through the Router (HTTP and a real WebSocket
+`kubectl get nodes`), Alice/Bob isolation holds, garbage/destroyed tokens are rejected, and the
+Router itself refuses any connection without the Gateway's client certificate. See that
+directory's README for exactly what is and isn't proven by a same-daemon simulation versus real
+separate hardware.
+
+## Cookie / auth production settings
+
+`infra/environments/production/*.env.template`: `COOKIE_SECURE=1`, `TRUST_PROXY=1`. Dev's
+insecure defaults (`COOKIE_SECURE` unset, no proxy trust) are never inherited — production env
+files are separate files, not a dev file with overrides layered on.

docs/PRODUCTION_VALIDATION_CHECKLIST.md

@@ -0,0 +1,58 @@
+# Production Validation Checklist
+
+Walk this once, top to bottom, before calling a production deployment live. Every item here
+needs the real hardware — this is the "DEPLOY LATER" half of Phase 4, not something this repo
+can prove in advance.
+
+## Hardware / Proxmox
+- [ ] Disk inspection done, layout chosen and matches what's actually in the server (`docs/RUNBOOK_DISK_FAILURE.md` inspection checklist, `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4)
+- [ ] Proxmox VE installed, reachable only from `management_cidr`/`proxmox_mgmt_cidr`
+- [ ] VM template built from real cloud-init (`infra/proxmox/README.md`)
+- [ ] `tofu apply` created both VMs; both boot and are SSH-reachable
+
+## Platform VM
+- [ ] `ansible-playbook site.yml` completed clean, all roles
+- [ ] `docker compose ps` shows every service healthy (postgres, provisioner, api, gateway, reverse-proxy, prometheus, alertmanager, grafana, loki, promtail, exporters)
+- [ ] `curl https://<origin>/healthz` (via Gateway) returns 200
+- [ ] `curl http://localhost:4300/healthz` (Platform API, from the VM) returns `{"ok":true,"db":"ok"}`
+- [ ] TLS certificate issued and valid for the real domain
+
+## Lab Host VM
+- [ ] Docker daemon TLS verified: `docker --tlsverify --tlscacert=... -H tcp://<lab_host_ip>:2376 info` succeeds from the Platform VM, fails from anywhere else
+- [ ] Lab Host Router mTLS verified: a request to `https://<lab_host_ip>:8443` with no client cert is rejected; with the Gateway's real client cert it isn't (mirrors `infra/scripts/simulate-two-host/test-remote-routing.js`'s "direct public access: NOT AVAILABLE" case, now against the real host)
+- [ ] Provisioner successfully creates a real Lab against the real Lab Host
+- [ ] Gateway can actually reach that Lab's terminal (HTTP UI load + WebSocket `kubectl` command) — application mechanism built and simulation-tested in Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1/§6/§7); this line is the real-hardware confirmation that the same result holds across the actual two VMs, not two Docker networks on one daemon
+- [ ] Egress proxy correctly restricts a Lab's outbound traffic (Provisioner's per-Lab `DOCKER-USER` chain)
+
+## Security
+- [ ] `infra/firewall/validate-firewall.sh` run against the real hosts with real `iptables-restore --test`, not just the structural lint
+- [ ] Confirm from an unauthorized IP: Postgres, Provisioner, Docker TLS port, Proxmox UI are all unreachable
+- [ ] Confirm Student Lab → another Lab is blocked; confirm Student Lab → Proxmox is blocked
+- [ ] `COOKIE_SECURE`/`TRUST_PROXY` active (check a real Set-Cookie header over HTTPS)
+
+## Academy (application-level smoke test)
+- [ ] Owner bootstrap login works
+- [ ] Instructor can create a cohort, roster a student
+- [ ] Student can log in, boot a Lab, complete a scenario, see progress
+- [ ] An assessment can be assigned, taken, and graded end-to-end
+- [ ] Instructor analytics dashboards populate with real data
+
+## Backup
+- [ ] Manual `backup-postgres.sh` run succeeds against real Postgres, real off-host target reachable
+- [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness — Phase 4.1's `infra/backup/test-destructive-restore-drill.sh` proved the full mechanism (realistic data, real storage destruction, restore, real Platform API boot) locally; this line is the real-hardware confirmation
+- [ ] Cron job confirmed scheduled (`crontab -l`)
+
+## Observability
+- [ ] Grafana dashboard shows real data for both VMs
+- [ ] Trigger one alert deliberately (e.g. stop a service) and confirm it fires and reaches the webhook
+- [ ] Loki has logs flowing from both VMs
+
+## Pre-production hardening (Phase 4.1 — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md)
+- [ ] `resource-policy.yaml`'s `admission_control.max_concurrent_labs` set from a real measured
+      `infra/scripts/capacity-model.js` run against the real Lab Host, not left disabled
+- [ ] A deliberate over-capacity attempt returns the clean 503 "Lab capacity is temporarily
+      full" message, not a 500 — confirms the real host's admission control matches the local test
+- [ ] `docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md`'s steps still make sense once capacity numbers are real
+
+Only once every box above is checked against the real server does the deployment graduate from
+"infrastructure package built" to "actually in production."

docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md

@@ -0,0 +1,85 @@
+# Remote & Self-Paced Delivery Architecture
+
+Phase 4.1 §12. This is an architecture review, not a new product — no self-registration,
+payments, or public enrollment is built here (explicit non-goal). The question this document
+answers: does the infrastructure and application assume "an instructor always provisions the
+Lab," and if so, where, and what would need to change for self-paced delivery to work.
+
+## The short answer: less of that assumption exists than expected
+
+RootNode Academy's Lab lifecycle was built student-driven from Phase 1 — `POST
+/api/environment/launch` is called by the STUDENT'S OWN session, not by an instructor action.
+"Prepare Labs" (`POST /api/admin/cohorts/:id/prepare-labs`) is a classroom **convenience** that
+front-loads several students' launches at once before a synchronous class session, not the only
+path to a running Lab. This was true before Phase 4.1 and confirmed still true by this phase's
+own regression suites (`platform/test/e2e-phase2a.js` roster's two students with `cohort_id:
+null` and takes them through login → curriculum → launch → scenario work with zero cohort
+involvement) and by two new Phase 4.1 drills built specifically to exercise the remote/returning
+-student narrative:
+
+- `infra/scripts/drills/test-lab-host-loss.js` — student launches, makes progress, their Lab is
+  destroyed out from under them (standing in for an idle-reaper destroy), Platform stays up,
+  progress is untouched, they launch again unprompted and continue.
+- `infra/scripts/drills/test-fake-cohort-rehearsal.js` — the same "returns later, fresh Lab,
+  curriculum continues" narrative as part of a fuller classroom rehearsal.
+
+Both pass today, with zero application code changes needed for the self-paced-relevant parts of
+this flow specifically — see the Phase 4.1 report for the full run results.
+
+## What already works, unmodified
+
+- **Student-initiated provisioning** — no instructor step is required between "student wants to
+  work" and "Lab exists." (`platform/api/src/server.js`'s `/api/environment/launch`.)
+- **Cohort-less students** — `cohort_id` is nullable on student creation
+  (`POST /api/admin/students`); a student with no cohort logs in, sees the curriculum, launches
+  Labs, and makes progress identically to a cohorted one. Confirmed by inspection AND by the
+  existing regression suite, not just by schema reading.
+- **Idle reaper → academic state survives → return → continue** — the exact lifecycle this
+  section describes as the target self-paced experience is what Phase 1's reaper +
+  Phase 2B's Postgres-backed academic data (separate from the disposable Lab) were already
+  designed to produce. Phase 4.1 proved it end-to-end, twice (see above).
+- **Curriculum content itself is not cohort-gated** — `GET /api/curriculum/path` reads the one
+  curriculum path file directly (`curriculum.loadCurriculumPath()`); it never checks
+  `curriculum_assignments` or cohort membership at all. A self-paced student sees the full
+  curriculum with no instructor action required to "unlock" it.
+
+## What's genuinely missing for a real self-paced product (Phase 5, not built here)
+
+1. **No self-registration / public enrollment.** A student row is created only by an instructor
+   (`POST /api/admin/students`, admin/owner RBAC-gated). There is no public "sign up" surface.
+   This is almost certainly the right call to leave alone for now — it's explicitly a Phase 5+
+   product/business decision (course commerce, is it free, who's allowed to enroll), not an
+   infrastructure gap.
+
+2. **No real student authentication.** A student session is created from a bare `student_id`
+   (`POST /api/login`, no password) — deliberately, since today every `student_id` is issued and
+   controlled by an instructor via the roster, not chosen or known by the public. A genuinely
+   remote/self-paced product where students access their account without an instructor in the
+   loop needs real credentials (email+password, or a magic link) — reusing the same
+   `password_hash`/`scrypt` machinery `instructors` already has (`src/auth/password.js`) would
+   be the natural extension, not a new auth system. **This is the one item on this list that
+   would actually block self-paced delivery today** — everything else already works or is a
+   product decision, not an infrastructure one.
+
+3. **`curriculum_assignments.student_id` is schema-ready but unwired.** The table has supported
+   a direct per-student curriculum assignment (independent of any cohort) since Phase 2B
+   (`CONSTRAINT curriculum_assignment_target CHECK (cohort_id IS NOT NULL OR student_id IS NOT
+   NULL)`) — but nothing in the application ever reads or writes that column today; there's only
+   one curriculum path, so nothing has needed to differentiate yet. Not a gap that blocks
+   self-paced delivery now (every student already sees the one path with no gate), but the
+   moment a second curriculum path exists, this is where per-individual assignment lands — the
+   schema was designed for exactly this and doesn't need a migration to support it.
+
+4. **No delivery-mode concept anywhere.** In-person, remote, and self-paced sessions aren't
+   distinguished in any data model today, and nothing currently needs them to be — capacity
+   planning, the reaper, and academic records all behave identically regardless of how a student
+   is attending. Worth being aware this is unmodeled, not worth building a taxonomy for before a
+   real product reason exists (YAGNI).
+
+## What Phase 4.1 deliberately did NOT do here
+
+No signup flow, no student password/email auth implementation, no per-student curriculum
+assignment route, no payments, no public course catalog, no held/premium scenario content. All
+explicit non-goals per this phase's authorization. This document exists so Phase 5 planning
+starts from an accurate picture of the gap (one real item: student authentication) instead of a
+guess.

docs/RUNBOOK_BACKUP_FAILURE.md

@@ -0,0 +1,31 @@
+# Runbook: Backup Failure
+
+Triggered by the `BackupFailed`/`BackupAging` alerts (`infra/monitoring/prometheus/rules/rootnode.yml`).
+
+```bash
+# 1. See what actually happened — the cron wrapper logs to syslog
+journalctl -u rootnode-backup --since "24 hours ago" -o cat
+# or, if the cron job never even ran:
+grep rootnode-postgres-backup /var/log/syslog
+
+# 2. Run it manually to see the failure live
+source /etc/rootnode/backup.env
+/opt/rootnode/backup/backup-postgres.sh
+```
+
+**Common causes:**
+- Disk full on `{{ rootnode_data_root }}/backups` — check `df -h`, prune more aggressively
+  (`BACKUP_RETENTION_DAYS`) or the ZFS dataset needs more space (see `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4).
+- Off-host target unreachable (`BACKUP_OFFSITE_TARGET`) — check the SSH key/credentials
+  haven't expired, the target host/bucket is reachable: `ssh <offsite-target-host> echo ok` or
+  the equivalent for whatever adapter is configured (`docs/BACKUP_DR.md`).
+- `BACKUP_ENCRYPTION_PASSPHRASE` missing/wrong in `/etc/rootnode/backup.env` — re-check against
+  the value in Ansible Vault (`vault_backup_encryption_passphrase`); a mismatch here means even
+  a "successful" old backup becomes unrestorable, so treat this specific failure mode as urgent.
+- The dump itself is suspiciously small (script's own sanity check tripped) — Postgres may be
+  unhealthy; check `docker exec rootnode-platform-postgres-1 pg_isready` first.
+
+**After fixing:** run the backup manually once to confirm success (metric updates immediately,
+alert clears within one Prometheus scrape interval), then confirm the cron job is still
+correctly scheduled (`crontab -l` as root, or re-run the `backup` Ansible role if the cron entry
+itself is suspect).

docs/RUNBOOK_DISK_FAILURE.md

@@ -0,0 +1,44 @@
+# Runbook: Disk Failure
+
+Applies once real hardware/ZFS exists — see `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4 for
+the storage decision framework this assumes (mirrored vdevs, never single-disk for Postgres).
+
+```bash
+# 1. Confirm the failure
+zpool status -v
+smartctl -a /dev/sdX   # the suspect drive
+
+# 2. If it's a mirrored vdev (the required layout for anything holding Postgres data — see §4),
+#    the pool stays online in DEGRADED state. No immediate data-loss risk, but no redundancy
+#    either until replaced — treat DEGRADED as urgent, not routine.
+zpool status   # confirm DEGRADED, not FAULTED/UNAVAIL (those mean act immediately, possibly
+                # restore from backup if the redundant mirror side is also compromised)
+
+# 3. Replace the physical drive, then:
+zpool replace <pool> <old-device> <new-device>
+zpool status   # watch the resilver progress
+
+# 4. If a whole pool was lost (both mirror sides failed simultaneously — the scenario mirroring
+#    exists to make vanishingly unlikely, but plan for it anyway):
+#    - rn-lab-host-01's disposable data: nothing to recover, just rebuild the VM
+#      (docs/RUNBOOK_VM_REBUILD.md)
+#    - rn-platform-01's Postgres data: restore from the latest off-host backup
+#      (docs/RUNBOOK_POSTGRES_RESTORE.md) — this is exactly the scenario off-host backup
+#      copies exist for; a local-only backup on the SAME failed pool would be useless here
+```
+
+**Pre-deployment disk inspection checklist** (run once when the real drives are known, before
+they go into service — `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4):
+
+```bash
+for d in /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1; do   # adjust to real device names
+  echo "== $d =="
+  smartctl -i "$d" | grep -E 'Model|Serial|Capacity|Firmware'
+  smartctl -A "$d" | grep -iE 'power_on_hours|reallocated|pending|uncorrectable|wear|percentage.used|temperature'
+  smartctl -H "$d" | grep -i overall
+done
+```
+Record model, size, power-on hours, wear/endurance percentage, reallocated/pending sector
+counts, temperature, and firmware version for every drive before committing to a ZFS layout —
+a drive already showing wear or reallocated sectors shouldn't be trusted with Postgres data
+even in a mirror.

docs/RUNBOOK_DOCKER_TLS_ROTATION.md

@@ -0,0 +1,31 @@
+# Runbook: Docker TLS Rotation
+
+Rotate on a schedule (recommend yearly, well before the ~2.25-year cert lifetime
+`generate-docker-tls.sh` defaults to) or immediately if a client cert may have leaked.
+
+```bash
+# 1. Generate a fresh bundle (new CA too — simplest, avoids any question about whether the old
+#    CA's private key was ever exposed)
+cd infra/scripts/docker-tls
+./generate-docker-tls.sh --out-dir ./out-$(date +%Y%m%d) --lab-host-ip <real-ip> --lab-host-dns rn-lab-host-01.internal
+./verify-docker-tls.sh ./out-$(date +%Y%m%d)
+
+# 2. Redeploy — this restarts dockerd on rn-lab-host-01 (brief interruption to any in-flight
+#    Docker API call from the Provisioner, NOT to already-running student Labs — their
+#    containers keep running through a dockerd restart)
+cd ../../ansible
+ROOTNODE_TLS_BUNDLE_DIR=../scripts/docker-tls/out-$(date +%Y%m%d) \
+  ansible-playbook site.yml -i inventory/production.ini --tags docker-tls --ask-vault-pass
+
+# 3. Verify the Provisioner can still reach rn-lab-host-01
+curl -s http://localhost:4100/health   # from rn-platform-01
+```
+
+**Troubleshooting a failed rotation:**
+- `x509: certificate signed by unknown authority` — the new ca.pem wasn't deployed to both
+  sides, or an old one is cached; re-run the docker-tls role on BOTH hosts.
+- Provisioner logs `ECONNREFUSED` on port 2376 — dockerd didn't actually restart with the new
+  daemon.json; check `systemctl status docker` on rn-lab-host-01 and the ExecStart override
+  (`infra/ansible/roles/docker-tls/templates/docker-override.conf.j2`) actually applied.
+- Firewall — confirm `infra/firewall/rn-lab-host-01.rules`' 2376 rule still matches
+  rn-platform-01's actual IP (didn't change as part of this rotation, but worth a sanity check).

docs/RUNBOOK_FAILED_MIGRATION.md

@@ -0,0 +1,35 @@
+# Runbook: Failed Migration
+
+Both Platform API and Provisioner apply their own migrations automatically on startup
+(`platform/api/src/db.js`, `platform/provisioner/src/db.js`), file-tracked in
+`schema_migrations`, wrapped in a transaction per file — a migration either fully applies or
+fully rolls back, it can't leave a table half-altered.
+
+**If a deploy fails because a migration errored:**
+
+```bash
+# 1. Check what actually happened
+docker logs rootnode-platform-api-1 --tail 50   # the error is right there — migration file + message
+docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
+  "SELECT * FROM schema_migrations ORDER BY applied_at DESC LIMIT 10;"
+
+# 2. Since the failing migration's transaction rolled back, the schema is still in its
+#    PREVIOUS valid state — the app just won't start until the migration succeeds. Fix the
+#    root cause (usually a real data issue the migration didn't anticipate, or the
+#    cross-service ordering dependency — Provisioner's migrations must run before Platform
+#    API's, since scenario_attempts has a real FK to environments; see
+#    docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md).
+
+# 3. Redeploy once fixed — the migration runner is idempotent and will just pick up where it
+#    left off; already-applied migrations are skipped.
+./infra/scripts/deploy-production.sh <same-or-fixed-tag>
+```
+
+**If you need to roll back the CODE but the migration already succeeded** (schema changed,
+code that depended on it is being reverted): rolling back application images
+(`docs/RUNBOOK_ROLLBACK.md`) does NOT undo a Postgres migration — there is no automatic
+down-migration mechanism (deliberate, matches the existing check-before-alter, additive-only
+migration convention documented in `docs/ENGINEERING_SOP.md` §6). If the old code genuinely
+can't run against the new schema, restore from the pre-migration backup instead
+(`docs/RUNBOOK_POSTGRES_RESTORE.md`) rather than trying to hand-write a down-migration under
+pressure.

docs/RUNBOOK_FIRST_DEPLOYMENT.md

@@ -0,0 +1,34 @@
+# Runbook: First Production Deployment
+
+Prerequisites: Proxmox installed (`infra/proxmox/README.md`), VMs created
+(`infra/terraform/`), DNS pointed at rn-platform-01's public IP.
+
+```bash
+# 1. Generate Docker TLS material (offline, on your own machine — never on either VM)
+cd infra/scripts/docker-tls
+./generate-docker-tls.sh --out-dir ./out --lab-host-ip <real-lab-host-ip> --lab-host-dns rn-lab-host-01.internal
+./verify-docker-tls.sh ./out
+
+# 2. Fill in real values
+cd ../../ansible
+cp group_vars/production.vault.yml.template group_vars/production.vault.yml
+$EDITOR group_vars/production.vault.yml   # real passwords/secrets — openssl rand -hex 32 etc.
+ansible-vault encrypt group_vars/production.vault.yml
+$EDITOR group_vars/production.yml         # real IPs, domain, CIDRs
+$EDITOR inventory/production.ini          # real IPs
+
+# 3. Deploy
+ROOTNODE_TLS_BUNDLE_DIR=../scripts/docker-tls/out \
+  ansible-playbook site.yml -i inventory/production.ini --ask-vault-pass
+
+# 4. Bootstrap the owner account (once, by a human)
+ssh rootnode-deploy@rn-platform-01
+docker exec -it rootnode-platform-api-1 node bootstrap-owner.js [email protected] "Your Name"
+# (password comes from BOOTSTRAP_OWNER_PASSWORD in the environment, or the CLI prompt — see
+# platform/api/README.md)
+
+# 5. Verify
+../../infra/scripts/production-smoke-test.sh
+```
+
+Then work through `docs/PRODUCTION_VALIDATION_CHECKLIST.md` in full before calling this done.

docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md

@@ -0,0 +1,29 @@
+# Runbook: High Lab Host Resource Pressure
+
+Triggered by the `LabHostHighMemory`/`HighDiskUsage` warning alerts.
+
+```bash
+# 1. How many Labs are actually running, and against what admission-control threshold?
+curl -s http://localhost:4100/environments | jq '[.[] | select(.status=="running")] | length'
+node infra/scripts/capacity-model.js   # recommended/warning/hard thresholds
+
+# 2. If genuinely over capacity: this is expected pressure from real usage, not a bug.
+#    - Destroy idle/abandoned Labs early via /admin (per-cohort "Destroy Labs" after a class
+#      session ends, rather than waiting for the idle reaper's default window)
+#    - If a specific cohort's class just ended, use /admin -> Cohorts -> [cohort] -> Destroy
+#      Cohort Labs immediately rather than waiting
+
+# 3. If pressure is high with FEWER Labs running than capacity-model.js's threshold: something
+#    is consuming more per-Lab resources than the model assumes (a scenario doing something
+#    unusually heavy, a memory leak in a long-running Lab, zombie processes from a crashed
+#    reset). Check per-container usage:
+docker stats --no-stream | grep rootnode-lab-
+
+# 4. Update infra/environments/production/capacity.yml with fresh measured numbers
+#    (infra/scripts/loadtest/) if the per-Lab footprint has genuinely changed, so the
+#    admission-control threshold reflects reality going forward.
+```
+
+Do not silently raise resource limits in `provisioner-config/resource-policy.yaml` to make an
+alert go away without understanding why usage is higher than modeled — that just moves the
+same pressure onto host RAM/CPU directly instead of onto individual Lab limits.

docs/RUNBOOK_NORMAL_DEPLOY.md

@@ -0,0 +1,17 @@
+# Runbook: Normal Deploy
+
+For a routine code change already merged to `main` on the canonical Gitea repo.
+
+1. CI (`infra/ci/gitea-workflows/build-and-test.yml`) runs automatically: tests, image build/tag/push, staging deploy, staging smoke test.
+2. Review the staging smoke test result and, ideally, poke around `https://staging.<domain>` yourself.
+3. Promote to production (manual, deliberate):
+
+```bash
+cd infra/scripts
+./deploy-production.sh sha-<12-char-sha>    # or a release tag, e.g. v1.3.0
+./production-smoke-test.sh
+```
+
+4. Watch Grafana (`/grafana/`) for 10-15 minutes — error rates, Postgres connections, Lab Host
+   memory — before considering the deploy "settled."
+5. If anything looks wrong: `docs/RUNBOOK_ROLLBACK.md`.

docs/RUNBOOK_OWNER_RECOVERY.md

@@ -0,0 +1,27 @@
+# Runbook: Owner Account Recovery
+
+**Locked out with zero working owner accounts** (all deactivated, forgot the only owner's
+password, etc.):
+
+```bash
+ssh rootnode-deploy@rn-platform-01
+docker exec -it rootnode-platform-api-1 sh
+node bootstrap-owner.js [email protected] "Your Name"
+# bootstrap-owner.js refuses to run if an active owner already exists UNLESS the target email
+# belongs to an EXISTING instructor row — in that case it promotes them to owner. Point it at
+# any known instructor email to recover through them, or a brand-new email to create a fresh
+# owner outright if the instructors table itself is in an unknown state.
+```
+
+**Forgot password, account still active and you have SSH access:**
+
+```bash
+docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
+  "SELECT instructor_id, email, role, status FROM instructors WHERE role='owner';"
+# Then use another owner's account to reset via /admin -> Instructors -> [account] -> Reset
+# Password, OR re-run bootstrap-owner.js targeting that email (also resets the password).
+```
+
+**Never** hand-edit `password_hash` in the database — it's `scrypt(salt):hash` in a specific
+format (`platform/api/src/auth/password.js`); a malformed value just breaks login rather than
+setting a usable password. Always go through `bootstrap-owner.js` or the admin UI's reset flow.

docs/RUNBOOK_POSTGRES_RESTORE.md

@@ -0,0 +1,45 @@
+# Runbook: Postgres Restore
+
+**When to use:** Postgres corruption/data loss on rn-platform-01, or verifying a backup is
+actually good (do this periodically even without an incident — an untested backup is a
+hypothesis, not a backup).
+
+```bash
+# 1. Find the backup to restore
+ls -la /srv/rootnode-data/backups/    # or wherever backup_offsite_target points, if local copies are gone
+
+# 2. Create a FRESH target database — never restore over the live one directly
+docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
+  "CREATE DATABASE rootnode_platform_restore_test;"
+
+# 3. Restore into it
+source /etc/rootnode/backup.env   # for BACKUP_ENCRYPTION_PASSPHRASE
+/opt/rootnode/backup/restore-postgres.sh \
+  /srv/rootnode-data/backups/rootnode-platform-<timestamp>.sql.gz.gpg \
+  "postgres://rootnode_platform:$PGPASS@localhost:5432/rootnode_platform_restore_test"
+# type RESTORE when prompted
+
+# 4. Verify
+/opt/rootnode/backup/verify-restore.sh \
+  "postgres://rootnode_platform:$PGPASS@localhost:5432/rootnode_platform_restore_test"
+
+# 5a. If this was a drill: drop the test database
+docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
+  "DROP DATABASE rootnode_platform_restore_test;"
+
+# 5b. If this was a REAL incident: stop the app stack, rename the broken DB, rename the
+#     restored one into its place, restart.
+systemctl stop rootnode-platform
+docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
+  "ALTER DATABASE rootnode_platform RENAME TO rootnode_platform_broken_$(date +%s);"
+docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
+  "ALTER DATABASE rootnode_platform_restore_test RENAME TO rootnode_platform;"
+systemctl start rootnode-platform
+```
+
+Local, non-production dry run of this exact flow: `infra/backup/test-backup-restore.sh` (uses
+disposable containers, safe to run anytime, no real data involved).
+
+**Recovery point objective:** up to 24 hours of academic data (daily backup cadence) — see
+`docs/BACKUP_DR.md` for the full RPO/RTO discussion and why that's the honest number, not an
+aspirational one.

docs/RUNBOOK_ROLLBACK.md

@@ -0,0 +1,18 @@
+# Runbook: Rollback
+
+```bash
+cd infra/scripts
+./rollback-production.sh              # rolls back to the previous deployed tag automatically
+# or, to go back further than one step:
+./rollback-production.sh sha-<older-sha>
+
+./production-smoke-test.sh
+```
+
+This rolls back **application images only**. If the bad deploy included a database migration
+that needs undoing too, schema changes aren't automatically reversible — see
+`docs/RUNBOOK_FAILED_MIGRATION.md` instead (or in addition).
+
+After rolling back: file an issue / note in the deploy log (`/opt/rootnode/deploy-history.log`
+already has the rollback recorded) about what went wrong before re-attempting the forward
+deploy.

docs/RUNBOOK_STUCK_LAB_CLEANUP.md

@@ -0,0 +1,40 @@
+# Runbook: Stuck Lab Cleanup
+
+A Lab that's stuck `provisioning`/`resetting` past its expected boot window, or a container
+that outlived its Postgres `environments` row (crash during teardown, etc.).
+
+**Via the admin UI (preferred, no host access needed):** `/admin` -> Students -> [student] ->
+Environment -> Destroy, then Provision again if they need a fresh one immediately.
+
+**Via the API directly** (same effect, useful for scripting a batch cleanup):
+```bash
+curl -s -b <owner-session-cookie> -X DELETE https://academy.example.com/api/admin/environments/<environment_id>
+```
+
+**A container the Provisioner itself has lost track of** (rare — e.g. a crash during
+provisioning left an orphaned container/network/firewall chain with no matching `environments`
+row, or vice versa):
+
+```bash
+# On rn-lab-host-01 — list what's actually running
+docker ps -a --format '{{.Names}}\t{{.Status}}' | grep '^rootnode-lab-'
+
+# Cross-reference against the Provisioner's own view
+curl -s http://localhost:4100/environments | jq '.[] | {environment_id, status, container_name}'
+
+# Provisioner already does this automatically on its own restart (reconcileOnStartup,
+# platform/provisioner/src/lifecycle.js) — a simple `systemctl restart rootnode-platform`
+# (which restarts the provisioner container as part of the stack) is often the fastest fix for
+# a batch of orphans rather than hand-cleaning each one.
+
+# Manual last resort — remove the container/network/firewall chain directly:
+docker rm -f rootnode-lab-<id>
+docker network rm rootnode-lab-net-<id>
+iptables -D DOCKER-USER -s <subnet> -j ROOTNODE-ENV-<id>   # remove the jump rule FIRST
+iptables -F ROOTNODE-ENV-<id> && iptables -X ROOTNODE-ENV-<id>
+```
+
+Never manually delete an `environments` row from Postgres directly unless you've confirmed the
+container/network/firewall chain are already gone — the Provisioner treats that table as
+authoritative and will behave unpredictably against real Docker/iptables state it doesn't know
+matches.

docs/RUNBOOK_TLS_RENEWAL.md

@@ -0,0 +1,19 @@
+# Runbook: Public TLS Certificate Renewal
+
+If using Caddy's built-in Let's Encrypt (the default in `infra/docker/platform/Caddyfile.j2`):
+**nothing to do** — Caddy renews automatically well before expiry, as long as port 80 stays
+reachable for the ACME HTTP-01 challenge (`infra/firewall/rn-platform-01.rules` already allows
+it). `infra/scripts/production-smoke-test.sh` checks the live expiry date on every run as an
+early warning if auto-renewal ever silently breaks.
+
+If using Cloudflare origin certs instead (see `docs/PRODUCTION_NETWORK_SECURITY.md` §8 for
+that decision): Cloudflare-issued origin certs are typically 15-year validity — this is a
+once-per-network-topology-change concern, not a recurring one. Re-issue from the Cloudflare
+dashboard (SSL/TLS -> Origin Server) and redeploy the cert/key via the `platform` Ansible role
+if it ever needs replacing (domain change, suspected compromise).
+
+**Manual check:**
+```bash
+echo | openssl s_client -connect academy.example.com:443 -servername academy.example.com 2>/dev/null \
+  | openssl x509 -noout -enddate
+```

docs/RUNBOOK_VM_REBUILD.md

@@ -0,0 +1,31 @@
+# Runbook: Platform VM / Lab Host VM Rebuild
+
+Both VMs are designed to be disposable and rebuildable from code — nothing irreplaceable
+should ever live only on either VM's disk outside of what's backed up (`docs/BACKUP_DR.md`).
+
+## rn-platform-01 lost
+
+Academic data survives (it's in Postgres backups, off-host). Instructor/student accounts and
+progress are recoverable; anything since the last backup is not (`docs/BACKUP_DR.md`'s RPO).
+
+```bash
+cd infra/terraform && terraform apply   # recreates the VM from the template
+cd ../ansible && ansible-playbook site.yml -i inventory/production.ini --limit platform --ask-vault-pass
+# Then restore Postgres — docs/RUNBOOK_POSTGRES_RESTORE.md, step 5b variant (fresh DB this time,
+# no need to rename since it's a fresh instance)
+./infra/scripts/production-smoke-test.sh
+```
+
+## rn-lab-host-01 lost
+
+**No academic data is lost** — this host only ever holds disposable Lab containers by design
+(`docs/BACKUP_DR.md`'s Tier distinction). Every enrolled student's Lab needs re-provisioning
+after this, which is a normal "Prepare Labs" click, not a recovery operation.
+
+```bash
+cd infra/terraform && terraform apply
+cd ../ansible && ansible-playbook site.yml -i inventory/production.ini --limit lab_host --ask-vault-pass
+# Re-generate Docker TLS material for the new host if its identity (IP/DNS) changed —
+# docs/RUNBOOK_DOCKER_TLS_ROTATION.md
+# From /admin: Prepare Labs again for any cohort that needs it.
+```

docs/SECURITY_SCANNING.md

@@ -0,0 +1,47 @@
+# Security Scanning (Phase 4.1 §24)
+
+`infra/ci/gitea-workflows/build-and-test.yml`'s `security-scan` job. All free, open-source,
+no account/signup required for any of them — no paid scanner was added.
+
+## What's scanned
+
+| Tool | Checks | Scope |
+|---|---|---|
+| `npm audit --audit-level=high` | Known-vulnerable dependencies | Every first-party Node service |
+| [gitleaks](https://github.com/gitleaks/gitleaks) | Committed secrets (keys, tokens, credentials) | Whole repo |
+| [hadolint](https://github.com/hadolint/hadolint) | Dockerfile best-practice/security lint | Every first-party Dockerfile |
+| [shellcheck](https://www.shellcheck.net/) | Shell script correctness/security | `infra/**/*.sh` |
+| [trivy](https://github.com/aquasecurity/trivy) | OS/library CVEs in a built container image | `platform-api` image (representative — same base/layers as the other Node services) |
+
+## Severity policy
+
+- **CRITICAL / HIGH** — fails the build (`security-scan` blocks `build-images`, which blocks
+  `deploy-staging`, which blocks production promotion — see the job dependency chain in the
+  workflow file).
+- **MEDIUM / LOW** — reported in the job output, never blocks. A build that's permanently red
+  over a low-severity transitive dependency finding nobody can immediately fix trains everyone
+  to ignore red builds, which is worse than not scanning at all.
+
+`npm audit --audit-level=high` and trivy's `--severity HIGH,CRITICAL` both implement this
+directly at the tool level, not via manual triage.
+
+## What's NOT in the automated job (and why)
+
+- **Dedicated IaC security linting** (checkov/tflint) — not added. `infra/scripts/validate-compose.sh`,
+  `infra/firewall/validate-firewall.sh`, `ansible-playbook --syntax-check`, and `tofu validate`
+  (all in the `regression` job) already catch structural/syntax problems in the infra code;
+  a dedicated security-policy linter for Terraform/Ansible is a reasonable Phase 5 addition once
+  there's a real cloud footprint to misconfigure, not before.
+- **DAST/penetration testing** — out of scope for CI; this is a build-time gate, not a running-system
+  test. Covered separately by `docs/PRODUCTION_VALIDATION_CHECKLIST.md`'s security section once
+  real hardware exists.
+- **SAST beyond hadolint/shellcheck** — the codebase is small enough (a handful of first-party
+  Node services) that `npm audit` + secret scanning + Dockerfile/shell lint cover the realistic
+  risk surface; a dedicated SAST tool (Semgrep, CodeQL) is a reasonable addition once the
+  codebase or team grows enough to justify the added CI time.
+
+## Local reproduction
+
+Every check above can be run locally with the exact same free tools — no CI-only dependency.
+See the `security-scan` job's steps in `infra/ci/gitea-workflows/build-and-test.yml` for the
+exact commands (they're copy-pasteable).

frontend/src/admin/Admin.module.css

@@ -57,3 +57,6 @@ tr:hover td { background: var(--surface2); }
 .linkBtn { background: none; border: none; color: var(--blue); cursor: pointer; font-size: 0.85rem; padding: 0; text-decoration: underline; }
 .errorBox { background: var(--red-dim); color: var(--red); border: 1px solid var(--red); border-radius: var(--radius); padding: 0.6rem 0.8rem; font-size: 0.85rem; margin-bottom: 1rem; }
 .muted { color: var(--text-2); font-size: 0.85rem; }
+
+/* Phase 4.1 §16 — shared pagination control (AuditView, StudentsView attempts). */
+.pager { display: flex; align-items: center; gap: 0.75rem; margin-top: 0.75rem; }

frontend/src/admin/AssessmentsView.jsx

@@ -29,7 +29,7 @@ export default function AssessmentsView({ role }) {
       {canAuthor && (
         <div className={styles.card}>
           <h2>Create assessment</h2>
-          <CreateAssessmentForm onCreated={reload} />
+          <CreateAssessmentForm onCreated={reload} role={role} />
         </div>
       )}
       <div className={styles.card}>
@@ -56,20 +56,86 @@ export default function AssessmentsView({ role }) {
   )
 }
 
-function CreateAssessmentForm({ onCreated }) {
-  const [form, setForm] = useState({ id: '', type: 'exam', title: '', duration_minutes: 60, hint_policy: 'disabled', pool_spec: '' })
+// Phase 4.1 §18 — pool_spec is built from structured picks (required scenarios + named random
+// pools), not hand-typed JSON. `catalog` ({ stages: [{ id, name, scenarios: [{id,title}] }] })
+// comes from GET /api/admin/scenario-catalog. An owner can still drop into a raw-JSON view for
+// anything the structured editor doesn't cover — see the toggle below.
+function PoolSpecEditor({ catalog, required, setRequired, pools, setPools }) {
+  const allScenarios = (catalog?.stages || []).flatMap((s) => s.scenarios.map((sc) => ({ ...sc, stageName: s.name })))
+  const titleFor = (id) => allScenarios.find((s) => s.id === id)?.title || id
+
+  const addPool = () => setPools([...pools, { count: 1, from: [] }])
+  const removePool = (i) => setPools(pools.filter((_, idx) => idx !== i))
+  const updatePool = (i, patch) => setPools(pools.map((p, idx) => (idx === i ? { ...p, ...patch } : p)))
+
+  return (
+    <div>
+      <label className={styles.label}>Required scenarios (always included)</label>
+      <select className={styles.select} multiple size={6} style={{ width: '100%' }}
+        value={required} onChange={(e) => setRequired([...e.target.selectedOptions].map((o) => o.value))}>
+        {catalog?.stages.map((stage) => (
+          <optgroup key={stage.id} label={stage.name}>
+            {stage.scenarios.map((sc) => <option key={sc.id} value={sc.id}>{sc.title}</option>)}
+          </optgroup>
+        ))}
+      </select>
+      <p className={styles.muted}>{required.length ? `${required.length} selected: ${required.map(titleFor).join(', ')}` : 'None selected — cmd/ctrl-click to pick multiple.'}</p>
+
+      <label className={styles.label} style={{ marginTop: '1rem', display: 'block' }}>Random pools (draw N at random from a set, every attempt)</label>
+      {pools.map((pool, i) => (
+        <div key={i} className={styles.card} style={{ marginBottom: '0.5rem' }}>
+          <div className={styles.formRow}>
+            <div style={{ maxWidth: '100px' }}><label className={styles.label}>Draw count</label>
+              <input className={styles.input} type="number" min="1" value={pool.count}
+                onChange={(e) => updatePool(i, { count: Number(e.target.value) })} /></div>
+            <div><button type="button" className={styles.linkBtn} onClick={() => removePool(i)}>Remove pool</button></div>
+          </div>
+          <label className={styles.label}>Draw from</label>
+          <select className={styles.select} multiple size={5} style={{ width: '100%' }}
+            value={pool.from} onChange={(e) => updatePool(i, { from: [...e.target.selectedOptions].map((o) => o.value) })}>
+            {catalog?.stages.map((stage) => (
+              <optgroup key={stage.id} label={stage.name}>
+                {stage.scenarios.map((sc) => <option key={sc.id} value={sc.id}>{sc.title}</option>)}
+              </optgroup>
+            ))}
+          </select>
+          {pool.count > pool.from.length && <p className={styles.muted} style={{ color: 'var(--amber, #b58900)' }}>Draw count ({pool.count}) exceeds the pool size ({pool.from.length}) — every draw will use every scenario, not a random subset.</p>}
+        </div>
+      ))}
+      <button type="button" className={styles.linkBtn} onClick={addPool}>+ Add random pool</button>
+    </div>
+  )
+}
+
+function CreateAssessmentForm({ onCreated, role }) {
+  const [form, setForm] = useState({ id: '', type: 'exam', title: '', duration_minutes: 60, hint_policy: 'disabled' })
+  const [catalog, setCatalog] = useState(null)
+  const [required, setRequired] = useState([])
+  const [pools, setPools] = useState([])
+  const [advanced, setAdvanced] = useState(false)
+  const [rawPoolSpec, setRawPoolSpec] = useState('')
   const [error, setError] = useState(null)
   const [busy, setBusy] = useState(false)
 
+  useEffect(() => { adminApi.scenarioCatalog().then(setCatalog).catch((e) => setError(e.message)) }, [])
+
   const submit = async (e) => {
     e.preventDefault()
     setBusy(true); setError(null)
     try {
-      const pool_spec = JSON.parse(form.pool_spec || '{"required":[],"random":[]}')
+      // Owner's raw-JSON view, when open, is the source of truth — otherwise built from the
+      // structured pickers above.
+      const pool_spec = advanced
+        ? JSON.parse(rawPoolSpec || '{"required":[],"random":[]}')
+        : { required, random: pools.filter((p) => p.count > 0 && p.from.length > 0) };
+      if (!advanced && !required.length && !pool_spec.random.length) {
+        throw new Error('Pick at least one required scenario or one random pool.')
+      }
       await adminApi.createAssessment({ ...form, duration_minutes: Number(form.duration_minutes), pool_spec })
-      setForm({ id: '', type: 'exam', title: '', duration_minutes: 60, hint_policy: 'disabled', pool_spec: '' })
+      setForm({ id: '', type: 'exam', title: '', duration_minutes: 60, hint_policy: 'disabled' })
+      setRequired([]); setPools([]); setRawPoolSpec('')
       onCreated()
-    } catch (e) { setError(e.message?.includes('JSON') ? 'pool_spec must be valid JSON' : e.message) } finally { setBusy(false) }
+    } catch (e) { setError(e.message?.includes('JSON') ? 'Raw pool_spec must be valid JSON' : e.message) } finally { setBusy(false) }
   }
 
   return (
@@ -91,26 +157,47 @@ function CreateAssessmentForm({ onCreated }) {
             <option value="disabled">disabled</option><option value="flagged">flagged</option><option value="free">free</option>
           </select></div>
       </div>
-      <label className={styles.label}>Pool spec (JSON)</label>
-      <textarea className={styles.input} style={{ width: '100%', fontFamily: 'var(--mono)', minHeight: '4rem' }}
-        placeholder='{"required":["scenario-id"],"random":[{"count":1,"from":["a","b"]}]}'
-        value={form.pool_spec} onChange={(e) => setForm({ ...form, pool_spec: e.target.value })} required />
+
+      {role === 'owner' && (
+        <p className={styles.muted}>
+          <button type="button" className={styles.linkBtn} onClick={() => setAdvanced(!advanced)}>
+            {advanced ? '← Back to the structured editor' : 'Advanced: edit pool_spec as raw JSON →'}
+          </button>
+        </p>
+      )}
+
+      {advanced ? (
+        <>
+          <label className={styles.label}>Pool spec (JSON)</label>
+          <textarea className={styles.input} style={{ width: '100%', fontFamily: 'var(--mono)', minHeight: '4rem' }}
+            placeholder='{"required":["scenario-id"],"random":[{"count":1,"from":["a","b"]}]}'
+            value={rawPoolSpec} onChange={(e) => setRawPoolSpec(e.target.value)} required />
+        </>
+      ) : !catalog ? <p className={styles.muted}>Loading scenario catalog…</p> : (
+        <PoolSpecEditor catalog={catalog} required={required} setRequired={setRequired} pools={pools} setPools={setPools} />
+      )}
+
       <button className={styles.btn} style={{ marginTop: '0.75rem' }} disabled={busy}>{busy ? 'Creating…' : 'Create'}</button>
     </form>
   )
 }
 
 function AssessmentDetail({ definition, cohorts, onBack }) {
-  const [sessions, setSessions] = useState(null)
+  // Phase 4.1 §16 — assessmentSessions is now paginated (was unbounded); `result` holds the
+  // full { items, page, total, total_pages } envelope, `sessions` stays the plain array for
+  // the table below so its render logic doesn't need to change.
+  const [result, setResult] = useState(null)
+  const [page, setPage] = useState(1)
   const [error, setError] = useState(null)
   const [notice, setNotice] = useState(null)
   const [cohortId, setCohortId] = useState(cohorts?.[0]?.cohort_id || '')
   const [busy, setBusy] = useState(false)
 
   const reload = useCallback(() => {
-    adminApi.assessmentSessions(definition.assessment_definition_id).then(setSessions).catch((e) => setError(e.message))
-  }, [definition])
+    adminApi.assessmentSessions(definition.assessment_definition_id, page).then(setResult).catch((e) => setError(e.message))
+  }, [definition, page])
   useEffect(reload, [reload])
+  const sessions = result?.items || null
 
   const assign = async () => {
     setBusy(true); setError(null); setNotice(null)
@@ -158,6 +245,13 @@ function AssessmentDetail({ definition, cohorts, onBack }) {
             </tbody>
           </table>
         )}
+        {result && result.total_pages > 1 && (
+          <div className={styles.pager}>
+            <button disabled={page <= 1} onClick={() => setPage((p) => p - 1)}>Previous</button>
+            <span className={styles.muted}>Page {result.page} of {result.total_pages} ({result.total} total)</span>
+            <button disabled={page >= result.total_pages} onClick={() => setPage((p) => p + 1)}>Next</button>
+          </div>
+        )}
       </div>
     </>
   )

frontend/src/admin/AuditView.jsx

@@ -2,36 +2,47 @@ import { useState, useEffect } from 'react'
 import { adminApi } from './api'
 import styles from './Admin.module.css'
 
-// admin/owner only (enforced server-side too). Latest 200 events, no pagination UI — this is a
-// pilot-scale audit log, not a SIEM; add pagination if that ever stops being true.
+// admin/owner only (enforced server-side too). Phase 4.1 §16 — real server-side pagination
+// (platform/api/src/pagination.js) replaces the old flat "latest 200, nothing older" cap.
 export default function AuditView() {
-  const [events, setEvents] = useState(null)
+  const [page, setPage] = useState(1)
+  const [result, setResult] = useState(null)
   const [error, setError] = useState(null)
 
   useEffect(() => {
-    adminApi.auditEvents().then(setEvents).catch((e) => setError(e.message))
-  }, [])
+    setResult(null)
+    adminApi.auditEvents(page).then(setResult).catch((e) => setError(e.message))
+  }, [page])
+
+  const events = result?.items || []
 
   return (
     <div className={styles.card}>
       <h2>Audit log</h2>
-      <p className={styles.muted}>Most recent 200 security/admin events.</p>
+      <p className={styles.muted}>{result ? `${result.total} total events` : 'Security/admin events, most recent first.'}</p>
       {error && <div className={styles.errorBox}>{error}</div>}
-      {!events ? <p className={styles.muted}>Loading…</p> : (
-        <table>
-          <thead><tr><th>When</th><th>Actor</th><th>Action</th><th>Resource</th><th>Result</th></tr></thead>
-          <tbody>
-            {events.map((e) => (
-              <tr key={e.audit_event_id}>
-                <td>{new Date(e.created_at).toLocaleString()}</td>
-                <td>{e.actor_type}{e.actor_id ? ` · ${e.actor_id.slice(0, 8)}` : ''}</td>
-                <td>{e.action}</td>
-                <td>{e.resource_type ? `${e.resource_type}${e.resource_id ? ` (${e.resource_id})` : ''}` : '—'}</td>
-                <td><span className={`${styles.pill} ${e.result === 'success' ? styles.pillOk : styles.pillErr}`}>{e.result}</span></td>
-              </tr>
-            ))}
-          </tbody>
-        </table>
+      {!result ? <p className={styles.muted}>Loading…</p> : (
+        <>
+          <table>
+            <thead><tr><th>When</th><th>Actor</th><th>Action</th><th>Resource</th><th>Result</th></tr></thead>
+            <tbody>
+              {events.map((e) => (
+                <tr key={e.audit_event_id}>
+                  <td>{new Date(e.created_at).toLocaleString()}</td>
+                  <td>{e.actor_type}{e.actor_id ? ` · ${e.actor_id.slice(0, 8)}` : ''}</td>
+                  <td>{e.action}</td>
+                  <td>{e.resource_type ? `${e.resource_type}${e.resource_id ? ` (${e.resource_id})` : ''}` : '—'}</td>
+                  <td><span className={`${styles.pill} ${e.result === 'success' ? styles.pillOk : styles.pillErr}`}>{e.result}</span></td>
+                </tr>
+              ))}
+            </tbody>
+          </table>
+          <div className={styles.pager}>
+            <button disabled={page <= 1} onClick={() => setPage((p) => p - 1)}>Previous</button>
+            <span className={styles.muted}>Page {result.page} of {result.total_pages}</span>
+            <button disabled={page >= result.total_pages} onClick={() => setPage((p) => p + 1)}>Next</button>
+          </div>
+        </>
       )}
     </div>
   )

frontend/src/admin/StudentsView.jsx

@@ -49,7 +49,10 @@ function StudentDetail({ studentId, onBack }) {
 
   const reload = useCallback(() => {
     adminApi.studentProgress(studentId).then(setProgress).catch(() => {})
-    adminApi.studentAttempts(studentId).then(setAttempts).catch(() => {})
+    // Phase 4.1 §16 — studentAttempts is now paginated; this panel only ever shows the most
+    // recent page (labeled "Recent attempts," not a full history browser — see
+    // /api/admin/students/:id/attempts's own pagination for the underlying capability).
+    adminApi.studentAttempts(studentId).then((r) => setAttempts(r.items)).catch(() => {})
     adminApi.studentEnvironment(studentId).then((rows) => setEnv(rows?.find((e) => e.status !== 'destroyed') || null)).catch(() => {})
   }, [studentId])
   useEffect(reload, [reload])

frontend/src/admin/api.js

@@ -52,7 +52,7 @@ export const adminApi = {
   createStudent: (display_name, cohort_id) => post('/api/admin/students', { display_name, cohort_id }),
   updateStudent: (id, fields) => patch(`/api/admin/students/${id}`, fields),
   studentProgress: (id) => get(`/api/admin/students/${id}/progress`),
-  studentAttempts: (id) => get(`/api/admin/students/${id}/attempts`),
+  studentAttempts: (id, page = 1) => get(`/api/admin/students/${id}/attempts?page=${page}`), // Phase 4.1 §16 — paginated, see pagination.js
   studentEnvironment: (id) => get(`/api/admin/students/${id}/environment`),
   provisionStudentLab: (id) => post(`/api/admin/students/${id}/environment/provision`),
   assignStudentCohort: (id, cohort_id) => post(`/api/admin/students/${id}/cohort`, { cohort_id }),
@@ -79,19 +79,20 @@ export const adminApi = {
   revokeInstructorSessions: (id) => post(`/api/admin/instructors/${id}/revoke-sessions`),
   resetInstructorPassword: (id, new_password) => post(`/api/admin/instructors/${id}/reset-password`, { new_password }),
 
-  auditEvents: () => get('/api/admin/audit-events'),
+  auditEvents: (page = 1) => get(`/api/admin/audit-events?page=${page}`), // Phase 4.1 §16 — paginated, see pagination.js
 
   // ── Phase 3B: learning analytics ────────────────────────────────────────────────────────
   cohortLearning: (id) => get(`/api/admin/cohorts/${id}/learning`),
   studentLearning: (id) => get(`/api/admin/students/${id}/learning`),
   scenarioAnalytics: (cohortId) => get(`/api/admin/scenario-analytics${cohortId ? `?cohort_id=${cohortId}` : ''}`),
   needsAttention: (cohortId) => get(`/api/admin/cohorts/${cohortId}/needs-attention`),
+  scenarioCatalog: () => get('/api/admin/scenario-catalog'), // Phase 4.1 §18 — structured assessment editor
 
   // ── Phase 3B: assessment (exam + capstone) management ───────────────────────────────────
   assessments: () => get('/api/admin/assessments'),
   createAssessment: (fields) => post('/api/admin/assessments', fields),
   updateAssessment: (id, fields) => patch(`/api/admin/assessments/${id}`, fields),
   assignAssessment: (id, cohort_id) => post(`/api/admin/assessments/${id}/assign`, { cohort_id }),
-  assessmentSessions: (id) => get(`/api/admin/assessments/${id}/sessions`),
+  assessmentSessions: (id, page = 1) => get(`/api/admin/assessments/${id}/sessions?page=${page}`), // Phase 4.1 §16 — paginated, see pagination.js
   cancelAssessmentSession: (sessionId) => post(`/api/admin/assessment-sessions/${sessionId}/cancel`),
 }

infra/.gitignore

@@ -0,0 +1,34 @@
+# Never commit filled secrets — only *.env.template files are tracked.
+*.env
+!*.env.template
+
+# Docker TLS material — CA/server/client certs and keys, generated by
+# infra/scripts/docker-tls/generate-ca.sh and friends. Templates/READMEs stay tracked.
+docker/tls/*.pem
+docker/tls/*.key
+docker/tls/*.crt
+docker/tls/*.srl
+!docker/tls/**/README.md
+
+# Ansible Vault password files and any decrypted vars
+ansible/.vault_pass
+ansible/**/vault_plain.yml
+
+# Terraform/OpenTofu state and generated vars (may contain resource IDs/IPs)
+# Phase 4.1: these patterns were previously written repo-root-relative (e.g.
+# "infra/terraform/...") inside THIS file, which git resolves relative to the .gitignore's own
+# directory (infra/) — so they silently matched nothing at all. Found via `git status` showing
+# a real .terraform/ provider cache as untracked when it should have been ignored. Fixed by
+# dropping the redundant leading "infra/" from every pattern below.
+terraform/**/*.tfstate
+terraform/**/*.tfstate.*
+terraform/**/.terraform/
+terraform/**/terraform.tfvars
+!terraform/**/terraform.tfvars.example
+
+# Backup output and restore scratch space
+backup/output/
+backup/restore-test/
+
+# Load test results (local artifacts, not source)
+scripts/loadtest/results/

infra/ansible/ansible.cfg

@@ -0,0 +1,10 @@
+[defaults]
+inventory = inventory/production.ini
+roles_path = roles
+host_key_checking = True
+retry_files_enabled = False
+stdout_callback = yaml
+
+[privilege_escalation]
+become = True
+become_method = sudo

infra/ansible/group_vars/production.vault.yml.template

@@ -0,0 +1,22 @@
+# Ansible Vault template — copy to production.vault.yml and encrypt it, NEVER commit the
+# plaintext or the encrypted file to git (infra/.gitignore already excludes
+# **/vault_plain.yml; the encrypted production.vault.yml itself is small/safe to commit IF
+# encrypted — see docs/PRODUCTION_DEPLOYMENT.md §10 for the actual decision made). Ansible
+# Vault chosen over SOPS/age here because it's already bundled with Ansible (no new tool to
+# install/trust) and this project's secret surface is small enough that Vault's simplicity
+# outweighs SOPS's per-key-value granularity — revisit if that changes.
+#
+#   cp production.vault.yml.template production.vault.yml
+#   # fill in real values below
+#   ansible-vault encrypt production.vault.yml
+#
+# Reference a vaulted value from group_vars/production.yml or a role like:
+#   postgres_password: "{{ vault_postgres_password }}"
+
+vault_postgres_password: __REPLACE__
+vault_provisioner_shared_secret: __REPLACE_openssl_rand_hex_32__
+vault_owner_bootstrap_password: __REPLACE__          # used once, by a human, per docs/RUNBOOK_FIRST_DEPLOYMENT.md
+vault_postgres_exporter_password: __REPLACE__
+vault_backup_encryption_passphrase: __REPLACE_openssl_rand_base64_32__
+vault_backup_offsite_credentials: __REPLACE_e.g._Hetzner_Storage_Box_ssh_key_passphrase_or_S3_secret_key__
+vault_grafana_admin_password: __REPLACE__

infra/ansible/group_vars/production.yml

@@ -0,0 +1,33 @@
+# Production variables — referenced by every {{ }} placeholder in infra/docker/, infra/firewall/,
+# and the roles below. Fill in the __PLACEHOLDER__ values once the real server exists
+# (docs/PRODUCTION_DEPLOYMENT.md §1). Nothing here is a secret — real secrets live in
+# infra/environments/production/*.env (gitignored) and Ansible Vault (see
+# infra/ansible/group_vars/production.vault.yml.template), never in this plain-vars file.
+
+env: production
+academy_domain: __REPLACE_e.g._academy.rootnode.example__
+
+# ── Network topology (Phase 4 §5 — placeholders until real hardware exists) ────────────────
+platform_ip: __PLATFORM_VM_IP__
+lab_host_ip: __LAB_HOST_VM_IP__
+management_cidr: __YOUR_ADMIN_ACCESS_CIDR__       # e.g. a VPN/WireGuard subnet, never 0.0.0.0/0
+proxmox_mgmt_cidr: __PROXMOX_HOST_MGMT_CIDR__
+lab_subnet_range: "10.42.0.0/16"                    # matches NO_PROXY_CIDRS already used by the app
+lab_router_port: "8443"                             # Gateway -> Lab Host Router (data plane) — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3
+
+# ── Storage (Phase 4 §4 — a decision framework, not a final layout; see docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §4)
+rootnode_data_root: /srv/rootnode-data   # expected to be a ZFS dataset mountpoint on the real host
+
+# ── Registry / images (infra/docker/IMAGE_STRATEGY.md) ─────────────────────────────────────
+registry: git.synthossystems.com
+image_tag: __GIT_SHA_OR_RELEASE_TAG__
+
+postgres_exporter_user: rootnode_exporter
+
+# ── Observability (infra/monitoring/) ───────────────────────────────────────────────────────
+prometheus_retention_days: 30
+alertmanager_webhook_url: __REPLACE_e.g._uptime_kuma_or_ntfy_webhook__
+
+# ── Backup (infra/backup/) ───────────────────────────────────────────────────────────────────
+backup_retention_days: 30
+backup_offsite_target: __REPLACE_e.g._Hetzner_Storage_Box_path_or_S3_bucket__

infra/ansible/group_vars/staging.yml

@@ -0,0 +1,22 @@
+# Staging variables — same shape as production.yml, lower stakes. See
+# infra/environments/staging/README.md.
+
+env: staging
+academy_domain: __REPLACE_e.g._staging.academy.rootnode.example_or_unset_for_no_public_TLS__
+
+platform_ip: __STAGING_PLATFORM_IP__
+lab_host_ip: __STAGING_LAB_HOST_IP__
+management_cidr: __YOUR_ADMIN_ACCESS_CIDR__
+proxmox_mgmt_cidr: __PROXMOX_HOST_MGMT_CIDR__
+lab_subnet_range: "10.42.0.0/16"
+
+rootnode_data_root: /srv/rootnode-data-staging
+
+registry: git.synthossystems.com
+image_tag: __STAGING_TAG_e.g._staging-<sha>__
+
+prometheus_retention_days: 7
+alertmanager_webhook_url: __OPTIONAL_staging_can_skip_alerting__
+
+backup_retention_days: 7
+backup_offsite_target: __OPTIONAL_staging_backups_are_lower_priority__

infra/ansible/inventory/production.ini

@@ -0,0 +1,16 @@
+# Production inventory. IPs/hostnames are placeholders — fill in once the real VMs exist
+# (docs/PRODUCTION_DEPLOYMENT.md §1). This file is safe to commit (no secrets, just topology).
+
+[platform]
+rn-platform-01 ansible_host=__PLATFORM_VM_IP__
+
+[lab_host]
+rn-lab-host-01 ansible_host=__LAB_HOST_VM_IP__
+
+[rootnode:children]
+platform
+lab_host
+
+[rootnode:vars]
+ansible_user=rootnode-deploy
+ansible_ssh_common_args='-o StrictHostKeyChecking=yes'

infra/ansible/inventory/staging.ini

@@ -0,0 +1,16 @@
+# Staging inventory — same group shape as production, pointed at homelab VMs/containers or a
+# resource-constrained slice of the dedicated server once it exists (infra/environments/staging/README.md).
+
+[platform]
+rn-platform-staging ansible_host=__STAGING_PLATFORM_IP__
+
+[lab_host]
+rn-lab-host-staging ansible_host=__STAGING_LAB_HOST_IP__
+
+[rootnode:children]
+platform
+lab_host
+
+[rootnode:vars]
+ansible_user=rootnode-deploy
+env=staging

infra/ansible/roles/backup/tasks/main.yml

@@ -0,0 +1,63 @@
+---
+# Deploys the backup script + a daily cron job on rn-platform-01. The script itself
+# (infra/backup/backup-postgres.sh) is identical to what infra/backup/test-backup-restore.sh
+# exercises locally — same code path, just pointed at real production values here.
+
+- name: Install postgresql-client + gpg (backup script dependencies)
+  ansible.builtin.apt:
+    name: [postgresql-client, gnupg]
+    state: present
+
+- name: Create backup directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0700"
+  loop:
+    - /opt/rootnode/backup
+    - "{{ rootnode_data_root }}/backups"
+
+- name: Deploy backup script
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../backup/backup-postgres.sh"
+    dest: /opt/rootnode/backup/backup-postgres.sh
+    mode: "0750"
+
+- name: Deploy restore + verify scripts (not run by cron, kept alongside for the runbook)
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../backup/{{ item }}"
+    dest: "/opt/rootnode/backup/{{ item }}"
+    mode: "0750"
+  loop:
+    - restore-postgres.sh
+    - verify-restore.sh
+
+- name: Deploy the backup env file (PLATFORM_DB_URL, encryption passphrase, offsite target)
+  ansible.builtin.template:
+    src: backup.env.j2
+    dest: /etc/rootnode/backup.env
+    owner: root
+    group: root
+    mode: "0600"
+
+- name: Deploy cron wrapper (sources the env file, writes to syslog too)
+  ansible.builtin.template:
+    src: run-backup.sh.j2
+    dest: /opt/rootnode/backup/run-backup.sh
+    mode: "0750"
+
+- name: Install daily backup cron job
+  ansible.builtin.cron:
+    name: rootnode-postgres-backup
+    user: root
+    minute: "15"
+    hour: "3"
+    job: /opt/rootnode/backup/run-backup.sh
+
+- name: Ensure node-exporter's textfile collector directory exists (backup metrics land here)
+  ansible.builtin.file:
+    path: /var/lib/node-exporter/textfile
+    state: directory
+    mode: "0755"

infra/ansible/roles/backup/templates/backup.env.j2

@@ -0,0 +1,6 @@
+PLATFORM_DB_URL=postgres://rootnode_platform:{{ vault_postgres_password }}@localhost:5432/rootnode_platform
+BACKUP_DIR={{ rootnode_data_root }}/backups
+BACKUP_ENCRYPTION_PASSPHRASE={{ vault_backup_encryption_passphrase }}
+BACKUP_RETENTION_DAYS={{ backup_retention_days }}
+BACKUP_OFFSITE_TARGET={{ backup_offsite_target }}
+METRICS_TEXTFILE_DIR=/var/lib/node-exporter/textfile

infra/ansible/roles/backup/templates/run-backup.sh.j2

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+# Cron wrapper — sources the deployed env file and runs the real backup script, logging to
+# syslog (visible via journalctl and shipped to Loki by promtail like every other unit) in
+# addition to whatever cron itself captures.
+set -euo pipefail
+set -a
+source /etc/rootnode/backup.env
+set +a
+exec /opt/rootnode/backup/backup-postgres.sh 2>&1 | systemd-cat -t rootnode-backup

infra/ansible/roles/common/defaults/main.yml

@@ -0,0 +1,3 @@
+---
+common_timezone: UTC
+common_unattended_upgrades: true

infra/ansible/roles/common/tasks/docker-engine.yml

@@ -0,0 +1,35 @@
+---
+# Shared Docker Engine install — used by both platform and lab-host roles (import_tasks from
+# common, not duplicated). Follows Docker's own official apt-repo installation method.
+
+- name: Add Docker's GPG key
+  ansible.builtin.get_url:
+    url: https://download.docker.com/linux/ubuntu/gpg
+    dest: /etc/apt/keyrings/docker.asc
+    mode: "0644"
+  register: docker_gpg
+  retries: 3
+  delay: 5
+  until: docker_gpg is succeeded
+
+- name: Add Docker apt repository
+  ansible.builtin.apt_repository:
+    repo: "deb [arch={{ ansible_architecture | replace('x86_64','amd64') }} signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable"
+    state: present
+    filename: docker
+
+- name: Install Docker Engine + Compose plugin
+  ansible.builtin.apt:
+    name:
+      - docker-ce
+      - docker-ce-cli
+      - containerd.io
+      - docker-compose-plugin
+    state: present
+    update_cache: true
+
+- name: Enable and start Docker
+  ansible.builtin.systemd:
+    name: docker
+    enabled: true
+    state: started

infra/ansible/roles/common/tasks/main.yml

@@ -0,0 +1,58 @@
+---
+# Baseline hardening applied to both rn-platform-01 and rn-lab-host-01 before anything
+# RootNode-specific — idempotent, safe to re-run (Phase 4 §20 requirement).
+
+- name: Set timezone
+  community.general.timezone:
+    name: "{{ common_timezone }}"
+
+- name: Install base packages
+  ansible.builtin.apt:
+    name:
+      - curl
+      - ca-certificates
+      - gnupg
+      - iptables
+      - iptables-persistent
+      - unattended-upgrades
+      - fail2ban
+      - jq
+    state: present
+    update_cache: true
+    cache_valid_time: 3600
+
+- name: Enable unattended security upgrades
+  ansible.builtin.copy:
+    dest: /etc/apt/apt.conf.d/20auto-upgrades
+    content: |
+      APT::Periodic::Update-Package-Lists "1";
+      APT::Periodic::Unattended-Upgrade "1";
+    mode: "0644"
+  when: common_unattended_upgrades
+
+- name: Create the rootnode-deploy service account (if not the connecting user already)
+  ansible.builtin.user:
+    name: rootnode-deploy
+    shell: /bin/bash
+    groups: docker
+    append: true
+    create_home: true
+  register: deploy_user_result
+  failed_when: false   # non-fatal — some hosts may already use this as ansible_user
+
+- name: Ensure fail2ban is enabled for SSH
+  ansible.builtin.service:
+    name: fail2ban
+    state: started
+    enabled: true
+
+- name: Create /opt/rootnode and /etc/rootnode base directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0750"
+  loop:
+    - /opt/rootnode
+    - /etc/rootnode

infra/ansible/roles/docker-tls/defaults/main.yml

@@ -0,0 +1,6 @@
+---
+# Local path (on the machine running ansible-playbook, NOT on either VM) holding the cert
+# bundle generated by infra/scripts/docker-tls/generate-docker-tls.sh. The CA private key
+# (ca-key.pem) is deliberately never copied to either host by this role — only its public
+# ca.pem, plus the one cert/key pair each host actually needs.
+docker_tls_local_bundle_dir: "{{ lookup('env', 'ROOTNODE_TLS_BUNDLE_DIR') | default('/does/not/exist', true) }}"

infra/ansible/roles/docker-tls/handlers/main.yml

@@ -0,0 +1,22 @@
+---
+- name: restart docker with tls
+  ansible.builtin.systemd:
+    name: docker
+    state: restarted
+    daemon_reload: true
+
+# Phase 4.1 — these restart the whole per-host compose wrapper unit (no per-container systemd
+# units exist in this project's convention, see the `platform`/`lab-host` roles), which is
+# heavier than a surgical single-container restart but consistent with how every other
+# config-changed-so-restart-the-stack case is already handled here.
+- name: restart lab-router
+  ansible.builtin.systemd:
+    name: rootnode-lab-host.service
+    state: restarted
+  when: inventory_hostname in groups['lab_host']
+
+- name: restart gateway
+  ansible.builtin.systemd:
+    name: rootnode-platform.service
+    state: restarted
+  when: inventory_hostname in groups['platform']

infra/ansible/roles/docker-tls/tasks/main.yml

@@ -0,0 +1,136 @@
+---
+# Deploys Docker TLS material generated OFFLINE by infra/scripts/docker-tls/generate-docker-tls.sh
+# — this role never generates key material itself, and the CA private key never touches either
+# VM (see infra/docker/tls/README.md). Fails loudly if the operator forgot to set
+# ROOTNODE_TLS_BUNDLE_DIR / run the generation script first, rather than silently skipping and
+# leaving the Provisioner unable to reach rn-lab-host-01.
+
+- name: Verify the local TLS bundle directory exists
+  ansible.builtin.stat:
+    path: "{{ docker_tls_local_bundle_dir }}"
+  register: bundle_dir
+  delegate_to: localhost
+  become: false
+
+- name: Fail fast if the TLS bundle wasn't generated
+  ansible.builtin.fail:
+    msg: >-
+      ROOTNODE_TLS_BUNDLE_DIR ({{ docker_tls_local_bundle_dir }}) does not exist. Run
+      infra/scripts/docker-tls/generate-docker-tls.sh first, then re-run with
+      ROOTNODE_TLS_BUNDLE_DIR=<output dir> in the environment.
+  when: not bundle_dir.stat.exists
+
+- name: Create /etc/rootnode/docker-tls
+  ansible.builtin.file:
+    path: /etc/rootnode/docker-tls
+    state: directory
+    owner: root
+    group: root
+    mode: "0700"
+
+- name: Deploy CA cert (public, both hosts)
+  ansible.builtin.copy:
+    src: "{{ docker_tls_local_bundle_dir }}/ca.pem"
+    dest: /etc/rootnode/docker-tls/ca.pem
+    owner: root
+    group: root
+    mode: "0444"
+
+- name: Deploy server cert/key (rn-lab-host-01 only)
+  ansible.builtin.copy:
+    src: "{{ docker_tls_local_bundle_dir }}/{{ item.src }}"
+    dest: "/etc/rootnode/docker-tls/{{ item.dest }}"
+    owner: root
+    group: root
+    mode: "{{ item.mode }}"
+  loop:
+    - { src: server-cert.pem, dest: server-cert.pem, mode: "0444" }
+    - { src: server-key.pem, dest: server-key.pem, mode: "0400" }
+  when: inventory_hostname in groups['lab_host']
+  notify: restart docker with tls
+
+- name: Deploy Provisioner client cert/key (rn-platform-01 only)
+  ansible.builtin.copy:
+    src: "{{ docker_tls_local_bundle_dir }}/{{ item.src }}"
+    dest: "/etc/rootnode/docker-tls/{{ item.dest }}"
+    owner: root
+    group: root
+    mode: "{{ item.mode }}"
+  loop:
+    - { src: provisioner-client-cert.pem, dest: provisioner-client-cert.pem, mode: "0444" }
+    - { src: provisioner-client-key.pem, dest: provisioner-client-key.pem, mode: "0400" }
+  when: inventory_hostname in groups['platform']
+
+- name: Configure dockerd for TLS (rn-lab-host-01 only)
+  ansible.builtin.template:
+    src: daemon.json.j2
+    dest: /etc/docker/daemon.json
+    owner: root
+    group: root
+    mode: "0644"
+  when: inventory_hostname in groups['lab_host']
+  notify: restart docker with tls
+
+- name: Override the Docker systemd unit's ExecStart (clears the default -H fd:// flag)
+  ansible.builtin.file:
+    path: /etc/systemd/system/docker.service.d
+    state: directory
+    mode: "0755"
+  when: inventory_hostname in groups['lab_host']
+
+- name: Deploy the ExecStart override
+  ansible.builtin.template:
+    src: docker-override.conf.j2
+    dest: /etc/systemd/system/docker.service.d/override.conf
+    owner: root
+    group: root
+    mode: "0644"
+  when: inventory_hostname in groups['lab_host']
+  notify: restart docker with tls
+
+# ── Phase 4.1: the DATA-plane mTLS pair (Gateway <-> Lab Host Router). Same CA as the
+# CONTROL-plane pair above, separate directory (/etc/rootnode/lab-router-tls) so the two
+# concerns stay bind-mount-separable — the `lab-router`/`gateway` containers only ever need
+# their own pair mounted in, never the Docker TLS one. See
+# docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3.
+- name: Create /etc/rootnode/lab-router-tls
+  ansible.builtin.file:
+    path: /etc/rootnode/lab-router-tls
+    state: directory
+    owner: root
+    group: root
+    mode: "0700"
+
+- name: Deploy CA cert for the data-plane pair (public, both hosts)
+  ansible.builtin.copy:
+    src: "{{ docker_tls_local_bundle_dir }}/ca.pem"
+    dest: /etc/rootnode/lab-router-tls/ca.pem
+    owner: root
+    group: root
+    mode: "0444"
+
+- name: Deploy Lab Host Router server cert/key (rn-lab-host-01 only)
+  ansible.builtin.copy:
+    src: "{{ docker_tls_local_bundle_dir }}/{{ item.src }}"
+    dest: "/etc/rootnode/lab-router-tls/{{ item.dest }}"
+    owner: root
+    group: root
+    mode: "{{ item.mode }}"
+  loop:
+    - { src: router-server-cert.pem, dest: router-server-cert.pem, mode: "0444" }
+    - { src: router-server-key.pem, dest: router-server-key.pem, mode: "0400" }
+  when: inventory_hostname in groups['lab_host']
+  notify: restart lab-router
+
+- name: Deploy Gateway client cert/key (rn-platform-01 only)
+  ansible.builtin.copy:
+    src: "{{ docker_tls_local_bundle_dir }}/{{ item.src }}"
+    dest: "/etc/rootnode/lab-router-tls/{{ item.dest }}"
+    owner: root
+    group: root
+    mode: "{{ item.mode }}"
+  loop:
+    - { src: gateway-client-cert.pem, dest: gateway-client-cert.pem, mode: "0444" }
+    - { src: gateway-client-key.pem, dest: gateway-client-key.pem, mode: "0400" }
+  when: inventory_hostname in groups['platform']
+  notify: restart gateway

infra/ansible/roles/docker-tls/templates/daemon.json.j2

@@ -0,0 +1,13 @@
+{
+  "tls": true,
+  "tlsverify": true,
+  "tlscacert": "/etc/rootnode/docker-tls/ca.pem",
+  "tlscert": "/etc/rootnode/docker-tls/server-cert.pem",
+  "tlskey": "/etc/rootnode/docker-tls/server-key.pem",
+  "hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2376"],
+
+  "storage-driver": "overlay2",
+  "default-address-pools": [
+    { "base": "10.42.0.0/16", "size": 24 }
+  ]
+}

infra/ansible/roles/docker-tls/templates/docker-override.conf.j2

@@ -0,0 +1,7 @@
+# Docker's systemd unit specifies `-H fd://` on ExecStart by default, which conflicts with
+# setting `hosts` in daemon.json ("specified both as a flag and in the configuration file").
+# This override clears ExecStart and re-specifies it with no -H flag, letting daemon.json's
+# `hosts` list be the only source of truth — the documented fix for this exact conflict.
+[Service]
+ExecStart=
+ExecStart=/usr/bin/dockerd

infra/ansible/roles/firewall/handlers/main.yml

@@ -0,0 +1,4 @@
+---
+- name: reload firewall
+  ansible.builtin.command: netfilter-persistent reload
+  changed_when: true

infra/ansible/roles/firewall/tasks/main.yml

@@ -0,0 +1,28 @@
+---
+# Deploys the matching infra/firewall/*.rules file for this host's group and loads it. Same
+# ruleset, same variables as infra/firewall/validate-firewall.sh renders locally — this task
+# just uses Ansible's own templating instead of sed, against the real inventory values.
+
+- name: Determine which ruleset applies to this host
+  ansible.builtin.set_fact:
+    firewall_ruleset_name: "{{ 'rn-platform-01' if inventory_hostname in groups['platform'] else 'rn-lab-host-01' }}"
+
+- name: Render firewall ruleset for this host
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../firewall/{{ firewall_ruleset_name }}.rules"
+    dest: /etc/iptables/rules.v4.rootnode
+    owner: root
+    group: root
+    mode: "0600"
+  notify: reload firewall
+
+- name: Point iptables-persistent at the RootNode ruleset
+  ansible.builtin.file:
+    src: /etc/iptables/rules.v4.rootnode
+    dest: /etc/iptables/rules.v4
+    state: link
+    force: true
+  notify: reload firewall
+
+- name: Flush handlers so the firewall is active before later roles assume it
+  ansible.builtin.meta: flush_handlers

infra/ansible/roles/lab-host/handlers/main.yml

@@ -0,0 +1,6 @@
+---
+- name: restart rootnode-lab-host
+  ansible.builtin.systemd:
+    name: rootnode-lab-host
+    state: restarted
+    daemon_reload: true

infra/ansible/roles/lab-host/tasks/main.yml

@@ -0,0 +1,72 @@
+---
+# rn-lab-host-01: Docker Engine (TLS config comes from the docker-tls role, run before this
+# one), the egress proxy + monitoring agents compose stack, kernel prerequisites for nested
+# k3s in privileged Lab containers, and Docker daemon storage/limits configuration.
+
+- name: Install Docker Engine
+  ansible.builtin.import_tasks: "{{ role_path }}/../common/tasks/docker-engine.yml"
+
+- name: Kernel prerequisites for nested k3s (privileged containers)
+  ansible.posix.sysctl:
+    name: "{{ item.name }}"
+    value: "{{ item.value }}"
+    sysctl_set: true
+    state: present
+    reload: true
+  loop:
+    - { name: net.ipv4.ip_forward, value: "1" }
+    - { name: fs.inotify.max_user_watches, value: "524288" }   # k3s/kubelet watch a lot of files
+    - { name: fs.inotify.max_user_instances, value: "512" }
+    - { name: vm.max_map_count, value: "262144" }               # matches upstream k3s/etcd guidance
+
+- name: Ensure cgroup v2 controllers needed by nested k3s are available
+  ansible.builtin.command: cat /sys/fs/cgroup/cgroup.controllers
+  register: cgroup_controllers
+  changed_when: false
+  failed_when: false
+
+- name: Warn if cgroup v2 controllers look incomplete (informational only)
+  ansible.builtin.debug:
+    msg: >-
+      cgroup.controllers reports: {{ cgroup_controllers.stdout | default('unreadable') }}.
+      Expect at least cpu, memory, pids, io for reliable nested k3s — verify manually if any
+      are missing (see docs/PRODUCTION_VALIDATION_CHECKLIST.md "Lab boot" section).
+  when: cgroup_controllers.rc == 0
+
+- name: Create data/config directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: docker
+    mode: "0750"
+  loop:
+    - /opt/rootnode/lab-host
+    - "{{ rootnode_data_root }}/lab-work"   # disposable — see docs/BACKUP_DR.md, never backed up
+
+- name: Render production docker-compose.yml
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../docker/lab-host/docker-compose.production.yml"
+    dest: /opt/rootnode/lab-host/docker-compose.yml
+    owner: root
+    group: root
+    mode: "0640"
+  notify: restart rootnode-lab-host
+
+- name: Deploy systemd unit
+  ansible.builtin.template:
+    src: rootnode-lab-host.service.j2
+    dest: /etc/systemd/system/rootnode-lab-host.service
+    mode: "0644"
+  notify: restart rootnode-lab-host
+
+- name: Enable and start rootnode-lab-host
+  ansible.builtin.systemd:
+    name: rootnode-lab-host
+    enabled: true
+    state: started
+    daemon_reload: true
+
+# Storage-driver/networking daemon options live in the docker-tls role's daemon.json.j2 —
+# Docker only ever reads ONE /etc/docker/daemon.json, so that's the single source of truth for
+# this host's whole daemon config, not just its TLS settings (see that role's template).

infra/ansible/roles/lab-host/templates/rootnode-lab-host.service.j2

@@ -0,0 +1,17 @@
+[Unit]
+Description=RootNode Academy Lab Host services (egress proxy + monitoring agents)
+Requires=docker.service
+After=docker.service network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+WorkingDirectory=/opt/rootnode/lab-host
+ExecStart=/usr/bin/docker compose -f docker-compose.yml up -d --remove-orphans
+ExecStop=/usr/bin/docker compose -f docker-compose.yml down
+ExecReload=/usr/bin/docker compose -f docker-compose.yml up -d --remove-orphans
+TimeoutStartSec=180
+
+[Install]
+WantedBy=multi-user.target

infra/ansible/roles/monitoring-agent/handlers/main.yml

@@ -0,0 +1,16 @@
+---
+- name: reload prometheus
+  ansible.builtin.command: docker compose -f /opt/rootnode/platform/docker-compose.yml restart prometheus
+  changed_when: true
+
+- name: reload alertmanager
+  ansible.builtin.command: docker compose -f /opt/rootnode/platform/docker-compose.yml restart alertmanager
+  changed_when: true
+
+- name: restart platform promtail
+  ansible.builtin.command: docker compose -f /opt/rootnode/platform/docker-compose.yml restart promtail
+  changed_when: true
+
+- name: restart lab-host promtail
+  ansible.builtin.command: docker compose -f /opt/rootnode/lab-host/docker-compose.yml restart promtail
+  changed_when: true

infra/ansible/roles/monitoring-agent/tasks/main.yml

@@ -0,0 +1,75 @@
+---
+# Deploys infra/monitoring/ configs to whichever host needs them. Named monitoring-agent for
+# historical reasons in this repo's role list, but it deploys both the "agent" pieces
+# (node-exporter/cadvisor/promtail configs, already referenced directly by the platform/
+# lab-host compose files) and the central stack's config (Prometheus/Alertmanager/Loki/Grafana
+# on rn-platform-01 only).
+
+- name: Create /opt/rootnode/monitoring (rn-platform-01)
+  ansible.builtin.file:
+    path: /opt/rootnode/monitoring/rules
+    state: directory
+    mode: "0755"
+  when: inventory_hostname in groups['platform']
+
+- name: Deploy Prometheus config
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../monitoring/prometheus/prometheus.yml.j2"
+    dest: /opt/rootnode/monitoring/prometheus.yml
+    mode: "0644"
+  when: inventory_hostname in groups['platform']
+  notify: reload prometheus
+
+- name: Deploy alert rules
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../monitoring/prometheus/rules/rootnode.yml"
+    dest: /opt/rootnode/monitoring/rules/rootnode.yml
+    mode: "0644"
+  when: inventory_hostname in groups['platform']
+  notify: reload prometheus
+
+- name: Deploy blackbox_exporter config
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../monitoring/prometheus/blackbox.yml"
+    dest: /opt/rootnode/monitoring/blackbox.yml
+    mode: "0644"
+  when: inventory_hostname in groups['platform']
+
+- name: Deploy Alertmanager config
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../monitoring/alertmanager/alertmanager.yml.j2"
+    dest: /opt/rootnode/monitoring/alertmanager.yml
+    mode: "0644"
+  when: inventory_hostname in groups['platform']
+  notify: reload alertmanager
+
+- name: Deploy Loki config
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../monitoring/loki/loki-config.yml"
+    dest: /opt/rootnode/monitoring/loki-config.yml
+    mode: "0644"
+  when: inventory_hostname in groups['platform']
+
+- name: Deploy Grafana provisioning + dashboards
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../monitoring/grafana/{{ item }}/"
+    dest: "/opt/rootnode/monitoring/grafana/{{ item }}/"
+    mode: "0644"
+  loop: [provisioning, dashboards]
+  when: inventory_hostname in groups['platform']
+
+- name: Deploy platform-side promtail config
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../monitoring/promtail/promtail-config.platform.yml"
+    dest: /opt/rootnode/monitoring/promtail-config.yml
+    mode: "0644"
+  when: inventory_hostname in groups['platform']
+  notify: restart platform promtail
+
+- name: Deploy lab-host promtail config
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../monitoring/promtail/promtail-config.lab-host.yml.j2"
+    dest: /opt/rootnode/lab-host/promtail-config.yml
+    mode: "0644"
+  when: inventory_hostname in groups['lab_host']
+  notify: restart lab-host promtail

infra/ansible/roles/platform/handlers/main.yml

@@ -0,0 +1,6 @@
+---
+- name: restart rootnode-platform
+  ansible.builtin.systemd:
+    name: rootnode-platform
+    state: restarted
+    daemon_reload: true

infra/ansible/roles/platform/tasks/main.yml

@@ -0,0 +1,85 @@
+---
+# rn-platform-01: Docker Engine, directory structure, rendered compose file + env files,
+# Postgres tuning overlay, reverse proxy config, and the systemd unit that runs it all.
+# Docker TLS material and the host firewall are separate roles (docker-tls, firewall) run
+# before this one in site.yml — this role assumes both are already in place.
+
+- name: Install Docker Engine
+  ansible.builtin.import_tasks: "{{ role_path }}/../common/tasks/docker-engine.yml"
+
+- name: Create data/config directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: docker
+    mode: "0750"
+  loop:
+    - "{{ rootnode_data_root }}/postgres"
+    - "{{ rootnode_data_root }}/backups"
+    - /opt/rootnode/platform
+    - /opt/rootnode/platform/postgresql.conf.d
+    - /etc/rootnode/provisioner-config
+
+- name: Deploy Postgres tuning overlay
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../docker/postgres/postgresql.conf.d/00-rootnode-baseline.conf"
+    dest: /opt/rootnode/platform/postgresql.conf.d/00-rootnode-baseline.conf
+    mode: "0644"
+  notify: restart rootnode-platform
+
+- name: Deploy provisioner-config overlay ({{ env }})
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../../environments/{{ env }}/provisioner-config/"
+    dest: /etc/rootnode/provisioner-config/
+    mode: "0644"
+  notify: restart rootnode-platform
+
+- name: Deploy per-service env files (from Ansible Vault-backed variables — see group_vars)
+  ansible.builtin.template:
+    src: "{{ item }}.env.j2"
+    dest: "/etc/rootnode/{{ item }}.env"
+    owner: root
+    group: root
+    mode: "0600"
+  loop:
+    - platform
+    - provisioner
+    - gateway
+    - postgres
+  notify: restart rootnode-platform
+
+- name: Render production docker-compose.yml
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../docker/platform/docker-compose.production.yml"
+    dest: /opt/rootnode/platform/docker-compose.yml
+    owner: root
+    group: root
+    mode: "0640"
+  notify: restart rootnode-platform
+
+- name: Render Caddyfile
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../docker/platform/Caddyfile.j2"
+    dest: /opt/rootnode/platform/Caddyfile
+    owner: root
+    group: root
+    mode: "0644"
+  notify: restart rootnode-platform
+
+- name: Deploy systemd unit
+  ansible.builtin.template:
+    src: rootnode-platform.service.j2
+    dest: /etc/systemd/system/rootnode-platform.service
+    mode: "0644"
+  notify: restart rootnode-platform
+
+# The `backup` role (applied after this one in site.yml) deploys the backup script and cron —
+# kept separate since it also has to run correctly on a host that ISN'T freshly deployed.
+
+- name: Enable and start rootnode-platform
+  ansible.builtin.systemd:
+    name: rootnode-platform
+    enabled: true
+    state: started
+    daemon_reload: true

infra/ansible/roles/platform/templates/gateway.env.j2

@@ -0,0 +1,13 @@
+# Rendered by Ansible — see platform.env.j2 for the pattern.
+PLATFORM_API_URL=http://api:4300
+PROVISIONER_URL=http://provisioner:4100
+PROVISIONER_SHARED_SECRET={{ vault_provisioner_shared_secret }}
+GATEWAY_PORT=4200
+# Phase 4.1 — mTLS client identity for the Lab Host Router (data plane). Only used when a
+# lab_host's routing_mode is 'router' (see platform/gateway/src/lab-endpoint.js); harmless if
+# every configured lab_host is still 'direct'. Paths are inside the container — mounted from
+# /etc/rootnode/lab-router-tls on the host by the `docker-tls` role, same pattern as the
+# Provisioner's Docker TLS client cert.
+LAB_ROUTER_TLS_CA=/tls/ca.pem
+LAB_ROUTER_TLS_CERT=/tls/gateway-client-cert.pem
+LAB_ROUTER_TLS_KEY=/tls/gateway-client-key.pem

infra/ansible/roles/platform/templates/platform.env.j2

@@ -0,0 +1,24 @@
+# Rendered by Ansible from infra/environments/{{ env }}/platform.env.template's shape —
+# real values come from group_vars/{{ env }}.yml + the encrypted vault file. Do not hand-edit
+# on the host; re-run the platform role instead.
+
+# service names, not localhost — production runs these on a named Docker bridge network
+# (rn-platform-internal), not network_mode: host like the dev compose file does.
+PLATFORM_DB_URL=postgres://rootnode_platform:{{ vault_postgres_password }}@postgres:5432/rootnode_platform
+PROVISIONER_URL=http://provisioner:4100
+PROVISIONER_SHARED_SECRET={{ vault_provisioner_shared_secret }}
+PLATFORM_API_PORT=4300
+
+STUDENT_SESSION_TTL_HOURS=8
+STUDENT_SESSION_IDLE_HOURS=2
+INSTRUCTOR_SESSION_TTL_HOURS=12
+INSTRUCTOR_SESSION_IDLE_HOURS=2
+
+COOKIE_SECURE=1
+TRUST_PROXY=1
+
+# Phase 4.1 §17 — instructor "needs attention" thresholds. Commented out (code defaults apply:
+# 3 failures, 7 stale days) — uncomment and override in group_vars/{{ env }}.yml only if a
+# cohort's real usage shows the defaults are wrong for it.
+# NEEDS_ATTENTION_FAIL_THRESHOLD=3
+# NEEDS_ATTENTION_STALE_DAYS=7

infra/ansible/roles/platform/templates/postgres.env.j2

@@ -0,0 +1,4 @@
+# Rendered by Ansible — see platform.env.j2 for the pattern.
+POSTGRES_USER=rootnode_platform
+POSTGRES_PASSWORD={{ vault_postgres_password }}
+POSTGRES_DB=rootnode_platform

infra/ansible/roles/platform/templates/provisioner.env.j2

@@ -0,0 +1,11 @@
+# Rendered by Ansible — see platform.env.j2 for the pattern.
+PLATFORM_DB_URL=postgres://rootnode_platform:{{ vault_postgres_password }}@postgres:5432/rootnode_platform
+PROVISIONER_SHARED_SECRET={{ vault_provisioner_shared_secret }}
+PROVISIONER_PORT=4100
+PROVISIONER_CONFIG_DIR=/app/config
+
+LAB_IMAGE_REF={{ registry }}/rootnode/lab:{{ image_tag }}
+
+EGRESS_PROXY_PORT=3128
+DNS_RESOLVER=127.0.0.11
+NO_PROXY_CIDRS=10.42.0.0/16,10.43.0.0/16

infra/ansible/roles/platform/templates/rootnode-platform.service.j2

@@ -0,0 +1,17 @@
+[Unit]
+Description=RootNode Academy Platform (docker compose)
+Requires=docker.service
+After=docker.service network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+WorkingDirectory=/opt/rootnode/platform
+ExecStart=/usr/bin/docker compose -f docker-compose.yml up -d --remove-orphans
+ExecStop=/usr/bin/docker compose -f docker-compose.yml down
+ExecReload=/usr/bin/docker compose -f docker-compose.yml up -d --remove-orphans
+TimeoutStartSec=300
+
+[Install]
+WantedBy=multi-user.target

infra/ansible/site.yml

@@ -0,0 +1,52 @@
+---
+# Top-level playbook — full production/staging deployment. Idempotent: re-running against an
+# already-deployed host converges it, doesn't rebuild from scratch (Phase 4 §20 requirement).
+#
+# Usage:
+#   ansible-playbook site.yml -i inventory/production.ini --vault-password-file .vault_pass
+#   ansible-playbook site.yml -i inventory/staging.ini -e env=staging
+#
+# Role order matters: firewall and docker-tls must land before platform/lab-host bring up
+# services that depend on the network policy / TLS material already being in place.
+
+- name: Base hardening — all RootNode hosts
+  hosts: rootnode
+  become: true
+  roles:
+    - common
+
+- name: Firewall policy — all RootNode hosts
+  hosts: rootnode
+  become: true
+  roles:
+    - firewall
+
+- name: Docker TLS — both hosts (server cert on lab_host, client cert on platform)
+  hosts: rootnode
+  become: true
+  roles:
+    - docker-tls
+
+- name: Deploy the Platform VM
+  hosts: platform
+  become: true
+  roles:
+    - platform
+
+- name: Deploy the Lab Host VM
+  hosts: lab_host
+  become: true
+  roles:
+    - lab-host
+
+- name: Deploy monitoring configuration — both hosts
+  hosts: rootnode
+  become: true
+  roles:
+    - monitoring-agent
+
+- name: Deploy backup automation — Platform VM only (that's where Postgres lives)
+  hosts: platform
+  become: true
+  roles:
+    - backup

infra/backup/backup-postgres.sh

@@ -0,0 +1,77 @@
+#!/usr/bin/env bash
+# Postgres backup — Tier 1 (irreplaceable) data per docs/BACKUP_DR.md. Runs via cron on
+# rn-platform-01 (deployed by the `backup` Ansible role), and locally testable against ANY
+# reachable Postgres (that's exactly what infra/backup/test-backup-restore.sh does).
+#
+# Produces one gzip-compressed pg_dump per run, encrypts it, prunes local copies past
+# BACKUP_RETENTION_DAYS, and (if BACKUP_OFFSITE_TARGET is set) rsyncs the encrypted copy
+# off-host. Writes a Prometheus textfile-collector metric so infra/monitoring/'s BackupFailed/
+# BackupAging alerts have real data to fire on.
+#
+# Required env: PLATFORM_DB_URL, BACKUP_DIR, BACKUP_ENCRYPTION_PASSPHRASE
+# Optional env: BACKUP_RETENTION_DAYS (default 30), BACKUP_OFFSITE_TARGET, METRICS_TEXTFILE_DIR
+set -euo pipefail
+
+: "${PLATFORM_DB_URL:?PLATFORM_DB_URL is required}"
+: "${BACKUP_DIR:?BACKUP_DIR is required}"
+: "${BACKUP_ENCRYPTION_PASSPHRASE:?BACKUP_ENCRYPTION_PASSPHRASE is required — see docs/BACKUP_DR.md}"
+BACKUP_RETENTION_DAYS="${BACKUP_RETENTION_DAYS:-30}"
+METRICS_TEXTFILE_DIR="${METRICS_TEXTFILE_DIR:-}"
+
+TS="$(date -u +%Y%m%dT%H%M%SZ)"
+mkdir -p "$BACKUP_DIR"
+DUMP_FILE="$BACKUP_DIR/rootnode-platform-$TS.sql.gz"
+ENC_FILE="$DUMP_FILE.gpg"
+
+write_metric() {
+  # $1 = metric line. Written atomically (tmp+rename) so Prometheus's node-exporter
+  # textfile-collector never scrapes a half-written file.
+  [ -n "$METRICS_TEXTFILE_DIR" ] || return 0
+  mkdir -p "$METRICS_TEXTFILE_DIR"
+  local tmp; tmp="$(mktemp "$METRICS_TEXTFILE_DIR/.rootnode_backup.XXXXXX")"
+  echo "$1" > "$tmp"
+  mv -f "$tmp" "$METRICS_TEXTFILE_DIR/rootnode_backup.prom"
+}
+
+fail() {
+  echo "BACKUP FAILED: $1" >&2
+  write_metric "rootnode_backup_last_run_success 0"
+  exit 1
+}
+trap 'fail "unexpected error at line $LINENO"' ERR
+
+echo "==> Dumping $PLATFORM_DB_URL to $DUMP_FILE"
+pg_dump --format=plain --no-owner --no-privileges "$PLATFORM_DB_URL" | gzip -9 > "$DUMP_FILE"
+
+# Fail loudly on a suspiciously small dump (empty/truncated) rather than "successfully"
+# backing up nothing — a real, if crude, sanity check.
+DUMP_SIZE=$(stat -c%s "$DUMP_FILE" 2>/dev/null || stat -f%z "$DUMP_FILE")
+[ "$DUMP_SIZE" -gt 1024 ] || fail "dump is suspiciously small ($DUMP_SIZE bytes) — refusing to treat this as a real backup"
+
+echo "==> Encrypting (AES-256, symmetric passphrase — see docs/BACKUP_DR.md for the recovery-key handling policy)"
+gpg --batch --yes --passphrase "$BACKUP_ENCRYPTION_PASSPHRASE" --symmetric --cipher-algo AES256 \
+  --output "$ENC_FILE" "$DUMP_FILE"
+rm -f "$DUMP_FILE"   # never leave the plaintext dump on disk once the encrypted copy exists
+
+echo "==> Local backup complete: $ENC_FILE ($(stat -c%s "$ENC_FILE" 2>/dev/null || stat -f%z "$ENC_FILE") bytes)"
+
+if [ -n "${BACKUP_OFFSITE_TARGET:-}" ]; then
+  echo "==> Copying off-host to $BACKUP_OFFSITE_TARGET"
+  # rsync over SSH works for both a Hetzner Storage Box (SSH/SFTP-native) and any other
+  # SSH-reachable target — the adapter point the Phase 4 spec asked for is this one variable,
+  # not a vendor-specific script. An S3-compatible target uses `aws s3 cp` / `rclone` instead;
+  # see docs/BACKUP_DR.md for both forms.
+  rsync -az --timeout=120 "$ENC_FILE" "$BACKUP_OFFSITE_TARGET/" \
+    || fail "off-host copy to $BACKUP_OFFSITE_TARGET failed"
+else
+  echo "==> BACKUP_OFFSITE_TARGET not set — local-only backup (fine for a local test run, NOT fine for production, see docs/BACKUP_DR.md)"
+fi
+
+echo "==> Pruning local backups older than $BACKUP_RETENTION_DAYS days"
+find "$BACKUP_DIR" -name 'rootnode-platform-*.sql.gz.gpg' -mtime "+$BACKUP_RETENTION_DAYS" -delete
+
+write_metric "rootnode_backup_last_run_success 1
+rootnode_backup_last_success_timestamp_seconds $(date +%s)
+rootnode_backup_size_bytes $(stat -c%s "$ENC_FILE" 2>/dev/null || stat -f%z "$ENC_FILE")"
+
+echo "==> Backup succeeded"

infra/backup/restore-postgres.sh

@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+# Restores an encrypted backup produced by backup-postgres.sh into a target database.
+# Deliberately requires an EXPLICIT target — never silently restores over the live production
+# database ("what would you do if this ran the wrong way" is a real disaster-recovery failure
+# mode, so making that impossible to trigger by accident matters more than convenience here).
+#
+# Usage: ./restore-postgres.sh <encrypted-backup-file> <target-db-url>
+#
+# For a REAL production restore, target-db-url should point at a FRESH database created for
+# this purpose (docs/RUNBOOK_POSTGRES_RESTORE.md walks through creating it) — restoring into a
+# database that already has data merges/conflicts rather than cleanly replacing it.
+set -euo pipefail
+
+ENC_FILE="${1:?usage: restore-postgres.sh <encrypted-backup-file> <target-db-url>}"
+TARGET_DB_URL="${2:?usage: restore-postgres.sh <encrypted-backup-file> <target-db-url>}"
+: "${BACKUP_ENCRYPTION_PASSPHRASE:?BACKUP_ENCRYPTION_PASSPHRASE is required to decrypt}"
+
+[ -f "$ENC_FILE" ] || { echo "No such file: $ENC_FILE" >&2; exit 1; }
+
+TMPDIR="$(mktemp -d)"
+trap 'rm -rf "$TMPDIR"' EXIT
+DUMP_FILE="$TMPDIR/restore.sql.gz"
+
+echo "==> Decrypting $ENC_FILE"
+gpg --batch --yes --passphrase "$BACKUP_ENCRYPTION_PASSPHRASE" --decrypt --output "$DUMP_FILE" "$ENC_FILE"
+
+echo "==> Confirming target is reachable"
+psql "$TARGET_DB_URL" -c "SELECT 1" > /dev/null
+
+echo "==> Target database currently has these tables (should be empty/unimportant before you continue):"
+psql "$TARGET_DB_URL" -c "\dt" || true
+read -r -p "Type RESTORE to proceed against $TARGET_DB_URL: " CONFIRM
+[ "$CONFIRM" = "RESTORE" ] || { echo "Aborted."; exit 1; }
+
+echo "==> Restoring"
+gunzip -c "$DUMP_FILE" | psql "$TARGET_DB_URL"
+
+echo "==> Restore complete. Verify with infra/backup/verify-restore.sh $TARGET_DB_URL"

infra/backup/seed-realistic-data.js

@@ -0,0 +1,94 @@
+#!/usr/bin/env node
+// Seeds realistic academy data for the Phase 4.1 §20 destructive restore drill — an owner, an
+// admin instructor, a cohort, 3 students, progress, attempts, an assessment definition +
+// session, and an achievement. Uses the REAL Platform API over HTTP for everything that
+// doesn't need a live Provisioner/k3s Lab (owner login, instructor/cohort/student/assessment
+// creation — genuinely exercises the real RBAC/creation code paths), and direct SQL inserts
+// only for progress/attempts/achievements (those need real scenario activity in the full app;
+// re-running the whole learning engine here would just re-prove what e2e-phase3b.js already
+// proves — this drill is about DATA DURABILITY, not re-testing scenario validation).
+//
+// Usage: PLATFORM_API_URL=http://localhost:PORT PLATFORM_DB_URL=postgres://... node seed-realistic-data.js
+
+const { Pool } = require('../../platform/api/node_modules/pg');
+
+const API = process.env.PLATFORM_API_URL || 'http://localhost:4390';
+const DB_URL = process.env.PLATFORM_DB_URL;
+if (!DB_URL) { console.error('Set PLATFORM_DB_URL'); process.exit(1); }
+
+function makeSession() {
+  const jar = new Map();
+  function cookieHeader() { return [...jar].map(([k, v]) => `${k}=${v}`).join('; '); }
+  function absorb(res) {
+    const cookies = typeof res.headers.getSetCookie === 'function' ? res.headers.getSetCookie() : [];
+    for (const c of cookies) {
+      const [pair] = c.split(';');
+      const eq = pair.indexOf('=');
+      if (eq === -1) continue;
+      jar.set(pair.slice(0, eq).trim(), pair.slice(eq + 1).trim());
+    }
+  }
+  return {
+    async req(method, path, body) {
+      const headers = { 'Content-Type': 'application/json' };
+      if (cookieHeader()) headers.Cookie = cookieHeader();
+      if (!['GET', 'HEAD'].includes(method)) headers['X-CSRF-Token'] = jar.get('rn_csrf_i') || '';
+      const res = await fetch(`${API}${path}`, { method, headers, body: body ? JSON.stringify(body) : undefined });
+      absorb(res);
+      const text = await res.text();
+      let json; try { json = JSON.parse(text); } catch { json = text; }
+      if (!res.ok) throw new Error(`${method} ${path} -> ${res.status}: ${JSON.stringify(json)}`);
+      return json;
+    },
+  };
+}
+
+async function main() {
+  const owner = makeSession();
+  await owner.req('POST', '/api/auth/instructor/login', { email: '[email protected]', password: 'DrillOwnerPass!2026' });
+  console.log('  owner login ok');
+
+  const admin = await owner.req('POST', '/api/admin/instructors', {
+    email: '[email protected]', display_name: 'Drill Admin', password: 'DrillAdminPass!2026', role: 'admin',
+  });
+  console.log('  admin instructor created:', admin.instructor_id);
+
+  const cohort = await owner.req('POST', '/api/admin/cohorts', { cohort_id: 'drill-cohort-01', name: 'Restore Drill Cohort' });
+  console.log('  cohort created');
+
+  const students = [];
+  for (const name of ['Drill Alice', 'Drill Bob', 'Drill Carlos']) {
+    const s = await owner.req('POST', '/api/admin/students', { display_name: name, cohort_id: 'drill-cohort-01' });
+    students.push(s.student_id);
+  }
+  console.log('  3 students rostered and enrolled:', students.join(', '));
+
+  const assessment = await owner.req('POST', '/api/admin/assessments', {
+    id: 'drill-exam-01', type: 'exam', title: 'Restore Drill Exam', duration_minutes: 30, hint_policy: 'disabled',
+    pool_spec: { required: ['pod-basics-mcq'], random: [] },
+  });
+  console.log('  assessment definition created:', assessment.assessment_definition_id || 'drill-exam-01');
+
+  // Direct SQL for progress/attempts/achievements — see file header for why.
+  const pool = new Pool({ connectionString: DB_URL });
+  for (const studentId of students) {
+    // environment_id is a real FK to the Provisioner's environments table (nullable) — left
+    // NULL here rather than faking a row that doesn't exist, same as how a genuinely old
+    // attempt whose environment has long since been cleaned up would look in real data.
+    await pool.query(
+      `INSERT INTO scenario_attempts (student_id, scenario_id, status, result, started_at, completed_at, validation_attempt_count, validation_pass_count)
+       VALUES ($1, 'pod-basics-mcq', 'completed', 'passed', now() - interval '1 hour', now(), 1, 1)`,
+      [studentId]
+    );
+    await pool.query(
+      `INSERT INTO achievements (student_id, achievement_type, stage_id, achieved_at) VALUES ($1, 'stage_completed', 'foundation', now())`,
+      [studentId]
+    );
+  }
+  await pool.end();
+  console.log('  progress/attempts/achievements seeded for all 3 students');
+
+  console.log('\nSeed complete.');
+}
+
+main().catch((e) => { console.error('SEED FAILED:', e.message); process.exit(1); });

infra/backup/test-backup-restore.sh

@@ -0,0 +1,133 @@
+#!/usr/bin/env bash
+# Local test (Phase 4 §29): a REAL end-to-end backup/restore cycle against disposable local
+# Postgres containers — not a mock. Applies the actual Platform API migration runner (the same
+# code path production uses, platform/api/src/db.js) so the schema under test is genuinely the
+# real one, inserts sample data, backs it up, restores into a SECOND empty Postgres, and
+# verifies the result. Proves the backup/restore MECHANISM works; does NOT prove anything about
+# the real off-host target (Storage Box/S3) — that needs real credentials, marked as
+# hardware/credential-dependent below.
+set -euo pipefail
+cd "$(dirname "$0")/../.."   # repo root
+
+SRC_CONTAINER=rootnode-backup-test-src
+DST_CONTAINER=rootnode-backup-test-dst
+SRC_PORT=55432
+DST_PORT=55433
+TESTDIR="$(mktemp -d)"
+export BACKUP_ENCRYPTION_PASSPHRASE="test-passphrase-not-for-production"
+
+cleanup() {
+  docker rm -f "$SRC_CONTAINER" "$DST_CONTAINER" >/dev/null 2>&1 || true
+  rm -rf "$TESTDIR"
+}
+trap cleanup EXIT
+
+# backup-postgres.sh/restore-postgres.sh assume real postgresql-client + gpg binaries — true on
+# a real deployment host (the `backup` Ansible role installs postgresql-client there). This dev
+# machine may not have psql/pg_dump installed at all, so for THIS TEST ONLY we shim them as
+# thin wrappers around `docker exec` into the two disposable test containers — the scripts
+# under test are otherwise completely unmodified from what production actually runs.
+if ! command -v psql >/dev/null 2>&1 || ! command -v pg_dump >/dev/null 2>&1; then
+  echo "==> psql/pg_dump not found locally — shimming both via docker exec for this test run"
+  SHIM_BIN="$TESTDIR/bin"
+  mkdir -p "$SHIM_BIN"
+  # Both wrappers pick a container from the target port (explicit -p, or embedded in a
+  # postgres://...:PORT/... URL), then re-exec psql/pg_dump INSIDE that container against ITS
+  # OWN localhost:5432 — the host-side port mapping (55432/55433) doesn't exist inside the
+  # container's network namespace, so the connection target itself must be replaced, not just
+  # which container runs the command.
+  cat > "$SHIM_BIN/psql" <<'EOF'
+#!/usr/bin/env bash
+args=(); port=""; want_port=0; skip_next=0
+for a in "$@"; do
+  if [ "$skip_next" = "1" ]; then
+    [ "$want_port" = "1" ] && port="$a"
+    skip_next=0; want_port=0; continue
+  fi
+  if [ "$a" = "-p" ]; then skip_next=1; want_port=1; continue; fi
+  if [ "$a" = "-h" ]; then skip_next=1; want_port=0; continue; fi
+  if [[ "$a" =~ ^postgres://[^:]+:[^@]+@[^:]+:([0-9]+)/ ]]; then
+    port="${BASH_REMATCH[1]}"
+    continue   # drop the URL — reconnect locally inside the container instead
+  fi
+  args+=("$a")
+done
+case "$port" in
+  55433) c="rootnode-backup-test-dst" ;;
+  *) c="rootnode-backup-test-src" ;;
+esac
+# Only forward local stdin (-i) for the one call that actually reads SQL from it (no -c/-tAc
+# in its args, i.e. `psql <target>` fed by a pipe) — every -c/-tAc call is self-contained and
+# must NOT hold docker exec's stdin open, or it silently steals bytes meant for a later `read`
+# elsewhere in the calling script (exactly what broke restore-postgres.sh's confirmation prompt
+# the first time this shim was written).
+needs_stdin=1
+for a in "${args[@]}"; do [ "$a" = "-c" ] || [ "$a" = "-tAc" ] && needs_stdin=0; done
+if [ "$needs_stdin" = "1" ]; then
+  exec docker exec -e PGPASSWORD=postgres -i "$c" psql -h localhost -U postgres "${args[@]}"
+else
+  exec docker exec -e PGPASSWORD=postgres "$c" psql -h localhost -U postgres "${args[@]}"
+fi
+EOF
+  cat > "$SHIM_BIN/pg_dump" <<'EOF'
+#!/usr/bin/env bash
+args=()
+for a in "$@"; do
+  if [[ "$a" =~ ^postgres:// ]]; then continue; fi   # source is always the src container here
+  args+=("$a")
+done
+exec docker exec -e PGPASSWORD=postgres -i rootnode-backup-test-src pg_dump -h localhost -U postgres "${args[@]}"
+EOF
+  chmod +x "$SHIM_BIN/psql" "$SHIM_BIN/pg_dump"
+  export PATH="$SHIM_BIN:$PATH"
+fi
+
+echo "==> Starting source Postgres (the one we'll back up)"
+docker run -d --name "$SRC_CONTAINER" -e POSTGRES_PASSWORD=postgres -p "$SRC_PORT:5432" postgres:16-alpine >/dev/null
+echo "==> Starting destination Postgres (the one we'll restore into)"
+docker run -d --name "$DST_CONTAINER" -e POSTGRES_PASSWORD=postgres -p "$DST_PORT:5432" postgres:16-alpine >/dev/null
+
+echo "==> Waiting for both to accept connections"
+for port in "$SRC_PORT" "$DST_PORT"; do
+  for i in $(seq 1 30); do
+    PGPASSWORD=postgres psql -h localhost -p "$port" -U postgres -c "SELECT 1" >/dev/null 2>&1 && break
+    sleep 1
+  done
+done
+
+SRC_DB_URL="postgres://postgres:postgres@localhost:$SRC_PORT/postgres"
+DST_DB_URL="postgres://postgres:postgres@localhost:$DST_PORT/postgres"
+
+echo "==> Applying the REAL Provisioner migrations first (Platform API's scenario_attempts has a real FK to Provisioner's environments table — documented cross-service ordering dependency, see docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md)"
+( cd platform/provisioner && [ -d node_modules ] || npm install --no-audit --no-fund >/dev/null
+  PLATFORM_DB_URL="$SRC_DB_URL" node -e "require('./src/db').runMigrations().then((m)=>console.log('Applied:', m.join(', ')))" )
+
+echo "==> Applying the REAL Platform API migrations against the source"
+( cd platform/api && [ -d node_modules ] || npm install --no-audit --no-fund >/dev/null
+  PLATFORM_DB_URL="$SRC_DB_URL" node -e "require('./src/db').runMigrations().then((m)=>console.log('Applied:', m.join(', ')))" )
+
+echo "==> Inserting sample data (a student, so the backup has something real to carry)"
+PGPASSWORD=postgres psql -h localhost -p "$SRC_PORT" -U postgres -c \
+  "INSERT INTO students (student_id, display_name) VALUES ('student-backup-test', 'Backup Test Student');"
+
+echo "==> Running backup-postgres.sh against the source"
+BACKUP_DIR="$TESTDIR/backups" PLATFORM_DB_URL="$SRC_DB_URL" ./infra/backup/backup-postgres.sh
+
+ENC_FILE=$(ls "$TESTDIR"/backups/*.sql.gz.gpg | head -1)
+[ -f "$ENC_FILE" ] || { echo "FAIL: no encrypted backup file produced"; exit 1; }
+echo "==> Backup produced: $ENC_FILE"
+
+echo "==> Restoring into the destination (non-interactive: piping the confirmation)"
+echo "RESTORE" | BACKUP_ENCRYPTION_PASSPHRASE="$BACKUP_ENCRYPTION_PASSPHRASE" ./infra/backup/restore-postgres.sh "$ENC_FILE" "$DST_DB_URL"
+
+echo "==> Verifying the restore"
+./infra/backup/verify-restore.sh "$DST_DB_URL"
+
+echo "==> Confirming the sample student actually round-tripped"
+RESTORED_NAME=$(PGPASSWORD=postgres psql -h localhost -p "$DST_PORT" -U postgres -tAc \
+  "SELECT display_name FROM students WHERE student_id = 'student-backup-test'")
+[ "$RESTORED_NAME" = "Backup Test Student" ] || { echo "FAIL: restored student data doesn't match"; exit 1; }
+
+echo
+echo "OK: full backup -> encrypt -> restore -> verify cycle succeeded against real local Postgres."
+echo "NOT TESTED HERE (needs real credentials/hardware): off-host copy to a real Storage Box/S3 target."

infra/backup/test-destructive-restore-drill.sh

@@ -0,0 +1,149 @@
+#!/usr/bin/env bash
+# Phase 4.1 §20 — the DESTRUCTIVE disaster-recovery drill, one level up from
+# test-backup-restore.sh's mechanism-only test: seeds realistic academy data through the REAL
+# Platform API (owner, admin instructor, cohort, 3 students, progress, attempts, an assessment
+# definition, achievements), backs it up, DESTROYS the disposable Postgres storage entirely
+# (docker rm — not just "restore into a second container," the actual source is gone),
+# recreates Postgres from nothing, restores, and boots the REAL Platform API again against the
+# restored database to prove the whole stack — not just the SQL — comes back.
+#
+# Never touches the developer's real persistent database — every Postgres here is a disposable
+# container on a throwaway port, created and destroyed entirely within this script.
+set -euo pipefail
+cd "$(dirname "$0")/../.."   # repo root
+
+CONTAINER=rootnode-drill-postgres
+PORT=55434
+API_PORT=4390
+TESTDIR="$(mktemp -d)"
+# curriculum.js defaults to platform/api/scenarios/{data,bundles} — only present inside the
+# Docker image build (COPY scenarios/ ./scenarios/). Running src/server.js directly from
+# source (as this drill does, to avoid needing a full container build) needs these pointed at
+# the repo's real scenario content instead.
+export SCENARIOS_DIR="$PWD/scenarios/data"
+export BUNDLES_DIR="$PWD/scenarios/bundles"
+export CURRICULUM_PATH_FILE="$PWD/scenarios/rootnode-path.json"
+export BACKUP_ENCRYPTION_PASSPHRASE="drill-passphrase-not-for-production"
+export PROVISIONER_SHARED_SECRET="drill-shared-secret-not-for-production"
+
+API_PID=""
+cleanup() {
+  [ -n "$API_PID" ] && kill "$API_PID" 2>/dev/null || true
+  docker rm -f "$CONTAINER" >/dev/null 2>&1 || true
+  rm -rf "$TESTDIR"
+}
+trap cleanup EXIT
+
+if ! command -v psql >/dev/null 2>&1 || ! command -v pg_dump >/dev/null 2>&1; then
+  echo "==> psql/pg_dump not found locally — reusing the same docker-exec shim as test-backup-restore.sh"
+  SHIM_BIN="$TESTDIR/bin"; mkdir -p "$SHIM_BIN"
+  # Same shim as test-backup-restore.sh (see that file's comment for the full "why") — the URL
+  # is DROPPED, not kept, since the host-side port it names doesn't exist inside the
+  # container's own network namespace; -tAc counts as a self-contained call (needs no stdin)
+  # the same way -c does, or the admin-verification queries below would hang waiting on stdin.
+  cat > "$SHIM_BIN/psql" <<'EOF'
+#!/usr/bin/env bash
+args=(); skip_next=0
+for a in "$@"; do
+  if [ "$skip_next" = "1" ]; then skip_next=0; continue; fi
+  if [ "$a" = "-p" ] || [ "$a" = "-h" ]; then skip_next=1; continue; fi
+  if [[ "$a" =~ ^postgres://[^:]+:[^@]+@[^:]+:([0-9]+)/ ]]; then continue; fi   # drop the URL — reconnect locally inside the container instead
+  args+=("$a")
+done
+container="rootnode-drill-postgres"
+needs_stdin=1
+for a in "${args[@]}"; do { [ "$a" = "-c" ] || [ "$a" = "-tAc" ]; } && needs_stdin=0; done
+if [ "$needs_stdin" = "1" ]; then
+  exec docker exec -e PGPASSWORD=postgres -i "$container" psql -h localhost -U postgres "${args[@]}"
+else
+  exec docker exec -e PGPASSWORD=postgres "$container" psql -h localhost -U postgres "${args[@]}"
+fi
+EOF
+  cat > "$SHIM_BIN/pg_dump" <<'EOF'
+#!/usr/bin/env bash
+args=()
+for a in "$@"; do
+  if [[ "$a" =~ ^postgres://[^:]+:[^@]+@[^:]+:([0-9]+)/(.*)$ ]]; then args+=("postgres://postgres:postgres@localhost:5432/${BASH_REMATCH[2]}"); continue; fi
+  args+=("$a")
+done
+exec docker exec -e PGPASSWORD=postgres rootnode-drill-postgres pg_dump "${args[@]}"
+EOF
+  chmod +x "$SHIM_BIN/psql" "$SHIM_BIN/pg_dump"
+  export PATH="$SHIM_BIN:$PATH"
+fi
+
+DB_URL="postgres://postgres:postgres@localhost:$PORT/postgres"
+
+echo "==> [1/9] Starting disposable Postgres"
+docker run -d --name "$CONTAINER" -e POSTGRES_PASSWORD=postgres -p "$PORT:5432" postgres:16-alpine >/dev/null
+for i in $(seq 1 30); do PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -c "SELECT 1" >/dev/null 2>&1 && break; sleep 1; done
+
+echo "==> [2/9] Applying real Provisioner + Platform API migrations"
+( cd platform/provisioner && [ -d node_modules ] || npm install --no-audit --no-fund >/dev/null
+  PLATFORM_DB_URL="$DB_URL" node -e "require('./src/db').runMigrations().then((m)=>console.log('  Provisioner:', m.join(', ')))" )
+( cd platform/api && [ -d node_modules ] || npm install --no-audit --no-fund >/dev/null
+  PLATFORM_DB_URL="$DB_URL" node -e "require('./src/db').runMigrations().then((m)=>console.log('  Platform API:', m.join(', ')))" )
+
+echo "==> [3/9] Bootstrapping the drill owner and starting a REAL Platform API"
+( cd platform/api && PLATFORM_DB_URL="$DB_URL" node bootstrap-owner.js [email protected] "Drill Owner" 'DrillOwnerPass!2026' )
+( cd platform/api && PLATFORM_DB_URL="$DB_URL" PROVISIONER_URL="http://localhost:1" PLATFORM_API_PORT="$API_PORT" node src/server.js > "$TESTDIR/api.log" 2>&1 & echo $! > "$TESTDIR/api.pid" )
+API_PID=$(cat "$TESTDIR/api.pid")
+for i in $(seq 1 30); do curl -sf "http://localhost:$API_PORT/healthz" >/dev/null 2>&1 && break; sleep 1; done
+curl -sf "http://localhost:$API_PORT/healthz" >/dev/null || { echo "FAIL: Platform API never came up"; cat "$TESTDIR/api.log"; exit 1; }
+echo "  Platform API up on :$API_PORT"
+
+echo "==> [4/9] Seeding realistic academy data through the real API (owner/admin/cohort/3 students/progress/attempts/assessment/achievements)"
+( cd infra/backup && PLATFORM_API_URL="http://localhost:$API_PORT" PLATFORM_DB_URL="$DB_URL" node seed-realistic-data.js )
+
+echo "==> [5/9] Backing up (real backup-postgres.sh)"
+BACKUP_DIR="$TESTDIR/backups" PLATFORM_DB_URL="$DB_URL" ./infra/backup/backup-postgres.sh
+ENC_FILE=$(ls "$TESTDIR"/backups/*.sql.gz.gpg | head -1)
+[ -f "$ENC_FILE" ] || { echo "FAIL: no encrypted backup produced"; exit 1; }
+
+echo "==> [6/9] DESTROYING the Postgres storage entirely (this is the destructive part — not restoring into a second container, the original is gone)"
+kill "$API_PID" 2>/dev/null || true; API_PID=""
+docker rm -f "$CONTAINER" >/dev/null
+echo "  source container + its storage are gone"
+
+echo "==> [7/9] Recreating Postgres from nothing"
+docker run -d --name "$CONTAINER" -e POSTGRES_PASSWORD=postgres -p "$PORT:5432" postgres:16-alpine >/dev/null
+for i in $(seq 1 30); do PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -c "SELECT 1" >/dev/null 2>&1 && break; sleep 1; done
+
+echo "==> [8/9] Restoring from the encrypted backup (schema + data both come from the dump — no separate migration step)"
+echo "RESTORE" | BACKUP_ENCRYPTION_PASSPHRASE="$BACKUP_ENCRYPTION_PASSPHRASE" ./infra/backup/restore-postgres.sh "$ENC_FILE" "$DB_URL"
+./infra/backup/verify-restore.sh "$DB_URL"
+
+echo "==> [9/9] Booting the REAL Platform API again against the RESTORED database"
+( cd platform/api && PLATFORM_DB_URL="$DB_URL" PROVISIONER_URL="http://localhost:1" PLATFORM_API_PORT="$API_PORT" node src/server.js > "$TESTDIR/api2.log" 2>&1 & echo $! > "$TESTDIR/api2.pid" )
+API_PID=$(cat "$TESTDIR/api2.pid")
+for i in $(seq 1 30); do curl -sf "http://localhost:$API_PORT/healthz" >/dev/null 2>&1 && break; sleep 1; done
+curl -sf "http://localhost:$API_PORT/healthz" >/dev/null || { echo "FAIL: Platform API did not come back up against the restored DB"; cat "$TESTDIR/api2.log"; exit 1; }
+echo "  Platform API is back up, against the RESTORED database — no fresh migration run needed"
+
+echo "==> Verifying accounts/RBAC/cohorts/students/progress/attempts/achievements survived"
+FAIL=0
+check() { local desc="$1" want="$2" got="$3"; if [ "$want" = "$got" ]; then echo "  ok — $desc"; else echo "  FAIL — $desc (want $want, got $got)"; FAIL=1; fi; }
+
+check "owner account (role) restored" "owner" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT role FROM instructors WHERE email='[email protected]'" | tr -d '[:space:]')"
+check "admin instructor restored" "admin" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT role FROM instructors WHERE email='[email protected]'" | tr -d '[:space:]')"
+check "cohort restored" "1" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM cohorts WHERE cohort_id='drill-cohort-01'" | tr -d '[:space:]')"
+check "3 students restored" "3" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM cohort_enrollments WHERE cohort_id='drill-cohort-01'" | tr -d '[:space:]')"
+check "3 completed attempts restored" "3" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM scenario_attempts WHERE result='passed' AND scenario_id='pod-basics-mcq'" | tr -d '[:space:]')"
+check "3 achievements restored" "3" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM achievements WHERE stage_id='foundation'" | tr -d '[:space:]')"
+check "assessment definition restored" "1" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM assessment_definitions WHERE assessment_definition_id='drill-exam-01'" | tr -d '[:space:]')"
+
+echo "==> Verifying RBAC still works against the restored DB (real login through the real API)"
+LOGIN_STATUS=$(curl -s -o /dev/null -w '%{http_code}' -X POST "http://localhost:$API_PORT/api/auth/instructor/login" \
+  -H 'Content-Type: application/json' -d '{"email":"[email protected]","password":"DrillAdminPass!2026"}')
+check "restored admin instructor can log in" "200" "$LOGIN_STATUS"
+
+if [ "$FAIL" = "1" ]; then
+  echo; echo "DRILL FAILED — see above."; exit 1
+fi
+
+echo
+echo "OK: full destructive restore drill passed — owner/admin/cohort/3 students/progress/attempts/"
+echo "assessment/achievements all survived a real Postgres storage loss + recreation + restore, and"
+echo "the real Platform API booted and authenticated against the restored database with no fresh"
+echo "migration run required."
+echo "NOT TESTED HERE (needs real credentials/hardware): the actual off-host Storage Box/S3 copy."

infra/backup/verify-restore.sh

@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+# Sanity-checks a restored database — confirms the expected Platform API schema landed and has
+# data, not just that psql exited 0 (a restore that silently imported zero rows would still
+# "succeed" by that measure). Used both by restore-postgres.sh's own suggestion and by
+# test-backup-restore.sh's automated local test.
+set -euo pipefail
+DB_URL="${1:?usage: verify-restore.sh <db-url>}"
+
+EXPECTED_TABLES=(students cohorts instructors sessions scenario_attempts assessment_definitions achievements audit_events)
+FAIL=0
+
+for t in "${EXPECTED_TABLES[@]}"; do
+  exists=$(psql "$DB_URL" -tAc "SELECT to_regclass('public.$t') IS NOT NULL")
+  if [ "$exists" != "t" ]; then
+    echo "MISSING TABLE: $t" >&2
+    FAIL=1
+  fi
+done
+
+if [ "$FAIL" = "1" ]; then
+  echo "FAIL: restored database is missing expected tables" >&2
+  exit 1
+fi
+
+MIGRATIONS_APPLIED=$(psql "$DB_URL" -tAc "SELECT count(*) FROM schema_migrations" 2>/dev/null || echo 0)
+echo "OK: all expected tables present. $MIGRATIONS_APPLIED migrations recorded in schema_migrations."

infra/ci/gitea-workflows/build-and-test.yml

@@ -0,0 +1,162 @@
+# Gitea Actions workflow — build/test on every push to main. Gitea Actions uses the same
+# workflow syntax as GitHub Actions (a compatible runner), so this is portable if the canonical
+# remote ever needs to move CI providers, without being GitHub Actions itself or requiring a
+# paid CI provider. Runs against the canonical Gitea repo only — see docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md
+# §18 for why this never touches the GitHub kubekosh fork.
+#
+# Deploy this file to .gitea/workflows/build-and-test.yml at the repo root to activate it (kept
+# under infra/ci/ in this PR so it's reviewed alongside the rest of the infra package first,
+# per the same "build now, activate deliberately" posture as everything else server-dependent
+# in this phase — enabling Gitea Actions on the instance is itself a one-time admin step that
+# may not be done yet).
+
+name: build-and-test
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  regression:
+    runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: postgres:16-alpine
+        env:
+          POSTGRES_PASSWORD: postgres
+        ports: ["5432:5432"]
+        options: >-
+          --health-cmd "pg_isready -U postgres"
+          --health-interval 10s --health-timeout 5s --health-retries 10
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Node
+        uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+
+      - name: Install dependencies (all first-party services)
+        run: |
+          for d in backend frontend platform/api platform/gateway platform/provisioner platform/lab-router; do
+            (cd "$d" && npm ci)
+          done
+
+      - name: Syntax-check every service
+        run: |
+          for d in backend frontend/src platform/api platform/gateway platform/provisioner platform/lab-router; do
+            find "$d" -name '*.js' -not -path '*/node_modules/*' -exec node --check {} \;
+          done
+
+      - name: Lab Host Router unit tests (Phase 4.1 — fast, no Docker daemon needed)
+        run: (cd platform/lab-router && npm test)
+
+      - name: Build frontend
+        run: (cd frontend && npm run build)
+
+      - name: Provisioner migrations against the service Postgres
+        run: |
+          cd platform/provisioner
+          PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5432/postgres \
+            node -e "require('./src/db').runMigrations().then(m=>console.log('Applied:',m.join(', ')))"
+
+      - name: Platform API migrations (depends on Provisioner's, same order as production)
+        run: |
+          cd platform/api
+          PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5432/postgres \
+            node -e "require('./src/db').runMigrations().then(m=>console.log('Applied:',m.join(', ')))"
+
+      - name: Validate infra configs (Phase 4 local tests — no hardware needed)
+        run: |
+          ./infra/scripts/validate-compose.sh
+          ./infra/firewall/validate-firewall.sh
+
+      # The real k3s-boot-dependent e2e suites (platform/test/e2e-phase*.js) need a full Docker
+      # Compose stack with --privileged Lab containers, which a shared CI runner typically
+      # can't grant safely — they stay a LOCAL/staging pre-promotion step
+      # (infra/scripts/staging-smoke-test.sh), not this job. See docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §18.
+
+  # Phase 4.1 §24 — free/open-source security scanning, no paid tooling, no signup required for
+  # any of these. Severity policy (docs/SECURITY_SCANNING.md): CRITICAL/HIGH findings fail the
+  # build; MEDIUM/LOW are reported but never block a merge — a build that's permanently red over
+  # an unrelated low-severity transitive dependency finding trains everyone to ignore red builds.
+  security-scan:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+
+      - name: npm audit (dependency vulnerabilities) — fails only on high/critical
+        run: |
+          FAIL=0
+          for d in backend frontend platform/api platform/gateway platform/provisioner platform/lab-router; do
+            echo "== $d =="
+            (cd "$d" && npm audit --audit-level=high) || FAIL=1
+          done
+          exit $FAIL
+
+      - name: Secret scanning (gitleaks — free, no account needed)
+        run: docker run --rm -v "$PWD:/repo" zricethezav/gitleaks:latest detect --source=/repo --no-git -v
+
+      - name: Dockerfile linting (hadolint — free)
+        run: |
+          FAIL=0
+          for f in Dockerfile platform/api/Dockerfile platform/gateway/Dockerfile platform/provisioner/Dockerfile platform/egress-proxy/Dockerfile platform/lab-router/Dockerfile; do
+            echo "== $f =="
+            docker run --rm -i hadolint/hadolint < "$f" || FAIL=1
+          done
+          exit $FAIL
+
+      # -S error, not warning — matches docs/SECURITY_SCANNING.md's severity policy. Real run
+      # against this tree at -S warning surfaces only style nits (unused loop vars, a missing
+      # `cd ... || exit` guard) on scripts that are otherwise correct; failing the build on
+      # those trains everyone to ignore the check.
+      - name: Shell linting (shellcheck — free, standard)
+        run: |
+          sudo apt-get update -qq && sudo apt-get install -y -qq shellcheck
+          find infra -name '*.sh' -exec shellcheck -S error {} \+
+
+      - name: Container image scanning (trivy — free, no account needed)
+        run: |
+          docker build -q -t rootnode-scan/platform-api -f platform/api/Dockerfile .
+          docker run --rm -v /var/run/docker.sock:/var/run/docker.sock aquasec/trivy:latest image \
+            --severity HIGH,CRITICAL --exit-code 1 rootnode-scan/platform-api
+
+  build-images:
+    needs: [regression, security-scan]
+    if: github.ref == 'refs/heads/main'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Build and tag images
+        run: ./infra/scripts/ci/tag-and-push.sh
+        env:
+          REGISTRY: ${{ vars.REGISTRY }}
+          REGISTRY_USER: ${{ secrets.REGISTRY_USER }}
+          REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
+
+  deploy-staging:
+    needs: build-images
+    if: github.ref == 'refs/heads/main'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Deploy to staging (automatic)
+        run: |
+          cd infra/ansible
+          ansible-playbook site.yml -i inventory/staging.ini \
+            -e "image_tag=staging-${GITHUB_SHA::12}" \
+            --vault-password-file <(echo "$VAULT_PASSWORD")
+        env:
+          VAULT_PASSWORD: ${{ secrets.ANSIBLE_VAULT_PASSWORD }}
+
+      - name: Staging smoke test
+        run: ./infra/scripts/staging-smoke-test.sh
+
+  # Production is NEVER automatic — a human runs infra/scripts/deploy-production.sh after
+  # reviewing the staging smoke test result, per the explicit "manual approval step" preference.
+  # No job here does that; see docs/RUNBOOK_NORMAL_DEPLOY.md for the actual command.

infra/docker/IMAGE_STRATEGY.md

@@ -0,0 +1,50 @@
+# Container image strategy
+
+## Images
+
+| Image | Built from | Registry path |
+|---|---|---|
+| `rootnode/platform-api` | `platform/api/Dockerfile` | `{{ registry }}/rootnode/platform-api` |
+| `rootnode/gateway` | `platform/gateway/Dockerfile` | `{{ registry }}/rootnode/gateway` |
+| `rootnode/provisioner` | `platform/provisioner/Dockerfile` | `{{ registry }}/rootnode/provisioner` |
+| `rootnode/egress-proxy` | `platform/egress-proxy/Dockerfile` | `{{ registry }}/rootnode/egress-proxy` |
+| `rootnode/lab-router` | `platform/lab-router/Dockerfile` (Phase 4.1) | `{{ registry }}/rootnode/lab-router` |
+| `rootnode/lab` | root `Dockerfile` (the student Lab image) | `{{ registry }}/rootnode/lab` |
+
+## Tagging
+
+**Never `:latest` in staging or production.** Every build is tagged with:
+
+- the full git SHA (`sha-<40 hex>`) — always, every build, the ground truth
+- `staging-<short-sha>` on every push to `main` that passes CI (moving pointer, staging only)
+- a semantic tag (`vX.Y.Z`) only on an explicit, human-triggered production release
+
+`infra/scripts/ci/tag-and-push.sh` implements this — see §18 in
+`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`.
+
+Deployment configs (`infra/environments/*/​*.env.template`, Ansible `group_vars`) reference the
+tag as a variable (`image_tag`), never a hardcoded value baked into a compose file — rolling
+back is changing one variable and re-running the `platform`/`lab-host` Ansible role, not editing
+a compose file by hand (see `docs/RUNBOOK_ROLLBACK.md`).
+
+## Registry
+
+No paid registry required. Two supported options, chosen at deploy time via `registry` in
+`infra/ansible/group_vars/production.yml`:
+
+1. **Gitea's own built-in container registry** (`git.synthossystems.com`) — zero new
+   infrastructure, already authenticated the same way the repo itself is. Preferred default.
+2. **A self-hosted registry** (e.g. a `registry:2` container on rn-platform-01 itself, or
+   elsewhere in the homelab) — a documented fallback if the Gitea registry isn't enabled on
+   this instance.
+
+Either way: private registry, credentials pulled the same way any other production secret is
+(`docs/PRODUCTION_DEPLOYMENT.md` §10), never a public Docker Hub push of anything beyond what's
+already public (this project doesn't publish anything to Docker Hub in Phase 4).
+
+## Rollback
+
+Every image is content-addressable by SHA tag, so "roll back" is always available as long as
+the previous tag hasn't been garbage-collected from the registry — see
+`docs/RUNBOOK_ROLLBACK.md` for the exact procedure (re-point `image_tag`, re-run the Ansible
+role, verify).

infra/docker/lab-host/docker-compose.production.yml

@@ -0,0 +1,91 @@
+# Production compose for rn-lab-host-01. Deployed by infra/ansible/roles/lab-host/ to
+# /opt/rootnode/lab-host/docker-compose.yml, run as a systemd unit
+# (rootnode-lab-host.service.j2). Runs the shared egress proxy every student Lab's outbound
+# traffic passes through (platform/egress-proxy/, unchanged since Phase 1) and, as of Phase
+# 4.1, the Lab Host Router (platform/lab-router/) that gives the Gateway on rn-platform-01 a
+# way to reach a running Lab without ever touching a Docker-internal address — see
+# docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3. Student Lab containers themselves are NOT
+# part of this compose file — they're created/destroyed individually by the Provisioner over
+# the remote Docker TLS endpoint (dockerd itself, not compose), same mechanism as dev, just
+# remote; the Router finds them by label, purely from local Docker state (see its own README).
+
+networks:
+  rn-egress:
+    driver: bridge
+
+services:
+  egress-proxy:
+    image: "{{ registry }}/rootnode/egress-proxy:{{ image_tag }}"
+    restart: unless-stopped
+    ports:
+      - "3128:3128"   # reachable from every per-Lab Docker network on this host — restricted
+                       # to those networks by the Provisioner's per-environment DOCKER-USER
+                       # chains, not by a published-port firewall rule (there is no sensible
+                       # single source IP to restrict this to; the egress proxy's OWN allowlist
+                       # is the actual control, see platform/egress-proxy/allowlist.txt)
+    networks: [rn-egress]
+
+  lab-router:
+    image: "{{ registry }}/rootnode/lab-router:{{ image_tag }}"
+    restart: unless-stopped
+    # Needs the host's own routing table to reach whichever per-Lab bridge network
+    # (rootnode-lab-net-<id>) the Provisioner created for a given container — those networks
+    # aren't known ahead of time and aren't attachable one-by-one, the same reason egress-proxy
+    # would need this too if it initiated connections outward instead of just accepting them.
+    # Restricted to platform_ip on port {{ lab_router_port | default('8443') }} at the host
+    # firewall (infra/firewall/rn-lab-host-01.rules) — this is the ONLY thing that makes host
+    # networking here safe, same defense-in-depth pattern as the Docker TLS port.
+    network_mode: host
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /etc/rootnode/lab-router-tls:/tls:ro
+    environment:
+      LAB_ROUTER_PORT: "{{ lab_router_port | default('8443') }}"
+      LAB_ROUTER_TLS_CA: /tls/ca.pem
+      LAB_ROUTER_TLS_CERT: /tls/router-server-cert.pem
+      LAB_ROUTER_TLS_KEY: /tls/router-server-key.pem
+
+  node-exporter:
+    image: prom/node-exporter:latest
+    restart: unless-stopped
+    network_mode: host
+    pid: host
+    volumes:
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/rootfs:ro
+    command:
+      - "--path.procfs=/host/proc"
+      - "--path.sysfs=/host/sys"
+      - "--path.rootfs=/rootfs"
+      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
+
+  cadvisor:
+    image: gcr.io/cadvisor/cadvisor:latest
+    restart: unless-stopped
+    privileged: true
+    devices:
+      - /dev/kmsg
+    volumes:
+      - /:/rootfs:ro
+      - /var/run:/var/run:ro
+      - /sys:/sys:ro
+      - /var/lib/docker/:/var/lib/docker:ro
+    ports:
+      - "8080:8080"   # scraped remotely by Prometheus on rn-platform-01 — restricted to that
+                       # one source IP by the host firewall (rn-lab-host-01.rules), not by
+                       # binding to localhost, since Prometheus isn't local on this host.
+
+  promtail:
+    image: grafana/promtail:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/lab-host/promtail-config.yml:/etc/promtail/config.yml:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /var/log/journal:/var/log/journal:ro
+      - /etc/machine-id:/etc/machine-id:ro
+    command: ["-config.file=/etc/promtail/config.yml"]
+    # host networking so promtail can reach Loki on rn-platform-01 over the network — this
+    # traffic is OUTBOUND from rn-lab-host-01 (log push, client-initiated), which the host
+    # firewall's default-ACCEPT OUTPUT chain already allows; nothing new needed there.
+    network_mode: host

infra/docker/platform/Caddyfile.j2

@@ -0,0 +1,38 @@
+# Caddy 2 — single stable RootNode Academy origin, automatic HTTPS (Let's Encrypt) unless
+# Cloudflare origin certs are used instead (see docs/PRODUCTION_NETWORK_SECURITY.md §8 for the
+# Cloudflare-vs-direct-LE decision). Rendered by the `platform` Ansible role from
+# infra/ansible/group_vars/production.yml's `academy_domain`.
+#
+# The student/instructor path proxies to the Gateway (platform/gateway) — the sole
+# student-facing entrypoint, unchanged architectural rule from Phase 2A onward. Nothing here
+# reaches the Provisioner, Postgres, or Docker TLS directly; those have no public route at all.
+# Grafana is reachable at /grafana/ but restricted to the management CIDR even though it has
+# its own login — internal observability tooling shouldn't be internet-facing at all, same
+# posture as everything else that isn't the student/instructor product surface.
+
+{{ academy_domain }} {
+	encode gzip zstd
+
+	handle_path /grafana/* {
+		@notmanagement not remote_ip {{ management_cidr }}
+		respond @notmanagement 403
+		reverse_proxy grafana:3000
+	}
+
+	# WebSocket terminal traffic (shell-ws) needs the connection upgraded, not buffered —
+	# Caddy's reverse_proxy handles this automatically for Upgrade: websocket requests, no
+	# special directive needed, but flush_interval -1 avoids buffering SSE/long-poll-shaped
+	# responses too. `gateway` is the Docker Compose service name (rn-platform-internal
+	# network), not localhost — Caddy no longer shares host networking with the Gateway in
+	# production, unlike the dev compose file.
+	reverse_proxy gateway:4200 {
+		flush_interval -1
+	}
+
+	log {
+		output file /var/log/caddy/access.log {
+			roll_size 100mb
+			roll_keep 7
+		}
+	}
+}

infra/docker/platform/docker-compose.production.yml

@@ -0,0 +1,221 @@
+# Production compose for rn-platform-01. Deployed by infra/ansible/roles/platform/ to
+# /opt/rootnode/platform/docker-compose.yml, run as a systemd-managed `docker compose` unit
+# (infra/ansible/roles/platform/templates/rootnode-platform.service.j2).
+#
+# Differs from platform/docker-compose.dev.yml in the ways that actually matter for
+# production, and nothing else:
+#   - images are pulled from a registry by DIGEST-PINNED TAG (never :latest, never built
+#     in-place) — see infra/docker/IMAGE_STRATEGY.md
+#   - Postgres data and backups live on a bind-mounted ZFS dataset, not an anonymous volume
+#   - a reverse proxy (Caddy) terminates TLS and is the only container with a port published to
+#     the internet; a few other services publish loopback-only or single-source-IP-restricted
+#     ports for specific operational needs (Postgres for the host's own backup cron, Loki for
+#     rn-lab-host-01's log shipper) — see each service's own comment
+#   - no `network_mode: host` on gateway/provisioner — production uses a named bridge network,
+#     since rn-platform-01 no longer needs to reach ephemeral container IPs directly (those live
+#     on rn-lab-host-01, reached over the network, not shared Docker networking)
+#   - the api/gateway/provisioner containers still don't reach the internet except through
+#     what's explicitly allowed by the host firewall (infra/firewall/), not Docker's own
+#     default bridge NAT behavior
+#
+# All image tags, ports, and paths below are Ansible Jinja2 variables — this file is a
+# template rendered by the `platform` role, not run directly. See
+# infra/ansible/group_vars/production.yml for the values.
+
+networks:
+  rn-platform-internal:
+    driver: bridge
+    internal: false   # postgres/api/gateway/provisioner need it false to reach rn-lab-host-01
+                       # over the network; the HOST firewall (not Docker) is what actually
+                       # restricts what that traffic can reach — see docs/PRODUCTION_NETWORK_SECURITY.md
+
+volumes:
+  postgres-data:
+    driver: local
+    driver_opts:
+      type: none
+      o: bind
+      device: "{{ rootnode_data_root }}/postgres"
+  caddy-data:
+  caddy-config:
+  prometheus-data:
+  alertmanager-data:
+  grafana-data:
+  loki-data:
+
+services:
+  postgres:
+    image: postgres:16-alpine
+    restart: unless-stopped
+    env_file: /etc/rootnode/postgres.env
+    volumes:
+      - postgres-data:/var/lib/postgresql/data
+      - /opt/rootnode/platform/postgresql.conf.d:/etc/postgresql/conf.d:ro
+    # include_dir appends our tuning files to the image's own auto-generated postgresql.conf —
+    # no need to replace or template the base config file at all.
+    command: ["postgres", "-c", "include_dir=/etc/postgresql/conf.d"]
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-rootnode_platform}"]
+      interval: 10s
+      timeout: 5s
+      retries: 10
+    networks: [rn-platform-internal]
+    ports:
+      - "127.0.0.1:5432:5432"   # loopback-only — reachable from other containers on
+                                 # rn-platform-internal AND from the host's own backup cron
+                                 # (infra/backup/), never from another host on the network, let
+                                 # alone the internet. This is NOT the same as the "Postgres
+                                 # never exposed on this VM's external interface" rule in
+                                 # rn-platform-01.rules — that rule (and the host firewall in
+                                 # general) still applies to this port; 127.0.0.1 binding is a
+                                 # second, independent reason nothing off-host can ever reach it.
+
+  provisioner:
+    image: "{{ registry }}/rootnode/provisioner:{{ image_tag }}"
+    restart: unless-stopped
+    env_file: /etc/rootnode/provisioner.env
+    volumes:
+      - /etc/rootnode/provisioner-config:/app/config:ro
+      - /etc/rootnode/docker-tls:/app/docker-tls:ro
+    depends_on:
+      postgres:
+        condition: service_healthy
+    networks: [rn-platform-internal]
+    cap_add: [NET_ADMIN, NET_RAW]   # only needed while a lab_host row is still a local Unix
+                                    # socket (dev); harmless no-op against a remote TLS endpoint
+
+  api:
+    image: "{{ registry }}/rootnode/platform-api:{{ image_tag }}"
+    restart: unless-stopped
+    env_file: /etc/rootnode/platform.env
+    environment:
+      # Phase 4.1 §25 — not a secret, just the deployed tag; set directly here rather than in
+      # the env template since it's the same value already used for the image tag above, not a
+      # separately-managed setting.
+      GIT_SHA: "{{ image_tag }}"
+    depends_on:
+      postgres:
+        condition: service_healthy
+      provisioner:
+        condition: service_started
+    networks: [rn-platform-internal]
+
+  gateway:
+    image: "{{ registry }}/rootnode/gateway:{{ image_tag }}"
+    restart: unless-stopped
+    env_file: /etc/rootnode/gateway.env
+    volumes:
+      # mTLS client identity for the Lab Host Router (data plane) — see gateway.env.j2's
+      # LAB_ROUTER_TLS_* vars. Read-only; the private key never leaves this one bind mount.
+      - /etc/rootnode/lab-router-tls:/tls:ro
+    depends_on: [provisioner, api]
+    networks: [rn-platform-internal]
+
+  reverse-proxy:
+    image: caddy:2-alpine
+    restart: unless-stopped
+    ports:
+      - "443:443"
+      - "80:80"   # ACME HTTP-01 challenge / plain-HTTP-to-HTTPS redirect only
+    volumes:
+      - /opt/rootnode/platform/Caddyfile:/etc/caddy/Caddyfile:ro
+      - caddy-data:/data
+      - caddy-config:/config
+    depends_on: [gateway]
+    networks: [rn-platform-internal]
+
+  # ── Observability agents (infra/monitoring/) ────────────────────────────────────────────
+  node-exporter:
+    image: prom/node-exporter:latest
+    restart: unless-stopped
+    network_mode: host   # needs host-level metrics; this is metrics-only, not app traffic
+    pid: host
+    volumes:
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/rootfs:ro
+      - /var/lib/node-exporter/textfile:/textfile:ro   # backup-postgres.sh writes its metric here
+    command:
+      - "--path.procfs=/host/proc"
+      - "--path.sysfs=/host/sys"
+      - "--path.rootfs=/rootfs"
+      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
+      - "--collector.textfile.directory=/textfile"
+
+  postgres-exporter:
+    image: prometheuscommunity/postgres-exporter:latest
+    restart: unless-stopped
+    environment:
+      DATA_SOURCE_NAME: "postgresql://{{ postgres_exporter_user }}:{{ vault_postgres_exporter_password }}@postgres:5432/rootnode_platform?sslmode=disable"
+    depends_on: [postgres]
+    networks: [rn-platform-internal]
+
+  blackbox-exporter:
+    image: prom/blackbox-exporter:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
+    networks: [rn-platform-internal]
+
+  prometheus:
+    image: prom/prometheus:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - /opt/rootnode/monitoring/rules:/etc/prometheus/rules:ro
+      - prometheus-data:/prometheus
+    command:
+      - "--config.file=/etc/prometheus/prometheus.yml"
+      - "--storage.tsdb.retention.time={{ prometheus_retention_days }}d"
+    networks: [rn-platform-internal]
+    ports:
+      - "127.0.0.1:9090:9090"   # bound to loopback only — access via SSH tunnel or Grafana, never public
+
+  alertmanager:
+    image: prom/alertmanager:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
+      - alertmanager-data:/alertmanager
+    networks: [rn-platform-internal]
+
+  grafana:
+    image: grafana/grafana-oss:latest
+    restart: unless-stopped
+    environment:
+      GF_SECURITY_ADMIN_PASSWORD: "{{ vault_grafana_admin_password }}"
+      GF_SERVER_ROOT_URL: "https://{{ academy_domain }}/grafana/"
+      GF_SERVER_SERVE_FROM_SUB_PATH: "true"
+      GF_USERS_ALLOW_SIGN_UP: "false"
+    volumes:
+      - grafana-data:/var/lib/grafana
+      - /opt/rootnode/monitoring/grafana/provisioning:/etc/grafana/provisioning:ro
+      - /opt/rootnode/monitoring/grafana/dashboards:/var/lib/grafana/dashboards:ro
+    networks: [rn-platform-internal]
+    # No published port — reached only via the reverse proxy's /grafana/ path (Caddyfile),
+    # never a direct public listener. See docs/PRODUCTION_NETWORK_SECURITY.md §8.
+
+  loki:
+    image: grafana/loki:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/loki-config.yml:/etc/loki/local-config.yaml:ro
+      - loki-data:/loki
+    command: ["-config.file=/etc/loki/local-config.yaml"]
+    networks: [rn-platform-internal]
+    ports:
+      - "3100:3100"   # reachable from rn-lab-host-01's promtail over the network — restricted
+                       # to that one source IP by the host firewall (rn-platform-01.rules), same
+                       # pattern as Docker TLS's 2376. Not published to the internet.
+
+  promtail:
+    image: grafana/promtail:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/promtail-config.yml:/etc/promtail/config.yml:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /var/log/journal:/var/log/journal:ro
+      - /etc/machine-id:/etc/machine-id:ro
+    command: ["-config.file=/etc/promtail/config.yml"]
+    networks: [rn-platform-internal]
+    depends_on: [loki]

infra/docker/postgres/postgresql.conf.d/00-rootnode-baseline.conf

@@ -0,0 +1,30 @@
+# Conservative production tuning baseline — deliberately NOT hand-tuned to a specific RAM
+# allocation before rn-platform-01's real resources are known (Phase 4 instruction: do not
+# over-tune before actual hardware exists). Sized for a modest ~4GB Postgres allocation, safe
+# defaults everywhere else. Revisit once VM RAM is final — see docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §11.
+
+shared_buffers = 1GB
+effective_cache_size = 3GB
+maintenance_work_mem = 256MB
+work_mem = 16MB
+
+max_connections = 100
+
+# WAL — safe defaults for a single-instance setup with pg_dump/pg_basebackup-style backups
+# (infra/backup/). Not configured for streaming replication (out of scope this phase — see
+# docs/BACKUP_DR.md's DR model for why pg_dump + periodic pg_basebackup is the chosen approach,
+# not a hot standby).
+wal_level = replica
+max_wal_size = 2GB
+min_wal_size = 512MB
+checkpoint_completion_target = 0.9
+
+# Query logging — slow queries only, never full statement logging (would capture bind
+# parameters that can include session tokens' hashes / PII in edge cases). See docs/OBSERVABILITY.md.
+log_min_duration_statement = 500
+log_line_prefix = '%m [%p] %q%u@%d '
+log_checkpoints = on
+log_connections = on
+log_disconnections = on
+
+# Autovacuum stays on defaults — no reason to disable it for this workload size.

infra/docker/tls/README.md

@@ -0,0 +1,19 @@
+# Docker TLS material (generated, not committed)
+
+Generate with `infra/scripts/docker-tls/generate-docker-tls.sh`. Deploy with the `docker-tls`
+Ansible role. Verify with `infra/scripts/docker-tls/verify-docker-tls.sh`.
+
+**Nothing in this directory is ever committed** — see `infra/.gitignore`. This README (and only
+this README) is tracked, so the directory itself exists in the repo as documentation of what
+belongs here operationally.
+
+Files that will exist here on a real deployment host (never in git):
+
+```
+ca.pem                              # public — safe to distribute to both VMs
+ca-key.pem                          # PRIVATE — offline/secrets-vault only, never on either VM
+server-cert.pem / server-key.pem                    # rn-lab-host-01's dockerd
+provisioner-client-cert.pem / provisioner-client-key.pem   # rn-platform-01's Provisioner
+```
+
+Rotation procedure: `docs/RUNBOOK_DOCKER_TLS_ROTATION.md`.

infra/environments/README.md

@@ -0,0 +1,21 @@
+# Environment configuration
+
+Three environments, explicitly separated so production never silently inherits a dev default:
+
+| | dev | staging | production |
+|---|---|---|---|
+| Where it runs | any dev machine, `platform/docker-compose.dev.yml` | homelab or a resource-limited slice of the dedicated server once it exists | rn-platform-01 / rn-lab-host-01 on Proxmox |
+| `COOKIE_SECURE` | unset (plain HTTP) | `1` if served over HTTPS, else unset | `1` — required |
+| `TRUST_PROXY` | unset | `1` if fronted by a proxy | `1` — required, real reverse proxy in front |
+| `ADMIN`/owner bootstrap | `platform/api/bootstrap-owner.js` manually | same, against staging DB | same, against production DB, run once by a human, credential handled per `docs/PRODUCTION_DEPLOYMENT.md` §10 |
+| Secrets | `platform/.env` (gitignored, throwaway) | `infra/environments/staging/*.env` (gitignored, real but low-stakes) | `infra/environments/production/*.env` (gitignored, real, backed up per `docs/BACKUP_DR.md`) |
+| Lab image | built locally, `:test` tag | tagged build from CI, `:staging-<sha>` | tagged release, `:<sha>` or `:vX.Y.Z`, never `:latest` |
+
+Each environment directory holds `*.env.template` files (tracked) — copy to `*.env` (gitignored
+via `infra/.gitignore`) and fill in real values. **Never commit a filled `.env` file.** See
+`docs/PRODUCTION_DEPLOYMENT.md` §10 for how production secrets are actually generated, stored,
+and rotated (not just "don't commit them").
+
+`dev/` is intentionally thin — it just points back at the existing
+`platform/docker-compose.dev.yml` workflow (Phase 1–3B, unchanged); there was nothing new to
+build there for Phase 4.

infra/environments/dev/README.md

@@ -0,0 +1,7 @@
+# Dev environment
+
+Nothing new here for Phase 4 — dev still means `platform/docker-compose.dev.yml` +
+`platform/.env` (gitignored), exactly as built in Phases 1–3B. See `platform/README.md`.
+
+This directory exists only so `infra/environments/` visibly has all three tiers, not because
+dev needed new infrastructure code.

infra/environments/production/capacity.yml.example

@@ -0,0 +1,18 @@
+# Capacity planning inputs (Phase 4 §25). Copy to capacity.yml and fill in REAL measured values
+# once the actual Hetzner host and a real load test (infra/scripts/loadtest/) exist. The
+# numbers below are illustrative placeholders for a ~16c/32t/128GB host — do not treat them as
+# a real recommendation; run infra/scripts/capacity-model.js against real ones once measured.
+_placeholder: true
+
+host_cpu_cores: 32          # threads, if the host does hyperthreading — matches how Docker/cgroups count them
+host_mem_mb: 131072          # 128 GB
+
+platform_reserved_cpu_cores: 4    # rn-platform-01's VM allocation, reserved off the Lab Host's own pool
+platform_reserved_mem_mb: 16384
+
+proxmox_overhead_pct: 5      # hypervisor + host OS overhead — a conservative starting estimate
+
+measured_lab_cpu_cores: 2    # from infra/scripts/loadtest/sample-resources.sh peak CPU per Lab
+measured_lab_mem_mb: 2048    # from the same — peak memory per Lab
+
+safety_margin_pct: 20        # headroom kept unallocated even at the "recommended" threshold

infra/environments/production/gateway.env.template

@@ -0,0 +1,8 @@
+# Production environment template for rn-platform-01's `gateway` service.
+# `api`/`provisioner` are Docker Compose service names, not localhost — production runs these
+# on a named bridge network (rn-platform-internal), not network_mode: host like dev.
+
+PLATFORM_API_URL=http://api:4300
+PROVISIONER_URL=http://provisioner:4100
+PROVISIONER_SHARED_SECRET=__REPLACE_same_value_as_platform.env_and_provisioner.env__
+GATEWAY_PORT=4200

infra/environments/production/platform.env.template

@@ -0,0 +1,38 @@
+# Production environment template for rn-platform-01's `api` service.
+# Copy to platform.env OUTSIDE git (see infra/backup/README.md "secrets" and
+# docs/PRODUCTION_DEPLOYMENT.md §10), fill in real values, never commit the filled file.
+# infra/.gitignore already excludes *.env (only *.env.template is tracked).
+
+# ── Postgres ────────────────────────────────────────────────────────────────────────────
+# `postgres`/`provisioner` are Docker Compose service names, not localhost — production runs
+# these on a named bridge network (rn-platform-internal), not network_mode: host like dev.
+PLATFORM_DB_URL=postgres://rootnode_platform:__REPLACE_WITH_STRONG_PASSWORD__@postgres:5432/rootnode_platform
+
+# ── Provisioner ─────────────────────────────────────────────────────────────────────────
+PROVISIONER_URL=http://provisioner:4100
+PROVISIONER_SHARED_SECRET=__REPLACE_openssl_rand_hex_32__
+
+# ── Listen ──────────────────────────────────────────────────────────────────────────────
+PLATFORM_API_PORT=4300
+
+# ── Sessions (production values — do NOT inherit dev defaults) ────────────────────────────
+STUDENT_SESSION_TTL_HOURS=8
+STUDENT_SESSION_IDLE_HOURS=2
+INSTRUCTOR_SESSION_TTL_HOURS=12
+INSTRUCTOR_SESSION_IDLE_HOURS=2
+
+# ── Cookies / proxy — MUST be set for production, unlike dev ──────────────────────────────
+# Secure requires the origin to actually be HTTPS end-to-end (see docs/PRODUCTION_NETWORK_SECURITY.md §8).
+COOKIE_SECURE=1
+# Only set to 1 once a trusted reverse proxy (Cloudflare/nginx/Caddy, see infra/docker/reverse-proxy/)
+# is actually in front of the Gateway — otherwise req.ip becomes spoofable via X-Forwarded-For.
+TRUST_PROXY=1
+
+# ── Frontend (baked into the platform/api image at build time; listed here for reference) ─
+# FRONTEND_DIST_DIR=/frontend-dist
+
+# ── Future OIDC (Phase 3A adapter — not wired yet, see docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md §16) ─
+# OIDC_ISSUER_URL=
+# OIDC_CLIENT_ID=
+# OIDC_CLIENT_SECRET=__do_not_commit__
+# OIDC_REDIRECT_URL=https://academy.example.com/api/auth/oidc/callback

infra/environments/production/postgres.env.template

@@ -0,0 +1,14 @@
+# Production environment template for rn-platform-01's `postgres` service.
+# Postgres itself binds only to localhost/the platform-internal Docker network — never exposed
+# beyond rn-platform-01, see docs/PRODUCTION_NETWORK_SECURITY.md §6.
+
+POSTGRES_USER=rootnode_platform
+POSTGRES_PASSWORD=__REPLACE_WITH_STRONG_PASSWORD__
+POSTGRES_DB=rootnode_platform
+
+# Tuning baseline — conservative, meant to be revisited once rn-platform-01's real allocated
+# RAM is known (see docs/production-postgres section in docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md
+# §11). Applied via infra/docker/postgres/postgresql.conf.d/ — do not hand-edit the container.
+# POSTGRES_SHARED_BUFFERS=1GB
+# POSTGRES_EFFECTIVE_CACHE_SIZE=3GB
+# POSTGRES_MAX_CONNECTIONS=100

infra/environments/production/provisioner-config/README.md

@@ -0,0 +1,15 @@
+# Production Provisioner config overlay
+
+Deployed by the `platform` Ansible role to `/etc/rootnode/provisioner-config/` on
+rn-platform-01 (`PROVISIONER_CONFIG_DIR`), replacing the dev-only files shipped in
+`platform/provisioner/config/`.
+
+- `lab-hosts.yaml` — points at the real rn-lab-host-01 VM over TLS instead of the dev Unix
+  socket. TLS paths reference material generated by `infra/scripts/docker-tls/` and deployed by
+  the `docker-tls` Ansible role — **never checked into this repo** (see
+  `infra/docker/tls/.gitignore`).
+- `resource-policy.yaml` — deliberately still the Phase 1 dev defaults, unchanged, with a
+  loud comment explaining why. Per the explicit Phase 4 instruction: do not hard-code final
+  per-Lab CPU/RAM numbers before the real host is benchmarked (`docs/CAPACITY_PLANNING.md`).
+  Whoever runs the first production load test (`infra/scripts/loadtest/`) updates this file
+  with real measured numbers, not this deployment.

infra/environments/production/provisioner-config/lab-hosts.yaml

@@ -0,0 +1,27 @@
+# Production lab_hosts seed — points at the real rn-lab-host-01 VM instead of Phase 1's dev
+# Unix socket. Same mechanism the Provisioner has had since Phase 1
+# (platform/provisioner/config/lab-hosts.seed.yaml), just pointed at real remote endpoints.
+#
+# Phase 4.1 correction: routing_mode/router_endpoint is REQUIRED here — Phase 4's original
+# claim that "no application code changes required" was true for the Provisioner's CONTROL
+# plane (Docker mTLS) but false for the Gateway's DATA plane, which used to assume
+# env.internal_ip was directly reachable. It is not, across two real VMs. See
+# docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1 and docs/PRODUCTION_NETWORK_SECURITY.md.
+#
+# {{ lab_host_ip }} is an Ansible template variable — see infra/ansible/group_vars/production.yml.
+# Rendered by the `platform` role; the raw file here keeps the placeholder for readability.
+
+lab_hosts:
+  - lab_host_id: labhost-01
+    display_name: "RootNode Lab Host VM 01 (production)"
+    # CONTROL plane: Provisioner -> dockerd, mTLS, port 2376.
+    endpoint: "tcp://{{ lab_host_ip }}:2376"
+    tls_ca_path: /etc/rootnode/docker-tls/ca.pem
+    tls_cert_path: /etc/rootnode/docker-tls/provisioner-client-cert.pem
+    tls_key_path: /etc/rootnode/docker-tls/provisioner-client-key.pem
+    enabled: true
+    # DATA plane: Gateway -> Lab Host Router, mTLS, router's own port (8443 by default —
+    # infra/docker/lab-host/docker-compose.production.yml). Gateway never touches a
+    # Docker-internal address once routing_mode is 'router'.
+    routing_mode: router
+    router_endpoint: "https://{{ lab_host_ip }}:8443"

infra/environments/production/provisioner-config/resource-policy.yaml

@@ -0,0 +1,25 @@
+# PRODUCTION OVERLAY — placeholder, deliberately unchanged from the Phase 1 dev defaults.
+# Do not hand-tune these before the real host exists (docs/CAPACITY_PLANNING.md, Phase 4 §25).
+# After the first real load test on rn-lab-host-01 (infra/scripts/loadtest/), replace this
+# comment block and the numbers below with actual measured values, then redeploy via the
+# `platform` Ansible role (idempotent — re-running just re-renders this file).
+#
+# Phase 1 resource policy — config-driven, not hard-coded (decision #4).
+# Target host class for planning is ~16 cores/32 threads, 128GB RAM, SSD/NVMe.
+
+default:
+  cpu_limit: 2.0          # vCPUs
+  mem_limit_mb: 4096
+  pids_limit: 512
+  storage_limit_mb: 8192  # only enforced if the host's Docker storage driver supports
+                           # --storage-opt size= (see known-limitations in the SOP)
+
+idle_threshold_minutes: 30       # past this: log a warning event only
+abandoned_threshold_minutes: 120 # past this: reaper destroys for real
+
+reaper_poll_interval_seconds: 60
+
+health_check:
+  path: /api/health
+  timeout_ms: 240000   # k3s cold start in a nested container can be slow — matches the
+                        # ~60-attempt wait already in scripts/entrypoint.sh

infra/environments/production/provisioner.env.template

@@ -0,0 +1,20 @@
+# Production environment template for rn-platform-01's `provisioner` service.
+# See platform.env.template for the secrets-handling convention.
+#
+# Docker-connectivity config (lab_hosts) and resource limits are NOT env vars — they're YAML
+# files the Provisioner already loads from PROVISIONER_CONFIG_DIR (Phase 1 design, unchanged).
+# Production overlays for both live at infra/environments/production/provisioner-config/ — see
+# that directory's README before touching resource-policy.yaml numbers.
+
+PLATFORM_DB_URL=postgres://rootnode_platform:__REPLACE_WITH_STRONG_PASSWORD__@localhost:5432/rootnode_platform
+PROVISIONER_SHARED_SECRET=__REPLACE_same_value_as_platform.env__
+PROVISIONER_PORT=4100
+PROVISIONER_CONFIG_DIR=/etc/rootnode/provisioner-config
+
+# Real Lab image, tagged (never :latest in production — see infra/docker/IMAGE_STRATEGY.md)
+LAB_IMAGE_REF=registry.example.com/rootnode/lab:__GIT_SHA_OR_TAG__
+
+# ── Egress proxy on rn-lab-host-01 ─────────────────────────────────────────────────────────
+EGRESS_PROXY_PORT=3128
+DNS_RESOLVER=127.0.0.11
+NO_PROXY_CIDRS=10.42.0.0/16,10.43.0.0/16

infra/environments/staging/README.md

@@ -0,0 +1,16 @@
+# Staging environment
+
+Same shape as production (`docs/PRODUCTION_DEPLOYMENT.md` applies to both — pass `-e env=staging`
+to the Ansible playbooks), lower stakes: separate database name/port, staging-tagged images, and
+`COOKIE_SECURE` only if staging actually sits behind HTTPS. See `docs/CI_CD` section in
+`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §18 for how a Gitea Actions build lands here
+automatically before a human promotes it to production.
+
+Staging is designed to run either in the homelab (a couple of low-resource VMs/containers) or,
+once the dedicated server exists, as a resource-constrained slice of it — see
+`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §23 for the sizing note. It does not require a
+second dedicated server.
+
+Staging acceptance = the same four things a person actually needs before promoting a build:
+app deploys, migrations run clean, one or two disposable Labs boot through the Gateway, and an
+assessment can be started and completed. See `infra/scripts/staging-smoke-test.sh`.

infra/environments/staging/gateway.env.template

@@ -0,0 +1,6 @@
+# Production environment template for rn-platform-01's `gateway` service.
+
+PLATFORM_API_URL=http://localhost:4300
+PROVISIONER_URL=http://localhost:4100
+PROVISIONER_SHARED_SECRET=__REPLACE_same_value_as_platform.env_and_provisioner.env__
+GATEWAY_PORT=4200

infra/environments/staging/platform.env.template

@@ -0,0 +1,36 @@
+# Production environment template for rn-platform-01's `api` service.
+# Copy to platform.env OUTSIDE git (see infra/backup/README.md "secrets" and
+# docs/PRODUCTION_DEPLOYMENT.md §10), fill in real values, never commit the filled file.
+# infra/.gitignore already excludes *.env (only *.env.template is tracked).
+
+# ── Postgres ────────────────────────────────────────────────────────────────────────────
+PLATFORM_DB_URL=postgres://rootnode_platform_staging:__REPLACE_WITH_STRONG_PASSWORD__@localhost:5433/rootnode_platform_staging
+
+# ── Provisioner ─────────────────────────────────────────────────────────────────────────
+PROVISIONER_URL=http://localhost:4100
+PROVISIONER_SHARED_SECRET=__REPLACE_openssl_rand_hex_32__
+
+# ── Listen ──────────────────────────────────────────────────────────────────────────────
+PLATFORM_API_PORT=4300
+
+# ── Sessions (production values — do NOT inherit dev defaults) ────────────────────────────
+STUDENT_SESSION_TTL_HOURS=8
+STUDENT_SESSION_IDLE_HOURS=2
+INSTRUCTOR_SESSION_TTL_HOURS=12
+INSTRUCTOR_SESSION_IDLE_HOURS=2
+
+# ── Cookies / proxy — MUST be set for production, unlike dev ──────────────────────────────
+# Secure requires the origin to actually be HTTPS end-to-end (see docs/PRODUCTION_NETWORK_SECURITY.md §8).
+COOKIE_SECURE=1  # unset if staging is not served over HTTPS
+# Only set to 1 once a trusted reverse proxy (Cloudflare/nginx/Caddy, see infra/docker/reverse-proxy/)
+# is actually in front of the Gateway — otherwise req.ip becomes spoofable via X-Forwarded-For.
+TRUST_PROXY=1
+
+# ── Frontend (baked into the platform/api image at build time; listed here for reference) ─
+# FRONTEND_DIST_DIR=/frontend-dist
+
+# ── Future OIDC (Phase 3A adapter — not wired yet, see docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md §16) ─
+# OIDC_ISSUER_URL=
+# OIDC_CLIENT_ID=
+# OIDC_CLIENT_SECRET=__do_not_commit__
+# OIDC_REDIRECT_URL=https://academy.example.com/api/auth/oidc/callback

infra/environments/staging/postgres.env.template

@@ -0,0 +1,14 @@
+# Production environment template for rn-platform-01's `postgres` service.
+# Postgres itself binds only to localhost/the platform-internal Docker network — never exposed
+# beyond rn-platform-01, see docs/PRODUCTION_NETWORK_SECURITY.md §6.
+
+POSTGRES_USER=rootnode_platform
+POSTGRES_PASSWORD=__REPLACE_WITH_STRONG_PASSWORD__
+POSTGRES_DB=rootnode_platform
+
+# Tuning baseline — conservative, meant to be revisited once rn-platform-01's real allocated
+# RAM is known (see docs/production-postgres section in docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md
+# §11). Applied via infra/docker/postgres/postgresql.conf.d/ — do not hand-edit the container.
+# POSTGRES_SHARED_BUFFERS=1GB
+# POSTGRES_EFFECTIVE_CACHE_SIZE=3GB
+# POSTGRES_MAX_CONNECTIONS=100

infra/environments/staging/provisioner-config/lab-hosts.yaml

@@ -0,0 +1,17 @@
+# Staging lab_hosts seed — see infra/environments/production/provisioner-config/lab-hosts.yaml
+# for the full explanation; this is the same shape, lower stakes. Staging's `lab_host_ip` is
+# distinct from production's own (infra/ansible/group_vars/staging.yml), so it needs the same
+# router routing_mode as production once staging genuinely runs on two hosts. If staging is
+# ever collapsed onto a single host instead, change routing_mode to `direct` and drop
+# router_endpoint — that's the only edit needed either way.
+
+lab_hosts:
+  - lab_host_id: labhost-01
+    display_name: "RootNode Lab Host VM 01 (staging)"
+    endpoint: "tcp://{{ lab_host_ip }}:2376"
+    tls_ca_path: /etc/rootnode/docker-tls/ca.pem
+    tls_cert_path: /etc/rootnode/docker-tls/provisioner-client-cert.pem
+    tls_key_path: /etc/rootnode/docker-tls/provisioner-client-key.pem
+    enabled: true
+    routing_mode: router
+    router_endpoint: "https://{{ lab_host_ip }}:8443"

infra/environments/staging/provisioner-config/resource-policy.yaml

@@ -0,0 +1,25 @@
+# PRODUCTION OVERLAY — placeholder, deliberately unchanged from the Phase 1 dev defaults.
+# Do not hand-tune these before the real host exists (docs/CAPACITY_PLANNING.md, Phase 4 §25).
+# After the first real load test on rn-lab-host-01 (infra/scripts/loadtest/), replace this
+# comment block and the numbers below with actual measured values, then redeploy via the
+# `platform` Ansible role (idempotent — re-running just re-renders this file).
+#
+# Phase 1 resource policy — config-driven, not hard-coded (decision #4).
+# Target host class for planning is ~16 cores/32 threads, 128GB RAM, SSD/NVMe.
+
+default:
+  cpu_limit: 2.0          # vCPUs
+  mem_limit_mb: 4096
+  pids_limit: 512
+  storage_limit_mb: 8192  # only enforced if the host's Docker storage driver supports
+                           # --storage-opt size= (see known-limitations in the SOP)
+
+idle_threshold_minutes: 30       # past this: log a warning event only
+abandoned_threshold_minutes: 120 # past this: reaper destroys for real
+
+reaper_poll_interval_seconds: 60
+
+health_check:
+  path: /api/health
+  timeout_ms: 240000   # k3s cold start in a nested container can be slow — matches the
+                        # ~60-attempt wait already in scripts/entrypoint.sh

infra/environments/staging/provisioner.env.template

@@ -0,0 +1,20 @@
+# Production environment template for rn-platform-01's `provisioner` service.
+# See platform.env.template for the secrets-handling convention.
+#
+# Docker-connectivity config (lab_hosts) and resource limits are NOT env vars — they're YAML
+# files the Provisioner already loads from PROVISIONER_CONFIG_DIR (Phase 1 design, unchanged).
+# Production overlays for both live at infra/environments/production/provisioner-config/ — see
+# that directory's README before touching resource-policy.yaml numbers.
+
+PLATFORM_DB_URL=postgres://rootnode_platform:__REPLACE_WITH_STRONG_PASSWORD__@localhost:5433/rootnode_platform
+PROVISIONER_SHARED_SECRET=__REPLACE_same_value_as_platform.env__
+PROVISIONER_PORT=4100
+PROVISIONER_CONFIG_DIR=/etc/rootnode/provisioner-config
+
+# Real Lab image, tagged (never :latest in production — see infra/docker/IMAGE_STRATEGY.md)
+LAB_IMAGE_REF=registry.example.com/rootnode/lab:staging-__GIT_SHA__
+
+# ── Egress proxy on rn-lab-host-01 ─────────────────────────────────────────────────────────
+EGRESS_PROXY_PORT=3128
+DNS_RESOLVER=127.0.0.11
+NO_PROXY_CIDRS=10.42.0.0/16,10.43.0.0/16

infra/firewall/README.md

@@ -0,0 +1,31 @@
+# Production firewall policy
+
+iptables (not nftables) — deliberately, to stay consistent with the Provisioner's own dynamic
+per-Lab egress rules, which already manipulate the `DOCKER-USER` chain via iptables
+(`platform/provisioner/src/firewall.js`, Phase 1). Mixing the nft and legacy iptables backends
+on the same host is a known source of silent rule-ordering bugs; the host-level policy below
+uses the same backend the app already depends on.
+
+Each `.rules` file is `iptables-restore` format — deployed by the matching Ansible role
+(`firewall` for the VMs; the Proxmox host ruleset is applied manually per
+`docs/PRODUCTION_DEPLOYMENT.md` §2, since Proxmox host firewalling is out of Ansible's blast
+radius by design — a mistake there can lock out the only way in).
+
+| File | Applies to | Deployed by |
+|---|---|---|
+| `proxmox-host.rules` | The Proxmox hypervisor itself | manual, `docs/PRODUCTION_DEPLOYMENT.md` §2 |
+| `rn-platform-01.rules` | Platform VM | `infra/ansible/roles/firewall/` |
+| `rn-lab-host-01.rules` | Lab Host VM | `infra/ansible/roles/firewall/` |
+
+All three encode the same policy table from `docs/PRODUCTION_NETWORK_SECURITY.md` §6 — this
+README doesn't restate it, that doc is the source of truth for *why*; these files are the
+*what*, in a form the host can actually load.
+
+**IP addresses/subnets are Ansible template variables** (`{{ platform_ip }}`,
+`{{ lab_host_ip }}`, `{{ management_cidr }}`, etc.) — see `infra/ansible/group_vars/production.yml`.
+Nothing here hardcodes a real production IP, per the explicit Phase 4 instruction not to guess
+those before the real host exists.
+
+`infra/firewall/validate-firewall.sh` renders each ruleset with dummy IPs and runs it through
+`iptables-restore --test` (syntax/semantic validation only — proves the rules are well-formed
+iptables, not that the real network topology is correct, which needs the real hardware).

infra/firewall/proxmox-host.rules

@@ -0,0 +1,42 @@
+# iptables-restore ruleset for the Proxmox VE hypervisor itself. Applied MANUALLY per
+# docs/PRODUCTION_DEPLOYMENT.md §2 — deliberately not Ansible-automated (a mistake in a
+# playbook run against the one host that IS the only way to reach every VM console is a much
+# worse failure mode than a manual, reviewed apply). Prefer configuring this through Proxmox's
+# own Datacenter -> Firewall UI/API where practical instead of hand-editing iptables on the
+# hypervisor — this file documents the equivalent policy either way.
+#
+# Policy summary (docs/PRODUCTION_NETWORK_SECURITY.md §5-6):
+#   - Proxmox web UI (8006) and SSH (22) reachable ONLY from the management CIDR — never from
+#     the internet, never from either VM's network, never from a student Lab subnet
+#   - VM traffic (rn-platform-01 <-> rn-lab-host-01, rn-platform-01 <-> internet via NAT/bridge)
+#     is FORWARDed by the bridge, not filtered by this hypervisor-level ruleset — that's the
+#     VMs' own INPUT chains' job (rn-platform-01.rules / rn-lab-host-01.rules)
+#   - no student Lab subnet, and no rn-lab-host-01 traffic at all, may ever reach Proxmox
+#     management — this is the literal "Proxmox management not reachable from student Lab
+#     networks" requirement, enforced at the hypervisor itself, not just hoped-for via VM
+#     network segmentation
+
+*filter
+:INPUT DROP [0:0]
+:FORWARD ACCEPT [0:0]
+:OUTPUT ACCEPT [0:0]
+
+-A INPUT -i lo -j ACCEPT
+-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
+-A INPUT -m conntrack --ctstate INVALID -j DROP
+-A INPUT -p icmp --icmp-type echo-request -m conntrack --ctstate NEW -j ACCEPT
+
+-A INPUT -p tcp --dport 22 -s {{ management_cidr }} -m conntrack --ctstate NEW -j ACCEPT
+-A INPUT -p tcp --dport 8006 -s {{ management_cidr }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Explicit deny + log for the Lab Host VM's network and the student Lab subnet range reaching
+# Proxmox management directly — the literal "Proxmox management denied from student Lab
+# networks" requirement.
+-A INPUT -s {{ lab_host_ip }} -p tcp -m multiport --dports 22,8006 -j LOG --log-prefix "ROOTNODE-DENY-LABHOST-TO-PROXMOX: " --log-level 4
+-A INPUT -s {{ lab_host_ip }} -p tcp -m multiport --dports 22,8006 -j DROP
+-A INPUT -s {{ lab_subnet_range }} -p tcp -m multiport --dports 22,8006 -j LOG --log-prefix "ROOTNODE-DENY-LAB-TO-PROXMOX: " --log-level 4
+-A INPUT -s {{ lab_subnet_range }} -p tcp -m multiport --dports 22,8006 -j DROP
+
+-A INPUT -j LOG --log-prefix "ROOTNODE-DENY-PROXMOX-DEFAULT: " --log-level 4
+
+COMMIT

infra/firewall/rn-lab-host-01.rules

@@ -0,0 +1,63 @@
+# iptables-restore ruleset for rn-lab-host-01 (Lab Host VM). Deployed by
+# infra/ansible/roles/firewall/. See docs/PRODUCTION_NETWORK_SECURITY.md §6 for full rationale.
+#
+# This is the security-critical host — it runs privileged, disposable, student-controlled
+# containers. Its own INPUT/FORWARD policy is only half the story: the Provisioner's dynamic
+# per-Lab DOCKER-USER chains (platform/provisioner/src/egress.js, unchanged since Phase 1)
+# already enforce egress-proxy-only outbound for every student subnet and implicitly deny
+# Lab-to-Lab and Lab-to-anything-not-allowlisted traffic. This file is the BASE policy under
+# that dynamic layer — it must never be more permissive than what the dynamic chains assume,
+# and it's what protects the host itself (SSH, Docker TLS) rather than inter-container traffic.
+#
+# Policy summary:
+#   - Docker TLS (2376) reachable ONLY from rn-platform-01 — this is the entire point of mTLS
+#     plus a firewall restriction, defense in depth (a leaked client cert alone still isn't
+#     enough without also being on this IP)
+#   - SSH only from the management CIDR
+#   - no port on this host is ever reachable from a student Lab's own Docker subnet — Labs only
+#     ever reach the egress proxy and DNS resolver (already enforced by the Provisioner's
+#     per-Lab chains); this file adds an explicit belt-and-suspenders DROP for the Lab subnet
+#     range as a whole, so even a Lab whose per-environment chain failed to install can't reach
+#     the host's own services
+#   - nothing on this host is ever reachable from the public internet at all
+
+*filter
+:INPUT DROP [0:0]
+:FORWARD DROP [0:0]
+:OUTPUT ACCEPT [0:0]
+
+-A INPUT -i lo -j ACCEPT
+-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
+-A INPUT -m conntrack --ctstate INVALID -j DROP
+-A INPUT -p icmp --icmp-type echo-request -m conntrack --ctstate NEW -j ACCEPT
+
+-A INPUT -p tcp --dport 22 -s {{ management_cidr }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Docker TLS — rn-platform-01 (the Provisioner) only. This is the ONLY control-plane flow
+# allowed into this host at all — see the "Platform Provisioner -> Lab Host Docker TLS
+# endpoint: ALLOW" example in the Phase 4 spec, encoded literally here.
+-A INPUT -p tcp --dport 2376 -s {{ platform_ip }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Lab Host Router (8443) — rn-platform-01 (the Gateway) only. This is the DATA-plane
+# equivalent of the Docker TLS rule above: the Gateway never reaches a Lab's Docker-internal
+# address directly (no per-Lab port publishing, no wide port range needed) — it reaches this
+# one mTLS-authenticated Router instead, which resolves the request to a local container on
+# ITS OWN host. See docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3 and
+# docs/PRODUCTION_NETWORK_SECURITY.md — this closes the "Gateway-to-Lab cross-VM reachability"
+# gap Phase 4 flagged as a required follow-up; it is no longer open.
+-A INPUT -p tcp --dport {{ lab_router_port | default('8443') }} -s {{ platform_ip }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Prometheus (rn-platform-01) scraping this host's node-exporter/cadvisor — metrics only,
+# read-only exposition endpoints, still restricted to the one legitimate scraper.
+-A INPUT -p tcp -m multiport --dports 9100,8080 -s {{ platform_ip }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Belt-and-suspenders: explicit deny + log for any student Lab subnet reaching THIS HOST
+# (not another container — the host's own listening ports: SSH, Docker TLS, anything else).
+# The Provisioner's per-Lab DOCKER-USER chains already stop this at the container-egress layer;
+# this is what still protects the host if that layer is ever misconfigured or mid-install.
+-A INPUT -s {{ lab_subnet_range }} -j LOG --log-prefix "ROOTNODE-DENY-LAB-TO-HOST: " --log-level 4
+-A INPUT -s {{ lab_subnet_range }} -j DROP
+
+-A INPUT -j LOG --log-prefix "ROOTNODE-DENY-LABHOST-DEFAULT: " --log-level 4
+
+COMMIT

infra/firewall/rn-platform-01.rules

@@ -0,0 +1,58 @@
+# iptables-restore ruleset for rn-platform-01 (Platform VM). Deployed by
+# infra/ansible/roles/firewall/. Variables ({{ }}) are Ansible Jinja2, rendered before this
+# file reaches the host — see infra/ansible/group_vars/production.yml.
+#
+# Policy summary (full rationale: docs/PRODUCTION_NETWORK_SECURITY.md §6):
+#   - only the reverse proxy's 443/80 are reachable from the internet
+#   - SSH only from the management CIDR (your home/office/VPN, never 0.0.0.0/0)
+#   - Docker TLS (2376) is a rn-lab-host-01 concern, not this host's — nothing to open here
+#   - Postgres, Provisioner's internal API, and the Platform API/Gateway's raw ports are never
+#     exposed on this VM's external interface at all — the reverse proxy is the only public
+#     listener; everything else binds to the docker-compose internal network only, so there is
+#     no host-level rule needed to "hide" them, only to make sure nothing ever publishes them
+#     by accident (this ruleset's default-deny INPUT catches that class of mistake too)
+#   - rn-lab-host-01 may reach exactly one port here: Loki (3100), for its own log shipper —
+#     the Provisioner otherwise calls OUT to rn-lab-host-01, never the other way around — and
+#     may NEVER reach Postgres or the Provisioner's internal API
+
+*filter
+:INPUT DROP [0:0]
+:FORWARD DROP [0:0]
+:OUTPUT ACCEPT [0:0]
+
+# Loopback and established/related — standard baseline
+-A INPUT -i lo -j ACCEPT
+-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
+-A INPUT -m conntrack --ctstate INVALID -j DROP
+
+# ICMP — keep basic reachability diagnostics working
+-A INPUT -p icmp --icmp-type echo-request -m conntrack --ctstate NEW -j ACCEPT
+
+# SSH — management CIDR only, never open to the internet
+-A INPUT -p tcp --dport 22 -s {{ management_cidr }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Public HTTPS/HTTP — the reverse proxy (Caddy), the only public listener on this host
+-A INPUT -p tcp --dport 443 -m conntrack --ctstate NEW -j ACCEPT
+-A INPUT -p tcp --dport 80 -m conntrack --ctstate NEW -j ACCEPT
+
+# Loki log ingestion — rn-lab-host-01's promtail only (that host's own log shipper, pushing its
+# container/journald logs here for centralized viewing in Grafana). Same "one specific source
+# IP, one specific port" pattern as Docker TLS on the other host.
+-A INPUT -p tcp --dport 3100 -s {{ lab_host_ip }} -m conntrack --ctstate NEW -j ACCEPT
+
+# Explicit deny + log for anything from rn-lab-host-01 hitting Postgres or the Provisioner's
+# internal port — defense in depth on top of "these ports were never published," and the one
+# rule in this file that most directly encodes the Lab->Platform boundary the whole platform
+# depends on. Logged (not silently dropped) so an attempted breach attempt is visible in
+# infra/monitoring/ alerts, not just absorbed by the default DROP.
+-A INPUT -s {{ lab_host_ip }} -p tcp --dport 5432 -j LOG --log-prefix "ROOTNODE-DENY-LABHOST-PG: " --log-level 4
+-A INPUT -s {{ lab_host_ip }} -p tcp --dport 5432 -j DROP
+-A INPUT -s {{ lab_host_ip }} -p tcp --dport 4100 -j LOG --log-prefix "ROOTNODE-DENY-LABHOST-PROVISIONER: " --log-level 4
+-A INPUT -s {{ lab_host_ip }} -p tcp --dport 4100 -j DROP
+
+# Default: everything else already falls to the chain policy (DROP), nothing further needed —
+# but an explicit final logged catch-all makes "what got denied and why" visible in logs rather
+# than silent.
+-A INPUT -j LOG --log-prefix "ROOTNODE-DENY-PLATFORM-DEFAULT: " --log-level 4
+
+COMMIT

infra/firewall/validate-firewall.sh

@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+# Local test (Phase 4 §29): renders each .rules file with dummy IPs and validates it.
+#
+# `iptables-restore --test` needs root (it talks to netfilter) — if this shell has passwordless
+# sudo, we use it for a real semantic check. Otherwise we fall back to a structural lint
+# (balanced *table/COMMIT blocks, every rule starts with a real flag, no unrendered {{ }}
+# placeholders left over) — good enough to catch a typo'd rule or a missing COMMIT, NOT a
+# substitute for the real `iptables-restore --test` a human/Ansible with actual root runs
+# before this ever reaches a live host. HARDWARE-DEPENDENT: proving the real IPs/CIDRs are
+# correct needs the actual production network, not this script.
+set -euo pipefail
+cd "$(dirname "$0")"
+
+TMPDIR="$(mktemp -d)"
+trap 'rm -rf "$TMPDIR"' EXIT
+
+render() {
+  sed \
+    -e 's/{{ management_cidr }}/203.0.113.0\/24/g' \
+    -e 's/{{ platform_ip }}/10.10.10.10/g' \
+    -e 's/{{ lab_host_ip }}/10.10.10.20/g' \
+    -e 's/{{ lab_subnet_range }}/10.42.0.0\/16/g' \
+    -e 's/{{ proxmox_mgmt_cidr }}/203.0.113.0\/24/g' \
+    -e "s/{{ lab_router_port | default('8443') }}/8443/g" \
+    "$1" > "$2"
+}
+
+structural_lint() {
+  local f="$1"
+  grep -q '^\*filter' "$f" || { echo "  missing *filter table declaration"; return 1; }
+  grep -q '^COMMIT' "$f" || { echo "  missing COMMIT"; return 1; }
+  grep -v '^#' "$f" | grep -q '{{' && { echo "  unrendered {{ }} placeholder left in a non-comment line"; return 1; }
+  # every -A/-I rule line should end in a real target (ACCEPT/DROP/REJECT/LOG/a custom chain)
+  while IFS= read -r line; do
+    [[ "$line" =~ ^-[AI]\  ]] || continue
+    [[ "$line" =~ -j\ [A-Za-z0-9_-]+ ]] || { echo "  rule missing a -j target: $line"; return 1; }
+  done < "$f"
+  return 0
+}
+
+HAVE_SUDO=0
+sudo -n true 2>/dev/null && HAVE_SUDO=1
+
+FAIL=0
+for f in proxmox-host.rules rn-platform-01.rules rn-lab-host-01.rules; do
+  render "$f" "$TMPDIR/$f"
+  echo "==> $f"
+  if [ "$HAVE_SUDO" = "1" ] && command -v iptables-restore >/dev/null 2>&1; then
+    if sudo -n iptables-restore --test < "$TMPDIR/$f"; then
+      echo "OK: $f passed a real iptables-restore --test"
+    else
+      echo "FAIL: $f did not pass iptables-restore --test" >&2; FAIL=1
+    fi
+  else
+    if structural_lint "$TMPDIR/$f"; then
+      echo "OK: $f passes structural lint (no root available for a real iptables-restore --test — see script header)"
+    else
+      echo "FAIL: $f failed structural lint" >&2; FAIL=1
+    fi
+  fi
+done
+
+exit $FAIL