docs(phase5a): student identity and remote classroom documentation

- PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md: the full design doc.
- RUNBOOK_STUDENT_ACCOUNT_LIFECYCLE.md, RUNBOOK_REMOTE_CLASSROOM.md: operational runbooks.
- REMOTE_AND_SELF_PACED_ARCHITECTURE.md: closes the 'no real student authentication' gap this
  doc previously flagged as the one item blocking self-paced delivery.
- ENGINEERING_SOP.md §5 rewritten (was describing a pre-Phase-2A state no longer true of any
  deployed version of this codebase) plus a new §14; roadmap item 3 updated.
- LAB_ARCHITECTURE.md, PRODUCTION_VALIDATION_CHECKLIST.md, README.md, platform/README.md:
  cross-references and a Phase 5A production-validation section.

.gitignore

@@ -4,3 +4,4 @@ frontend/dist/
 *.db-journal
 /data/
 .env
+.ansible/

README.md

@@ -150,12 +150,19 @@ its core:
   the browser (`docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md`, session model in
   `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md` §3, hint gating in
   `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` §3).
-- The student flow (roster login → curriculum dashboard → Launch Lab → practice/assessments/
+- The student flow (real login (email or instructor-assigned login ID) or legacy pilot roster
-  capstone → achievements) lives in `frontend/src/platform/`; the instructor/admin portal
+  entry → curriculum dashboard → Launch Lab → practice/assessments/capstone → achievements →
-  (per-instructor login, cohorts, students, instructor accounts, audit log, classroom
+  live classroom → profile) lives in `frontend/src/platform/`; the instructor/admin portal
-  "Prepare Labs," environment controls, learning analytics, assessment/capstone management)
+  (per-instructor login, cohorts, students with credential/CSV-import management, a remote
-  lives in `frontend/src/admin/` at the `/admin` route — both additions alongside the existing
+  classroom control center, instructor accounts, audit log, environment controls, learning
-  Lab UI, not a rewrite of it.
+  analytics, assessment/capstone management) lives in `frontend/src/admin/` at the `/admin`
+  route — both additions alongside the existing Lab UI, not a rewrite of it.
+- Students authenticate with real credentials as of Phase 5A (email+password or
+  login-identifier+password, first-login/reset via a single-use token an instructor relays) —
+  see `docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md`. Instructors run live, remote-capable
+  classes from `/admin` (presence, Lab/learning status, help-request queue, announcements,
+  guidance-only current-activity) — RootNode owns identity/coordination, an external service
+  (Meet/Zoom/Teams/etc.) owns the actual video/voice.
 
 **Never publish the raw privileged container** — front it with isolation (that's what the
 Gateway is). The one-container `docker run --privileged -p 7554:80 ...` command above stays
@@ -192,7 +199,16 @@ Key properties of the platform (not this image):
 
 Production topology and what's built vs. not yet: `docs/LAB_ARCHITECTURE.md`,
 `docs/PHASE1_PLATFORM_FOUNDATION.md`, `docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md`,
-`docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md`, `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`.
+`docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md`, `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`,
+`docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md`.
+
+The production infrastructure package (Proxmox/Ansible/Terraform/firewall/monitoring/backup/CI)
+that runs all of this on real hardware lives in `infra/` — see
+`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` and `docs/PHASE4_1_PREPRODUCTION_HARDENING.md` (the
+Gateway-to-Lab cross-VM routing fix, plus application hardening — real pagination, abandonment
+handling, a Postgres-backed rate limiter, a structured assessment editor, admission control).
+Built and locally validated now; deployment is pending purchase of the target Hetzner dedicated
+server.
 
 ---
 

docs/BACKUP_DR.md

@@ -0,0 +1,74 @@
+# Backup & Disaster Recovery
+
+## What's backed up (Tier 1) vs. not
+
+| | Backed up | Why |
+|---|---|---|
+| Postgres data | Yes, daily | The only irreplaceable data — student records, cohort/session state, scenario attempts, assessments, achievements, audit events |
+| `{{ rootnode_data_root }}` config/secrets (Vault files, env templates) | Yes, via the infra repo + Ansible Vault itself | Small, static, versioned |
+| Infra code (`infra/`, Ansible, Terraform state) | Yes — it's in git; Terraform state should be backed up alongside Postgres if not using a remote backend | Needed to rebuild either VM from scratch |
+| Disposable student Labs | **No, intentionally** | Ephemeral by design — a Lab is meant to be destroyed and recreated; backing it up would contradict the whole isolation/disposability model |
+| Prometheus/Loki metrics & log history | No | Operational telemetry, not source-of-truth data; losing 30 days of dashboards on a rebuild is an inconvenience, not data loss |
+
+## Backup mechanism
+
+`infra/backup/backup-postgres.sh` (deployed to `/opt/rootnode/backup/` by the `backup` Ansible
+role, run daily at 03:15 via cron):
+
+1. `pg_dump --format=plain --no-owner --no-privileges | gzip -9`
+2. Sanity check: reject if the compressed dump is under 1KB (almost certainly means Postgres
+   was unreachable or empty, not a real backup).
+3. `gpg --symmetric --cipher-algo AES256` — encrypt, then delete the plaintext dump immediately.
+4. Optional off-host copy via `rsync -az` over SSH — works for Hetzner Storage Box or any SSH
+   target; that's the extension point for S3-compatible alternatives (swap the rsync line for
+   `aws s3 cp`/`rclone` if a different target is chosen later).
+5. Retention pruning (`find ... -mtime +{{ backup_retention_days }} -delete`, default 30 days).
+6. Writes Prometheus textfile-collector metrics
+   (`rootnode_backup_last_run_success`, `..._last_success_timestamp_seconds`, `..._size_bytes`)
+   atomically (tmp file + rename) so a half-written metric file is never scraped.
+
+## Restore
+
+`infra/backup/restore-postgres.sh` — requires an explicit target DB URL (never infers or
+defaults to production, to prevent an accidental overwrite), shows the target's current tables,
+requires typing `RESTORE` to confirm, then decrypts and pipes into `psql`.
+`infra/backup/verify-restore.sh` — confirms all 8 expected tables exist plus the
+`schema_migrations` row count, so a "successful" restore that silently missed a table is caught.
+
+**Genuinely tested**, not just written: `infra/backup/test-backup-restore.sh` runs the full
+cycle against two disposable local Postgres containers, with the actual production migrations
+applied (3 Provisioner + Platform API, in the correct cross-service order), a
+sample row inserted, backed up, restored into the second container, and verified byte-for-byte.
+Confirmed passing, exit 0.
+
+**Phase 4.1's destructive drill goes one step further**
+(`infra/backup/test-destructive-restore-drill.sh`): realistic data (owner, admin instructor,
+cohort, 3 students, progress, attempts, an assessment definition, achievements) seeded through
+the REAL Platform API, backed up, the source Postgres **container genuinely destroyed**
+(`docker rm`, not just left running alongside a second one), a fresh Postgres created from
+nothing, restored, and the real Platform API booted again against the restored database and
+used to log in as the restored admin instructor over HTTP — proving the whole stack recovers,
+not just the SQL. Never touches the developer's real persistent database. Confirmed passing.
+
+## Disaster recovery scenarios
+
+| Scenario | RPO | RTO (estimate) | What happens |
+|---|---|---|---|
+| Platform VM lost, disks intact | 0 | ~30 min | `docs/RUNBOOK_VM_REBUILD.md` — Ansible re-run against the same disks |
+| Platform VM + its disks lost | Up to 24h (since last backup) | ~1-2h | Rebuild VM (Terraform), re-run Ansible, restore Postgres from the latest off-host backup (`docs/RUNBOOK_POSTGRES_RESTORE.md`) |
+| Lab Host VM lost, any reason | 0 (intentional) | ~30 min | Rebuild VM, re-run Ansible — no data to restore, all Labs were disposable and get recreated on next student login |
+| Single disk failure in a mirror | 0 | 0 (no outage) | `docs/RUNBOOK_DISK_FAILURE.md` — pool stays online DEGRADED, replace and resilver |
+| Whole storage pool lost (both mirror sides) | Up to 24h | ~1-2h | Same as "Platform VM + disks lost" above — this is precisely the scenario the off-host copy exists for; a local-only backup on the same failed pool is useless here |
+| Encryption passphrase lost | N/A | Unrecoverable | The single most important secret to protect and back up separately (e.g., in the same secrets manager as other Vault secrets, not only in `production.vault.yml`) — losing it makes every existing backup file permanently unreadable |
+
+RPO is bounded by backup cadence (daily) — a mid-day incident can lose up to a day of student
+activity. If that's ever unacceptable, the fix is a shorter cron interval or WAL archiving for
+point-in-time recovery, not decided or built here since daily has not yet been proven
+insufficient against real usage.
+
+## What backup does NOT cover
+
+Application code and container images — those live in Gitea and the image registry, both
+already durable and versioned independently of this backup mechanism. TLS certificates — those
+are regenerable (`docs/RUNBOOK_DOCKER_TLS_ROTATION.md`, `docs/RUNBOOK_TLS_RENEWAL.md`), not
+backed up.

docs/CAPACITY_PLANNING.md

@@ -0,0 +1,59 @@
+# Capacity Planning
+
+How many concurrent student Labs `rn-lab-host-01` can actually run — determined from measured
+data after deployment, not guessed or marketed now.
+
+## The model
+
+`infra/scripts/capacity-model.js`:
+
+```
+usable        = hostTotal * (1 - overheadPct/100) - platformReserved
+withMargin    = usable * (1 - marginPct/100)
+perLabLimit   = floor(withMargin / perLabUsage)
+recommended   = min(cpuBoundLimit, memoryBoundLimit)     # whichever runs out first
+warning       = floor(recommended * 0.8)
+hardAdmission = recommended
+```
+
+Computed once for CPU, once for memory; the tighter (smaller) of the two constrains the actual
+recommendation — a host can be memory-starved long before its CPU threads are exhausted, or
+vice versa, and the model doesn't pretend otherwise.
+
+**Inputs** (`infra/environments/production/capacity.yml.example` — explicitly placeholder data
+until real numbers exist):
+- `host_cpu_threads`, `host_memory_mb` — from the real Lab Host VM, once sized
+- `overhead_pct` — Proxmox/hypervisor + OS overhead reserved off the top
+- `platform_reserved_cpu`/`platform_reserved_memory_mb` — headroom for the egress proxy,
+  node-exporter, cadvisor, promtail themselves (they run on the Lab Host too)
+- `per_lab_cpu`/`per_lab_memory_mb` — **measured**, from `infra/scripts/loadtest/`, not assumed
+- `safety_margin_pct` — extra headroom below the raw math, for burst/variance
+
+**Outputs**: recommended concurrent Labs, an 80%-of-recommended warning threshold (for
+alerting — `LabHostHighMemory` in `docs/OBSERVABILITY.md`), and a hard admission-control
+threshold the Provisioner's resource policy should enforce so it refuses to boot a Lab past that
+point rather than degrading every running Lab.
+
+## Load-test tooling that produces the measured inputs
+
+`infra/scripts/loadtest/`:
+- `boot-labs.js` — reuses `platform/test/session-helper.js` (built in Phase 3A) to drive the
+  real admin Prepare-Labs endpoint, booting N Labs and recording boot time/failure rate per Lab
+  into a JSON report.
+- `sample-resources.sh` — samples real `docker stats` for every `rootnode-lab-*` container every
+  5 seconds to CSV (timestamp, name, CPU%, mem usage, mem%, net I/O, block I/O, PIDs).
+
+Together these produce the `per_lab_cpu`/`per_lab_memory_mb` this model needs, plus data on how
+boot time and failure rate change as concurrency rises — the dimensions this phase was told to
+cover: single Lab boot, multiple simultaneous boots, steady-state active Labs, terminal
+WebSocket load, scenario validation load, Gateway throughput, API request volume, Postgres
+connection activity.
+
+## What this document deliberately does not claim
+
+No concurrency number in this repo (`capacity.yml.example`'s numbers included) is a real,
+validated figure — every one is a placeholder until run against the real Lab Host VM with real
+per-Lab measurements. Treating a placeholder as a marketing/sales number would be actively
+misleading; the whole point of the model is that the real number gets filled in once, from
+measurement, after deployment (`docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md` step 4 is the
+trigger to re-measure and update it if the per-Lab footprint later changes).

docs/ENGINEERING_SOP.md

@@ -90,35 +90,34 @@ To add/reorder a stage: edit `rootnode-path.json` directly, referencing existing
 
 ## 5. Student identity — what's built, what isn't
 
-**Built (backend only, as of this doc):**
+**Current state (Phase 5A — see `docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md` for the
-- `students` table: `student_id` (`student-NNN`, sequential), `display_name`, `cohort`,
+full design):**
-  `created_at`, `last_active_at`.
+- `students.student_id` (`student-NNN`, sequential) is still the FK every other table hangs off
-- Login model: **instructor roster, pick-your-name, no password.** `POST /api/roster` adds
+  of and never changes across an account's lifecycle, including conversion.
-  a student (instructor-only action, currently unauthenticated at the API level — see
+- Real credentials: `email` and `login_identifier` (independently optional/unique — email for
-  "not built" below). `GET /api/roster?cohort=X` lists names for a dropdown.
+  older/self-paced students, an instructor-assigned login ID for younger ones), `password_hash`
-  `POST /api/login` just validates the id exists and returns it; the frontend is expected to
+  reusing `instructors`' exact `scrypt` machinery (`src/auth/password.js`). A student with
-  hold onto `student_id` and send it back as an `X-Student-Id` header on every request.
+  neither field set is a plain **pilot** account, unchanged since Phase 2A: `POST /api/login
-- `progress` table now keyed on `(student_id, scenario_id)` instead of `scenario_id` alone —
+  { student_id }`, no password, instructor-roster-only.
-  each student's completion/attempts/hints are isolated. Verified by smoke test: two students
+- First login/reset is one shared, single-use, hashed, time-limited token mechanism
-  answering the same scenario don't affect each other's state, and pre-migration data was
+  (`students-auth.js`) — instructors never see or set a student's actual password.
-  preserved under `student_id = 'unassigned'`.
+- Sessions are the same Postgres-backed `sessions` table instructors use (opaque, HttpOnly,
-- Exam `sessions` now carry `student_id` (set at session start), so exam history and the
+  hashed, revocable, CSRF-protected); login is rate-limited the same Postgres-backed way as
-  timer-reset-on-submit logic are correctly scoped per student instead of leaking across
+  instructor login (Phase 4.1), now also applied to students.
-  whoever's browser hits the API next.
+- Per-student `progress`/`scenario_attempts`/exam `sessions` isolation (below) and real
+  concurrent multi-student Lab isolation (Phase 1's Provisioner/k3s-per-student model) were both
+  already in place well before this — Phase 5A only changed how a student proves who they are,
+  not how their data or Labs are isolated.
 
-**Not built yet — do not assume these exist:**
+**Not built — deliberate non-goals, not gaps (Phase 5A §37):**
-- **No frontend UI.** No login screen, no name picker, no roster-add form. The API works;
+- No public self-registration — accounts are still instructor-created/converted/imported only.
-  nothing calls it yet. This is the next slice.
+- No email delivery infrastructure — password reset works via an instructor-issued token today;
-- **No auth on `/api/roster` POST.** Anyone who can reach the API can add a "student."
+  self-service reset returns a real token only in non-production, by design.
-  Fine for a dev instance, not fine once this is reachable by actual students — needs an
+- No Parent/Guardian portal — the schema doesn't block adding one later, but none exists.
-  instructor-only gate before that happens (ties into the instructor dashboard, roadmap
+
-  item 3).
+**Historical note, kept for context:** earlier drafts of this section described a pre-Phase-2A
-- **No concurrency / isolation.** Exam sessions are still global-single-active
+state (`X-Student-Id` header, unauthenticated roster POST, one shared cluster) that no longer
-  (`WHERE status='active' LIMIT 1` — see `server.js`), and there is one shared k3s cluster
+reflects any deployed version of this codebase; that content has been superseded by the above.
-  for the whole instance. Per-student identity ≠ per-student environment. **Multiple students
-  cannot yet use this platform at the same time without stepping on each other.** That's
-  roadmap item 4 (isolation) and it gates real classroom/minors use, same as the plan called
-  out — this SOP is not claiming that gate is cleared.
 
 ## 6. Schema migration pattern
 
@@ -146,9 +145,11 @@ Order agreed 2026-08-17/18. Status as of this doc:
    (§12), student-facing tabs (My Learning/Practice/Assessments/Capstone/Achievements) at
    `frontend/src/platform/`.
 3. Student accounts / persistent progress — **done**, Postgres-authoritative as of Phase 2B
-   (§9), sessions persistent as of Phase 3A (§10). Roster name-picking is still classroom/pilot
+   (§9), sessions persistent as of Phase 3A (§10). Real student credentials (email or
-   identification, not strong student auth — documented, not silently glossed over
+   login-identifier + password, first-login/reset token flow) closed the remaining "pick your
-   (docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md §7).
+   name" gap as of Phase 5A — see §5 above and
+   docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md. Pilot (no-login) accounts still work
+   unchanged for anyone not yet converted.
 4. Instructor dashboard — **done** (§9/§10). Full admin portal at `/admin`: dashboard, cohort
    and student management, classroom Prepare/Destroy Labs, per-instructor accounts with
    owner/admin/instructor roles, audit log.
@@ -280,3 +281,79 @@ Full design: `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`.
 - **Known limitations:** see the Phase 3B doc §20 — no automatic abandoned-attempt detection, no
   per-cohort needs-attention threshold configuration, raw-JSON pool_spec authoring in the admin
   UI (correct, not yet a guided form).
+
+## 13. Production Infrastructure (Phase 4)
+
+Full design: `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`, plus detail docs
+`PRODUCTION_DEPLOYMENT.md`, `PRODUCTION_NETWORK_SECURITY.md`, `BACKUP_DR.md`,
+`OBSERVABILITY.md`, `CAPACITY_PLANNING.md`, `PRODUCTION_VALIDATION_CHECKLIST.md`, and 12
+`RUNBOOK_*.md` files, all under `docs/`. Everything lives in `infra/` (Ansible, Terraform,
+Compose, firewall, monitoring, backup, CI, load-test tooling).
+
+- **What it is:** the infrastructure-as-code needed to run rn-platform-01/rn-lab-host-01 on a
+  real Hetzner/Proxmox host — buildable and locally testable now, even though the dedicated
+  server hasn't been purchased. Status is explicitly **PHASE 4 BUILD COMPLETE / PRODUCTION
+  DEPLOYMENT PENDING HARDWARE**, not "deployed."
+- **Where the boundary held:** the Platform VM / Lab Host VM security split from
+  `docs/LAB_ARCHITECTURE.md` is codified in firewall rulesets and mutual-TLS Docker config, not
+  weakened anywhere.
+- **Known gap surfaced, then resolved in Phase 4.1:** the Gateway's Lab-reachability mechanism
+  didn't survive the move to two real separate VMs — flagged explicitly rather than silently
+  fixed, then actually built and simulation-tested (a Lab Host Router,
+  `platform/lab-router/`) in Phase 4.1. See §13 below.
+- **Validated without the hardware:** real `tofu validate`, `ansible-playbook --syntax-check`,
+  `docker compose config`, `caddy validate`, `promtool`/`amtool`, and a genuine end-to-end
+  backup→encrypt→restore→verify cycle against disposable local Postgres with the real
+  production migrations applied. None of this proves the real Hetzner network boundary — that
+  needs `docs/PRODUCTION_VALIDATION_CHECKLIST.md` walked against the real server.
+
+## 13. Pre-Production Hardening (Phase 4.1)
+
+Full design: `docs/PHASE4_1_PREPRODUCTION_HARDENING.md`, plus
+`docs/PRODUCTION_NETWORK_SECURITY.md` (routing resolution), `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`,
+`docs/SECURITY_SCANNING.md`.
+
+- **What it is:** closes the Gateway-to-Lab cross-VM gap Phase 4 flagged (a Lab Host Router,
+  `platform/lab-router/`, plus a routing abstraction in the Gateway,
+  `platform/gateway/src/lab-endpoint.js`), then hardens the application layer found along the
+  way: abandoned-attempt handling (`scenario_attempts.status = 'abandoned'`, never a punitive
+  failure), a Postgres-backed login rate limiter (survives a restart, ready to be
+  shared/multi-instance), real pagination on audit/attempt/assessment-session history, a
+  structured assessment authoring UI (owner-only raw JSON remains as a fallback), a minimal
+  `role_in_cohort` enforcement (`teacher`/`viewer`), a `/api/version` endpoint, and admission
+  control (a clean 503 "capacity full" instead of a 500 once a configured cap is reached).
+- **Tested, not just designed:** a real two-Docker-network routing simulation
+  (`infra/scripts/simulate-two-host/`, 13/13), a destructive Postgres restore drill
+  (`infra/backup/test-destructive-restore-drill.sh`), a Lab Host loss/rebuild drill
+  (`infra/scripts/drills/test-lab-host-loss.js`), a deployment rollback drill, a CI-fails-closed
+  drill (which found and fixed a real pre-existing `set -e` bug in both smoke-test scripts —
+  see that drill's own output), and a full fake-cohort dress rehearsal
+  (`infra/scripts/drills/test-fake-cohort-rehearsal.js`) — all passing, all in
+  `infra/scripts/drills/` or `infra/backup/`.
+- **Known limitation carried forward:** real remote/self-paced delivery needs actual student
+  authentication (today's `student_id`-only login relies on instructors controlling who gets a
+  valid id) — see `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md` for the full accounting of what
+  already works vs. what's genuinely Phase 5 scope. **Closed in Phase 5A, below.**
+
+## 14. Student Identity & Remote Classroom (Phase 5A)
+
+Full design: `docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md`, plus
+`docs/RUNBOOK_STUDENT_ACCOUNT_LIFECYCLE.md` and `docs/RUNBOOK_REMOTE_CLASSROOM.md`.
+
+- **What it is:** two deliverables built on the Phase 4.1 milestone. (A) Real student accounts —
+  email-or-login-identifier + password, reusing `instructors`' exact credential machinery, a
+  single first-login/reset token mechanism, deactivation enforced per-request — replacing pilot
+  "pick your name" identification without touching a single existing pilot student's progress.
+  (B) A remote classroom control center — scheduled/live/ended sessions, derived (not stored)
+  presence via heartbeat, a live instructor dashboard combining real Provisioner Lab state and
+  real academic records, a help-request queue, guidance-only announcements/current-activity, and
+  a generic external meeting-URL field. Explicitly **not** video/voice/chat/shell-takeover —
+  RootNode owns identity and coordination, an external service owns the call.
+- **Tested, not just designed:** `platform/test/e2e-phase5a.js` (21 checks: the two-student RBAC
+  matrix and full password-security checklist, including a real Platform API restart proving
+  rate-limit state survives it) and `infra/scripts/drills/test-phase5a-rehearsal.js` (15 checks:
+  5 pilot students with seeded real history, a live classroom, help requests, **a real
+  mid-rehearsal API restart** proving classroom/session/presence recovery, then converting two
+  students and confirming zero orphaned data) — both against a live `docker-compose.dev.yml`
+  stack, not mocks. The full pre-existing regression suite was re-run given how much shared code
+  this phase touched (`server.js`, `admin-routes.js`, `sessions.js`, `rbac.js`).

docs/LAB_ARCHITECTURE.md

@@ -486,3 +486,24 @@ All four open questions this doc originally ended with are resolved — see the
 at the top and §2–§5 above. Next: `docs/PHASE1_PLATFORM_FOUNDATION.md` — the concrete build
 plan for the platform foundation (Postgres environment registry, provisioner skeleton,
 networking, egress, resource policy, reaper) against these decisions.
+
+## Phase 4: this boundary in production
+
+Phase 4 (`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`) codifies the Platform/Lab-Host split this
+document designed as two real, separately-firewalled Proxmox VMs (`rn-platform-01`,
+`rn-lab-host-01`) connected by two separate mTLS channels (Provisioner↔dockerd CONTROL plane,
+Gateway↔Lab Host Router DATA plane) and a tightly scoped firewall allowance — see
+`docs/PRODUCTION_NETWORK_SECURITY.md`. Phase 4 surfaced a gap in the Gateway's `env.internal_ip`
+mechanism for reaching a Lab container (it assumes Gateway and Lab share a Docker daemon, which
+breaks once they're genuinely on separate VMs); Phase 4.1
+(`docs/PHASE4_1_PREPRODUCTION_HARDENING.md`) built and simulation-tested the fix, a Lab Host
+Router (`platform/lab-router/`).
+
+## Phase 5A: reused, not modified
+
+Phase 5A (`docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md`) added real student accounts and a
+remote classroom control center on top of everything this document describes, without changing
+any of it — the Provisioner, the `environments` state machine, the Lab Host Router, disposable
+Lab lifecycle, and the Platform/Lab-Host VM boundary are all untouched. The classroom dashboard
+reads Lab status the exact same way every other admin view does (`provisioner.listForStudent`,
+the same state enum) rather than inventing a second one.

docs/OBSERVABILITY.md

@@ -0,0 +1,64 @@
+# Observability
+
+Self-hosted Prometheus + Grafana + Alertmanager + Loki/Promtail, deployed via
+`infra/docker/platform/docker-compose.production.yml` on rn-platform-01, with lightweight agents
+(node-exporter, cadvisor, promtail) also on rn-lab-host-01. No paid SaaS.
+
+This is operational observability — host/container/database health, uptime, resource pressure.
+It is deliberately separate from the application's own instructor learning analytics
+(Phase 3B's cohort/scenario/student dashboards) — different audience (ops vs. instructors),
+different data (infrastructure vs. learning outcomes), and mixing them would make both worse.
+
+## What's real today
+
+- **Host metrics** (both VMs): node-exporter — CPU, memory, disk, network.
+- **Container metrics** (Lab Host): cadvisor — per-container CPU/memory/network, including
+  student Lab containers (`rootnode-lab-*`).
+- **Postgres metrics**: postgres-exporter — connections, transaction rate, table sizes.
+- **Uptime/liveness**: blackbox-exporter probing `/healthz` on the Gateway
+  (`platform/gateway/src/server.js`) and the Platform API (`platform/api/src/server.js`), both
+  added this phase specifically for this purpose, plus the Provisioner's pre-existing `/health`.
+- **Logs**: Loki + Promtail, scraping Docker container logs and journald from both VMs, with a
+  redaction pipeline stage that strips `password|token|secret=`-pattern values before they're
+  stored (`infra/monitoring/promtail/`).
+- **Dashboards**: Grafana, reachable only via `https://<origin>/grafana/` and only from
+  `management_cidr` (see `docs/PRODUCTION_NETWORK_SECURITY.md`). One starter dashboard,
+  `infra/monitoring/grafana/provisioning/dashboards/rootnode-overview.json` — 8 panels (Platform
+  + Lab Host CPU/memory/disk, Postgres up/connections, running-Lab-count via cadvisor label
+  match [best-effort, caveated], uptime probes).
+
+## What's NOT built yet (honest accounting, not silently assumed)
+
+Application-level metrics — active student count, Lab provisioning success/failure rate,
+Gateway request latency/error rate, per-endpoint API metrics — would need real instrumentation
+added to the Platform API/Gateway/Provisioner (e.g. a `/metrics` endpoint via `prom-client`).
+`infra/monitoring/prometheus/prometheus.yml.j2` has a commented-out scrape job as the landing
+spot for this, explicitly marked as a follow-up, not wired to anything real yet.
+
+## Alerting
+
+`infra/monitoring/prometheus/rules/rootnode.yml` — two severities:
+
+**Critical** (fires fast, pages): PlatformAPIDown, GatewayDown, PostgresDown, LabHostDown,
+BackupFailed, DiskCriticallyFull.
+
+**Warning** (informational, slower repeat): LabHostHighMemory, HighDiskUsage, BackupAging.
+(RepeatedProvisioningFailures and ExcessiveFailedLogins are present as inert placeholders —
+they need the not-yet-built application metrics above before they can fire on anything real.)
+
+Never alerts on an individual student's scenario failure or a single Lab crash — those are
+expected, routine outcomes of normal usage, not operational incidents.
+
+`infra/monitoring/alertmanager/alertmanager.yml.j2` — critical alerts route separately with a
+30-minute repeat interval, everything else at 4 hours, one webhook receiver
+(`vault_alertmanager_webhook_url` — points at whatever channel gets configured at deploy time,
+e.g. a Slack/Discord/Matrix webhook), and one inhibit rule so a LabHostDown critical suppresses
+its own dependent warnings instead of paging twice for the same root cause.
+
+## Validation performed (local, real tooling — not hardware-dependent)
+
+`infra/monitoring/validate-monitoring.sh` wraps: `promtool check config` (Prometheus config),
+`promtool check rules` (alert rules — 11 rules found, valid), `amtool check-config`
+(Alertmanager — 1 inhibit rule, 2 receivers, valid). These prove syntactic/semantic correctness
+of the configs, not that the real network paths between rn-platform-01 and rn-lab-host-01 work —
+that requires the real hardware and is covered by `docs/PRODUCTION_VALIDATION_CHECKLIST.md`.

docs/PHASE4_1_PREPRODUCTION_HARDENING.md

@@ -0,0 +1,254 @@
+# Phase 4.1 — Pre-Production Hardening
+
+**Status: PHASE 4.1 COMPLETE. PRE-PRODUCTION HARDENING COMPLETE. READY FOR HARDWARE DEPLOYMENT.**
+
+Phase 4 built the production infrastructure package with the real Hetzner/Proxmox hardware not
+yet purchased, and flagged one genuine architectural gap: the Gateway's mechanism for reaching
+a student Lab assumed Gateway and Lab shared a Docker daemon, which is false the moment Labs run
+on a genuinely separate `rn-lab-host-01` VM. Phase 4.1's job was to close that gap and harden
+everything else discovered along the way, still without the real hardware.
+
+## 1. The core fix: a Lab Host Router
+
+**Before:** Gateway → `http://${env.internal_ip}:${env.internal_port}` directly — a Docker
+bridge-local address, unreachable across a real VM boundary.
+
+**After:** Gateway → (mode-aware) → either `direct` (unchanged, dev/single-host) or `router`
+(production): a small new service, `platform/lab-router/`, runs on the Lab Host itself,
+authenticates the Gateway via mTLS (the same private CA the Docker TLS control plane already
+used, extended with two new leaf certs), resolves an opaque routing token to a local container
+purely from `docker ps` label state (no Postgres access, no call back to the Provisioner), and
+reverse-proxies HTTP and WebSocket traffic to it.
+
+The decision of which mode a given `lab_host` uses is data-driven
+(`lab_hosts.routing_mode`/`router_endpoint`, `platform/provisioner/migrations/004_lab_hosts_routing.sql`),
+resolved once per request in `platform/gateway/src/lab-endpoint.js`'s `labTarget()` — the only
+place in the Gateway that knows this distinction exists. Every proxy/fetch call site in
+`platform/gateway/src/server.js` goes through it.
+
+**Exact `env.internal_ip` replacement:** `labTarget(env, token)` returns `{ url, headers, agent,
+mode }`. In `direct` mode, `url` is exactly the old `http://internal_ip:internal_port`. In
+`router` mode, `url` is the lab_host's `router_endpoint`, `headers` carries
+`X-RootNode-Route-Token: <token>` (computed server-side from the Provisioner-verified
+environment row — never trusted from the browser), and `agent` is an `https.Agent` configured
+with the Gateway's mTLS client identity.
+
+## 2. No per-student host port
+
+Deliberately not solved by publishing a host port per Lab (the originally-flagged, rejected
+approach). The Router resolves entirely from Docker's own label state on its own host — no
+port-per-student anything, no dynamic firewall rule per Lab.
+
+## 3. Lab Host Router — design
+
+See `platform/lab-router/src/server.js`'s header comment for the full "narrow by design" list:
+one route (proxy-by-token), requires mTLS (no bearer-token fallback), never exposes Docker API/
+container IDs/network info to its caller, never provides student-to-student routing (each
+request resolves to exactly the one container matching its own token).
+
+## 4. Multi-Lab-Host ready
+
+Routing is resolved per `lab_host_id` — adding `rn-lab-host-02` later is one new `lab_hosts` row
+(its own `routing_mode`/`router_endpoint`/TLS material), no Gateway code change. No scheduler
+was built (explicitly out of scope) — the Provisioner still picks "the one enabled lab_host"
+(`selectLabHost`), unchanged from Phase 1.
+
+## 5. Local development compatibility
+
+`dev-local` (the Unix-socket lab_host used by `docker-compose.dev.yml`) stays `routing_mode:
+direct` — zero behavior change for the T480/any local dev setup, confirmed by the full existing
+regression suite passing unmodified (§29 below) and by every new drill in this phase running
+against the same dev stack. The standalone single-container Lab image build/boot is unaffected
+(it never touches the Gateway or Provisioner at all).
+
+## 6-9. Simulation, HTTP/WS validation, security tests, routing token security
+
+`infra/scripts/simulate-two-host/` — a real two-Docker-network simulation (not mocks): Gateway
+attached ONLY to an isolated bridge with no route to any Lab's bridge network, Lab Host Router
+host-networked (mirrors `egress-proxy`'s existing pattern) so it can reach whatever per-Lab
+bridge the Provisioner creates. 13/13 real assertions pass, including:
+
+- Gateway genuinely cannot open a TCP connection to a Lab's Docker-internal IP (proven from
+  inside the Gateway container's own network namespace via a raw connect attempt).
+- Gateway reaches the Lab correctly through the Router — HTTP AND a real WebSocket terminal
+  session running a genuine `kubectl get nodes` inside the Lab.
+- Alice/Bob isolation (own lab ALLOW, other's DENY — HTTP and WebSocket both), garbage tokens,
+  anonymous requests, and destroyed-environment tokens are all rejected through the real code
+  path.
+- The Router refuses any connection with no client certificate — mTLS is the real
+  authentication mechanism.
+
+Routing token security reviewed: already CSPRNG since Phase 1 (`generateRoutingToken`), never
+logged in plaintext (the Router's `tokenFingerprint` truncates), redacted in every API response
+except the two narrow internal reads that legitimately need it, and — critically — the header
+the Router trusts is computed server-side from the Provisioner-verified environment row, never
+read from the browser's own request, so a browser cannot forge routing by supplying its own
+`X-RootNode-Route-Token`.
+
+## 10. Control plane vs data plane
+
+Kept strictly separate — see `docs/PRODUCTION_NETWORK_SECURITY.md`'s dedicated section. Student
+browser traffic is never proxied through the Docker API at any point.
+
+## 11. Firewall update
+
+`infra/firewall/rn-lab-host-01.rules`'s old speculative wide port-range allowance replaced with
+one narrow rule: `platform_ip` → the Router's port only (`8443` default, configurable).
+
+## 12. Self-paced / remote delivery architecture check
+
+`docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`. Short version: student-driven provisioning,
+cohort-less students, and "idle reaper destroys → progress survives → student returns → fresh
+Lab → curriculum continues" all already worked before this phase and are now proven end-to-end
+by two new drills. The one real gap for actual remote/self-paced delivery is student
+authentication (today's bare `student_id` login assumes an instructor-controlled roster) — not
+built here, explicitly Phase 5 scope.
+
+## 13. Abandoned attempt cleanup
+
+`scenario_attempts.status = 'abandoned'` (already a valid value since Phase 3B, never
+previously set) now gets set on: environment destroy/reset (student- or instructor-triggered),
+logout, session expiry (piggybacked on the existing hourly session sweeper — no new background
+job), and switching to a different scenario while one is still open. Never a punitive marker —
+`result` stays NULL, history is preserved, instructors see an honest "abandoned (reason)"
+instead of a phantom "still in progress."
+
+## 14. Gateway session cache
+
+Reviewed — already a tight 2-second TTL since Phase 3A, specifically to bound the
+revocation-propagation window. Decision: **keep it, don't add push-based invalidation** — the
+added coupling (Platform API would need to know how to reach the Gateway) isn't justified by
+shrinking an already-2-second window further. Documented reasoning, not a silently-accepted gap.
+
+## 15. Rate limiting
+
+Login rate limiting moved from an in-memory `Map` (reset on every restart, not shareable across
+instances) to Postgres (`login_attempts` table, `migrations/016_login_attempts.sql`) —
+`platform/api/src/security.js`'s `tooManyAttempts()`. No Redis added. Restart behavior
+confirmed for real: `platform/test/e2e-phase3a.js`'s rate-limiting test passes after a genuine
+Platform API restart in the same run.
+
+## 16. Pagination
+
+Real server-side pagination (`platform/api/src/pagination.js`) on: the audit log (was a flat
+`LIMIT 200`), student attempt history (was `LIMIT 100`), and assessment session history (was
+**fully unbounded** — the one genuine growth risk found). Scenario analytics was deliberately
+left as-is — grouped by scenario id, naturally bounded around ~100 rows, no real growth risk to
+paginate against. Frontend (`AuditView.jsx`, `AssessmentsView.jsx`, `StudentsView.jsx`) updated
+to match; `frontend` build confirmed clean.
+
+## 17. Needs-attention configuration
+
+`NEEDS_ATTENTION_FAIL_THRESHOLD`/`NEEDS_ATTENTION_STALE_DAYS` moved from hardcoded constants to
+env-var-configurable (matching the existing session-TTL pattern), sensible defaults unchanged.
+
+## 18. Assessment authoring UX
+
+Raw-JSON `pool_spec` textarea replaced with a structured editor
+(`frontend/src/admin/AssessmentsView.jsx`'s `PoolSpecEditor`): required scenarios and random
+pools picked from a real scenario catalog (`GET /api/admin/scenario-catalog`, new), grouped by
+curriculum stage, via native multi-select — no new UI framework. An owner-only "advanced" raw
+JSON view remains for anything the structured editor doesn't cover.
+
+## 19. Cohort role-in-cohort
+
+`cohort_instructors.role_in_cohort` existed since Phase 2B but was never read. Implemented
+minimally: `teacher` (unchanged full capability) vs `viewer` (read-only — dashboards/analytics
+still visible, but Prepare/Destroy Labs and provisioning a student's environment are now 403'd
+server-side, `rbac.js`'s `requireCohortWriteAccess`/`requireStudentCohortWriteAccess`).
+
+## 20. Backup / restore destructive drill
+
+`infra/backup/test-destructive-restore-drill.sh` — real owner/admin/cohort/3-students/progress/
+attempts/assessment/achievements seeded through the real Platform API, encrypted-backed-up, the
+disposable Postgres container **genuinely destroyed** (not just restored into a second
+container), a fresh Postgres created from nothing, restored, and the real Platform API booted
+again against the restored database and used to log in as the restored admin over HTTP. All
+data verified present; never touches the developer's real database.
+
+## 21. Lab Host rebuild drill
+
+`infra/scripts/drills/test-lab-host-loss.js` — a student's Lab container is destroyed out from
+under the Provisioner (no clean destroy call, simulating total Lab Host loss). Proves: Platform
+stays up throughout, the stale environment is never presented as healthy once the Provisioner's
+real `reconcileOnStartup` reconciliation runs, academic progress is untouched, and the student
+can launch a genuinely new Lab and continue.
+
+## 22. Deployment rollback test
+
+`infra/scripts/drills/test-deployment-rollback.sh` — builds a known-good and a deliberately
+broken gateway image, "deploys" each via the real deterministic-tag mechanism, proves the smoke
+test catches the regression, computes the rollback target with the exact same
+`deploy-history.log` logic `rollback-production.sh` uses, and proves the rollback recovers.
+Documents explicitly that image rollback never implies a DB migration rollback.
+
+## 23. CI/CD failure tests
+
+`infra/scripts/drills/test-ci-fails-closed.sh` — proves all six required failure modes (test/
+syntax failure, frontend build failure, image build failure, config validation failure,
+migration preflight failure, staging smoke test failure) genuinely exit nonzero. **Found and
+fixed two real pre-existing bugs**: `staging-smoke-test.sh` and `production-smoke-test.sh` both
+had a classic `set -e` gotcha (`curl ... && echo "OK"` as one statement — a failing command
+that isn't the last in an `&&` chain doesn't trigger `-e`) that let an unreachable/broken
+endpoint fall through silently instead of failing the smoke test. Fixed in both files.
+
+## 24. Security scanning
+
+`infra/ci/gitea-workflows/build-and-test.yml`'s new `security-scan` job — npm audit, gitleaks
+(secrets), hadolint (Dockerfiles), shellcheck (shell scripts), trivy (container image CVEs).
+All free/open-source. Severity policy documented in `docs/SECURITY_SCANNING.md`: CRITICAL/HIGH
+blocks the build, MEDIUM/LOW is reported only. `security-scan` gates `build-images` the same
+way `regression` does.
+
+## 25. Version / release visibility
+
+`GET /api/version` (Platform API) — app version (`package.json`), deployed Git SHA
+(`GIT_SHA` env var, set to the same value as the image tag in the production compose file), and
+the most recently applied migration as a schema-version stand-in. Unauthenticated, minimal, no
+infrastructure detail.
+
+## 26. Admission control failure mode
+
+`resource-policy.yaml`'s new (disabled-by-default) `admission_control.max_concurrent_labs` —
+when set, the Provisioner checks it BEFORE touching Docker at all, so a full host fails cleanly
+(503, `CAPACITY_EXCEEDED`) instead of failing mid-provision. Relayed by the Platform API as a
+clean "Lab capacity is temporarily full. Try again shortly." — never a raw 500. A new
+`GET /capacity` Provisioner endpoint feeds the instructor dashboard's capacity display.
+
+## 27. Fake cohort dress rehearsal
+
+`infra/scripts/drills/test-fake-cohort-rehearsal.js` — one continuous narrative: instructor
+login, cohort + 4 students, curriculum assignment, Prepare Labs, students working different
+scenarios, hints, a deliberate failed validation, instructor analytics, a Lab reset, a Lab
+reaped (same consequence as the real idle reaper — see the drill's own comment on why the real
+multi-minute timer isn't waited out), the student returning and continuing, an exam
+assigned/taken, and end-of-class Lab destruction — progress verified intact after every
+destructive step. 14/14 passing.
+
+## 28. Documentation
+
+This file, `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`, `docs/SECURITY_SCANNING.md` (new).
+Updated: `docs/LAB_ARCHITECTURE.md`, `docs/PRODUCTION_NETWORK_SECURITY.md`,
+`docs/PRODUCTION_DEPLOYMENT.md`, `docs/PRODUCTION_VALIDATION_CHECKLIST.md`, `docs/BACKUP_DR.md`,
+`docs/ENGINEERING_SOP.md`, `README.md`, `platform/README.md`.
+
+## 29. Regression
+
+Re-ran the full existing suite after every application-code change in this phase:
+`e2e-phase2a.js` (10/10), `e2e-phase2b.js` (12/12), `e2e-phase3a.js` (21/21 — including a real
+audit-log pagination-shape fix and confirming rate-limit persistence across a genuine restart),
+`e2e-phase3b.js` (24/24). Frontend production build clean. Real Lab image builds/boots (both
+dev-compose and standalone single-container). `tofu validate`, `ansible-playbook
+--syntax-check`, production Compose config (both platform and lab-host), `caddy validate`,
+`promtool`/`amtool`, and the backup/restore mechanism all re-validated clean after the
+infrastructure changes in this phase (new `lab-router` service, new firewall rule, new Ansible
+tasks). No knowingly-unexplained regression was left in this codebase at the end of this phase.
+
+## What Phase 4.1 deliberately did NOT build
+
+Production deployment itself, Hetzner/Proxmox installation, public course signup, payments,
+subscriptions, parent portal, marketing site, full LMS, video hosting, email automation,
+certificates, course marketplace, mobile app, a multi-host scheduler, an AI tutor, invasive
+proctoring, held scenario content, student email/password authentication (documented as the one
+real gap for self-paced delivery, not built). All explicit non-goals per this phase's
+authorization.

docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md

@@ -0,0 +1,187 @@
+# Phase 4 — Production Infrastructure Package
+
+**Status: PHASE 4 BUILD COMPLETE. PRODUCTION DEPLOYMENT PENDING HARDWARE.**
+
+The dedicated Hetzner server this is designed for hasn't been purchased yet. Everything in this
+document and the `infra/` tree it describes is buildable, testable (against local
+Docker/Postgres/disposable certs), and reviewable **now** — none of it requires the real
+hardware to exist. What genuinely can't happen until the hardware exists is called out
+explicitly throughout, never silently assumed to already work.
+
+This document is the map. Detail docs: `docs/PRODUCTION_DEPLOYMENT.md` (step-by-step),
+`docs/PRODUCTION_NETWORK_SECURITY.md` (network/firewall/TLS design), `docs/BACKUP_DR.md`
+(backup/restore/disaster recovery), `docs/OBSERVABILITY.md` (metrics/logs/alerts),
+`docs/CAPACITY_PLANNING.md` (sizing model), and the `docs/RUNBOOK_*.md` files (14 topics,
+command-oriented).
+
+## 1. What was built
+
+```
+infra/
+├── proxmox/          manual bare-metal install docs + cloud-init template definition
+├── terraform/         VM creation (rn-platform-01, rn-lab-host-01) — real `tofu validate` pass
+├── ansible/            all host configuration — real `ansible-playbook --syntax-check` pass
+│   ├── roles/          common, firewall, docker-tls, platform, lab-host, monitoring-agent, backup
+│   ├── group_vars/     production.yml, staging.yml, vault template
+│   └── site.yml        top-level playbook, correct role ordering
+├── docker/            production Compose files, Caddyfile, Postgres tuning — real `docker
+│                       compose config` + `caddy validate` pass
+├── firewall/           iptables-restore rulesets for all 3 hosts — structural lint pass
+├── monitoring/          Prometheus/Alertmanager/Loki/Promtail/Grafana config — real `promtool`
+│                        + `amtool` validation pass
+├── backup/              Postgres backup/restore/verify scripts — REAL end-to-end test against
+│                        disposable local Postgres, including the actual production migrations
+├── scripts/             deploy/rollback/smoke-test/CI/loadtest/capacity-model tooling
+├── ci/                  Gitea Actions workflow (build/test/stage automatic, prod manual)
+└── environments/        dev/staging/production config templates, no secrets committed
+```
+
+18 new docs (this file + 5 detail docs + 12 runbooks), 4 updated (`LAB_ARCHITECTURE.md`,
+`ENGINEERING_SOP.md`, `README.md`, `platform/README.md`).
+
+## 2. Architecture (unchanged from the design, now codified)
+
+```
+Hetzner Dedicated Server
+└── Proxmox VE
+    ├── rn-platform-01   (frontend, Platform API, Gateway, Provisioner, Postgres,
+    │                      Prometheus/Alertmanager/Grafana/Loki, backup cron, Caddy)
+    └── rn-lab-host-01   (Docker Engine w/ TLS, egress proxy, disposable student Labs,
+                           node-exporter/cadvisor/promtail)
+```
+
+The Platform VM / Lab Host VM security boundary from `docs/LAB_ARCHITECTURE.md` is the one
+thing this phase was explicitly told never to weaken — it's codified, not just described, in
+`infra/firewall/` and `infra/ansible/roles/docker-tls/`.
+
+## 3. VM specifications
+
+| | rn-platform-01 | rn-lab-host-01 |
+|---|---|---|
+| Baseline (default in `infra/terraform/variables.tf`) | 4 cores, 16 GB RAM, 100 GB disk | 24 cores, 96 GB RAM, 500 GB disk |
+| Recommended (once real host is ~16c/32t/128GB) | same — Postgres/API/Gateway/Provisioner don't need much | majority of remaining host resources |
+| Max-growth | scale RAM with Postgres `shared_buffers` if instructor/cohort count grows a lot | scale with measured concurrent-Lab capacity (`docs/CAPACITY_PLANNING.md`) |
+| Configurable via | `terraform.tfvars`' `platform_vm`/`lab_host_vm` objects | same |
+
+NUMA/CPU pinning: not configured — not useful at this scale (a handful of VMs on one physical
+host, not a dozen contending for cache locality). Revisit only if real load testing shows a
+reason to.
+
+## 4. Storage decision framework
+
+Not a final answer — `infra/environments/production/provisioner-config/README.md` and this
+section exist so the final ZFS layout is a deployment-time decision made against real disks,
+not guessed now. Candidate layouts (`infra/terraform/variables.tf`'s `vm_disk_datastore` names
+whichever one gets chosen):
+
+- **4× NVMe**: mirrored vdevs (RAID10-equivalent) — two mirror pairs, one for
+  Postgres/platform data (I/O-sensitive, small), one for Lab Host disposable data
+  (throughput-sensitive, large, never needs redundancy since nothing on it is precious).
+- **2× SSD/NVMe**: one mirrored vdev, logical dataset separation (ZFS datasets, not separate
+  pools) between platform (`{{ rootnode_data_root }}/postgres`, `/backups`) and Lab Host data.
+- **Mixed SSD/HDD**: only if the purchased server's actual disks require it — HDD for
+  disposable Lab scratch space is acceptable (it's disposable), never for Postgres.
+
+**Never single-disk for Postgres data**, in any layout — a mirror is the floor, not a nice-to-have.
+
+Pre-deployment disk inspection checklist (model, size, SMART, power-on hours, wear/endurance,
+sector errors, temperature, firmware): `docs/RUNBOOK_DISK_FAILURE.md`'s inspection section —
+run once, against the real drives, before committing to a layout.
+
+## 5-8. Network, firewall, Docker TLS, public ingress
+
+Full detail: `docs/PRODUCTION_NETWORK_SECURITY.md`. Summary: two private VM networks, Docker TLS
+(mutual, already application-supported since Phase 1 — this phase only needed the certificate
+generation/deployment automation, not new app code), a single public origin (Caddy, automatic
+HTTPS) proxying only to the Gateway, and one genuinely new architectural item this phase
+surfaced and addressed at the network-policy layer: **Gateway-to-Lab reachability across two
+real VMs** — see that doc's dedicated section for what's built vs. what still needs a small,
+reviewed Provisioner/Gateway code change before real go-live.
+
+## 9-10. Production auth/cookie config, secrets
+
+`infra/environments/production/*.env.template` set `COOKIE_SECURE=1`/`TRUST_PROXY=1` explicitly
+(dev's unsafe defaults are never inherited). Secrets: Ansible Vault (already bundled with
+Ansible — no new tool to install/trust for this project's secret surface), template at
+`infra/ansible/group_vars/production.vault.yml.template`. Nothing is committed —
+`infra/.gitignore` covers `*.env`, TLS private key material, vault plaintext, Terraform state.
+
+## 11-13. Postgres, backup, DR
+
+Full detail: `docs/BACKUP_DR.md`. Summary: conservative tuning baseline (not hand-tuned before
+real RAM is known), daily encrypted backups with an off-host adapter point (Storage Box/S3/any
+SSH target), and a **real, tested** backup→encrypt→restore→verify cycle
+(`infra/backup/test-backup-restore.sh`) — run against disposable local Postgres with the actual
+production migrations applied, not a mock.
+
+## 14-17. Observability, logging, alerting, uptime
+
+Full detail: `docs/OBSERVABILITY.md`. Prometheus + Grafana + Loki/Promtail (the user's stated
+Prometheus/Grafana preference, honored) + Alertmanager, all self-hosted, no paid dependency.
+Deliberately does not duplicate the application's own instructor learning analytics
+(Phase 3B) — operational observability and learning analytics stay separate concerns with
+separate audiences, per the explicit instruction. Application-level metrics (active
+students/Labs, provisioning failures, Gateway error rate) are a documented, **not-yet-built**
+follow-up — see that doc's honest accounting of what's real today (host/container/Postgres
+metrics, HTTP healthz probes) vs. what needs app instrumentation later.
+
+## 18-19. CI/CD, image strategy
+
+`infra/ci/gitea-workflows/build-and-test.yml` — automatic test/build/tag/staging-deploy,
+**manual** production promotion (`infra/scripts/deploy-production.sh`), matching the stated
+preference. SHA-tagged images always, semantic tags on deliberate releases, never `:latest` in
+staging/production. Full detail: `infra/docker/IMAGE_STRATEGY.md`.
+
+## 20-21. Ansible, Terraform
+
+`infra/ansible/site.yml` — idempotent roles, doesn't assume the VMs already exist (Terraform
+creates them first). Real `ansible-playbook --syntax-check` pass. `infra/terraform/` — real
+`tofu validate` pass against the actual `bpg/proxmox` provider schema. Neither installs Proxmox
+itself or builds the base VM template — both stay deliberate manual, documented, one-time steps
+(`infra/proxmox/README.md`), per the explicit "don't over-automate the fragile one-time step"
+instruction.
+
+## 22-23. Environment config, staging
+
+`infra/environments/{dev,staging,production}/` — explicit separation, production never
+inherits dev defaults. Staging is designed to run in the homelab or a resource-constrained
+slice of the real server later — no second dedicated server required.
+
+## 24-25. Load testing, capacity model
+
+`infra/scripts/loadtest/` (real Prepare-Labs-driven boot test + `docker stats` sampling) and
+`infra/scripts/capacity-model.js` (a real, working worksheet — tested against example inputs,
+produces CPU-vs-memory-bound recommended/warning/hard-admission-control thresholds). No
+concurrency number is hard-coded as a marketing figure; the example config in
+`infra/environments/production/capacity.yml.example` is explicitly flagged as placeholder data.
+
+## 26-27. Validation checklist, runbooks
+
+`docs/PRODUCTION_VALIDATION_CHECKLIST.md` — one master checklist spanning hardware through
+academy-workflow verification. 12 runbooks (`docs/RUNBOOK_*.md`) — first deployment, normal
+deploy, rollback, Postgres restore, VM rebuild (both hosts), Docker TLS rotation, TLS renewal,
+owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host
+resource pressure.
+
+## What this phase deliberately did NOT touch
+
+Learning system design, curriculum content, RBAC model, session architecture — all Phase 3A/3B,
+unchanged. No parent portal, billing, held scenarios, or production OIDC provider integration.
+RootNode production infrastructure is dedicated to RootNode Academy — never folded into
+existing Synthos production servers, and canonical source stays Gitea, never GitHub `kubekosh`.
+
+## Known gap surfaced during this phase (not silently papered over) — RESOLVED in Phase 4.1
+
+Building the real production Compose/network config surfaced that the Gateway's existing
+mechanism for reaching a Lab container (`env.internal_ip`, a Docker-bridge-local address) only
+works when Gateway and Lab share a Docker daemon — true in every dev/single-host setup so far,
+**not** true once Labs run on a genuinely separate `rn-lab-host-01`. At the time this phase
+closed, the network-policy half was built but the application half was flagged as an explicit,
+not-yet-built follow-up rather than guessed at mid-infra-phase.
+
+**Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1) built and proved the application
+half** — a Lab Host Router (`platform/lab-router/`) plus a `labTarget()` routing abstraction in
+the Gateway, real-tested against a genuine two-network Docker simulation
+(`infra/scripts/simulate-two-host/`), not just designed. See
+`docs/PRODUCTION_NETWORK_SECURITY.md`'s "Gateway-to-Lab reachability" section for the full
+resolution. This item no longer blocks real two-VM go-live.

docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md

@@ -0,0 +1,195 @@
+# Phase 5A — Student Identity & Remote Classroom
+
+**Status: PHASE 5A COMPLETE. STUDENT IDENTITY COMPLETE. REMOTE CLASSROOM FOUNDATION COMPLETE.**
+
+Phase 4.1 closed the last pre-production infrastructure gap without real hardware. Phase 5A's
+job — still without real hardware — was the last real *product* gap flagged along the way: pilot
+students identify themselves by picking a name from a list, and there was no way for an
+instructor to run a live, remote-capable class through the platform itself. This phase replaces
+pilot identification with real accounts and adds a remote classroom control center. It does
+**not** add video, voice, chat, or any A/V — those stay Meet/Zoom/Teams's job; RootNode owns
+identity, presence, curriculum guidance, Lab status, help requests, and instructor control.
+
+## 1. Student identity model
+
+`students` (migration `017_student_credentials.sql`) gained `email`, `login_identifier`,
+`password_hash`, `last_login_at`, and a single credential-token triple
+(`credential_token_hash`/`_purpose`/`_expires_at`) — all nullable, all additive. `email` and
+`login_identifier` are independently unique (partial unique indexes, `WHERE ... IS NOT NULL`)
+and both optional: an older/self-paced student gets an email, a younger instructor-managed
+student gets an instructor-assigned login ID, neither is required, and nothing here blocks a
+future `guardian_id` FK for a Parent Portal — deliberately not built this phase (§37).
+
+A student with **neither** field set is unchanged from before Phase 5A: a plain pilot row,
+identified with `POST /api/login { student_id }`. Setting either field (via account creation or
+`/convert`) is what makes an account "real." `password_hash` reuses `platform/api/src/auth/
+password.js` (scrypt, `salt:hash`) verbatim — the same function instructors use, no second
+hashing scheme.
+
+## 2. Credential lifecycle — one token mechanism, not two
+
+Activation ("first login") and password reset are the same problem: a time-limited, single-use
+secret that lets someone set a new password. `students-auth.js`'s `issueCredentialToken`/
+`consumeCredentialToken` implement it once (SHA-256-hashed at rest, plaintext returned exactly
+once in the API response, never logged), with `purpose` (`activation` | `reset`) distinguishing
+the two callers. Activation tokens live 7 days, reset tokens 1 hour. `consumeCredentialToken` is
+atomic: token verification, password hashing, `password_hash` write, and token-clearing happen
+in one statement — there is no window where a token is "spent" but the password isn't set.
+
+Instructors never see or choose a student's password (§5's "instructor creates account →
+temporary credential" flow): `POST /api/admin/students`, `/convert`, and `/reset-password` all
+return a token (activation or reset), which the instructor relays out of band (printed, read
+aloud, handed over). The admin UI surfaces the resulting `?activate=<token>`/`?reset=<token>`
+link once and never again — there is no "view credentials" screen anywhere.
+
+## 3. Password reset without email delivery
+
+No email-sending infrastructure exists yet, and this phase deliberately doesn't fake one.
+`POST /api/student/request-password-reset` always returns an identical generic response
+regardless of whether the identifier exists (anti-enumeration) and never echoes the token in
+production. In non-production (`isProduction` false, i.e. `COOKIE_SECURE !== '1'`) it also
+returns `dev_reset_token` so local/CI flows can complete the loop without a real inbox — a
+documented dev/test path, not a production feature. The realistic production path today is
+instructor-triggered: `POST /api/admin/students/:id/reset-password`. Real email delivery is a
+named, not-yet-built follow-up, not something silently promised.
+
+## 4. Session and login security — reused, not duplicated
+
+Students and instructors share the exact same `sessions` table and mechanism
+(`sessions.js`: opaque tokens, HttpOnly, SameSite, hashed at rest, revocable), differentiated
+only by `session_type`/`student_id` vs `instructor_id`. `studentSession` middleware now
+re-verifies `status = 'active'` on **every** request (previously only checked session
+existence) — deactivating a student takes effect immediately, matching `instructorSession`'s
+existing behavior, not just at next login (verified in `e2e-phase5a.js`).
+
+Rate limiting reuses Phase 4.1's Postgres-backed `tooManyAttempts` (`security.js`) with a
+`student:` key prefix — 8 attempts / 5 minutes, survives a Platform API restart because it's a
+table, not an in-memory Map (verified with a real restart in both `e2e-phase5a.js` and the
+rehearsal drill). CSRF reuses the existing double-submit cookie pattern (`rn_csrf_s`).
+
+**A real bug this closed:** the legacy `POST /api/login` (bare `student_id`, no password) only
+checked `password_hash` to decide whether an account had "moved on" from pilot mode. An account
+mid-conversion — instructor has issued an activation token, student hasn't set a password yet —
+has `credential_token_hash` set but `password_hash` still `NULL`, and was reachable through the
+old insecure path for that whole window. Found via testing, fixed by checking
+`student.password_hash || student.credential_token_hash` instead (`server.js`).
+
+## 5. Pilot migration
+
+No destructive migration. `017_student_credentials.sql` only adds nullable columns — every
+existing pilot row, its progress, attempts, achievements, and cohort membership are completely
+untouched at migration time. `POST /api/admin/students/:id/convert` assigns email/login_
+identifier and issues an activation token without touching `student_id` (the FK every other
+table hangs off of) or anything downstream of it. Verified directly, not just asserted: the
+Phase 5A rehearsal drill (`infra/scripts/drills/test-phase5a-rehearsal.js`) seeds real pre-
+existing progress/attempt/achievement rows for a pilot student, converts them, activates them,
+logs in the new way, and confirms `/api/progress` shows the exact same completed scenario —
+nothing orphaned, nothing re-created.
+
+## 6. Remote classroom — a fourth session concept
+
+Distinct from cohort membership, assessment sessions, and Lab environments: a scheduled,
+bounded teaching event. `018_classroom_sessions.sql` adds `classroom_sessions` (lifecycle
+`scheduled → live → ended`, or `cancelled`), `classroom_participants`, `classroom_events`
+(append-only audit), `classroom_announcements`, and `help_requests`. A partial unique index
+(`WHERE status = 'live'`) enforces at most one live classroom per cohort at the database level —
+the app layer catches the resulting `23505` and returns a clean `ALREADY_LIVE` (409), not a
+raw constraint error (verified in `e2e-phase5a.js`).
+
+## 7. Presence — derived, not stored
+
+`classroom.derivePresence(lastSeenAt, leftAt)` is a pure function computing `online` /
+`recently_active` / `offline` from a timestamp at read time — there's no status column that can
+go stale or need a background job to sweep. Heartbeats (`POST /api/classroom/:id/heartbeat`,
+client fires every 20-30s while the Classroom tab is open and joined) are throttled server-side
+(`WHERE last_seen_at IS NULL OR last_seen_at < now() - interval`), never depend on Lab terminal
+activity, and stop the moment the tab is closed or the student leaves — no invasive tracking.
+
+## 8. The classroom dashboard — real data, not a second state system
+
+`GET /api/admin/classroom-sessions/:id` is the one place that reaches beyond `classroom.js`: it
+joins participant/presence data with the Provisioner's actual `environments` state (same enum
+every other Lab-status view uses — `none`/`requested`/`provisioning`/`running`/`resetting`/
+`error`/`destroying`/`destroyed`) and a lightweight academic snapshot from the existing
+`progress`/`scenario_attempts` tables (current stage, most recent scenario, validation-fail
+count, hints used). Never inspects terminal command history. Nothing here is a second Lab-state
+or academic-state system — it's the existing ones, read the same way every other admin view
+already reads them.
+
+## 9. Help requests — a queue, not a chat
+
+`POST /api/classroom/:id/help-request { category, message }` → `open` → (instructor)
+`acknowledge` → `resolve`. That's the entire surface — explicitly no chat, no video, no remote
+shell takeover (§37). The instructor dashboard's help queue polls the same dashboard endpoint
+everything else does.
+
+## 10. Guidance, never control
+
+`setCurrentActivity` (current scenario) and announcements are exactly that — a link the student
+can choose to open, a message they can read. Nothing server-side redirects a student's browser
+or marks work complete on their behalf. The external meeting URL (`classroom_sessions.
+meeting_url`) is a generic field the student sees as a link — RootNode never proxies or embeds
+Meet/Zoom/Teams/anything else, so it isn't coupled to a single vendor.
+
+## 11. CSV roster import — whole-file, transactional
+
+`student-csv.js` hand-parses (no CSV dependency — the file format here is small and simple
+enough that pulling in a library would just be one more thing to audit) and validates the
+**entire** file before any database write. `POST /api/admin/students/import` then does every
+insert inside one Postgres transaction — one bad row rolls back the whole import rather than
+leaving a half-imported roster, which is the more dangerous failure mode of the two. Never
+accepts a `password`/`password_hash`/`credential_token_hash` column — activation tokens are
+always generated server-side, never taken from input.
+
+## 12. Observability
+
+`GET /metrics` is a hand-rolled Prometheus text-exposition endpoint (no `prom-client`
+dependency — a handful of counters/gauges don't need one) with real in-process counters for
+login success/failure and on-demand gauges for live-classroom/help-request/Lab-state counts.
+No student or instructor ID ever appears as a label value — every label is a bounded enum
+(status, presence, result), per the explicit no-PII/no-high-cardinality-labels instruction. The
+Prometheus scrape job for the Platform API (`infra/monitoring/prometheus/prometheus.yml.j2`),
+previously commented out, is now active.
+
+## 13. Frontend
+
+**Student side** (`frontend/src/platform/`): `LoginScreen` (credentialed, now the default entry
+point) with links to the legacy pilot entry and forgot-password; `SetPasswordScreen` (shared
+activate/reset UI, driven by `?activate=`/`?reset=` query params — no router library, matching
+this app's existing state-machine-per-shell convention); `ForgotPasswordScreen`; a `Classroom`
+tab (live-class banner, join, guidance-only current-activity link, own Lab status/launch reusing
+the My Learning tab's exact API, announcements, Need Help form); a `Profile` tab (identity,
+cohort, progress, achievements, active-session count, change-password).
+
+**Admin side** (`frontend/src/admin/`): `StudentsView` gained credential status in the roster
+table, a create-student form (email/login-ID/cohort), convert/reset-password/revoke-sessions
+controls per student, and a bulk CSV import form. A new `ClassroomView` is the control center:
+cohort picker → schedule/list classes → a live dashboard (participants with presence/Lab-status/
+current-work, help queue with acknowledge/resolve, current-activity setter, announcement
+poster, start/end/cancel, Prepare Labs).
+
+## 14. Testing — real, not asserted
+
+- `platform/test/e2e-phase5a.js` (21 checks): activation single-use, password security (§32 —
+  correct/wrong/unknown-identifier/deactivated/immediate-deactivation/rate-limit/rate-limit-
+  survives-restart/change-password/session-revocation/forgot-password anti-enumeration/admin
+  reset/CSRF), and the two-student RBAC matrix (§31 — student can't reach `/api/admin/*`,
+  assigned/unassigned/viewer instructor access), plus a classroom lifecycle smoke pass.
+- `infra/scripts/drills/test-phase5a-rehearsal.js` (15 checks): 5 pilot students, real seeded
+  progress/attempt/achievement history, a live classroom, help requests, announcements, **a real
+  Platform API restart mid-rehearsal** proving classroom/session/presence recovery, then
+  converting two students mid-rehearsal and confirming every bit of their prior history survived
+  under the new login.
+- Both run against a live `docker-compose.dev.yml` stack over real HTTP, not mocks.
+- The full pre-existing regression suite (Phase 1/2A/2B/3A/3B/4.1) was re-run unmodified given
+  how much shared code (`server.js`, `admin-routes.js`, `sessions.js`, `rbac.js`) this phase
+  touched — see the Phase 5A completion report for results.
+
+## 15. What was deliberately not built
+
+Per §37: video/voice conferencing, student-to-student chat, remote instructor shell takeover,
+screen recording or proctoring, public registration or course catalog, payments/Stripe, a Parent
+Portal, school/SIS integrations, real email delivery infrastructure, certificates, an AI tutor,
+a mobile app, a multi-Lab-host scheduler, and production deployment itself. Each is a plausible
+"while we're in here" addition and each was left alone on purpose — the schema and API surface
+built this phase don't block any of them later.

docs/PRODUCTION_DEPLOYMENT.md

@@ -0,0 +1,90 @@
+# Production Deployment
+
+Step-by-step, for when the real Hetzner host exists. See
+`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` for the architecture this executes.
+
+## 0. One-time, manual, not automated (deliberately)
+
+1. Install Proxmox VE on the bare-metal host — `infra/proxmox/README.md`.
+2. Inspect the real disks, pick a ZFS layout — `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4,
+   `docs/RUNBOOK_DISK_FAILURE.md`'s inspection checklist.
+3. Build the cloud-init VM template Terraform will clone —
+   `infra/proxmox/cloud-init/user-data.yaml` + `infra/proxmox/README.md`.
+4. Fill in `infra/terraform/terraform.tfvars` (copy from `.tfvars.example`) with the real
+   Proxmox API endpoint, node name, storage/datastore names, and chosen VM sizes.
+5. Fill in `infra/ansible/inventory/production.ini` with the real VM IPs, and
+   `infra/ansible/group_vars/production.yml` with `platform_ip`, `lab_host_ip`,
+   `management_cidr`, `proxmox_mgmt_cidr`.
+6. Copy `infra/ansible/group_vars/production.vault.yml.template` →
+   `production.vault.yml`, fill in every `vault_*` secret with real generated values, then
+   `ansible-vault encrypt group_vars/production.vault.yml`. Never commit the plaintext.
+
+## 1. Create the VMs
+
+```bash
+cd infra/terraform
+tofu init
+tofu plan    # review before applying against real hardware
+tofu apply
+```
+
+Creates `rn-platform-01` and `rn-lab-host-01` from the cloud-init template. Confirm both boot
+and are SSH-reachable from the management network before continuing.
+
+## 2. Generate TLS certificates (offline, once)
+
+```bash
+export ROOTNODE_TLS_BUNDLE_DIR=/secure/offline/path   # never inside the repo
+./infra/scripts/docker-tls/generate-docker-tls.sh
+./infra/scripts/docker-tls/verify-docker-tls.sh
+```
+
+Generates FOUR leaf certs under one CA (Phase 4.1 extended this from two): the CONTROL-plane
+pair (`server-*`/`provisioner-client-*`, for Provisioner↔dockerd) and the DATA-plane pair
+(`router-server-*`/`gateway-client-*`, for Gateway↔Lab Host Router — see
+`docs/PRODUCTION_NETWORK_SECURITY.md`'s "control plane vs data plane" section). The CA private
+key stays in `ROOTNODE_TLS_BUNDLE_DIR` and never touches either VM — only the CA cert (public)
+and each host's own leaf cert+key go to each host, both pairs deployed by the same `docker-tls`
+Ansible role into two separate directories (`/etc/rootnode/docker-tls/` and
+`/etc/rootnode/lab-router-tls/`).
+
+## 3. Configure both hosts
+
+```bash
+cd infra/ansible
+ansible-playbook -i inventory/production.ini site.yml --ask-vault-pass
+```
+
+Role order matters and is fixed in `site.yml`: `common` → `firewall` → `docker-tls` →
+`platform`/`lab-host` → `monitoring-agent` → `backup`. Firewall and TLS land before anything
+that depends on them.
+
+## 4. First application deploy
+
+```bash
+./infra/scripts/deploy-production.sh <image-tag>
+```
+
+Pulls the tagged images, runs the compose stack, applies Postgres migrations (automatic, on
+Platform API/Provisioner startup — see `docs/RUNBOOK_FAILED_MIGRATION.md` if one fails).
+
+## 5. Verify
+
+```bash
+./infra/scripts/production-smoke-test.sh
+```
+
+Curls `/healthz` on the Gateway and Platform API, confirms Postgres is reachable, confirms
+Prometheus/Grafana/Alertmanager are up. Then walk
+`docs/PRODUCTION_VALIDATION_CHECKLIST.md` in full before calling the deployment live.
+
+## Normal (non-first) deploys
+
+`docs/RUNBOOK_NORMAL_DEPLOY.md`. Rollback: `docs/RUNBOOK_ROLLBACK.md`.
+
+## What this document does NOT cover
+
+DNS/Cloudflare setup for the public origin (a one-time manual step outside this repo's scope —
+point the A/AAAA record at the real host and confirm Caddy issues its certificate). Which
+storage layout to pick (decision framework only, §4 of the master doc — the real answer depends
+on the real disks).

docs/PRODUCTION_NETWORK_SECURITY.md

@@ -0,0 +1,130 @@
+# Production Network & Security
+
+The Platform VM / Lab Host VM boundary is the core production control. This document is where
+every network decision that touches it gets written down. Nothing here weakens it.
+
+## Topology
+
+```
+Internet ──HTTPS(443)──▶ Caddy (rn-platform-01) ──▶ Gateway (internal bridge)
+                                                        │
+management_cidr ──SSH──▶ rn-platform-01                │
+management_cidr ──SSH──▶ rn-lab-host-01                 │
+proxmox_mgmt_cidr ──22/8006──▶ Proxmox host             │
+                                                        │
+rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd)          [CONTROL plane]
+rn-platform-01 (Gateway) ──mTLS:8443──▶ rn-lab-host-01 (Lab Host Router)     [DATA plane]
+rn-lab-host-01 (promtail) ──3100──▶ rn-platform-01 (Loki)
+rn-lab-host-01 (node-exporter/cadvisor) ◀──scrape── rn-platform-01 (Prometheus)
+```
+
+Firewall source of truth: `infra/firewall/{proxmox-host,rn-platform-01,rn-lab-host-01}.rules`
+(iptables-restore format, deployed by the `firewall` Ansible role). Validate with
+`infra/firewall/validate-firewall.sh`.
+
+## What is explicitly denied
+
+- Student Lab → another student Lab (no student-to-student traffic, ever).
+- Student Lab → Proxmox management (both the host firewall and the `lab_subnet_range` DENY
+  rule in `rn-lab-host-01.rules`).
+- Student Lab → Platform Postgres / Platform API / Gateway internals (logged DENY in
+  `rn-platform-01.rules` for anything from `lab_host_ip` to ports 5432/4100).
+- rn-lab-host-01 → Postgres (same rule — the Lab Host itself has no legitimate reason to reach
+  Postgres; only the Provisioner on rn-platform-01 does, and it talks to Postgres over the
+  internal Docker bridge, never across the VM boundary).
+- Any student egress except through the approved egress proxy (enforced by the Provisioner's
+  existing per-Lab `DOCKER-USER` chain manipulation from Phase 1 — unchanged by this phase).
+- Direct public exposure of the Provisioner, Postgres, Docker TLS port, Proxmox UI, or Grafana
+  (Grafana is reachable only via Caddy's `/grafana/*` path, itself restricted to
+  `management_cidr` by a `caddy` matcher — see `infra/docker/platform/Caddyfile.j2`).
+
+## Control plane vs data plane (Phase 4.1 §10)
+
+Two separate, never-conflated mTLS connections between the two VMs, deliberately kept apart
+both conceptually and operationally:
+
+| | Control plane | Data plane |
+|---|---|---|
+| What | Provisioner manages Lab containers | Gateway serves student browser traffic |
+| Path | Provisioner → dockerd (port 2376) | Gateway → Lab Host Router (port 8443) |
+| Who initiates | Provisioner only | Gateway only |
+| What it can do | Create/start/stop/remove containers, full Docker API | Reverse-proxy one HTTP/WS request to one already-running container by opaque token — nothing else |
+| Certs | `provisioner-client-*` / `server-*` | `gateway-client-*` / `router-server-*` |
+| Exposes Docker API to caller? | Yes (that's its job — Provisioner IS a Docker client) | **Never** — the Router never exposes Docker API, container IDs, or network info to its caller |
+
+The student browser's traffic is proxied at the Gateway/Router layer and never touches the
+Docker API at any point — the Router deliberately has no route that echoes back anything
+Docker-internal (see `platform/lab-router/src/server.js`'s header comment for the full "narrow
+by design" list).
+
+## Docker TLS (Provisioner ↔ Lab Host)
+
+Mutual TLS on port 2376, restricted at the firewall to `platform_ip` only. Application support
+already existed since Phase 1 (`platform/provisioner/src/docker.js`'s `buildClient()`); this
+phase added the operational half: `infra/scripts/docker-tls/generate-docker-tls.sh` (CA +
+server + client cert generation), the `docker-tls` Ansible role (deploys certs, configures
+`dockerd`'s `daemon.json`), and `docs/RUNBOOK_DOCKER_TLS_ROTATION.md` for renewal. The CA
+private key is generated offline and never committed or deployed to either VM — only the CA
+cert (to verify the peer) and each host's own leaf cert+key.
+
+## Public ingress
+
+One origin, Caddy, automatic HTTPS. `reverse_proxy gateway:4200` — Caddy talks only to the
+Gateway; nothing else is proxied. WebSocket upgrade works automatically (Caddy's default
+`reverse_proxy` behavior, no special config needed for the terminal WebSocket path). Whether to
+front this with Cloudflare or expose the origin directly is left as a deployment-time decision
+(both work with Caddy's ACME flow; Cloudflare adds DDoS absorption at the cost of Cloudflare
+being able to see plaintext unless "Full (strict)" mode is used) — not decided here since it
+doesn't affect anything else in this document.
+
+## Gateway-to-Lab reachability across two real VMs — RESOLVED in Phase 4.1
+
+Phase 4 flagged this as the single most important open item; Phase 4.1
+(`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1) closed it. Kept here as the historical record of
+what the gap was and how it was closed, since the reasoning matters for anyone extending this
+later (e.g. adding a second Lab Host).
+
+**The gap, as originally found:** the Gateway used to reach a running Lab container via
+`env.internal_ip` — a Docker bridge-local address recorded by the Provisioner when it creates
+the container. That only works when Gateway and the Lab share a Docker daemon, true in dev and
+any single-host setup, false the moment Labs run on a genuinely separate `rn-lab-host-01`.
+
+**The fix — a Lab Host Router (`platform/lab-router/`):** a small, narrow service that runs on
+rn-lab-host-01 (host-networked, like `egress-proxy`, so it can reach whatever per-Lab bridge
+network the Provisioner creates) and does exactly one thing: given an opaque routing token over
+mTLS, find the ONE local container labeled with that token (`docker ps --filter
+label=rootnode.routing-token=<token>` — resolved purely from local Docker state, no Postgres,
+no call back to the Provisioner) and reverse-proxy the request to it, HTTP and WebSocket both.
+
+The Gateway now decides `direct` vs `router` per `lab_host` (`lab_hosts.routing_mode` +
+`router_endpoint`, `platform/provisioner/migrations/004_lab_hosts_routing.sql`) via
+`platform/gateway/src/lab-endpoint.js` — a single function, `labTarget(env, token)`, that every
+proxy/fetch call site in `server.js` goes through instead of building
+`http://${env.internal_ip}:${env.internal_port}` itself. `dev-local` stays `direct` (T480/local
+dev unaffected, zero behavior change there); a real `rn-lab-host-01` is configured `router`.
+
+**Firewall (`infra/firewall/rn-lab-host-01.rules`):** the old speculative
+`lab_gateway_port_range` allowance (which assumed option (a), publishing per-Lab ports) was
+replaced with a single, narrow rule: `platform_ip` → port `{{ lab_router_port | default('8443')
+}}` only. No wide port range, no per-Lab port publishing at all — the Router is the only thing
+the Gateway ever talks to on rn-lab-host-01 for Lab traffic.
+
+**mTLS:** the Gateway↔Router pair uses the SAME private CA as the Provisioner↔dockerd pair
+(`infra/scripts/docker-tls/generate-docker-tls.sh`, extended in Phase 4.1 to also issue
+`router-server`/`gateway-client` leaf certs), deployed to a separate directory
+(`/etc/rootnode/lab-router-tls/`) by the same `docker-tls` Ansible role.
+
+**Proven, not just designed:** `infra/scripts/simulate-two-host/` builds a real two-network
+Docker simulation (Gateway genuinely isolated on its own bridge, no route to any Lab's bridge)
+and runs 13 real assertions against it — Gateway cannot reach a Lab's Docker-internal IP
+directly, but reaches it correctly through the Router (HTTP and a real WebSocket
+`kubectl get nodes`), Alice/Bob isolation holds, garbage/destroyed tokens are rejected, and the
+Router itself refuses any connection without the Gateway's client certificate. See that
+directory's README for exactly what is and isn't proven by a same-daemon simulation versus real
+separate hardware.
+
+## Cookie / auth production settings
+
+`infra/environments/production/*.env.template`: `COOKIE_SECURE=1`, `TRUST_PROXY=1`. Dev's
+insecure defaults (`COOKIE_SECURE` unset, no proxy trust) are never inherited — production env
+files are separate files, not a dev file with overrides layered on.

docs/PRODUCTION_VALIDATION_CHECKLIST.md

@@ -0,0 +1,71 @@
+# Production Validation Checklist
+
+Walk this once, top to bottom, before calling a production deployment live. Every item here
+needs the real hardware — this is the "DEPLOY LATER" half of Phase 4, not something this repo
+can prove in advance.
+
+## Hardware / Proxmox
+- [ ] Disk inspection done, layout chosen and matches what's actually in the server (`docs/RUNBOOK_DISK_FAILURE.md` inspection checklist, `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4)
+- [ ] Proxmox VE installed, reachable only from `management_cidr`/`proxmox_mgmt_cidr`
+- [ ] VM template built from real cloud-init (`infra/proxmox/README.md`)
+- [ ] `tofu apply` created both VMs; both boot and are SSH-reachable
+
+## Platform VM
+- [ ] `ansible-playbook site.yml` completed clean, all roles
+- [ ] `docker compose ps` shows every service healthy (postgres, provisioner, api, gateway, reverse-proxy, prometheus, alertmanager, grafana, loki, promtail, exporters)
+- [ ] `curl https://<origin>/healthz` (via Gateway) returns 200
+- [ ] `curl http://localhost:4300/healthz` (Platform API, from the VM) returns `{"ok":true,"db":"ok"}`
+- [ ] TLS certificate issued and valid for the real domain
+
+## Lab Host VM
+- [ ] Docker daemon TLS verified: `docker --tlsverify --tlscacert=... -H tcp://<lab_host_ip>:2376 info` succeeds from the Platform VM, fails from anywhere else
+- [ ] Lab Host Router mTLS verified: a request to `https://<lab_host_ip>:8443` with no client cert is rejected; with the Gateway's real client cert it isn't (mirrors `infra/scripts/simulate-two-host/test-remote-routing.js`'s "direct public access: NOT AVAILABLE" case, now against the real host)
+- [ ] Provisioner successfully creates a real Lab against the real Lab Host
+- [ ] Gateway can actually reach that Lab's terminal (HTTP UI load + WebSocket `kubectl` command) — application mechanism built and simulation-tested in Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1/§6/§7); this line is the real-hardware confirmation that the same result holds across the actual two VMs, not two Docker networks on one daemon
+- [ ] Egress proxy correctly restricts a Lab's outbound traffic (Provisioner's per-Lab `DOCKER-USER` chain)
+
+## Security
+- [ ] `infra/firewall/validate-firewall.sh` run against the real hosts with real `iptables-restore --test`, not just the structural lint
+- [ ] Confirm from an unauthorized IP: Postgres, Provisioner, Docker TLS port, Proxmox UI are all unreachable
+- [ ] Confirm Student Lab → another Lab is blocked; confirm Student Lab → Proxmox is blocked
+- [ ] `COOKIE_SECURE`/`TRUST_PROXY` active (check a real Set-Cookie header over HTTPS)
+
+## Academy (application-level smoke test)
+- [ ] Owner bootstrap login works
+- [ ] Instructor can create a cohort, roster a student
+- [ ] Student can log in, boot a Lab, complete a scenario, see progress
+- [ ] An assessment can be assigned, taken, and graded end-to-end
+- [ ] Instructor analytics dashboards populate with real data
+
+## Backup
+- [ ] Manual `backup-postgres.sh` run succeeds against real Postgres, real off-host target reachable
+- [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness — Phase 4.1's `infra/backup/test-destructive-restore-drill.sh` proved the full mechanism (realistic data, real storage destruction, restore, real Platform API boot) locally; this line is the real-hardware confirmation
+- [ ] Cron job confirmed scheduled (`crontab -l`)
+
+## Observability
+- [ ] Grafana dashboard shows real data for both VMs
+- [ ] Trigger one alert deliberately (e.g. stop a service) and confirm it fires and reaches the webhook
+- [ ] Loki has logs flowing from both VMs
+
+## Pre-production hardening (Phase 4.1 — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md)
+- [ ] `resource-policy.yaml`'s `admission_control.max_concurrent_labs` set from a real measured
+      `infra/scripts/capacity-model.js` run against the real Lab Host, not left disabled
+- [ ] A deliberate over-capacity attempt returns the clean 503 "Lab capacity is temporarily
+      full" message, not a 500 — confirms the real host's admission control matches the local test
+- [ ] `docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md`'s steps still make sense once capacity numbers are real
+
+## Student identity & remote classroom (Phase 5A — see docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md)
+- [ ] `COOKIE_SECURE=1` in the real production environment — confirms `/api/student/
+      request-password-reset` never returns `dev_reset_token` outside of dev/CI
+- [ ] A real activation/reset link (`?activate=`/`?reset=`) round-trips against the production
+      domain, not just `localhost` — query-param parsing, no router library in play
+- [ ] Student login rate limiting still 429s after a restart against the real production
+      Postgres, same as the local `e2e-phase5a.js` check
+- [ ] `GET /metrics` is reachable by the real Prometheus instance (`infra/monitoring/
+      prometheus/prometheus.yml.j2`'s `rootnode-platform-api` job) and no student/instructor ID
+      appears in any label value
+- [ ] `docs/RUNBOOK_STUDENT_ACCOUNT_LIFECYCLE.md` and `docs/RUNBOOK_REMOTE_CLASSROOM.md`'s steps
+      still make sense walked against the real admin portal
+
+Only once every box above is checked against the real server does the deployment graduate from
+"infrastructure package built" to "actually in production."

docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md

@@ -0,0 +1,83 @@
+# Remote & Self-Paced Delivery Architecture
+
+Phase 4.1 §12. This is an architecture review, not a new product — no self-registration,
+payments, or public enrollment is built here (explicit non-goal). The question this document
+answers: does the infrastructure and application assume "an instructor always provisions the
+Lab," and if so, where, and what would need to change for self-paced delivery to work.
+
+## The short answer: less of that assumption exists than expected
+
+RootNode Academy's Lab lifecycle was built student-driven from Phase 1 — `POST
+/api/environment/launch` is called by the STUDENT'S OWN session, not by an instructor action.
+"Prepare Labs" (`POST /api/admin/cohorts/:id/prepare-labs`) is a classroom **convenience** that
+front-loads several students' launches at once before a synchronous class session, not the only
+path to a running Lab. This was true before Phase 4.1 and confirmed still true by this phase's
+own regression suites (`platform/test/e2e-phase2a.js` roster's two students with `cohort_id:
+null` and takes them through login → curriculum → launch → scenario work with zero cohort
+involvement) and by two new Phase 4.1 drills built specifically to exercise the remote/returning
+-student narrative:
+
+- `infra/scripts/drills/test-lab-host-loss.js` — student launches, makes progress, their Lab is
+  destroyed out from under them (standing in for an idle-reaper destroy), Platform stays up,
+  progress is untouched, they launch again unprompted and continue.
+- `infra/scripts/drills/test-fake-cohort-rehearsal.js` — the same "returns later, fresh Lab,
+  curriculum continues" narrative as part of a fuller classroom rehearsal.
+
+Both pass today, with zero application code changes needed for the self-paced-relevant parts of
+this flow specifically — see the Phase 4.1 report for the full run results.
+
+## What already works, unmodified
+
+- **Student-initiated provisioning** — no instructor step is required between "student wants to
+  work" and "Lab exists." (`platform/api/src/server.js`'s `/api/environment/launch`.)
+- **Cohort-less students** — `cohort_id` is nullable on student creation
+  (`POST /api/admin/students`); a student with no cohort logs in, sees the curriculum, launches
+  Labs, and makes progress identically to a cohorted one. Confirmed by inspection AND by the
+  existing regression suite, not just by schema reading.
+- **Idle reaper → academic state survives → return → continue** — the exact lifecycle this
+  section describes as the target self-paced experience is what Phase 1's reaper +
+  Phase 2B's Postgres-backed academic data (separate from the disposable Lab) were already
+  designed to produce. Phase 4.1 proved it end-to-end, twice (see above).
+- **Curriculum content itself is not cohort-gated** — `GET /api/curriculum/path` reads the one
+  curriculum path file directly (`curriculum.loadCurriculumPath()`); it never checks
+  `curriculum_assignments` or cohort membership at all. A self-paced student sees the full
+  curriculum with no instructor action required to "unlock" it.
+
+## What's genuinely missing for a real self-paced product (Phase 5, not built here)
+
+1. **No self-registration / public enrollment.** A student row is created only by an instructor
+   (`POST /api/admin/students`, admin/owner RBAC-gated). There is no public "sign up" surface.
+   This is almost certainly the right call to leave alone for now — it's explicitly a Phase 5+
+   product/business decision (course commerce, is it free, who's allowed to enroll), not an
+   infrastructure gap.
+
+2. ~~**No real student authentication.**~~ **Closed in Phase 5A.** Students now have real
+   credentials (email+password or login-identifier+password, `password_hash`/`scrypt` reusing
+   `instructors`' exact `src/auth/password.js`), a first-login/reset token flow, and Postgres-
+   backed rate-limited/CSRF-protected sessions on the same `sessions` table instructors use — see
+   `docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md`. The old bare-`student_id` `POST
+   /api/login` still works for not-yet-converted pilot rows only; it's rejected (409) the moment
+   an account has real credentials or a pending activation token.
+
+3. **`curriculum_assignments.student_id` is schema-ready but unwired.** The table has supported
+   a direct per-student curriculum assignment (independent of any cohort) since Phase 2B
+   (`CONSTRAINT curriculum_assignment_target CHECK (cohort_id IS NOT NULL OR student_id IS NOT
+   NULL)`) — but nothing in the application ever reads or writes that column today; there's only
+   one curriculum path, so nothing has needed to differentiate yet. Not a gap that blocks
+   self-paced delivery now (every student already sees the one path with no gate), but the
+   moment a second curriculum path exists, this is where per-individual assignment lands — the
+   schema was designed for exactly this and doesn't need a migration to support it.
+
+4. **No delivery-mode concept anywhere.** In-person, remote, and self-paced sessions aren't
+   distinguished in any data model today, and nothing currently needs them to be — capacity
+   planning, the reaper, and academic records all behave identically regardless of how a student
+   is attending. Worth being aware this is unmodeled, not worth building a taxonomy for before a
+   real product reason exists (YAGNI).
+
+## What Phase 4.1 deliberately did NOT do here
+
+No signup flow, no student password/email auth implementation, no per-student curriculum
+assignment route, no payments, no public course catalog, no held/premium scenario content. All
+explicit non-goals per this phase's authorization. Student authentication (item 2, above) was
+closed in Phase 5A; the rest remain open, unmodeled by design (YAGNI) until a real product
+reason exists.

docs/RUNBOOK_BACKUP_FAILURE.md

@@ -0,0 +1,31 @@
+# Runbook: Backup Failure
+
+Triggered by the `BackupFailed`/`BackupAging` alerts (`infra/monitoring/prometheus/rules/rootnode.yml`).
+
+```bash
+# 1. See what actually happened — the cron wrapper logs to syslog
+journalctl -u rootnode-backup --since "24 hours ago" -o cat
+# or, if the cron job never even ran:
+grep rootnode-postgres-backup /var/log/syslog
+
+# 2. Run it manually to see the failure live
+source /etc/rootnode/backup.env
+/opt/rootnode/backup/backup-postgres.sh
+```
+
+**Common causes:**
+- Disk full on `{{ rootnode_data_root }}/backups` — check `df -h`, prune more aggressively
+  (`BACKUP_RETENTION_DAYS`) or the ZFS dataset needs more space (see `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4).
+- Off-host target unreachable (`BACKUP_OFFSITE_TARGET`) — check the SSH key/credentials
+  haven't expired, the target host/bucket is reachable: `ssh <offsite-target-host> echo ok` or
+  the equivalent for whatever adapter is configured (`docs/BACKUP_DR.md`).
+- `BACKUP_ENCRYPTION_PASSPHRASE` missing/wrong in `/etc/rootnode/backup.env` — re-check against
+  the value in Ansible Vault (`vault_backup_encryption_passphrase`); a mismatch here means even
+  a "successful" old backup becomes unrestorable, so treat this specific failure mode as urgent.
+- The dump itself is suspiciously small (script's own sanity check tripped) — Postgres may be
+  unhealthy; check `docker exec rootnode-platform-postgres-1 pg_isready` first.
+
+**After fixing:** run the backup manually once to confirm success (metric updates immediately,
+alert clears within one Prometheus scrape interval), then confirm the cron job is still
+correctly scheduled (`crontab -l` as root, or re-run the `backup` Ansible role if the cron entry
+itself is suspect).

docs/RUNBOOK_DISK_FAILURE.md

@@ -0,0 +1,44 @@
+# Runbook: Disk Failure
+
+Applies once real hardware/ZFS exists — see `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4 for
+the storage decision framework this assumes (mirrored vdevs, never single-disk for Postgres).
+
+```bash
+# 1. Confirm the failure
+zpool status -v
+smartctl -a /dev/sdX   # the suspect drive
+
+# 2. If it's a mirrored vdev (the required layout for anything holding Postgres data — see §4),
+#    the pool stays online in DEGRADED state. No immediate data-loss risk, but no redundancy
+#    either until replaced — treat DEGRADED as urgent, not routine.
+zpool status   # confirm DEGRADED, not FAULTED/UNAVAIL (those mean act immediately, possibly
+                # restore from backup if the redundant mirror side is also compromised)
+
+# 3. Replace the physical drive, then:
+zpool replace <pool> <old-device> <new-device>
+zpool status   # watch the resilver progress
+
+# 4. If a whole pool was lost (both mirror sides failed simultaneously — the scenario mirroring
+#    exists to make vanishingly unlikely, but plan for it anyway):
+#    - rn-lab-host-01's disposable data: nothing to recover, just rebuild the VM
+#      (docs/RUNBOOK_VM_REBUILD.md)
+#    - rn-platform-01's Postgres data: restore from the latest off-host backup
+#      (docs/RUNBOOK_POSTGRES_RESTORE.md) — this is exactly the scenario off-host backup
+#      copies exist for; a local-only backup on the SAME failed pool would be useless here
+```
+
+**Pre-deployment disk inspection checklist** (run once when the real drives are known, before
+they go into service — `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4):
+
+```bash
+for d in /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1; do   # adjust to real device names
+  echo "== $d =="
+  smartctl -i "$d" | grep -E 'Model|Serial|Capacity|Firmware'
+  smartctl -A "$d" | grep -iE 'power_on_hours|reallocated|pending|uncorrectable|wear|percentage.used|temperature'
+  smartctl -H "$d" | grep -i overall
+done
+```
+Record model, size, power-on hours, wear/endurance percentage, reallocated/pending sector
+counts, temperature, and firmware version for every drive before committing to a ZFS layout —
+a drive already showing wear or reallocated sectors shouldn't be trusted with Postgres data
+even in a mirror.

docs/RUNBOOK_DOCKER_TLS_ROTATION.md

@@ -0,0 +1,31 @@
+# Runbook: Docker TLS Rotation
+
+Rotate on a schedule (recommend yearly, well before the ~2.25-year cert lifetime
+`generate-docker-tls.sh` defaults to) or immediately if a client cert may have leaked.
+
+```bash
+# 1. Generate a fresh bundle (new CA too — simplest, avoids any question about whether the old
+#    CA's private key was ever exposed)
+cd infra/scripts/docker-tls
+./generate-docker-tls.sh --out-dir ./out-$(date +%Y%m%d) --lab-host-ip <real-ip> --lab-host-dns rn-lab-host-01.internal
+./verify-docker-tls.sh ./out-$(date +%Y%m%d)
+
+# 2. Redeploy — this restarts dockerd on rn-lab-host-01 (brief interruption to any in-flight
+#    Docker API call from the Provisioner, NOT to already-running student Labs — their
+#    containers keep running through a dockerd restart)
+cd ../../ansible
+ROOTNODE_TLS_BUNDLE_DIR=../scripts/docker-tls/out-$(date +%Y%m%d) \
+  ansible-playbook site.yml -i inventory/production.ini --tags docker-tls --ask-vault-pass
+
+# 3. Verify the Provisioner can still reach rn-lab-host-01
+curl -s http://localhost:4100/health   # from rn-platform-01
+```
+
+**Troubleshooting a failed rotation:**
+- `x509: certificate signed by unknown authority` — the new ca.pem wasn't deployed to both
+  sides, or an old one is cached; re-run the docker-tls role on BOTH hosts.
+- Provisioner logs `ECONNREFUSED` on port 2376 — dockerd didn't actually restart with the new
+  daemon.json; check `systemctl status docker` on rn-lab-host-01 and the ExecStart override
+  (`infra/ansible/roles/docker-tls/templates/docker-override.conf.j2`) actually applied.
+- Firewall — confirm `infra/firewall/rn-lab-host-01.rules`' 2376 rule still matches
+  rn-platform-01's actual IP (didn't change as part of this rotation, but worth a sanity check).

docs/RUNBOOK_FAILED_MIGRATION.md

@@ -0,0 +1,35 @@
+# Runbook: Failed Migration
+
+Both Platform API and Provisioner apply their own migrations automatically on startup
+(`platform/api/src/db.js`, `platform/provisioner/src/db.js`), file-tracked in
+`schema_migrations`, wrapped in a transaction per file — a migration either fully applies or
+fully rolls back, it can't leave a table half-altered.
+
+**If a deploy fails because a migration errored:**
+
+```bash
+# 1. Check what actually happened
+docker logs rootnode-platform-api-1 --tail 50   # the error is right there — migration file + message
+docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
+  "SELECT * FROM schema_migrations ORDER BY applied_at DESC LIMIT 10;"
+
+# 2. Since the failing migration's transaction rolled back, the schema is still in its
+#    PREVIOUS valid state — the app just won't start until the migration succeeds. Fix the
+#    root cause (usually a real data issue the migration didn't anticipate, or the
+#    cross-service ordering dependency — Provisioner's migrations must run before Platform
+#    API's, since scenario_attempts has a real FK to environments; see
+#    docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md).
+
+# 3. Redeploy once fixed — the migration runner is idempotent and will just pick up where it
+#    left off; already-applied migrations are skipped.
+./infra/scripts/deploy-production.sh <same-or-fixed-tag>
+```
+
+**If you need to roll back the CODE but the migration already succeeded** (schema changed,
+code that depended on it is being reverted): rolling back application images
+(`docs/RUNBOOK_ROLLBACK.md`) does NOT undo a Postgres migration — there is no automatic
+down-migration mechanism (deliberate, matches the existing check-before-alter, additive-only
+migration convention documented in `docs/ENGINEERING_SOP.md` §6). If the old code genuinely
+can't run against the new schema, restore from the pre-migration backup instead
+(`docs/RUNBOOK_POSTGRES_RESTORE.md`) rather than trying to hand-write a down-migration under
+pressure.

docs/RUNBOOK_FIRST_DEPLOYMENT.md

@@ -0,0 +1,34 @@
+# Runbook: First Production Deployment
+
+Prerequisites: Proxmox installed (`infra/proxmox/README.md`), VMs created
+(`infra/terraform/`), DNS pointed at rn-platform-01's public IP.
+
+```bash
+# 1. Generate Docker TLS material (offline, on your own machine — never on either VM)
+cd infra/scripts/docker-tls
+./generate-docker-tls.sh --out-dir ./out --lab-host-ip <real-lab-host-ip> --lab-host-dns rn-lab-host-01.internal
+./verify-docker-tls.sh ./out
+
+# 2. Fill in real values
+cd ../../ansible
+cp group_vars/production.vault.yml.template group_vars/production.vault.yml
+$EDITOR group_vars/production.vault.yml   # real passwords/secrets — openssl rand -hex 32 etc.
+ansible-vault encrypt group_vars/production.vault.yml
+$EDITOR group_vars/production.yml         # real IPs, domain, CIDRs
+$EDITOR inventory/production.ini          # real IPs
+
+# 3. Deploy
+ROOTNODE_TLS_BUNDLE_DIR=../scripts/docker-tls/out \
+  ansible-playbook site.yml -i inventory/production.ini --ask-vault-pass
+
+# 4. Bootstrap the owner account (once, by a human)
+ssh rootnode-deploy@rn-platform-01
+docker exec -it rootnode-platform-api-1 node bootstrap-owner.js [email protected] "Your Name"
+# (password comes from BOOTSTRAP_OWNER_PASSWORD in the environment, or the CLI prompt — see
+# platform/api/README.md)
+
+# 5. Verify
+../../infra/scripts/production-smoke-test.sh
+```
+
+Then work through `docs/PRODUCTION_VALIDATION_CHECKLIST.md` in full before calling this done.

docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md

@@ -0,0 +1,29 @@
+# Runbook: High Lab Host Resource Pressure
+
+Triggered by the `LabHostHighMemory`/`HighDiskUsage` warning alerts.
+
+```bash
+# 1. How many Labs are actually running, and against what admission-control threshold?
+curl -s http://localhost:4100/environments | jq '[.[] | select(.status=="running")] | length'
+node infra/scripts/capacity-model.js   # recommended/warning/hard thresholds
+
+# 2. If genuinely over capacity: this is expected pressure from real usage, not a bug.
+#    - Destroy idle/abandoned Labs early via /admin (per-cohort "Destroy Labs" after a class
+#      session ends, rather than waiting for the idle reaper's default window)
+#    - If a specific cohort's class just ended, use /admin -> Cohorts -> [cohort] -> Destroy
+#      Cohort Labs immediately rather than waiting
+
+# 3. If pressure is high with FEWER Labs running than capacity-model.js's threshold: something
+#    is consuming more per-Lab resources than the model assumes (a scenario doing something
+#    unusually heavy, a memory leak in a long-running Lab, zombie processes from a crashed
+#    reset). Check per-container usage:
+docker stats --no-stream | grep rootnode-lab-
+
+# 4. Update infra/environments/production/capacity.yml with fresh measured numbers
+#    (infra/scripts/loadtest/) if the per-Lab footprint has genuinely changed, so the
+#    admission-control threshold reflects reality going forward.
+```
+
+Do not silently raise resource limits in `provisioner-config/resource-policy.yaml` to make an
+alert go away without understanding why usage is higher than modeled — that just moves the
+same pressure onto host RAM/CPU directly instead of onto individual Lab limits.

docs/RUNBOOK_NORMAL_DEPLOY.md

@@ -0,0 +1,17 @@
+# Runbook: Normal Deploy
+
+For a routine code change already merged to `main` on the canonical Gitea repo.
+
+1. CI (`infra/ci/gitea-workflows/build-and-test.yml`) runs automatically: tests, image build/tag/push, staging deploy, staging smoke test.
+2. Review the staging smoke test result and, ideally, poke around `https://staging.<domain>` yourself.
+3. Promote to production (manual, deliberate):
+
+```bash
+cd infra/scripts
+./deploy-production.sh sha-<12-char-sha>    # or a release tag, e.g. v1.3.0
+./production-smoke-test.sh
+```
+
+4. Watch Grafana (`/grafana/`) for 10-15 minutes — error rates, Postgres connections, Lab Host
+   memory — before considering the deploy "settled."
+5. If anything looks wrong: `docs/RUNBOOK_ROLLBACK.md`.

docs/RUNBOOK_OWNER_RECOVERY.md

@@ -0,0 +1,27 @@
+# Runbook: Owner Account Recovery
+
+**Locked out with zero working owner accounts** (all deactivated, forgot the only owner's
+password, etc.):
+
+```bash
+ssh rootnode-deploy@rn-platform-01
+docker exec -it rootnode-platform-api-1 sh
+node bootstrap-owner.js [email protected] "Your Name"
+# bootstrap-owner.js refuses to run if an active owner already exists UNLESS the target email
+# belongs to an EXISTING instructor row — in that case it promotes them to owner. Point it at
+# any known instructor email to recover through them, or a brand-new email to create a fresh
+# owner outright if the instructors table itself is in an unknown state.
+```
+
+**Forgot password, account still active and you have SSH access:**
+
+```bash
+docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
+  "SELECT instructor_id, email, role, status FROM instructors WHERE role='owner';"
+# Then use another owner's account to reset via /admin -> Instructors -> [account] -> Reset
+# Password, OR re-run bootstrap-owner.js targeting that email (also resets the password).
+```
+
+**Never** hand-edit `password_hash` in the database — it's `scrypt(salt):hash` in a specific
+format (`platform/api/src/auth/password.js`); a malformed value just breaks login rather than
+setting a usable password. Always go through `bootstrap-owner.js` or the admin UI's reset flow.

docs/RUNBOOK_POSTGRES_RESTORE.md

@@ -0,0 +1,45 @@
+# Runbook: Postgres Restore
+
+**When to use:** Postgres corruption/data loss on rn-platform-01, or verifying a backup is
+actually good (do this periodically even without an incident — an untested backup is a
+hypothesis, not a backup).
+
+```bash
+# 1. Find the backup to restore
+ls -la /srv/rootnode-data/backups/    # or wherever backup_offsite_target points, if local copies are gone
+
+# 2. Create a FRESH target database — never restore over the live one directly
+docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
+  "CREATE DATABASE rootnode_platform_restore_test;"
+
+# 3. Restore into it
+source /etc/rootnode/backup.env   # for BACKUP_ENCRYPTION_PASSPHRASE
+/opt/rootnode/backup/restore-postgres.sh \
+  /srv/rootnode-data/backups/rootnode-platform-<timestamp>.sql.gz.gpg \
+  "postgres://rootnode_platform:$PGPASS@localhost:5432/rootnode_platform_restore_test"
+# type RESTORE when prompted
+
+# 4. Verify
+/opt/rootnode/backup/verify-restore.sh \
+  "postgres://rootnode_platform:$PGPASS@localhost:5432/rootnode_platform_restore_test"
+
+# 5a. If this was a drill: drop the test database
+docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
+  "DROP DATABASE rootnode_platform_restore_test;"
+
+# 5b. If this was a REAL incident: stop the app stack, rename the broken DB, rename the
+#     restored one into its place, restart.
+systemctl stop rootnode-platform
+docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
+  "ALTER DATABASE rootnode_platform RENAME TO rootnode_platform_broken_$(date +%s);"
+docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
+  "ALTER DATABASE rootnode_platform_restore_test RENAME TO rootnode_platform;"
+systemctl start rootnode-platform
+```
+
+Local, non-production dry run of this exact flow: `infra/backup/test-backup-restore.sh` (uses
+disposable containers, safe to run anytime, no real data involved).
+
+**Recovery point objective:** up to 24 hours of academic data (daily backup cadence) — see
+`docs/BACKUP_DR.md` for the full RPO/RTO discussion and why that's the honest number, not an
+aspirational one.

docs/RUNBOOK_REMOTE_CLASSROOM.md

@@ -0,0 +1,53 @@
+# Runbook: Running a Remote Classroom
+
+Covers scheduling, running, and recovering a live class through `/admin` → Classroom. See
+`docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md` for the underlying design. RootNode does
+**not** provide video/voice — the meeting URL field just links out to Meet/Zoom/Teams/whatever
+you already use for that.
+
+**Before class: schedule it**
+
+`/admin` → Classroom → pick the cohort → Schedule a class. Title is required; scheduled start,
+description, and a meeting URL are optional. Only one class can be `live` per cohort at a time —
+scheduling a second one is fine, starting it while another is live is rejected.
+
+**Right before class: Prepare Labs**
+
+Open the classroom → Prepare Labs. This provisions/warms every enrolled student's Lab ahead of
+time (same underlying flow as the cohort-level Prepare Labs button) so nobody is waiting on a
+cold start once class begins. If a Lab host is at capacity, that student shows as
+`capacity_waiting` in the results rather than failing silently — check back or ease off other
+concurrent load.
+
+**Start class**
+
+Open the classroom → Start class. Status moves to `live`; students now see a "LIVE NOW" banner
+on their Classroom tab and can join.
+
+**During class**
+
+- **Participants table**: presence (`online`/`recently_active`/`offline`, derived from each
+  student's last heartbeat — not a hard real-time signal, allow ~30s of lag), Lab status, and a
+  one-line current-work snapshot (stage, scenario, validation failures, hints used).
+- **Set current activity**: enter a scenario ID and Set. This only shows students a link — it
+  never redirects their browser or marks anything complete. Use it to say "everyone open this
+  next," not to force navigation.
+- **Post an announcement**: short messages, shown newest-first on the student side.
+- **Help queue**: students flag `lab_problem` / `dont_understand` / `validation_failing` /
+  `other` with an optional note. Acknowledge when you've seen it, Resolve when it's handled.
+  There is no chat here — if it needs back-and-forth, use your actual video call.
+
+**End of class**
+
+Open the classroom → End class. Status moves to `ended`; this does not touch Labs or student
+progress — destroy Labs separately from the cohort view if you're done with them for the day.
+
+**If you need to cancel a scheduled (not yet started) class:** Cancel, from the classroom
+detail view. A `live` class can only be ended, not cancelled.
+
+**If the Platform API restarts mid-class** (deploy, crash, host reboot): nothing to do. Sessions,
+the live classroom row, and participant presence all live in Postgres, not in the API process's
+memory — students' browsers keep their existing session and heartbeat right through it, and the
+dashboard shows the same state a few seconds after the API comes back. This is exercised for
+real in `infra/scripts/drills/test-phase5a-rehearsal.js`, which restarts the API mid-rehearsal
+and asserts on exactly this.

docs/RUNBOOK_ROLLBACK.md

@@ -0,0 +1,18 @@
+# Runbook: Rollback
+
+```bash
+cd infra/scripts
+./rollback-production.sh              # rolls back to the previous deployed tag automatically
+# or, to go back further than one step:
+./rollback-production.sh sha-<older-sha>
+
+./production-smoke-test.sh
+```
+
+This rolls back **application images only**. If the bad deploy included a database migration
+that needs undoing too, schema changes aren't automatically reversible — see
+`docs/RUNBOOK_FAILED_MIGRATION.md` instead (or in addition).
+
+After rolling back: file an issue / note in the deploy log (`/opt/rootnode/deploy-history.log`
+already has the rollback recorded) about what went wrong before re-attempting the forward
+deploy.

docs/RUNBOOK_STUCK_LAB_CLEANUP.md

@@ -0,0 +1,40 @@
+# Runbook: Stuck Lab Cleanup
+
+A Lab that's stuck `provisioning`/`resetting` past its expected boot window, or a container
+that outlived its Postgres `environments` row (crash during teardown, etc.).
+
+**Via the admin UI (preferred, no host access needed):** `/admin` -> Students -> [student] ->
+Environment -> Destroy, then Provision again if they need a fresh one immediately.
+
+**Via the API directly** (same effect, useful for scripting a batch cleanup):
+```bash
+curl -s -b <owner-session-cookie> -X DELETE https://academy.example.com/api/admin/environments/<environment_id>
+```
+
+**A container the Provisioner itself has lost track of** (rare — e.g. a crash during
+provisioning left an orphaned container/network/firewall chain with no matching `environments`
+row, or vice versa):
+
+```bash
+# On rn-lab-host-01 — list what's actually running
+docker ps -a --format '{{.Names}}\t{{.Status}}' | grep '^rootnode-lab-'
+
+# Cross-reference against the Provisioner's own view
+curl -s http://localhost:4100/environments | jq '.[] | {environment_id, status, container_name}'
+
+# Provisioner already does this automatically on its own restart (reconcileOnStartup,
+# platform/provisioner/src/lifecycle.js) — a simple `systemctl restart rootnode-platform`
+# (which restarts the provisioner container as part of the stack) is often the fastest fix for
+# a batch of orphans rather than hand-cleaning each one.
+
+# Manual last resort — remove the container/network/firewall chain directly:
+docker rm -f rootnode-lab-<id>
+docker network rm rootnode-lab-net-<id>
+iptables -D DOCKER-USER -s <subnet> -j ROOTNODE-ENV-<id>   # remove the jump rule FIRST
+iptables -F ROOTNODE-ENV-<id> && iptables -X ROOTNODE-ENV-<id>
+```
+
+Never manually delete an `environments` row from Postgres directly unless you've confirmed the
+container/network/firewall chain are already gone — the Provisioner treats that table as
+authoritative and will behave unpredictably against real Docker/iptables state it doesn't know
+matches.

docs/RUNBOOK_STUDENT_ACCOUNT_LIFECYCLE.md

@@ -0,0 +1,44 @@
+# Runbook: Student Account Lifecycle
+
+Covers creating, converting, resetting, and revoking student accounts. See
+`docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md` for the underlying design.
+
+**Create a new credentialed student** (`/admin` → Students → Add student):
+
+Fill in display name plus **either** email (older/self-paced) or login ID (younger/instructor-
+managed) — leave both blank for a plain pilot account instead. Submitting shows an activation
+link **once**: `https://<domain>/?activate=<token>`. Copy it immediately (print it, read it
+aloud, hand it over) — it is never shown again and never stored in plaintext anywhere. The
+student opens the link and sets their own password; the instructor never sees it.
+
+**Convert an existing pilot student to a real account** (Students → \[student\] → Convert):
+
+Same as creation, but for a student who already exists with progress/attempts/achievements —
+none of that is touched. Give an email or login ID, get a one-time activation link the same way.
+
+**Bulk import via CSV** (Students → Bulk import):
+
+Columns: `display_name, email, login_identifier, cohort_id` (only `display_name` is required per
+row). The whole file is validated before anything is written — one bad row rejects the entire
+file with a list of row errors, rather than importing some rows and skipping others. Never
+include a password column; it will always be rejected if present. Activation links for every
+created row are shown once after a successful import.
+
+**Reset a student's password** (Students → \[student\] → Reset password):
+
+Only for already-credentialed accounts. Issues a fresh reset link (`?reset=<token>`, valid 1
+hour) and immediately revokes every existing session for that student — the instructor still
+never sees or sets the actual password.
+
+**Revoke a student's sessions** (Students → \[student\] → Revoke all sessions):
+
+Immediately invalidates every active login for that student without touching their password.
+Use this if a device is lost/shared or a session looks compromised.
+
+**Deactivate a student** (Students → \[student\] → PATCH status: inactive, via the existing
+students list — deactivation takes effect on their very next request, not just their next
+login, because `studentSession` middleware re-checks `status='active'` every time.
+
+**If a student says their activation/reset link "doesn't work":** tokens are single-use and
+time-limited (activation 7 days, reset 1 hour). Just issue a new one via Convert or Reset
+password — there's no way to "resend" the same token, and that's intentional.

docs/RUNBOOK_TLS_RENEWAL.md

@@ -0,0 +1,19 @@
+# Runbook: Public TLS Certificate Renewal
+
+If using Caddy's built-in Let's Encrypt (the default in `infra/docker/platform/Caddyfile.j2`):
+**nothing to do** — Caddy renews automatically well before expiry, as long as port 80 stays
+reachable for the ACME HTTP-01 challenge (`infra/firewall/rn-platform-01.rules` already allows
+it). `infra/scripts/production-smoke-test.sh` checks the live expiry date on every run as an
+early warning if auto-renewal ever silently breaks.
+
+If using Cloudflare origin certs instead (see `docs/PRODUCTION_NETWORK_SECURITY.md` §8 for
+that decision): Cloudflare-issued origin certs are typically 15-year validity — this is a
+once-per-network-topology-change concern, not a recurring one. Re-issue from the Cloudflare
+dashboard (SSL/TLS -> Origin Server) and redeploy the cert/key via the `platform` Ansible role
+if it ever needs replacing (domain change, suspected compromise).
+
+**Manual check:**
+```bash
+echo | openssl s_client -connect academy.example.com:443 -servername academy.example.com 2>/dev/null \
+  | openssl x509 -noout -enddate
+```

docs/RUNBOOK_VM_REBUILD.md

@@ -0,0 +1,31 @@
+# Runbook: Platform VM / Lab Host VM Rebuild
+
+Both VMs are designed to be disposable and rebuildable from code — nothing irreplaceable
+should ever live only on either VM's disk outside of what's backed up (`docs/BACKUP_DR.md`).
+
+## rn-platform-01 lost
+
+Academic data survives (it's in Postgres backups, off-host). Instructor/student accounts and
+progress are recoverable; anything since the last backup is not (`docs/BACKUP_DR.md`'s RPO).
+
+```bash
+cd infra/terraform && terraform apply   # recreates the VM from the template
+cd ../ansible && ansible-playbook site.yml -i inventory/production.ini --limit platform --ask-vault-pass
+# Then restore Postgres — docs/RUNBOOK_POSTGRES_RESTORE.md, step 5b variant (fresh DB this time,
+# no need to rename since it's a fresh instance)
+./infra/scripts/production-smoke-test.sh
+```
+
+## rn-lab-host-01 lost
+
+**No academic data is lost** — this host only ever holds disposable Lab containers by design
+(`docs/BACKUP_DR.md`'s Tier distinction). Every enrolled student's Lab needs re-provisioning
+after this, which is a normal "Prepare Labs" click, not a recovery operation.
+
+```bash
+cd infra/terraform && terraform apply
+cd ../ansible && ansible-playbook site.yml -i inventory/production.ini --limit lab_host --ask-vault-pass
+# Re-generate Docker TLS material for the new host if its identity (IP/DNS) changed —
+# docs/RUNBOOK_DOCKER_TLS_ROTATION.md
+# From /admin: Prepare Labs again for any cohort that needs it.
+```

docs/SECURITY_SCANNING.md

@@ -0,0 +1,47 @@
+# Security Scanning (Phase 4.1 §24)
+
+`infra/ci/gitea-workflows/build-and-test.yml`'s `security-scan` job. All free, open-source,
+no account/signup required for any of them — no paid scanner was added.
+
+## What's scanned
+
+| Tool | Checks | Scope |
+|---|---|---|
+| `npm audit --audit-level=high` | Known-vulnerable dependencies | Every first-party Node service |
+| [gitleaks](https://github.com/gitleaks/gitleaks) | Committed secrets (keys, tokens, credentials) | Whole repo |
+| [hadolint](https://github.com/hadolint/hadolint) | Dockerfile best-practice/security lint | Every first-party Dockerfile |
+| [shellcheck](https://www.shellcheck.net/) | Shell script correctness/security | `infra/**/*.sh` |
+| [trivy](https://github.com/aquasecurity/trivy) | OS/library CVEs in a built container image | `platform-api` image (representative — same base/layers as the other Node services) |
+
+## Severity policy
+
+- **CRITICAL / HIGH** — fails the build (`security-scan` blocks `build-images`, which blocks
+  `deploy-staging`, which blocks production promotion — see the job dependency chain in the
+  workflow file).
+- **MEDIUM / LOW** — reported in the job output, never blocks. A build that's permanently red
+  over a low-severity transitive dependency finding nobody can immediately fix trains everyone
+  to ignore red builds, which is worse than not scanning at all.
+
+`npm audit --audit-level=high` and trivy's `--severity HIGH,CRITICAL` both implement this
+directly at the tool level, not via manual triage.
+
+## What's NOT in the automated job (and why)
+
+- **Dedicated IaC security linting** (checkov/tflint) — not added. `infra/scripts/validate-compose.sh`,
+  `infra/firewall/validate-firewall.sh`, `ansible-playbook --syntax-check`, and `tofu validate`
+  (all in the `regression` job) already catch structural/syntax problems in the infra code;
+  a dedicated security-policy linter for Terraform/Ansible is a reasonable Phase 5 addition once
+  there's a real cloud footprint to misconfigure, not before.
+- **DAST/penetration testing** — out of scope for CI; this is a build-time gate, not a running-system
+  test. Covered separately by `docs/PRODUCTION_VALIDATION_CHECKLIST.md`'s security section once
+  real hardware exists.
+- **SAST beyond hadolint/shellcheck** — the codebase is small enough (a handful of first-party
+  Node services) that `npm audit` + secret scanning + Dockerfile/shell lint cover the realistic
+  risk surface; a dedicated SAST tool (Semgrep, CodeQL) is a reasonable addition once the
+  codebase or team grows enough to justify the added CI time.
+
+## Local reproduction
+
+Every check above can be run locally with the exact same free tools — no CI-only dependency.
+See the `security-scan` job's steps in `infra/ci/gitea-workflows/build-and-test.yml` for the
+exact commands (they're copy-pasteable).

frontend/src/admin/Admin.module.css

@@ -57,3 +57,6 @@ tr:hover td { background: var(--surface2); }
 .linkBtn { background: none; border: none; color: var(--blue); cursor: pointer; font-size: 0.85rem; padding: 0; text-decoration: underline; }
 .errorBox { background: var(--red-dim); color: var(--red); border: 1px solid var(--red); border-radius: var(--radius); padding: 0.6rem 0.8rem; font-size: 0.85rem; margin-bottom: 1rem; }
 .muted { color: var(--text-2); font-size: 0.85rem; }
+
+/* Phase 4.1 §16 — shared pagination control (AuditView, StudentsView attempts). */
+.pager { display: flex; align-items: center; gap: 0.75rem; margin-top: 0.75rem; }

frontend/src/admin/AdminPortal.jsx

@@ -8,6 +8,7 @@ import AuditView from './AuditView'
 import AccountView from './AccountView'
 import LearningView from './LearningView'
 import AssessmentsView from './AssessmentsView'
+import ClassroomView from './ClassroomView'
 import styles from './Admin.module.css'
 
 // Instructor/Admin portal — clearly separate from the student experience (/admin route,
@@ -20,6 +21,7 @@ const NAV = [
   { key: 'dashboard', label: 'Dashboard' },
   { key: 'cohorts', label: 'Cohorts' },
   { key: 'students', label: 'Students' },
+  { key: 'classroom', label: 'Classroom' },
   { key: 'learning', label: 'Learning' },
   { key: 'assessments', label: 'Assessments' },
   { key: 'instructors', label: 'Instructors', roles: ['owner'] },
@@ -59,6 +61,7 @@ export default function AdminPortal() {
         {view === 'dashboard' && <AdminDashboard />}
         {view === 'cohorts' && <CohortsView role={instructor.role} />}
         {view === 'students' && <StudentsView role={instructor.role} />}
+        {view === 'classroom' && <ClassroomView />}
         {view === 'learning' && <LearningView />}
         {view === 'assessments' && <AssessmentsView role={instructor.role} />}
         {view === 'instructors' && instructor.role === 'owner' && <InstructorsView currentInstructorId={instructor.instructor_id} />}

frontend/src/admin/AssessmentsView.jsx

@@ -29,7 +29,7 @@ export default function AssessmentsView({ role }) {
       {canAuthor && (
         <div className={styles.card}>
           <h2>Create assessment</h2>
-          <CreateAssessmentForm onCreated={reload} />
+          <CreateAssessmentForm onCreated={reload} role={role} />
         </div>
       )}
       <div className={styles.card}>
@@ -56,20 +56,86 @@ export default function AssessmentsView({ role }) {
   )
 }
 
-function CreateAssessmentForm({ onCreated }) {
+// Phase 4.1 §18 — pool_spec is built from structured picks (required scenarios + named random
-  const [form, setForm] = useState({ id: '', type: 'exam', title: '', duration_minutes: 60, hint_policy: 'disabled', pool_spec: '' })
+// pools), not hand-typed JSON. `catalog` ({ stages: [{ id, name, scenarios: [{id,title}] }] })
+// comes from GET /api/admin/scenario-catalog. An owner can still drop into a raw-JSON view for
+// anything the structured editor doesn't cover — see the toggle below.
+function PoolSpecEditor({ catalog, required, setRequired, pools, setPools }) {
+  const allScenarios = (catalog?.stages || []).flatMap((s) => s.scenarios.map((sc) => ({ ...sc, stageName: s.name })))
+  const titleFor = (id) => allScenarios.find((s) => s.id === id)?.title || id
+
+  const addPool = () => setPools([...pools, { count: 1, from: [] }])
+  const removePool = (i) => setPools(pools.filter((_, idx) => idx !== i))
+  const updatePool = (i, patch) => setPools(pools.map((p, idx) => (idx === i ? { ...p, ...patch } : p)))
+
+  return (
+    <div>
+      <label className={styles.label}>Required scenarios (always included)</label>
+      <select className={styles.select} multiple size={6} style={{ width: '100%' }}
+        value={required} onChange={(e) => setRequired([...e.target.selectedOptions].map((o) => o.value))}>
+        {catalog?.stages.map((stage) => (
+          <optgroup key={stage.id} label={stage.name}>
+            {stage.scenarios.map((sc) => <option key={sc.id} value={sc.id}>{sc.title}</option>)}
+          </optgroup>
+        ))}
+      </select>
+      <p className={styles.muted}>{required.length ? `${required.length} selected: ${required.map(titleFor).join(', ')}` : 'None selected — cmd/ctrl-click to pick multiple.'}</p>
+
+      <label className={styles.label} style={{ marginTop: '1rem', display: 'block' }}>Random pools (draw N at random from a set, every attempt)</label>
+      {pools.map((pool, i) => (
+        <div key={i} className={styles.card} style={{ marginBottom: '0.5rem' }}>
+          <div className={styles.formRow}>
+            <div style={{ maxWidth: '100px' }}><label className={styles.label}>Draw count</label>
+              <input className={styles.input} type="number" min="1" value={pool.count}
+                onChange={(e) => updatePool(i, { count: Number(e.target.value) })} /></div>
+            <div><button type="button" className={styles.linkBtn} onClick={() => removePool(i)}>Remove pool</button></div>
+          </div>
+          <label className={styles.label}>Draw from</label>
+          <select className={styles.select} multiple size={5} style={{ width: '100%' }}
+            value={pool.from} onChange={(e) => updatePool(i, { from: [...e.target.selectedOptions].map((o) => o.value) })}>
+            {catalog?.stages.map((stage) => (
+              <optgroup key={stage.id} label={stage.name}>
+                {stage.scenarios.map((sc) => <option key={sc.id} value={sc.id}>{sc.title}</option>)}
+              </optgroup>
+            ))}
+          </select>
+          {pool.count > pool.from.length && <p className={styles.muted} style={{ color: 'var(--amber, #b58900)' }}>Draw count ({pool.count}) exceeds the pool size ({pool.from.length}) — every draw will use every scenario, not a random subset.</p>}
+        </div>
+      ))}
+      <button type="button" className={styles.linkBtn} onClick={addPool}>+ Add random pool</button>
+    </div>
+  )
+}
+
+function CreateAssessmentForm({ onCreated, role }) {
+  const [form, setForm] = useState({ id: '', type: 'exam', title: '', duration_minutes: 60, hint_policy: 'disabled' })
+  const [catalog, setCatalog] = useState(null)
+  const [required, setRequired] = useState([])
+  const [pools, setPools] = useState([])
+  const [advanced, setAdvanced] = useState(false)
+  const [rawPoolSpec, setRawPoolSpec] = useState('')
   const [error, setError] = useState(null)
   const [busy, setBusy] = useState(false)
 
+  useEffect(() => { adminApi.scenarioCatalog().then(setCatalog).catch((e) => setError(e.message)) }, [])
+
   const submit = async (e) => {
     e.preventDefault()
     setBusy(true); setError(null)
     try {
-      const pool_spec = JSON.parse(form.pool_spec || '{"required":[],"random":[]}')
+      // Owner's raw-JSON view, when open, is the source of truth — otherwise built from the
+      // structured pickers above.
+      const pool_spec = advanced
+        ? JSON.parse(rawPoolSpec || '{"required":[],"random":[]}')
+        : { required, random: pools.filter((p) => p.count > 0 && p.from.length > 0) };
+      if (!advanced && !required.length && !pool_spec.random.length) {
+        throw new Error('Pick at least one required scenario or one random pool.')
+      }
       await adminApi.createAssessment({ ...form, duration_minutes: Number(form.duration_minutes), pool_spec })
-      setForm({ id: '', type: 'exam', title: '', duration_minutes: 60, hint_policy: 'disabled', pool_spec: '' })
+      setForm({ id: '', type: 'exam', title: '', duration_minutes: 60, hint_policy: 'disabled' })
+      setRequired([]); setPools([]); setRawPoolSpec('')
       onCreated()
-    } catch (e) { setError(e.message?.includes('JSON') ? 'pool_spec must be valid JSON' : e.message) } finally { setBusy(false) }
+    } catch (e) { setError(e.message?.includes('JSON') ? 'Raw pool_spec must be valid JSON' : e.message) } finally { setBusy(false) }
   }
 
   return (
@@ -91,26 +157,47 @@ function CreateAssessmentForm({ onCreated }) {
             <option value="disabled">disabled</option><option value="flagged">flagged</option><option value="free">free</option>
           </select></div>
       </div>
+
+      {role === 'owner' && (
+        <p className={styles.muted}>
+          <button type="button" className={styles.linkBtn} onClick={() => setAdvanced(!advanced)}>
+            {advanced ? '← Back to the structured editor' : 'Advanced: edit pool_spec as raw JSON →'}
+          </button>
+        </p>
+      )}
+
+      {advanced ? (
+        <>
           <label className={styles.label}>Pool spec (JSON)</label>
           <textarea className={styles.input} style={{ width: '100%', fontFamily: 'var(--mono)', minHeight: '4rem' }}
             placeholder='{"required":["scenario-id"],"random":[{"count":1,"from":["a","b"]}]}'
-        value={form.pool_spec} onChange={(e) => setForm({ ...form, pool_spec: e.target.value })} required />
+            value={rawPoolSpec} onChange={(e) => setRawPoolSpec(e.target.value)} required />
+        </>
+      ) : !catalog ? <p className={styles.muted}>Loading scenario catalog…</p> : (
+        <PoolSpecEditor catalog={catalog} required={required} setRequired={setRequired} pools={pools} setPools={setPools} />
+      )}
+
       <button className={styles.btn} style={{ marginTop: '0.75rem' }} disabled={busy}>{busy ? 'Creating…' : 'Create'}</button>
     </form>
   )
 }
 
 function AssessmentDetail({ definition, cohorts, onBack }) {
-  const [sessions, setSessions] = useState(null)
+  // Phase 4.1 §16 — assessmentSessions is now paginated (was unbounded); `result` holds the
+  // full { items, page, total, total_pages } envelope, `sessions` stays the plain array for
+  // the table below so its render logic doesn't need to change.
+  const [result, setResult] = useState(null)
+  const [page, setPage] = useState(1)
   const [error, setError] = useState(null)
   const [notice, setNotice] = useState(null)
   const [cohortId, setCohortId] = useState(cohorts?.[0]?.cohort_id || '')
   const [busy, setBusy] = useState(false)
 
   const reload = useCallback(() => {
-    adminApi.assessmentSessions(definition.assessment_definition_id).then(setSessions).catch((e) => setError(e.message))
+    adminApi.assessmentSessions(definition.assessment_definition_id, page).then(setResult).catch((e) => setError(e.message))
-  }, [definition])
+  }, [definition, page])
   useEffect(reload, [reload])
+  const sessions = result?.items || null
 
   const assign = async () => {
     setBusy(true); setError(null); setNotice(null)
@@ -158,6 +245,13 @@ function AssessmentDetail({ definition, cohorts, onBack }) {
             </tbody>
           </table>
         )}
+        {result && result.total_pages > 1 && (
+          <div className={styles.pager}>
+            <button disabled={page <= 1} onClick={() => setPage((p) => p - 1)}>Previous</button>
+            <span className={styles.muted}>Page {result.page} of {result.total_pages} ({result.total} total)</span>
+            <button disabled={page >= result.total_pages} onClick={() => setPage((p) => p + 1)}>Next</button>
+          </div>
+        )}
       </div>
     </>
   )

frontend/src/admin/AuditView.jsx

@@ -2,22 +2,27 @@ import { useState, useEffect } from 'react'
 import { adminApi } from './api'
 import styles from './Admin.module.css'
 
-// admin/owner only (enforced server-side too). Latest 200 events, no pagination UI — this is a
+// admin/owner only (enforced server-side too). Phase 4.1 §16 — real server-side pagination
-// pilot-scale audit log, not a SIEM; add pagination if that ever stops being true.
+// (platform/api/src/pagination.js) replaces the old flat "latest 200, nothing older" cap.
 export default function AuditView() {
-  const [events, setEvents] = useState(null)
+  const [page, setPage] = useState(1)
+  const [result, setResult] = useState(null)
   const [error, setError] = useState(null)
 
   useEffect(() => {
-    adminApi.auditEvents().then(setEvents).catch((e) => setError(e.message))
+    setResult(null)
-  }, [])
+    adminApi.auditEvents(page).then(setResult).catch((e) => setError(e.message))
+  }, [page])
+
+  const events = result?.items || []
 
   return (
     <div className={styles.card}>
       <h2>Audit log</h2>
-      <p className={styles.muted}>Most recent 200 security/admin events.</p>
+      <p className={styles.muted}>{result ? `${result.total} total events` : 'Security/admin events, most recent first.'}</p>
       {error && <div className={styles.errorBox}>{error}</div>}
-      {!events ? <p className={styles.muted}>Loading…</p> : (
+      {!result ? <p className={styles.muted}>Loading…</p> : (
+        <>
           <table>
             <thead><tr><th>When</th><th>Actor</th><th>Action</th><th>Resource</th><th>Result</th></tr></thead>
             <tbody>
@@ -32,6 +37,12 @@ export default function AuditView() {
               ))}
             </tbody>
           </table>
+          <div className={styles.pager}>
+            <button disabled={page <= 1} onClick={() => setPage((p) => p - 1)}>Previous</button>
+            <span className={styles.muted}>Page {result.page} of {result.total_pages}</span>
+            <button disabled={page >= result.total_pages} onClick={() => setPage((p) => p + 1)}>Next</button>
+          </div>
+        </>
       )}
     </div>
   )

frontend/src/admin/ClassroomView.jsx

@@ -0,0 +1,216 @@
+import { useState, useEffect, useCallback } from 'react'
+import { adminApi } from './api'
+import styles from './Admin.module.css'
+
+const DASHBOARD_POLL_MS = 5000
+
+// Phase 5A §14/§21 — remote classroom control center. Cohort picker -> that cohort's
+// classroom sessions -> a live dashboard once one is open. No video/chat/shell-takeover here
+// by design (§37) — this is identity/presence/Lab-status/help-queue/announcements only.
+export default function ClassroomView() {
+  const [cohorts, setCohorts] = useState(null)
+  const [cohortId, setCohortId] = useState('')
+  const [classrooms, setClassrooms] = useState(null)
+  const [selected, setSelected] = useState(null)
+  const [error, setError] = useState(null)
+
+  useEffect(() => { adminApi.cohorts().then((rows) => { setCohorts(rows); if (rows.length) setCohortId(rows[0].cohort_id) }).catch((e) => setError(e.message)) }, [])
+
+  const reload = useCallback(() => {
+    if (!cohortId) return
+    adminApi.classrooms(cohortId).then(setClassrooms).catch((e) => setError(e.message))
+  }, [cohortId])
+  useEffect(reload, [reload])
+
+  if (selected) return <ClassroomDashboard classroomId={selected} onBack={() => { setSelected(null); reload() }} />
+
+  return (
+    <>
+      {error && <div className={styles.errorBox}>{error}</div>}
+      <div className={styles.card}>
+        <h2>Classroom</h2>
+        <div className={styles.formRow}>
+          <div><label className={styles.label}>Cohort</label>
+            <select className={styles.select} value={cohortId} onChange={(e) => setCohortId(e.target.value)}>
+              {cohorts?.map((c) => <option key={c.cohort_id} value={c.cohort_id}>{c.name}</option>)}
+            </select></div>
+        </div>
+      </div>
+
+      {cohortId && (
+        <div className={styles.card}>
+          <h3>Schedule a class</h3>
+          <CreateClassroomForm cohortId={cohortId} onCreated={reload} />
+        </div>
+      )}
+
+      <div className={styles.card}>
+        <h3>Classes</h3>
+        {!classrooms ? <p className={styles.muted}>Loading…</p> : classrooms.length === 0 ? <p className={styles.muted}>None yet.</p> : (
+          <table>
+            <thead><tr><th>Title</th><th>Status</th><th>Scheduled</th><th /></tr></thead>
+            <tbody>
+              {classrooms.map((c) => (
+                <tr key={c.classroom_session_id}>
+                  <td>{c.title}</td>
+                  <td><span className={`${styles.pill} ${c.status === 'live' ? styles.pillOk : c.status === 'cancelled' ? styles.pillErr : styles.pillMuted}`}>{c.status}</span></td>
+                  <td>{c.scheduled_start ? new Date(c.scheduled_start).toLocaleString() : '—'}</td>
+                  <td><button className={styles.linkBtn} onClick={() => setSelected(c.classroom_session_id)}>Open</button></td>
+                </tr>
+              ))}
+            </tbody>
+          </table>
+        )}
+      </div>
+    </>
+  )
+}
+
+function CreateClassroomForm({ cohortId, onCreated }) {
+  const [title, setTitle] = useState('')
+  const [description, setDescription] = useState('')
+  const [scheduledStart, setScheduledStart] = useState('')
+  const [meetingUrl, setMeetingUrl] = useState('')
+  const [busy, setBusy] = useState(false)
+  const [error, setError] = useState(null)
+
+  const submit = async (e) => {
+    e.preventDefault()
+    setBusy(true); setError(null)
+    try {
+      await adminApi.createClassroom(cohortId, {
+        title: title.trim(), description: description.trim() || undefined,
+        scheduled_start: scheduledStart || undefined, meeting_url: meetingUrl.trim() || undefined,
+      })
+      setTitle(''); setDescription(''); setScheduledStart(''); setMeetingUrl('')
+      onCreated()
+    } catch (e) { setError(e.message) } finally { setBusy(false) }
+  }
+
+  return (
+    <form onSubmit={submit}>
+      {error && <div className={styles.errorBox}>{error}</div>}
+      <div className={styles.formRow}>
+        <div><label className={styles.label}>Title</label>
+          <input className={styles.input} value={title} onChange={(e) => setTitle(e.target.value)} required /></div>
+        <div><label className={styles.label}>Scheduled start</label>
+          <input className={styles.input} type="datetime-local" value={scheduledStart} onChange={(e) => setScheduledStart(e.target.value)} /></div>
+        <div><label className={styles.label}>Meeting URL (Meet/Zoom/Teams — optional)</label>
+          <input className={styles.input} value={meetingUrl} onChange={(e) => setMeetingUrl(e.target.value)} /></div>
+      </div>
+      <label className={styles.label}>Description (optional)</label>
+      <input className={styles.input} value={description} onChange={(e) => setDescription(e.target.value)} />
+      <button className={styles.btn} style={{ marginTop: '0.75rem' }} disabled={busy || !title.trim()}>{busy ? 'Scheduling…' : 'Schedule'}</button>
+    </form>
+  )
+}
+
+function ClassroomDashboard({ classroomId, onBack }) {
+  const [data, setData] = useState(null)
+  const [error, setError] = useState(null)
+  const [notice, setNotice] = useState(null)
+  const [busy, setBusy] = useState(false)
+  const [announcement, setAnnouncement] = useState('')
+  const [activityScenario, setActivityScenario] = useState('')
+
+  const reload = useCallback(() => {
+    adminApi.classroomDashboard(classroomId).then(setData).catch((e) => setError(e.message))
+  }, [classroomId])
+  useEffect(() => {
+    reload()
+    const t = setInterval(reload, DASHBOARD_POLL_MS) // presence/help-queue/Lab-status all move live — poll rather than a second push channel
+    return () => clearInterval(t)
+  }, [reload])
+
+  const withBusy = (fn) => async () => {
+    setBusy(true); setError(null); setNotice(null)
+    try { await fn(); reload() } catch (e) { setError(e.message) } finally { setBusy(false) }
+  }
+
+  if (!data) return <>{error ? <div className={styles.errorBox}>{error}</div> : <p className={styles.muted}>Loading…</p>}</>
+  const { classroom, students, announcements, help_requests: helpRequests } = data
+  const openHelp = helpRequests.filter((h) => h.status !== 'resolved')
+
+  return (
+    <>
+      <button className={styles.linkBtn} onClick={onBack}>&larr; Back to classes</button>
+      <div className={styles.card} style={{ marginTop: '1rem' }}>
+        <h2>{classroom.title} <span className={`${styles.pill} ${classroom.status === 'live' ? styles.pillOk : styles.pillMuted}`}>{classroom.status}</span></h2>
+        {error && <div className={styles.errorBox}>{error}</div>}
+        {notice && <p className={styles.muted}>{notice}</p>}
+        <div className={styles.formRow}>
+          {classroom.status === 'scheduled' && <button className={styles.btn} disabled={busy} onClick={withBusy(() => adminApi.startClassroom(classroomId))}>Start class</button>}
+          {classroom.status === 'live' && <button className={styles.btnSecondary} disabled={busy} onClick={withBusy(() => adminApi.endClassroom(classroomId))}>End class</button>}
+          {classroom.status === 'scheduled' && <button className={styles.btnDanger} disabled={busy} onClick={withBusy(() => adminApi.cancelClassroom(classroomId))}>Cancel</button>}
+          <button className={styles.btnSecondary} disabled={busy} onClick={withBusy(async () => {
+            const { results } = await adminApi.prepareClassroomLabs(classroomId)
+            setNotice(`Labs prepared: ${results.length} student(s).`)
+          })}>Prepare Labs</button>
+        </div>
+      </div>
+
+      <div className={styles.card}>
+        <h3>Participants ({students.length})</h3>
+        {students.length === 0 ? <p className={styles.muted}>No one enrolled in this cohort yet.</p> : (
+          <table>
+            <thead><tr><th>Student</th><th>Presence</th><th>Lab</th><th>Current work</th><th>Hints</th></tr></thead>
+            <tbody>
+              {students.map((s) => (
+                <tr key={s.student_id}>
+                  <td>{s.display_name}</td>
+                  <td><span className={`${styles.pill} ${s.presence === 'online' ? styles.pillOk : s.presence === 'recently_active' ? styles.pillWarn : styles.pillMuted}`}>{s.presence}</span></td>
+                  <td><span className={`${styles.pill} ${s.lab_status === 'running' ? styles.pillOk : s.lab_status === 'error' ? styles.pillErr : styles.pillMuted}`}>{s.lab_status}</span></td>
+                  <td className={styles.muted}>{s.current_scenario || '—'} {s.attempt_status ? `(${s.attempt_status}, ${s.validation_failures} fails)` : ''}</td>
+                  <td className={styles.muted}>{s.hints_used}</td>
+                </tr>
+              ))}
+            </tbody>
+          </table>
+        )}
+      </div>
+
+      <div className={styles.card}>
+        <h3>Help queue {openHelp.length > 0 && <span className={`${styles.pill} ${styles.pillWarn}`}>{openHelp.length} open</span>}</h3>
+        {helpRequests.length === 0 ? <p className={styles.muted}>No help requests.</p> : (
+          <table>
+            <thead><tr><th>Student</th><th>Category</th><th>Message</th><th>Status</th><th /></tr></thead>
+            <tbody>
+              {helpRequests.map((h) => (
+                <tr key={h.help_request_id}>
+                  <td>{h.display_name || h.student_id}</td>
+                  <td className={styles.muted}>{h.category}</td>
+                  <td className={styles.muted}>{h.message || '—'}</td>
+                  <td><span className={`${styles.pill} ${h.status === 'resolved' ? styles.pillOk : h.status === 'acknowledged' ? styles.pillWarn : styles.pillErr}`}>{h.status}</span></td>
+                  <td>
+                    {h.status === 'open' && <button className={styles.linkBtn} onClick={withBusy(() => adminApi.acknowledgeHelpRequest(h.help_request_id))}>Acknowledge</button>}
+                    {h.status !== 'resolved' && <button className={styles.linkBtn} onClick={withBusy(() => adminApi.resolveHelpRequest(h.help_request_id))}>Resolve</button>}
+                  </td>
+                </tr>
+              ))}
+            </tbody>
+          </table>
+        )}
+      </div>
+
+      <div className={styles.card}>
+        <h3>Current activity</h3>
+        <p className={styles.muted}>Guidance only — students see this as a link, nothing auto-redirects them (§20).</p>
+        <div className={styles.formRow}>
+          <div><input className={styles.input} placeholder="scenario id" value={activityScenario} onChange={(e) => setActivityScenario(e.target.value)} /></div>
+          <div><button className={styles.btn} disabled={busy || !activityScenario.trim()} onClick={withBusy(async () => { await adminApi.setClassroomActivity(classroomId, activityScenario.trim()); setActivityScenario('') })}>Set</button></div>
+        </div>
+      </div>
+
+      <div className={styles.card}>
+        <h3>Announcements</h3>
+        {announcements.length === 0 ? <p className={styles.muted}>None yet.</p> : (
+          <ul>{announcements.map((a) => <li key={a.announcement_id} className={styles.muted}>{a.message}</li>)}</ul>
+        )}
+        <div className={styles.formRow} style={{ marginTop: '0.5rem' }}>
+          <div><input className={styles.input} placeholder="Post an announcement" value={announcement} onChange={(e) => setAnnouncement(e.target.value)} /></div>
+          <div><button className={styles.btn} disabled={busy || !announcement.trim()} onClick={withBusy(async () => { await adminApi.postClassroomAnnouncement(classroomId, announcement.trim()); setAnnouncement('') })}>Post</button></div>
+        </div>
+      </div>
+    </>
+  )
+}

frontend/src/admin/CohortsView.jsx

@@ -101,7 +101,7 @@ function CohortDetail({ cohortId, canManage, onBack }) {
     if (!newStudentName.trim()) return
     setBusy(true); setError(null)
     try {
-      await adminApi.createStudent(newStudentName.trim(), cohortId)
+      await adminApi.createStudent({ display_name: newStudentName.trim(), cohort_id: cohortId })
       setNewStudentName('')
       reload()
     } catch (e) { setError(e.message) } finally { setBusy(false) }

frontend/src/admin/StudentsView.jsx

@@ -3,30 +3,53 @@ import { adminApi } from './api'
 import { StatusPill } from './CohortsView'
 import styles from './Admin.module.css'
 
-export default function StudentsView() {
+// Phase 5A §4-7: student roster now shows credential status alongside the pilot fields that
+// were already here. canManage gates create/convert/reset/revoke/import the same way
+// CohortsView gates cohort management — the server (requireAdminOrOwner) is the real boundary,
+// this just avoids showing controls a plain instructor would get a 403 from.
+export default function StudentsView({ role }) {
   const [students, setStudents] = useState(null)
   const [selected, setSelected] = useState(null)
   const [error, setError] = useState(null)
+  const canManage = role !== 'instructor'
 
   const reload = useCallback(() => {
     adminApi.students().then(setStudents).catch((e) => setError(e.message))
   }, [])
   useEffect(reload, [reload])
 
-  if (selected) return <StudentDetail studentId={selected} onBack={() => { setSelected(null); reload() }} />
+  if (selected) return <StudentDetail studentId={selected} canManage={canManage} onBack={() => { setSelected(null); reload() }} />
 
   return (
+    <>
+      {error && <div className={styles.errorBox}>{error}</div>}
+      {canManage && (
+        <div className={styles.card}>
+          <h2>Add student</h2>
+          <CreateStudentForm onCreated={reload} />
+        </div>
+      )}
+      {canManage && (
+        <div className={styles.card}>
+          <h2>Bulk import (CSV)</h2>
+          <ImportForm onImported={reload} />
+        </div>
+      )}
       <div className={styles.card}>
         <h2>Students ({students?.length ?? '…'})</h2>
-      {error && <div className={styles.errorBox}>{error}</div>}
         {students && (
           <table>
-          <thead><tr><th>Name</th><th>ID</th><th>Status</th><th>Last active</th><th /></tr></thead>
+            <thead><tr><th>Name</th><th>ID</th><th>Login</th><th>Status</th><th>Last active</th><th /></tr></thead>
             <tbody>
               {students.map((s) => (
                 <tr key={s.student_id}>
                   <td>{s.display_name}</td>
                   <td className={styles.muted}>{s.student_id}</td>
+                  <td>
+                    {s.has_credentials
+                      ? <span className={`${styles.pill} ${styles.pillOk}`}>{s.email || s.login_identifier}</span>
+                      : <span className={`${styles.pill} ${styles.pillMuted}`}>pilot (no login)</span>}
+                  </td>
                   <td><span className={`${styles.pill} ${s.status === 'active' ? styles.pillOk : styles.pillMuted}`}>{s.status}</span></td>
                   <td>{s.last_active_at ? new Date(s.last_active_at).toLocaleString() : 'never'}</td>
                   <td><button className={styles.linkBtn} onClick={() => setSelected(s.student_id)}>Open</button></td>
@@ -36,21 +59,127 @@ export default function StudentsView() {
           </table>
         )}
       </div>
+    </>
   )
 }
 
-function StudentDetail({ studentId, onBack }) {
+function CreateStudentForm({ onCreated }) {
+  const [displayName, setDisplayName] = useState('')
+  const [email, setEmail] = useState('')
+  const [loginIdentifier, setLoginIdentifier] = useState('')
+  const [cohortId, setCohortId] = useState('')
+  const [cohorts, setCohorts] = useState(null)
+  const [busy, setBusy] = useState(false)
+  const [error, setError] = useState(null)
+  const [created, setCreated] = useState(null) // { student_id, activation_token } — token shown exactly once
+
+  useEffect(() => { adminApi.cohorts().then(setCohorts).catch(() => {}) }, [])
+
+  const submit = async (e) => {
+    e.preventDefault()
+    setBusy(true); setError(null); setCreated(null)
+    try {
+      const result = await adminApi.createStudent({
+        display_name: displayName.trim(), cohort_id: cohortId || undefined,
+        email: email.trim() || undefined, login_identifier: loginIdentifier.trim() || undefined,
+      })
+      setCreated(result)
+      setDisplayName(''); setEmail(''); setLoginIdentifier('')
+      onCreated()
+    } catch (e) { setError(e.message) } finally { setBusy(false) }
+  }
+
+  return (
+    <form onSubmit={submit}>
+      {error && <div className={styles.errorBox}>{error}</div>}
+      {created?.activation_token && (
+        <div className={styles.errorBox} style={{ background: 'var(--green-dim)', color: 'var(--green)', borderColor: 'var(--green)' }}>
+          Account created ({created.student_id}). Activation link (share it out of band, shown once):<br />
+          <code>{`${window.location.origin}/?activate=${created.activation_token}`}</code>
+        </div>
+      )}
+      <div className={styles.formRow}>
+        <div><label className={styles.label}>Display name</label>
+          <input className={styles.input} value={displayName} onChange={(e) => setDisplayName(e.target.value)} required /></div>
+        <div><label className={styles.label}>Email (older/self-paced)</label>
+          <input className={styles.input} type="email" value={email} onChange={(e) => setEmail(e.target.value)} /></div>
+        <div><label className={styles.label}>Login ID (younger/managed)</label>
+          <input className={styles.input} value={loginIdentifier} onChange={(e) => setLoginIdentifier(e.target.value)} /></div>
+        <div><label className={styles.label}>Cohort (optional)</label>
+          <select className={styles.select} value={cohortId} onChange={(e) => setCohortId(e.target.value)}>
+            <option value="">— none —</option>
+            {cohorts?.map((c) => <option key={c.cohort_id} value={c.cohort_id}>{c.name}</option>)}
+          </select></div>
+      </div>
+      <p className={styles.muted}>Leave email and login ID both blank to add a pilot (no-login) student.</p>
+      <button className={styles.btn} disabled={busy || !displayName.trim()}>{busy ? 'Adding…' : 'Add student'}</button>
+    </form>
+  )
+}
+
+function ImportForm({ onImported }) {
+  const [csv, setCsv] = useState('')
+  const [busy, setBusy] = useState(false)
+  const [error, setError] = useState(null)
+  const [rowErrors, setRowErrors] = useState(null)
+  const [created, setCreated] = useState(null)
+
+  const submit = async (e) => {
+    e.preventDefault()
+    setBusy(true); setError(null); setRowErrors(null); setCreated(null)
+    try {
+      const { created: rows } = await adminApi.importStudents(csv)
+      setCreated(rows); setCsv('')
+      onImported()
+    } catch (e) {
+      if (e.row_errors) setRowErrors(e.row_errors) // not thrown this way by the json() helper today; message covers it below
+      setError(e.message)
+    } finally { setBusy(false) }
+  }
+
+  return (
+    <form onSubmit={submit}>
+      <p className={styles.muted}>Columns: <code>display_name, email, login_identifier, cohort_id</code> (email/login_identifier/cohort_id optional per row). Never include a password column — the whole file is validated before anything is imported.</p>
+      {error && <div className={styles.errorBox}>{error}</div>}
+      {rowErrors && (
+        <div className={styles.errorBox}>{rowErrors.map((re, i) => <div key={i}>{re}</div>)}</div>
+      )}
+      {created && (
+        <div className={styles.errorBox} style={{ background: 'var(--green-dim)', color: 'var(--green)', borderColor: 'var(--green)' }}>
+          Imported {created.length} student(s). Activation links (shown once — copy before navigating away):
+          {created.filter((c) => c.activation_token).map((c) => (
+            <div key={c.student_id}><code>{c.display_name}: {window.location.origin}/?activate={c.activation_token}</code></div>
+          ))}
+        </div>
+      )}
+      <textarea className={styles.input} style={{ width: '100%', fontFamily: 'var(--mono)', minHeight: '6rem' }}
+        placeholder={'display_name,email,login_identifier,cohort_id\nJane Doe,[email protected],,'}
+        value={csv} onChange={(e) => setCsv(e.target.value)} required />
+      <button className={styles.btn} style={{ marginTop: '0.5rem' }} disabled={busy || !csv.trim()}>{busy ? 'Importing…' : 'Import'}</button>
+    </form>
+  )
+}
+
+function StudentDetail({ studentId, canManage, onBack }) {
+  const [student, setStudent] = useState(null)
   const [progress, setProgress] = useState(null)
   const [attempts, setAttempts] = useState(null)
   const [env, setEnv] = useState(null)
   const [events, setEvents] = useState(null)
+  const [sessions, setSessions] = useState(null)
   const [error, setError] = useState(null)
+  const [notice, setNotice] = useState(null)
   const [busy, setBusy] = useState(false)
 
   const reload = useCallback(() => {
+    adminApi.students().then((rows) => setStudent(rows.find((s) => s.student_id === studentId) || null)).catch(() => {})
     adminApi.studentProgress(studentId).then(setProgress).catch(() => {})
-    adminApi.studentAttempts(studentId).then(setAttempts).catch(() => {})
+    // Phase 4.1 §16 — studentAttempts is now paginated; this panel only ever shows the most
+    // recent page (labeled "Recent attempts," not a full history browser — see
+    // /api/admin/students/:id/attempts's own pagination for the underlying capability).
+    adminApi.studentAttempts(studentId).then((r) => setAttempts(r.items)).catch(() => {})
     adminApi.studentEnvironment(studentId).then((rows) => setEnv(rows?.find((e) => e.status !== 'destroyed') || null)).catch(() => {})
+    adminApi.studentSessions(studentId).then(setSessions).catch(() => {})
   }, [studentId])
   useEffect(reload, [reload])
 
@@ -69,6 +198,27 @@ function StudentDetail({ studentId, onBack }) {
       <button className={styles.linkBtn} onClick={onBack}>&larr; Back to students</button>
       <h2 style={{ margin: '1rem 0' }}>{studentId}</h2>
       {error && <div className={styles.errorBox}>{error}</div>}
+      {notice && <p className={styles.muted}>{notice}</p>}
+
+      {canManage && (
+        <div className={styles.card}>
+          <h3>Credentials</h3>
+          {student?.has_credentials ? (
+            <>
+              <p className={styles.muted}>Logs in with {student.email || student.login_identifier}. {sessions ? `${sessions.length} active session(s).` : ''}</p>
+              <div className={styles.formRow}>
+                <button className={styles.btnSecondary} disabled={busy} onClick={withBusy(async () => {
+                  const { reset_token } = await adminApi.resetStudentPassword(studentId)
+                  setNotice(`Password reset. Link (shown once): ${window.location.origin}/?reset=${reset_token}`)
+                })}>Reset password</button>
+                <button className={styles.btnDanger} disabled={busy || !sessions?.length} onClick={withBusy(() => adminApi.revokeStudentSessions(studentId))}>Revoke all sessions</button>
+              </div>
+            </>
+          ) : (
+            <ConvertForm studentId={studentId} onDone={(msg) => { setNotice(msg); reload() }} />
+          )}
+        </div>
+      )}
 
       <div className={styles.card}>
         <h3>Environment</h3>
@@ -131,3 +281,35 @@ function StudentDetail({ studentId, onBack }) {
     </>
   )
 }
+
+function ConvertForm({ studentId, onDone }) {
+  const [email, setEmail] = useState('')
+  const [loginIdentifier, setLoginIdentifier] = useState('')
+  const [busy, setBusy] = useState(false)
+  const [error, setError] = useState(null)
+
+  const submit = async (e) => {
+    e.preventDefault()
+    setBusy(true); setError(null)
+    try {
+      const { activation_token } = await adminApi.convertStudent(studentId, {
+        email: email.trim() || undefined, login_identifier: loginIdentifier.trim() || undefined,
+      })
+      onDone(`Converted. Activation link (shown once): ${window.location.origin}/?activate=${activation_token}`)
+    } catch (e) { setError(e.message) } finally { setBusy(false) }
+  }
+
+  return (
+    <form onSubmit={submit}>
+      <p className={styles.muted}>Still a pilot account (no login). Give it an email or login ID to convert it to a real, password-protected account.</p>
+      {error && <div className={styles.errorBox}>{error}</div>}
+      <div className={styles.formRow}>
+        <div><label className={styles.label}>Email</label>
+          <input className={styles.input} type="email" value={email} onChange={(e) => setEmail(e.target.value)} /></div>
+        <div><label className={styles.label}>Login ID</label>
+          <input className={styles.input} value={loginIdentifier} onChange={(e) => setLoginIdentifier(e.target.value)} /></div>
+        <div><button className={styles.btn} disabled={busy || (!email.trim() && !loginIdentifier.trim())}>{busy ? 'Converting…' : 'Convert'}</button></div>
+      </div>
+    </form>
+  )
+}

frontend/src/admin/api.js

@@ -16,7 +16,9 @@ function getCookie(name) {
 async function json(res) {
   if (!res.ok) {
     const body = await res.json().catch(() => ({}))
-    throw new Error(body.message || body.error || `Request failed (${res.status})`)
+    const err = new Error(body.message || body.error || `Request failed (${res.status})`)
+    if (body.row_errors) err.row_errors = body.row_errors // Phase 5A §29 — CSV import row-level errors
+    throw err
   }
   return res.json()
 }
@@ -49,10 +51,16 @@ export const adminApi = {
   dashboard: () => get('/api/admin/dashboard'),
 
   students: () => get('/api/admin/students'),
-  createStudent: (display_name, cohort_id) => post('/api/admin/students', { display_name, cohort_id }),
+  createStudent: (fields) => post('/api/admin/students', fields),
   updateStudent: (id, fields) => patch(`/api/admin/students/${id}`, fields),
+  // ── Phase 5A §4-7/§29: credential lifecycle + CSV import ────────────────────────────────
+  convertStudent: (id, fields) => post(`/api/admin/students/${id}/convert`, fields),
+  resetStudentPassword: (id) => post(`/api/admin/students/${id}/reset-password`),
+  revokeStudentSessions: (id) => post(`/api/admin/students/${id}/revoke-sessions`),
+  studentSessions: (id) => get(`/api/admin/students/${id}/sessions`),
+  importStudents: (csv) => post('/api/admin/students/import', { csv }),
   studentProgress: (id) => get(`/api/admin/students/${id}/progress`),
-  studentAttempts: (id) => get(`/api/admin/students/${id}/attempts`),
+  studentAttempts: (id, page = 1) => get(`/api/admin/students/${id}/attempts?page=${page}`), // Phase 4.1 §16 — paginated, see pagination.js
   studentEnvironment: (id) => get(`/api/admin/students/${id}/environment`),
   provisionStudentLab: (id) => post(`/api/admin/students/${id}/environment/provision`),
   assignStudentCohort: (id, cohort_id) => post(`/api/admin/students/${id}/cohort`, { cohort_id }),
@@ -79,19 +87,33 @@ export const adminApi = {
   revokeInstructorSessions: (id) => post(`/api/admin/instructors/${id}/revoke-sessions`),
   resetInstructorPassword: (id, new_password) => post(`/api/admin/instructors/${id}/reset-password`, { new_password }),
 
-  auditEvents: () => get('/api/admin/audit-events'),
+  auditEvents: (page = 1) => get(`/api/admin/audit-events?page=${page}`), // Phase 4.1 §16 — paginated, see pagination.js
 
   // ── Phase 3B: learning analytics ────────────────────────────────────────────────────────
   cohortLearning: (id) => get(`/api/admin/cohorts/${id}/learning`),
   studentLearning: (id) => get(`/api/admin/students/${id}/learning`),
   scenarioAnalytics: (cohortId) => get(`/api/admin/scenario-analytics${cohortId ? `?cohort_id=${cohortId}` : ''}`),
   needsAttention: (cohortId) => get(`/api/admin/cohorts/${cohortId}/needs-attention`),
+  scenarioCatalog: () => get('/api/admin/scenario-catalog'), // Phase 4.1 §18 — structured assessment editor
 
   // ── Phase 3B: assessment (exam + capstone) management ───────────────────────────────────
   assessments: () => get('/api/admin/assessments'),
   createAssessment: (fields) => post('/api/admin/assessments', fields),
   updateAssessment: (id, fields) => patch(`/api/admin/assessments/${id}`, fields),
   assignAssessment: (id, cohort_id) => post(`/api/admin/assessments/${id}/assign`, { cohort_id }),
-  assessmentSessions: (id) => get(`/api/admin/assessments/${id}/sessions`),
+  assessmentSessions: (id, page = 1) => get(`/api/admin/assessments/${id}/sessions?page=${page}`), // Phase 4.1 §16 — paginated, see pagination.js
   cancelAssessmentSession: (sessionId) => post(`/api/admin/assessment-sessions/${sessionId}/cancel`),
+
+  // ── Phase 5A §14/§21: remote classroom control center ───────────────────────────────────
+  classrooms: (cohortId) => get(`/api/admin/cohorts/${cohortId}/classroom-sessions`),
+  createClassroom: (cohortId, fields) => post(`/api/admin/cohorts/${cohortId}/classroom-sessions`, fields),
+  classroomDashboard: (id) => get(`/api/admin/classroom-sessions/${id}`),
+  startClassroom: (id) => post(`/api/admin/classroom-sessions/${id}/start`),
+  endClassroom: (id) => post(`/api/admin/classroom-sessions/${id}/end`),
+  cancelClassroom: (id) => post(`/api/admin/classroom-sessions/${id}/cancel`),
+  prepareClassroomLabs: (id) => post(`/api/admin/classroom-sessions/${id}/prepare-labs`),
+  setClassroomActivity: (id, scenario_id) => post(`/api/admin/classroom-sessions/${id}/activity`, { scenario_id }),
+  postClassroomAnnouncement: (id, message) => post(`/api/admin/classroom-sessions/${id}/announcements`, { message }),
+  acknowledgeHelpRequest: (id) => post(`/api/admin/help-requests/${id}/acknowledge`),
+  resolveHelpRequest: (id, note) => post(`/api/admin/help-requests/${id}/resolve`, { note }),
 }

frontend/src/platform/ClassroomTab.jsx

@@ -0,0 +1,146 @@
+import { useState, useEffect, useCallback, useRef } from 'react'
+import { api } from './api'
+import styles from './Platform.module.css'
+
+const HEARTBEAT_MS = 25000  // §15 — "browser heartbeat every 20-30 seconds"
+const POLL_MS = 5000
+
+const HELP_CATEGORIES = [
+  { value: 'lab_problem', label: 'Lab problem' },
+  { value: 'dont_understand', label: "Don't understand the task" },
+  { value: 'validation_failing', label: 'Validation failing' },
+  { value: 'other', label: 'Other' },
+]
+
+// Phase 5A §22 — the live classroom experience. No live classroom -> a quiet "nothing live
+// right now" state. Live -> title, current activity (guidance only — a link into the student's
+// own Lab, never an auto-redirect), announcements, Need Help, and the student's own Lab status
+// (reusing the exact same /api/environment/* the My Learning tab uses — no second Lab-status
+// system). Heartbeats fire only while this tab is mounted and a classroom is actually joined.
+export default function ClassroomTab() {
+  const [live, setLive] = useState(null)
+  const [detail, setDetail] = useState(null)
+  const [envStatus, setEnvStatus] = useState(null)
+  const [joined, setJoined] = useState(false)
+  const [helpCategory, setHelpCategory] = useState('lab_problem')
+  const [helpMessage, setHelpMessage] = useState('')
+  const [helpSent, setHelpSent] = useState(false)
+  const [error, setError] = useState(null)
+  const heartbeatRef = useRef(null)
+  const pollRef = useRef(null)
+
+  const loadLive = useCallback(() => {
+    api.liveClassroom().then((r) => setLive(r.classroom)).catch((e) => setError(e.message))
+  }, [])
+  useEffect(() => { loadLive() }, [loadLive])
+
+  const loadDetail = useCallback((id) => {
+    api.classroom(id).then(setDetail).catch((e) => setError(e.message))
+    api.environmentStatus().then(setEnvStatus).catch(() => {})
+  }, [])
+
+  useEffect(() => {
+    if (!live) return
+    loadDetail(live.classroom_session_id)
+    pollRef.current = setInterval(() => loadDetail(live.classroom_session_id), POLL_MS)
+    return () => clearInterval(pollRef.current)
+  }, [live, loadDetail])
+
+  useEffect(() => {
+    if (!joined || !live) return
+    heartbeatRef.current = setInterval(() => api.classroomHeartbeat(live.classroom_session_id).catch(() => {}), HEARTBEAT_MS)
+    api.classroomHeartbeat(live.classroom_session_id).catch(() => {}) // one immediately on join, not just after the first interval
+    return () => clearInterval(heartbeatRef.current)
+  }, [joined, live])
+
+  const join = async () => {
+    try { await api.joinClassroom(live.classroom_session_id); setJoined(true) }
+    catch (e) { setError(e.message) }
+  }
+
+  const sendHelp = async (e) => {
+    e.preventDefault()
+    try {
+      await api.requestHelp(live.classroom_session_id, helpCategory, helpMessage)
+      setHelpSent(true); setHelpMessage('')
+    } catch (e) { setError(e.message) }
+  }
+
+  const launchLab = async () => {
+    try { setEnvStatus(await api.launchEnvironment()) } catch (e) { setError(e.message) }
+  }
+
+  if (!live) {
+    return (
+      <div className={styles.card}>
+        <h2>Classroom</h2>
+        <p className={styles.muted}>No live class right now. Check back at your scheduled class time.</p>
+      </div>
+    )
+  }
+
+  return (
+    <div>
+      <div className={styles.card}>
+        <p className={styles.livePill}>LIVE NOW</p>
+        <h2 style={{ margin: '0.4rem 0' }}>{live.title}</h2>
+        {live.description && <p className={styles.muted}>{live.description}</p>}
+        {live.meeting_url && (
+          <p><a className={styles.primaryBtn} href={live.meeting_url} target="_blank" rel="noreferrer">Join Video Class</a></p>
+        )}
+        {!joined ? (
+          <button className={styles.primaryBtn} onClick={join}>Join Classroom</button>
+        ) : (
+          <p className={styles.muted}>You're in this classroom.</p>
+        )}
+      </div>
+
+      {joined && detail && (
+        <>
+          {detail.classroom.current_scenario_id && (
+            <div className={styles.card}>
+              <h3>Current class activity</h3>
+              <p>{detail.classroom.current_scenario_id}</p>
+              <p className={styles.muted}>Open it from the My Learning tab when you're ready — nothing here jumps you there automatically.</p>
+            </div>
+          )}
+
+          <div className={styles.card}>
+            <h3>Your Lab</h3>
+            {envStatus?.status === 'running'
+              ? <p><a className={styles.primaryBtn} href={envStatus.launch_path}>Open Lab</a></p>
+              : <button className={styles.primaryBtn} onClick={launchLab} disabled={envStatus?.status === 'provisioning'}>
+                  {envStatus?.status === 'provisioning' ? 'Starting…' : 'Launch Lab'}
+                </button>}
+          </div>
+
+          <div className={styles.card}>
+            <h3>Announcements</h3>
+            {!detail.announcements.length ? <p className={styles.muted}>None yet.</p> : (
+              <ul>
+                {detail.announcements.map((a) => <li key={a.announcement_id}>{a.message}</li>)}
+              </ul>
+            )}
+          </div>
+
+          <div className={styles.card}>
+            <h3>Need help?</h3>
+            {helpSent ? <p className={styles.muted}>Help requested — your instructor has been notified.</p> : (
+              <form onSubmit={sendHelp}>
+                <label className={styles.label}>What's going on?</label>
+                <select className={styles.select} value={helpCategory} onChange={(e) => setHelpCategory(e.target.value)}>
+                  {HELP_CATEGORIES.map((c) => <option key={c.value} value={c.value}>{c.label}</option>)}
+                </select>
+                <label className={styles.label}>Details (optional)</label>
+                <input className={styles.input} value={helpMessage} onChange={(e) => setHelpMessage(e.target.value)} />
+                <button className={styles.primaryBtn} type="submit">Need Help</button>
+              </form>
+            )}
+          </div>
+        </>
+      )}
+
+      {error && <div className={styles.errorBox}>{error}</div>}
+    </div>
+  )
+}

frontend/src/platform/ForgotPasswordScreen.jsx

@@ -0,0 +1,59 @@
+import { useState } from 'react'
+import { api } from './api'
+import styles from './Platform.module.css'
+
+// Phase 5A §6/§9 — self-service reset REQUEST. No email delivery exists yet
+// (docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md §5) — this always shows the same generic
+// message regardless of whether the identifier exists (anti-enumeration, matches the server's
+// own response shape exactly). In dev/test, the server includes dev_reset_token in its
+// response so this screen can offer a direct link — a real deployment omits it, and completing
+// the flow needs an instructor to run the admin-side reset instead (see
+// docs/RUNBOOK_RESET_STUDENT_PASSWORD.md).
+export default function ForgotPasswordScreen({ onCancel, onHaveToken }) {
+  const [identifier, setIdentifier] = useState('')
+  const [sent, setSent] = useState(null)
+  const [loading, setLoading] = useState(false)
+
+  const submit = async (e) => {
+    e.preventDefault()
+    setLoading(true)
+    try {
+      const result = await api.requestPasswordReset(identifier)
+      setSent(result)
+    } finally {
+      setLoading(false)
+    }
+  }
+
+  return (
+    <div className={styles.entryWrap}>
+      <div className={styles.entryCard}>
+        <div className={styles.brand}>
+          <span className={styles.brandIcon}>⎈</span> RootNode Academy
+        </div>
+        <p className={styles.tagline}>Reset your password</p>
+
+        {!sent ? (
+          <form onSubmit={submit}>
+            <label className={styles.label}>Email or login</label>
+            <input className={styles.input} type="text" value={identifier} onChange={(e) => setIdentifier(e.target.value)} required />
+            <button className={styles.primaryBtn} type="submit" disabled={loading}>{loading ? 'Sending…' : 'Send reset link'}</button>
+          </form>
+        ) : (
+          <>
+            <p style={{ color: 'var(--text-2)', fontSize: '0.9rem' }}>{sent.message}</p>
+            {sent.dev_reset_token && (
+              <p style={{ fontSize: '0.8rem', color: 'var(--text-3)' }}>
+                Dev/test only (never shown in production):{' '}
+                <button className={styles.linkBtn} onClick={() => onHaveToken(sent.dev_reset_token)}>use this reset token</button>
+              </p>
+            )}
+          </>
+        )}
+        <p style={{ marginTop: '1rem', textAlign: 'center' }}>
+          <button className={styles.linkBtn} onClick={onCancel}>Back to sign in</button>
+        </p>
+      </div>
+    </div>
+  )
+}

frontend/src/platform/LoginScreen.jsx

@@ -0,0 +1,63 @@
+import { useState } from 'react'
+import { api } from './api'
+import styles from './Platform.module.css'
+
+// Phase 5A §6 — the real, credentialed student login. This is now the default entry point
+// (see PlatformShell.jsx); EntryScreen (pilot, no password) is still reachable for
+// not-yet-converted accounts via the link at the bottom, not removed — see
+// docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md §7.
+export default function LoginScreen({ onLogin, onShowPilotEntry, onShowForgotPassword }) {
+  const [identifier, setIdentifier] = useState('')
+  const [password, setPassword] = useState('')
+  const [loading, setLoading] = useState(false)
+  const [error, setError] = useState(null)
+
+  const handleLogin = async (e) => {
+    e.preventDefault()
+    if (!identifier || !password) return
+    setLoading(true)
+    setError(null)
+    try {
+      const student = await api.studentLogin(identifier, password)
+      onLogin(student)
+    } catch (err) {
+      setError(err.message)
+    } finally {
+      setLoading(false)
+    }
+  }
+
+  return (
+    <div className={styles.entryWrap}>
+      <div className={styles.entryCard}>
+        <div className={styles.brand}>
+          <span className={styles.brandIcon}>⎈</span> RootNode Academy
+        </div>
+        <p className={styles.tagline}>Sign in to continue your learning.</p>
+
+        {error && <div className={styles.errorBox}>{error}</div>}
+
+        <form onSubmit={handleLogin}>
+          <label className={styles.label}>Email or login</label>
+          <input className={styles.input} type="text" autoComplete="username" value={identifier}
+            onChange={(e) => setIdentifier(e.target.value)} required />
+
+          <label className={styles.label}>Password</label>
+          <input className={styles.input} type="password" autoComplete="current-password" value={password}
+            onChange={(e) => setPassword(e.target.value)} required />
+
+          <button className={styles.primaryBtn} type="submit" disabled={loading}>
+            {loading ? 'Signing in…' : 'Sign in'}
+          </button>
+        </form>
+
+        <p style={{ marginTop: '1rem', textAlign: 'center' }}>
+          <button className={styles.linkBtn} onClick={onShowForgotPassword}>Forgot password?</button>
+        </p>
+        <p style={{ marginTop: '0.5rem', textAlign: 'center' }}>
+          <button className={styles.linkBtn} onClick={onShowPilotEntry}>Pick your name instead (pilot cohorts)</button>
+        </p>
+      </div>
+    </div>
+  )
+}

frontend/src/platform/Platform.module.css

@@ -36,7 +36,7 @@
 
 .label { display: block; font-size: 0.8rem; color: var(--text-2); margin: 0.9rem 0 0.3rem; }
 
-.select {
+.select, .input {
   width: 100%;
   padding: 0.55rem 0.6rem;
   background: var(--surface2);
@@ -45,6 +45,7 @@
   color: var(--text);
   font-family: var(--sans);
   font-size: 0.9rem;
+  box-sizing: border-box;
 }
 
 .primaryBtn {
@@ -179,5 +180,12 @@ table.dataTable { width: 100%; border-collapse: collapse; font-size: 0.85rem; }
 .pillErr { background: var(--red-dim); color: var(--red); }
 .pillMuted { background: var(--surface3); color: var(--text-2); }
 
+/* Phase 5A — classroom/profile screens */
+.livePill {
+  display: inline-block; background: var(--red-dim); color: var(--red); font-family: var(--mono);
+  font-size: 0.72rem; font-weight: 700; letter-spacing: 0.05em; padding: 0.2rem 0.6rem; border-radius: 999px;
+}
+.pager { display: flex; align-items: center; gap: 0.75rem; margin-top: 0.75rem; }
+
 .lockedBox { text-align: center; padding: 2rem 1rem; color: var(--text-2); }
 .checklistItem { display: flex; align-items: center; gap: 0.5rem; padding: 0.35rem 0; font-size: 0.88rem; }

frontend/src/platform/PlatformShell.jsx

@@ -1,17 +1,28 @@
 import { useState, useEffect } from 'react'
 import { api, getCachedDisplay } from './api'
+import LoginScreen from './LoginScreen'
 import EntryScreen from './EntryScreen'
+import SetPasswordScreen from './SetPasswordScreen'
+import ForgotPasswordScreen from './ForgotPasswordScreen'
 import StudentDashboard from './StudentDashboard'
 
-// Top-level Platform Shell: entry screen if no valid session, dashboard otherwise. Session
+// Top-level Platform Shell. Phase 5A: the credentialed LoginScreen is now the default entry
-// validity is decided by the server (GET /api/session/whoami against the rn_session cookie),
+// point (was EntryScreen's pilot name-picker); an activation/reset link
-// not by anything read from localStorage/sessionStorage — those only cache the display name
+// (?activate=<token> or ?reset=<token>) short-circuits straight to SetPasswordScreen before
-// for a snappier first paint, per api.js.
+// any session check, since neither flow requires an existing session. Session validity is
+// still decided by the server (GET /api/session/whoami against the rn_session cookie), not by
+// anything read from localStorage/sessionStorage.
 export default function PlatformShell() {
   const [student, setStudent] = useState(null)
   const [checking, setChecking] = useState(true)
+  const params = new URLSearchParams(window.location.search)
+  const [view, setView] = useState(
+    params.get('activate') ? 'activate' : params.get('reset') ? 'reset' : 'login'
+  )
+  const [resetToken, setResetToken] = useState(params.get('reset') || null)
 
   useEffect(() => {
+    if (view === 'activate' || view === 'reset') { setChecking(false); return }
     const cached = getCachedDisplay()
     if (cached) setStudent(cached) // optimistic paint while we confirm with the server
     api.whoami()
@@ -20,9 +31,29 @@ export default function PlatformShell() {
       })
       .catch(() => setStudent(null))
       .finally(() => setChecking(false))
-  }, [])
+  }, [view])
 
   if (checking) return null
-  if (!student) return <EntryScreen onLogin={setStudent} />
+
-  return <StudentDashboard student={student} onLogout={() => setStudent(null)} />
+  if (view === 'activate') {
+    return <SetPasswordScreen token={params.get('activate')} purpose="activation"
+      onDone={(s) => { setStudent(s); setView('dashboard'); window.history.replaceState({}, '', '/') }}
+      onCancel={() => { setView('login'); window.history.replaceState({}, '', '/') }} />
+  }
+  if (view === 'reset') {
+    return <SetPasswordScreen token={resetToken} purpose="reset"
+      onDone={() => { setView('login'); window.history.replaceState({}, '', '/') }}
+      onCancel={() => { setView('login'); window.history.replaceState({}, '', '/') }} />
+  }
+  if (view === 'forgot') {
+    return <ForgotPasswordScreen onCancel={() => setView('login')} onHaveToken={(t) => { setResetToken(t); setView('reset') }} />
+  }
+  if (view === 'pilot') {
+    return <EntryScreen onLogin={(s) => { setStudent(s); setView('dashboard') }} />
+  }
+  if (!student) {
+    return <LoginScreen onLogin={(s) => { setStudent(s); setView('dashboard') }}
+      onShowPilotEntry={() => setView('pilot')} onShowForgotPassword={() => setView('forgot')} />
+  }
+  return <StudentDashboard student={student} onLogout={() => { setStudent(null); setView('login') }} />
 }

frontend/src/platform/ProfileTab.jsx

@@ -0,0 +1,91 @@
+import { useState, useEffect } from 'react'
+import { api } from './api'
+import styles from './Platform.module.css'
+
+// Phase 5A §10 — minimal account/profile page. Deliberately narrow: identity, cohort, progress
+// summary, achievements, active session count, and change-password. No social/profile features.
+export default function ProfileTab() {
+  const [profile, setProfile] = useState(null)
+  const [error, setError] = useState(null)
+
+  useEffect(() => { api.profile().then(setProfile).catch((e) => setError(e.message)) }, [])
+
+  if (error) return <div className={styles.errorBox}>{error}</div>
+  if (!profile) return <p className={styles.muted}>Loading…</p>
+
+  return (
+    <div>
+      <div className={styles.card}>
+        <h2>Account</h2>
+        <p><strong>{profile.student.display_name}</strong></p>
+        <p className={styles.muted}>{profile.student.email || profile.student.login_identifier || 'No login identifier set'}</p>
+        <p className={styles.muted}>
+          {profile.cohorts.length ? profile.cohorts.map((c) => c.name).join(', ') : 'Not enrolled in a cohort yet'}
+        </p>
+        <p className={styles.muted}>{profile.active_sessions} active session{profile.active_sessions === 1 ? '' : 's'}</p>
+      </div>
+
+      <div className={styles.card}>
+        <h2>Progress</h2>
+        <p>{profile.progress_summary.completed} / {profile.progress_summary.total} scenarios completed</p>
+        <div className={styles.progressBarTrack}>
+          <div className={styles.progressBarFill} style={{ width: `${profile.progress_summary.total ? Math.round(100 * profile.progress_summary.completed / profile.progress_summary.total) : 0}%` }} />
+        </div>
+      </div>
+
+      <div className={styles.card}>
+        <h2>Achievements</h2>
+        {!profile.achievements.length ? <p className={styles.muted}>None yet — keep going!</p> : (
+          <ul>
+            {profile.achievements.map((a, i) => (
+              <li key={i}>{a.achievement_type.replace(/_/g, ' ')}{a.stage_id ? ` — ${a.stage_id}` : ''}</li>
+            ))}
+          </ul>
+        )}
+      </div>
+
+      <ChangePasswordCard hasCredentials={profile.student.has_credentials} />
+    </div>
+  )
+}
+
+function ChangePasswordCard({ hasCredentials }) {
+  const [oldPassword, setOldPassword] = useState('')
+  const [newPassword, setNewPassword] = useState('')
+  const [busy, setBusy] = useState(false)
+  const [notice, setNotice] = useState(null)
+  const [error, setError] = useState(null)
+
+  if (!hasCredentials) {
+    return (
+      <div className={styles.card}>
+        <h2>Change password</h2>
+        <p className={styles.muted}>This is still a pilot account with no password set. Ask your instructor to convert it to a real account first.</p>
+      </div>
+    )
+  }
+
+  const submit = async (e) => {
+    e.preventDefault()
+    setBusy(true); setError(null); setNotice(null)
+    try {
+      await api.changePassword(oldPassword, newPassword)
+      setNotice('Password updated.'); setOldPassword(''); setNewPassword('')
+    } catch (err) { setError(err.message) } finally { setBusy(false) }
+  }
+
+  return (
+    <div className={styles.card}>
+      <h2>Change password</h2>
+      {error && <div className={styles.errorBox}>{error}</div>}
+      {notice && <p className={styles.muted}>{notice}</p>}
+      <form onSubmit={submit}>
+        <label className={styles.label}>Current password</label>
+        <input className={styles.input} type="password" autoComplete="current-password" value={oldPassword} onChange={(e) => setOldPassword(e.target.value)} required />
+        <label className={styles.label}>New password</label>
+        <input className={styles.input} type="password" autoComplete="new-password" value={newPassword} onChange={(e) => setNewPassword(e.target.value)} required minLength={10} />
+        <button className={styles.primaryBtn} type="submit" disabled={busy}>{busy ? 'Saving…' : 'Update password'}</button>
+      </form>
+    </div>
+  )
+}

frontend/src/platform/SetPasswordScreen.jsx

@@ -0,0 +1,76 @@
+import { useState, useEffect } from 'react'
+import { api } from './api'
+import styles from './Platform.module.css'
+
+// Phase 5A §5/§6 — one screen for both flows the credential token mechanism serves:
+// activation (first login, purpose='activation') and forgot-password (purpose='reset'). Same
+// UI, different API call and success copy.
+export default function SetPasswordScreen({ token, purpose, onDone, onCancel }) {
+  const [preview, setPreview] = useState(null)
+  const [previewError, setPreviewError] = useState(null)
+  const [password, setPassword] = useState('')
+  const [confirm, setConfirm] = useState('')
+  const [loading, setLoading] = useState(false)
+  const [error, setError] = useState(null)
+
+  useEffect(() => {
+    api.credentialTokenPreview(purpose, token).then(setPreview).catch((e) => setPreviewError(e.message))
+  }, [token, purpose])
+
+  const submit = async (e) => {
+    e.preventDefault()
+    if (password.length < 10) { setError('Password must be at least 10 characters.'); return }
+    if (password !== confirm) { setError('Passwords do not match.'); return }
+    setLoading(true)
+    setError(null)
+    try {
+      if (purpose === 'activation') {
+        const student = await api.activate(token, password)
+        onDone(student)
+      } else {
+        await api.resetPassword(token, password)
+        onDone(null) // reset doesn't log the browser in — every existing session was revoked, sign in fresh
+      }
+    } catch (err) {
+      setError(err.message)
+    } finally {
+      setLoading(false)
+    }
+  }
+
+  return (
+    <div className={styles.entryWrap}>
+      <div className={styles.entryCard}>
+        <div className={styles.brand}>
+          <span className={styles.brandIcon}>⎈</span> RootNode Academy
+        </div>
+        <p className={styles.tagline}>{purpose === 'activation' ? 'Set up your account' : 'Choose a new password'}</p>
+
+        {previewError && (
+          <div className={styles.errorBox}>
+            This link is invalid or has expired. Ask your instructor for a new one.
+          </div>
+        )}
+        {preview && <p style={{ color: 'var(--text-2)', fontSize: '0.9rem' }}>Hi, {preview.display_name}.</p>}
+        {error && <div className={styles.errorBox}>{error}</div>}
+
+        {!previewError && (
+          <form onSubmit={submit}>
+            <label className={styles.label}>New password</label>
+            <input className={styles.input} type="password" autoComplete="new-password" value={password}
+              onChange={(e) => setPassword(e.target.value)} required minLength={10} />
+            <label className={styles.label}>Confirm password</label>
+            <input className={styles.input} type="password" autoComplete="new-password" value={confirm}
+              onChange={(e) => setConfirm(e.target.value)} required minLength={10} />
+            <button className={styles.primaryBtn} type="submit" disabled={loading}>
+              {loading ? 'Saving…' : purpose === 'activation' ? 'Activate my account' : 'Set new password'}
+            </button>
+          </form>
+        )}
+        <p style={{ marginTop: '1rem', textAlign: 'center' }}>
+          <button className={styles.linkBtn} onClick={onCancel}>Back to sign in</button>
+        </p>
+      </div>
+    </div>
+  )
+}

frontend/src/platform/StudentDashboard.jsx

@@ -6,13 +6,17 @@ import PracticeTab from './PracticeTab'
 import AssessmentsTab from './AssessmentsTab'
 import CapstoneTab from './CapstoneTab'
 import AchievementsTab from './AchievementsTab'
+import ClassroomTab from './ClassroomTab'
+import ProfileTab from './ProfileTab'
 
 const TABS = [
   { key: 'learning', label: 'My Learning' },
+  { key: 'classroom', label: 'Classroom' },
   { key: 'practice', label: 'Practice' },
   { key: 'assessments', label: 'Assessments' },
   { key: 'capstone', label: 'Capstone' },
   { key: 'achievements', label: 'Achievements' },
+  { key: 'profile', label: 'Profile' },
 ]
 
 // Student home — tabbed as of Phase 3B (docs/PHASE3B_LEARNING_AND_ASSESSMENT.md §10): My
@@ -41,10 +45,12 @@ export default function StudentDashboard({ student, onLogout }) {
       </nav>
 
       {tab === 'learning' && <MyLearningTab />}
+      {tab === 'classroom' && <ClassroomTab />}
       {tab === 'practice' && <PracticeTab />}
       {tab === 'assessments' && <AssessmentsTab />}
       {tab === 'capstone' && <CapstoneTab />}
       {tab === 'achievements' && <AchievementsTab />}
+      {tab === 'profile' && <ProfileTab />}
     </div>
   )
 }

frontend/src/platform/api.js

@@ -83,4 +83,40 @@ export const api = {
   capstone: () => fetch('/api/capstone').then(json),
 
   achievements: () => fetch('/api/achievements').then(json),
+
+  // ── Phase 5A: real student accounts ─────────────────────────────────────────────────────
+  // No CSRF token for login/activate/reset-request/reset — same reasoning as the pilot login
+  // above: nothing authenticated exists yet to forge into.
+  studentLogin: async (identifier, password) => {
+    const { student } = await fetch('/api/student/login', {
+      method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ identifier, password }),
+    }).then(json)
+    cacheDisplay(student)
+    return student
+  },
+  activate: async (token, password) => {
+    const { student } = await fetch('/api/student/activate', {
+      method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ token, password }),
+    }).then(json)
+    cacheDisplay(student)
+    return student
+  },
+  credentialTokenPreview: (purpose, token) => fetch(`/api/student/credential-token/${purpose}?token=${encodeURIComponent(token)}`).then(json),
+  requestPasswordReset: (identifier) => fetch('/api/student/request-password-reset', {
+    method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ identifier }),
+  }).then(json),
+  resetPassword: (token, password) => fetch('/api/student/reset-password', {
+    method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ token, password }),
+  }).then(json),
+  changePassword: (old_password, new_password) => mutate('/api/student/change-password', { old_password, new_password }),
+  profile: () => fetch('/api/student/profile').then(json),
+
+  // ── Phase 5A: remote classroom ───────────────────────────────────────────────────────────
+  liveClassroom: () => fetch('/api/classroom/live').then(json),
+  classroom: (id) => fetch(`/api/classroom/${id}`).then(json),
+  classroomAnnouncements: (id) => fetch(`/api/classroom/${id}/announcements`).then(json),
+  joinClassroom: (id) => mutate(`/api/classroom/${id}/join`),
+  leaveClassroom: (id) => mutate(`/api/classroom/${id}/leave`),
+  classroomHeartbeat: (id) => mutate(`/api/classroom/${id}/heartbeat`),
+  requestHelp: (id, category, message) => mutate(`/api/classroom/${id}/help-request`, { category, message }),
 }

infra/.gitignore

@@ -0,0 +1,34 @@
+# Never commit filled secrets — only *.env.template files are tracked.
+*.env
+!*.env.template
+
+# Docker TLS material — CA/server/client certs and keys, generated by
+# infra/scripts/docker-tls/generate-ca.sh and friends. Templates/READMEs stay tracked.
+docker/tls/*.pem
+docker/tls/*.key
+docker/tls/*.crt
+docker/tls/*.srl
+!docker/tls/**/README.md
+
+# Ansible Vault password files and any decrypted vars
+ansible/.vault_pass
+ansible/**/vault_plain.yml
+
+# Terraform/OpenTofu state and generated vars (may contain resource IDs/IPs)
+# Phase 4.1: these patterns were previously written repo-root-relative (e.g.
+# "infra/terraform/...") inside THIS file, which git resolves relative to the .gitignore's own
+# directory (infra/) — so they silently matched nothing at all. Found via `git status` showing
+# a real .terraform/ provider cache as untracked when it should have been ignored. Fixed by
+# dropping the redundant leading "infra/" from every pattern below.
+terraform/**/*.tfstate
+terraform/**/*.tfstate.*
+terraform/**/.terraform/
+terraform/**/terraform.tfvars
+!terraform/**/terraform.tfvars.example
+
+# Backup output and restore scratch space
+backup/output/
+backup/restore-test/
+
+# Load test results (local artifacts, not source)
+scripts/loadtest/results/

infra/ansible/ansible.cfg

@@ -0,0 +1,10 @@
+[defaults]
+inventory = inventory/production.ini
+roles_path = roles
+host_key_checking = True
+retry_files_enabled = False
+stdout_callback = yaml
+
+[privilege_escalation]
+become = True
+become_method = sudo

infra/ansible/group_vars/production.vault.yml.template

@@ -0,0 +1,22 @@
+# Ansible Vault template — copy to production.vault.yml and encrypt it, NEVER commit the
+# plaintext or the encrypted file to git (infra/.gitignore already excludes
+# **/vault_plain.yml; the encrypted production.vault.yml itself is small/safe to commit IF
+# encrypted — see docs/PRODUCTION_DEPLOYMENT.md §10 for the actual decision made). Ansible
+# Vault chosen over SOPS/age here because it's already bundled with Ansible (no new tool to
+# install/trust) and this project's secret surface is small enough that Vault's simplicity
+# outweighs SOPS's per-key-value granularity — revisit if that changes.
+#
+#   cp production.vault.yml.template production.vault.yml
+#   # fill in real values below
+#   ansible-vault encrypt production.vault.yml
+#
+# Reference a vaulted value from group_vars/production.yml or a role like:
+#   postgres_password: "{{ vault_postgres_password }}"
+
+vault_postgres_password: __REPLACE__
+vault_provisioner_shared_secret: __REPLACE_openssl_rand_hex_32__
+vault_owner_bootstrap_password: __REPLACE__          # used once, by a human, per docs/RUNBOOK_FIRST_DEPLOYMENT.md
+vault_postgres_exporter_password: __REPLACE__
+vault_backup_encryption_passphrase: __REPLACE_openssl_rand_base64_32__
+vault_backup_offsite_credentials: __REPLACE_e.g._Hetzner_Storage_Box_ssh_key_passphrase_or_S3_secret_key__
+vault_grafana_admin_password: __REPLACE__

infra/ansible/group_vars/production.yml

@@ -0,0 +1,33 @@
+# Production variables — referenced by every {{ }} placeholder in infra/docker/, infra/firewall/,
+# and the roles below. Fill in the __PLACEHOLDER__ values once the real server exists
+# (docs/PRODUCTION_DEPLOYMENT.md §1). Nothing here is a secret — real secrets live in
+# infra/environments/production/*.env (gitignored) and Ansible Vault (see
+# infra/ansible/group_vars/production.vault.yml.template), never in this plain-vars file.
+
+env: production
+academy_domain: __REPLACE_e.g._academy.rootnode.example__
+
+# ── Network topology (Phase 4 §5 — placeholders until real hardware exists) ────────────────
+platform_ip: __PLATFORM_VM_IP__
+lab_host_ip: __LAB_HOST_VM_IP__
+management_cidr: __YOUR_ADMIN_ACCESS_CIDR__       # e.g. a VPN/WireGuard subnet, never 0.0.0.0/0
+proxmox_mgmt_cidr: __PROXMOX_HOST_MGMT_CIDR__
+lab_subnet_range: "10.42.0.0/16"                    # matches NO_PROXY_CIDRS already used by the app
+lab_router_port: "8443"                             # Gateway -> Lab Host Router (data plane) — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3
+
+# ── Storage (Phase 4 §4 — a decision framework, not a final layout; see docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §4)
+rootnode_data_root: /srv/rootnode-data   # expected to be a ZFS dataset mountpoint on the real host
+
+# ── Registry / images (infra/docker/IMAGE_STRATEGY.md) ─────────────────────────────────────
+registry: git.synthossystems.com
+image_tag: __GIT_SHA_OR_RELEASE_TAG__
+
+postgres_exporter_user: rootnode_exporter
+
+# ── Observability (infra/monitoring/) ───────────────────────────────────────────────────────
+prometheus_retention_days: 30
+alertmanager_webhook_url: __REPLACE_e.g._uptime_kuma_or_ntfy_webhook__
+
+# ── Backup (infra/backup/) ───────────────────────────────────────────────────────────────────
+backup_retention_days: 30
+backup_offsite_target: __REPLACE_e.g._Hetzner_Storage_Box_path_or_S3_bucket__

infra/ansible/group_vars/staging.yml

@@ -0,0 +1,22 @@
+# Staging variables — same shape as production.yml, lower stakes. See
+# infra/environments/staging/README.md.
+
+env: staging
+academy_domain: __REPLACE_e.g._staging.academy.rootnode.example_or_unset_for_no_public_TLS__
+
+platform_ip: __STAGING_PLATFORM_IP__
+lab_host_ip: __STAGING_LAB_HOST_IP__
+management_cidr: __YOUR_ADMIN_ACCESS_CIDR__
+proxmox_mgmt_cidr: __PROXMOX_HOST_MGMT_CIDR__
+lab_subnet_range: "10.42.0.0/16"
+
+rootnode_data_root: /srv/rootnode-data-staging
+
+registry: git.synthossystems.com
+image_tag: __STAGING_TAG_e.g._staging-<sha>__
+
+prometheus_retention_days: 7
+alertmanager_webhook_url: __OPTIONAL_staging_can_skip_alerting__
+
+backup_retention_days: 7
+backup_offsite_target: __OPTIONAL_staging_backups_are_lower_priority__

infra/ansible/inventory/production.ini

@@ -0,0 +1,16 @@
+# Production inventory. IPs/hostnames are placeholders — fill in once the real VMs exist
+# (docs/PRODUCTION_DEPLOYMENT.md §1). This file is safe to commit (no secrets, just topology).
+
+[platform]
+rn-platform-01 ansible_host=__PLATFORM_VM_IP__
+
+[lab_host]
+rn-lab-host-01 ansible_host=__LAB_HOST_VM_IP__
+
+[rootnode:children]
+platform
+lab_host
+
+[rootnode:vars]
+ansible_user=rootnode-deploy
+ansible_ssh_common_args='-o StrictHostKeyChecking=yes'

infra/ansible/inventory/staging.ini

@@ -0,0 +1,16 @@
+# Staging inventory — same group shape as production, pointed at homelab VMs/containers or a
+# resource-constrained slice of the dedicated server once it exists (infra/environments/staging/README.md).
+
+[platform]
+rn-platform-staging ansible_host=__STAGING_PLATFORM_IP__
+
+[lab_host]
+rn-lab-host-staging ansible_host=__STAGING_LAB_HOST_IP__
+
+[rootnode:children]
+platform
+lab_host
+
+[rootnode:vars]
+ansible_user=rootnode-deploy
+env=staging

infra/ansible/roles/backup/tasks/main.yml

@@ -0,0 +1,63 @@
+---
+# Deploys the backup script + a daily cron job on rn-platform-01. The script itself
+# (infra/backup/backup-postgres.sh) is identical to what infra/backup/test-backup-restore.sh
+# exercises locally — same code path, just pointed at real production values here.
+
+- name: Install postgresql-client + gpg (backup script dependencies)
+  ansible.builtin.apt:
+    name: [postgresql-client, gnupg]
+    state: present
+
+- name: Create backup directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0700"
+  loop:
+    - /opt/rootnode/backup
+    - "{{ rootnode_data_root }}/backups"
+
+- name: Deploy backup script
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../backup/backup-postgres.sh"
+    dest: /opt/rootnode/backup/backup-postgres.sh
+    mode: "0750"
+
+- name: Deploy restore + verify scripts (not run by cron, kept alongside for the runbook)
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../backup/{{ item }}"
+    dest: "/opt/rootnode/backup/{{ item }}"
+    mode: "0750"
+  loop:
+    - restore-postgres.sh
+    - verify-restore.sh
+
+- name: Deploy the backup env file (PLATFORM_DB_URL, encryption passphrase, offsite target)
+  ansible.builtin.template:
+    src: backup.env.j2
+    dest: /etc/rootnode/backup.env
+    owner: root
+    group: root
+    mode: "0600"
+
+- name: Deploy cron wrapper (sources the env file, writes to syslog too)
+  ansible.builtin.template:
+    src: run-backup.sh.j2
+    dest: /opt/rootnode/backup/run-backup.sh
+    mode: "0750"
+
+- name: Install daily backup cron job
+  ansible.builtin.cron:
+    name: rootnode-postgres-backup
+    user: root
+    minute: "15"
+    hour: "3"
+    job: /opt/rootnode/backup/run-backup.sh
+
+- name: Ensure node-exporter's textfile collector directory exists (backup metrics land here)
+  ansible.builtin.file:
+    path: /var/lib/node-exporter/textfile
+    state: directory
+    mode: "0755"

infra/ansible/roles/backup/templates/backup.env.j2

@@ -0,0 +1,6 @@
+PLATFORM_DB_URL=postgres://rootnode_platform:{{ vault_postgres_password }}@localhost:5432/rootnode_platform
+BACKUP_DIR={{ rootnode_data_root }}/backups
+BACKUP_ENCRYPTION_PASSPHRASE={{ vault_backup_encryption_passphrase }}
+BACKUP_RETENTION_DAYS={{ backup_retention_days }}
+BACKUP_OFFSITE_TARGET={{ backup_offsite_target }}
+METRICS_TEXTFILE_DIR=/var/lib/node-exporter/textfile

infra/ansible/roles/backup/templates/run-backup.sh.j2

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+# Cron wrapper — sources the deployed env file and runs the real backup script, logging to
+# syslog (visible via journalctl and shipped to Loki by promtail like every other unit) in
+# addition to whatever cron itself captures.
+set -euo pipefail
+set -a
+source /etc/rootnode/backup.env
+set +a
+exec /opt/rootnode/backup/backup-postgres.sh 2>&1 | systemd-cat -t rootnode-backup

infra/ansible/roles/common/defaults/main.yml

@@ -0,0 +1,3 @@
+---
+common_timezone: UTC
+common_unattended_upgrades: true

infra/ansible/roles/common/tasks/docker-engine.yml

@@ -0,0 +1,35 @@
+---
+# Shared Docker Engine install — used by both platform and lab-host roles (import_tasks from
+# common, not duplicated). Follows Docker's own official apt-repo installation method.
+
+- name: Add Docker's GPG key
+  ansible.builtin.get_url:
+    url: https://download.docker.com/linux/ubuntu/gpg
+    dest: /etc/apt/keyrings/docker.asc
+    mode: "0644"
+  register: docker_gpg
+  retries: 3
+  delay: 5
+  until: docker_gpg is succeeded
+
+- name: Add Docker apt repository
+  ansible.builtin.apt_repository:
+    repo: "deb [arch={{ ansible_architecture | replace('x86_64','amd64') }} signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable"
+    state: present
+    filename: docker
+
+- name: Install Docker Engine + Compose plugin
+  ansible.builtin.apt:
+    name:
+      - docker-ce
+      - docker-ce-cli
+      - containerd.io
+      - docker-compose-plugin
+    state: present
+    update_cache: true
+
+- name: Enable and start Docker
+  ansible.builtin.systemd:
+    name: docker
+    enabled: true
+    state: started

infra/ansible/roles/common/tasks/main.yml

@@ -0,0 +1,58 @@
+---
+# Baseline hardening applied to both rn-platform-01 and rn-lab-host-01 before anything
+# RootNode-specific — idempotent, safe to re-run (Phase 4 §20 requirement).
+
+- name: Set timezone
+  community.general.timezone:
+    name: "{{ common_timezone }}"
+
+- name: Install base packages
+  ansible.builtin.apt:
+    name:
+      - curl
+      - ca-certificates
+      - gnupg
+      - iptables
+      - iptables-persistent
+      - unattended-upgrades
+      - fail2ban
+      - jq
+    state: present
+    update_cache: true
+    cache_valid_time: 3600
+
+- name: Enable unattended security upgrades
+  ansible.builtin.copy:
+    dest: /etc/apt/apt.conf.d/20auto-upgrades
+    content: |
+      APT::Periodic::Update-Package-Lists "1";
+      APT::Periodic::Unattended-Upgrade "1";
+    mode: "0644"
+  when: common_unattended_upgrades
+
+- name: Create the rootnode-deploy service account (if not the connecting user already)
+  ansible.builtin.user:
+    name: rootnode-deploy
+    shell: /bin/bash
+    groups: docker
+    append: true
+    create_home: true
+  register: deploy_user_result
+  failed_when: false   # non-fatal — some hosts may already use this as ansible_user
+
+- name: Ensure fail2ban is enabled for SSH
+  ansible.builtin.service:
+    name: fail2ban
+    state: started
+    enabled: true
+
+- name: Create /opt/rootnode and /etc/rootnode base directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0750"
+  loop:
+    - /opt/rootnode
+    - /etc/rootnode

infra/ansible/roles/docker-tls/defaults/main.yml

@@ -0,0 +1,6 @@
+---
+# Local path (on the machine running ansible-playbook, NOT on either VM) holding the cert
+# bundle generated by infra/scripts/docker-tls/generate-docker-tls.sh. The CA private key
+# (ca-key.pem) is deliberately never copied to either host by this role — only its public
+# ca.pem, plus the one cert/key pair each host actually needs.
+docker_tls_local_bundle_dir: "{{ lookup('env', 'ROOTNODE_TLS_BUNDLE_DIR') | default('/does/not/exist', true) }}"

infra/ansible/roles/docker-tls/handlers/main.yml

@@ -0,0 +1,22 @@
+---
+- name: restart docker with tls
+  ansible.builtin.systemd:
+    name: docker
+    state: restarted
+    daemon_reload: true
+
+# Phase 4.1 — these restart the whole per-host compose wrapper unit (no per-container systemd
+# units exist in this project's convention, see the `platform`/`lab-host` roles), which is
+# heavier than a surgical single-container restart but consistent with how every other
+# config-changed-so-restart-the-stack case is already handled here.
+- name: restart lab-router
+  ansible.builtin.systemd:
+    name: rootnode-lab-host.service
+    state: restarted
+  when: inventory_hostname in groups['lab_host']
+
+- name: restart gateway
+  ansible.builtin.systemd:
+    name: rootnode-platform.service
+    state: restarted
+  when: inventory_hostname in groups['platform']

infra/ansible/roles/docker-tls/tasks/main.yml

@@ -0,0 +1,136 @@
+---
+# Deploys Docker TLS material generated OFFLINE by infra/scripts/docker-tls/generate-docker-tls.sh
+# — this role never generates key material itself, and the CA private key never touches either
+# VM (see infra/docker/tls/README.md). Fails loudly if the operator forgot to set
+# ROOTNODE_TLS_BUNDLE_DIR / run the generation script first, rather than silently skipping and
+# leaving the Provisioner unable to reach rn-lab-host-01.
+
+- name: Verify the local TLS bundle directory exists
+  ansible.builtin.stat:
+    path: "{{ docker_tls_local_bundle_dir }}"
+  register: bundle_dir
+  delegate_to: localhost
+  become: false
+
+- name: Fail fast if the TLS bundle wasn't generated
+  ansible.builtin.fail:
+    msg: >-
+      ROOTNODE_TLS_BUNDLE_DIR ({{ docker_tls_local_bundle_dir }}) does not exist. Run
+      infra/scripts/docker-tls/generate-docker-tls.sh first, then re-run with
+      ROOTNODE_TLS_BUNDLE_DIR=<output dir> in the environment.
+  when: not bundle_dir.stat.exists
+
+- name: Create /etc/rootnode/docker-tls
+  ansible.builtin.file:
+    path: /etc/rootnode/docker-tls
+    state: directory
+    owner: root
+    group: root
+    mode: "0700"
+
+- name: Deploy CA cert (public, both hosts)
+  ansible.builtin.copy:
+    src: "{{ docker_tls_local_bundle_dir }}/ca.pem"
+    dest: /etc/rootnode/docker-tls/ca.pem
+    owner: root
+    group: root
+    mode: "0444"
+
+- name: Deploy server cert/key (rn-lab-host-01 only)
+  ansible.builtin.copy:
+    src: "{{ docker_tls_local_bundle_dir }}/{{ item.src }}"
+    dest: "/etc/rootnode/docker-tls/{{ item.dest }}"
+    owner: root
+    group: root
+    mode: "{{ item.mode }}"
+  loop:
+    - { src: server-cert.pem, dest: server-cert.pem, mode: "0444" }
+    - { src: server-key.pem, dest: server-key.pem, mode: "0400" }
+  when: inventory_hostname in groups['lab_host']
+  notify: restart docker with tls
+
+- name: Deploy Provisioner client cert/key (rn-platform-01 only)
+  ansible.builtin.copy:
+    src: "{{ docker_tls_local_bundle_dir }}/{{ item.src }}"
+    dest: "/etc/rootnode/docker-tls/{{ item.dest }}"
+    owner: root
+    group: root
+    mode: "{{ item.mode }}"
+  loop:
+    - { src: provisioner-client-cert.pem, dest: provisioner-client-cert.pem, mode: "0444" }
+    - { src: provisioner-client-key.pem, dest: provisioner-client-key.pem, mode: "0400" }
+  when: inventory_hostname in groups['platform']
+
+- name: Configure dockerd for TLS (rn-lab-host-01 only)
+  ansible.builtin.template:
+    src: daemon.json.j2
+    dest: /etc/docker/daemon.json
+    owner: root
+    group: root
+    mode: "0644"
+  when: inventory_hostname in groups['lab_host']
+  notify: restart docker with tls
+
+- name: Override the Docker systemd unit's ExecStart (clears the default -H fd:// flag)
+  ansible.builtin.file:
+    path: /etc/systemd/system/docker.service.d
+    state: directory
+    mode: "0755"
+  when: inventory_hostname in groups['lab_host']
+
+- name: Deploy the ExecStart override
+  ansible.builtin.template:
+    src: docker-override.conf.j2
+    dest: /etc/systemd/system/docker.service.d/override.conf
+    owner: root
+    group: root
+    mode: "0644"
+  when: inventory_hostname in groups['lab_host']
+  notify: restart docker with tls
+
+# ── Phase 4.1: the DATA-plane mTLS pair (Gateway <-> Lab Host Router). Same CA as the
+# CONTROL-plane pair above, separate directory (/etc/rootnode/lab-router-tls) so the two
+# concerns stay bind-mount-separable — the `lab-router`/`gateway` containers only ever need
+# their own pair mounted in, never the Docker TLS one. See
+# docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3.
+- name: Create /etc/rootnode/lab-router-tls
+  ansible.builtin.file:
+    path: /etc/rootnode/lab-router-tls
+    state: directory
+    owner: root
+    group: root
+    mode: "0700"
+
+- name: Deploy CA cert for the data-plane pair (public, both hosts)
+  ansible.builtin.copy:
+    src: "{{ docker_tls_local_bundle_dir }}/ca.pem"
+    dest: /etc/rootnode/lab-router-tls/ca.pem
+    owner: root
+    group: root
+    mode: "0444"
+
+- name: Deploy Lab Host Router server cert/key (rn-lab-host-01 only)
+  ansible.builtin.copy:
+    src: "{{ docker_tls_local_bundle_dir }}/{{ item.src }}"
+    dest: "/etc/rootnode/lab-router-tls/{{ item.dest }}"
+    owner: root
+    group: root
+    mode: "{{ item.mode }}"
+  loop:
+    - { src: router-server-cert.pem, dest: router-server-cert.pem, mode: "0444" }
+    - { src: router-server-key.pem, dest: router-server-key.pem, mode: "0400" }
+  when: inventory_hostname in groups['lab_host']
+  notify: restart lab-router
+
+- name: Deploy Gateway client cert/key (rn-platform-01 only)
+  ansible.builtin.copy:
+    src: "{{ docker_tls_local_bundle_dir }}/{{ item.src }}"
+    dest: "/etc/rootnode/lab-router-tls/{{ item.dest }}"
+    owner: root
+    group: root
+    mode: "{{ item.mode }}"
+  loop:
+    - { src: gateway-client-cert.pem, dest: gateway-client-cert.pem, mode: "0444" }
+    - { src: gateway-client-key.pem, dest: gateway-client-key.pem, mode: "0400" }
+  when: inventory_hostname in groups['platform']
+  notify: restart gateway

infra/ansible/roles/docker-tls/templates/daemon.json.j2

@@ -0,0 +1,13 @@
+{
+  "tls": true,
+  "tlsverify": true,
+  "tlscacert": "/etc/rootnode/docker-tls/ca.pem",
+  "tlscert": "/etc/rootnode/docker-tls/server-cert.pem",
+  "tlskey": "/etc/rootnode/docker-tls/server-key.pem",
+  "hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2376"],
+
+  "storage-driver": "overlay2",
+  "default-address-pools": [
+    { "base": "10.42.0.0/16", "size": 24 }
+  ]
+}

infra/ansible/roles/docker-tls/templates/docker-override.conf.j2

@@ -0,0 +1,7 @@
+# Docker's systemd unit specifies `-H fd://` on ExecStart by default, which conflicts with
+# setting `hosts` in daemon.json ("specified both as a flag and in the configuration file").
+# This override clears ExecStart and re-specifies it with no -H flag, letting daemon.json's
+# `hosts` list be the only source of truth — the documented fix for this exact conflict.
+[Service]
+ExecStart=
+ExecStart=/usr/bin/dockerd

infra/ansible/roles/firewall/handlers/main.yml

@@ -0,0 +1,4 @@
+---
+- name: reload firewall
+  ansible.builtin.command: netfilter-persistent reload
+  changed_when: true

infra/ansible/roles/firewall/tasks/main.yml

@@ -0,0 +1,28 @@
+---
+# Deploys the matching infra/firewall/*.rules file for this host's group and loads it. Same
+# ruleset, same variables as infra/firewall/validate-firewall.sh renders locally — this task
+# just uses Ansible's own templating instead of sed, against the real inventory values.
+
+- name: Determine which ruleset applies to this host
+  ansible.builtin.set_fact:
+    firewall_ruleset_name: "{{ 'rn-platform-01' if inventory_hostname in groups['platform'] else 'rn-lab-host-01' }}"
+
+- name: Render firewall ruleset for this host
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../firewall/{{ firewall_ruleset_name }}.rules"
+    dest: /etc/iptables/rules.v4.rootnode
+    owner: root
+    group: root
+    mode: "0600"
+  notify: reload firewall
+
+- name: Point iptables-persistent at the RootNode ruleset
+  ansible.builtin.file:
+    src: /etc/iptables/rules.v4.rootnode
+    dest: /etc/iptables/rules.v4
+    state: link
+    force: true
+  notify: reload firewall
+
+- name: Flush handlers so the firewall is active before later roles assume it
+  ansible.builtin.meta: flush_handlers

infra/ansible/roles/lab-host/handlers/main.yml

@@ -0,0 +1,6 @@
+---
+- name: restart rootnode-lab-host
+  ansible.builtin.systemd:
+    name: rootnode-lab-host
+    state: restarted
+    daemon_reload: true

infra/ansible/roles/lab-host/tasks/main.yml

@@ -0,0 +1,72 @@
+---
+# rn-lab-host-01: Docker Engine (TLS config comes from the docker-tls role, run before this
+# one), the egress proxy + monitoring agents compose stack, kernel prerequisites for nested
+# k3s in privileged Lab containers, and Docker daemon storage/limits configuration.
+
+- name: Install Docker Engine
+  ansible.builtin.import_tasks: "{{ role_path }}/../common/tasks/docker-engine.yml"
+
+- name: Kernel prerequisites for nested k3s (privileged containers)
+  ansible.posix.sysctl:
+    name: "{{ item.name }}"
+    value: "{{ item.value }}"
+    sysctl_set: true
+    state: present
+    reload: true
+  loop:
+    - { name: net.ipv4.ip_forward, value: "1" }
+    - { name: fs.inotify.max_user_watches, value: "524288" }   # k3s/kubelet watch a lot of files
+    - { name: fs.inotify.max_user_instances, value: "512" }
+    - { name: vm.max_map_count, value: "262144" }               # matches upstream k3s/etcd guidance
+
+- name: Ensure cgroup v2 controllers needed by nested k3s are available
+  ansible.builtin.command: cat /sys/fs/cgroup/cgroup.controllers
+  register: cgroup_controllers
+  changed_when: false
+  failed_when: false
+
+- name: Warn if cgroup v2 controllers look incomplete (informational only)
+  ansible.builtin.debug:
+    msg: >-
+      cgroup.controllers reports: {{ cgroup_controllers.stdout | default('unreadable') }}.
+      Expect at least cpu, memory, pids, io for reliable nested k3s — verify manually if any
+      are missing (see docs/PRODUCTION_VALIDATION_CHECKLIST.md "Lab boot" section).
+  when: cgroup_controllers.rc == 0
+
+- name: Create data/config directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: docker
+    mode: "0750"
+  loop:
+    - /opt/rootnode/lab-host
+    - "{{ rootnode_data_root }}/lab-work"   # disposable — see docs/BACKUP_DR.md, never backed up
+
+- name: Render production docker-compose.yml
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../docker/lab-host/docker-compose.production.yml"
+    dest: /opt/rootnode/lab-host/docker-compose.yml
+    owner: root
+    group: root
+    mode: "0640"
+  notify: restart rootnode-lab-host
+
+- name: Deploy systemd unit
+  ansible.builtin.template:
+    src: rootnode-lab-host.service.j2
+    dest: /etc/systemd/system/rootnode-lab-host.service
+    mode: "0644"
+  notify: restart rootnode-lab-host
+
+- name: Enable and start rootnode-lab-host
+  ansible.builtin.systemd:
+    name: rootnode-lab-host
+    enabled: true
+    state: started
+    daemon_reload: true
+
+# Storage-driver/networking daemon options live in the docker-tls role's daemon.json.j2 —
+# Docker only ever reads ONE /etc/docker/daemon.json, so that's the single source of truth for
+# this host's whole daemon config, not just its TLS settings (see that role's template).

infra/ansible/roles/lab-host/templates/rootnode-lab-host.service.j2

@@ -0,0 +1,17 @@
+[Unit]
+Description=RootNode Academy Lab Host services (egress proxy + monitoring agents)
+Requires=docker.service
+After=docker.service network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+WorkingDirectory=/opt/rootnode/lab-host
+ExecStart=/usr/bin/docker compose -f docker-compose.yml up -d --remove-orphans
+ExecStop=/usr/bin/docker compose -f docker-compose.yml down
+ExecReload=/usr/bin/docker compose -f docker-compose.yml up -d --remove-orphans
+TimeoutStartSec=180
+
+[Install]
+WantedBy=multi-user.target

infra/ansible/roles/monitoring-agent/handlers/main.yml

@@ -0,0 +1,16 @@
+---
+- name: reload prometheus
+  ansible.builtin.command: docker compose -f /opt/rootnode/platform/docker-compose.yml restart prometheus
+  changed_when: true
+
+- name: reload alertmanager
+  ansible.builtin.command: docker compose -f /opt/rootnode/platform/docker-compose.yml restart alertmanager
+  changed_when: true
+
+- name: restart platform promtail
+  ansible.builtin.command: docker compose -f /opt/rootnode/platform/docker-compose.yml restart promtail
+  changed_when: true
+
+- name: restart lab-host promtail
+  ansible.builtin.command: docker compose -f /opt/rootnode/lab-host/docker-compose.yml restart promtail
+  changed_when: true

infra/ansible/roles/monitoring-agent/tasks/main.yml

@@ -0,0 +1,75 @@
+---
+# Deploys infra/monitoring/ configs to whichever host needs them. Named monitoring-agent for
+# historical reasons in this repo's role list, but it deploys both the "agent" pieces
+# (node-exporter/cadvisor/promtail configs, already referenced directly by the platform/
+# lab-host compose files) and the central stack's config (Prometheus/Alertmanager/Loki/Grafana
+# on rn-platform-01 only).
+
+- name: Create /opt/rootnode/monitoring (rn-platform-01)
+  ansible.builtin.file:
+    path: /opt/rootnode/monitoring/rules
+    state: directory
+    mode: "0755"
+  when: inventory_hostname in groups['platform']
+
+- name: Deploy Prometheus config
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../monitoring/prometheus/prometheus.yml.j2"
+    dest: /opt/rootnode/monitoring/prometheus.yml
+    mode: "0644"
+  when: inventory_hostname in groups['platform']
+  notify: reload prometheus
+
+- name: Deploy alert rules
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../monitoring/prometheus/rules/rootnode.yml"
+    dest: /opt/rootnode/monitoring/rules/rootnode.yml
+    mode: "0644"
+  when: inventory_hostname in groups['platform']
+  notify: reload prometheus
+
+- name: Deploy blackbox_exporter config
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../monitoring/prometheus/blackbox.yml"
+    dest: /opt/rootnode/monitoring/blackbox.yml
+    mode: "0644"
+  when: inventory_hostname in groups['platform']
+
+- name: Deploy Alertmanager config
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../monitoring/alertmanager/alertmanager.yml.j2"
+    dest: /opt/rootnode/monitoring/alertmanager.yml
+    mode: "0644"
+  when: inventory_hostname in groups['platform']
+  notify: reload alertmanager
+
+- name: Deploy Loki config
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../monitoring/loki/loki-config.yml"
+    dest: /opt/rootnode/monitoring/loki-config.yml
+    mode: "0644"
+  when: inventory_hostname in groups['platform']
+
+- name: Deploy Grafana provisioning + dashboards
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../monitoring/grafana/{{ item }}/"
+    dest: "/opt/rootnode/monitoring/grafana/{{ item }}/"
+    mode: "0644"
+  loop: [provisioning, dashboards]
+  when: inventory_hostname in groups['platform']
+
+- name: Deploy platform-side promtail config
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../monitoring/promtail/promtail-config.platform.yml"
+    dest: /opt/rootnode/monitoring/promtail-config.yml
+    mode: "0644"
+  when: inventory_hostname in groups['platform']
+  notify: restart platform promtail
+
+- name: Deploy lab-host promtail config
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../monitoring/promtail/promtail-config.lab-host.yml.j2"
+    dest: /opt/rootnode/lab-host/promtail-config.yml
+    mode: "0644"
+  when: inventory_hostname in groups['lab_host']
+  notify: restart lab-host promtail

infra/ansible/roles/platform/handlers/main.yml

@@ -0,0 +1,6 @@
+---
+- name: restart rootnode-platform
+  ansible.builtin.systemd:
+    name: rootnode-platform
+    state: restarted
+    daemon_reload: true

infra/ansible/roles/platform/tasks/main.yml

@@ -0,0 +1,85 @@
+---
+# rn-platform-01: Docker Engine, directory structure, rendered compose file + env files,
+# Postgres tuning overlay, reverse proxy config, and the systemd unit that runs it all.
+# Docker TLS material and the host firewall are separate roles (docker-tls, firewall) run
+# before this one in site.yml — this role assumes both are already in place.
+
+- name: Install Docker Engine
+  ansible.builtin.import_tasks: "{{ role_path }}/../common/tasks/docker-engine.yml"
+
+- name: Create data/config directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: docker
+    mode: "0750"
+  loop:
+    - "{{ rootnode_data_root }}/postgres"
+    - "{{ rootnode_data_root }}/backups"
+    - /opt/rootnode/platform
+    - /opt/rootnode/platform/postgresql.conf.d
+    - /etc/rootnode/provisioner-config
+
+- name: Deploy Postgres tuning overlay
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../docker/postgres/postgresql.conf.d/00-rootnode-baseline.conf"
+    dest: /opt/rootnode/platform/postgresql.conf.d/00-rootnode-baseline.conf
+    mode: "0644"
+  notify: restart rootnode-platform
+
+- name: Deploy provisioner-config overlay ({{ env }})
+  ansible.builtin.copy:
+    src: "{{ role_path }}/../../../../environments/{{ env }}/provisioner-config/"
+    dest: /etc/rootnode/provisioner-config/
+    mode: "0644"
+  notify: restart rootnode-platform
+
+- name: Deploy per-service env files (from Ansible Vault-backed variables — see group_vars)
+  ansible.builtin.template:
+    src: "{{ item }}.env.j2"
+    dest: "/etc/rootnode/{{ item }}.env"
+    owner: root
+    group: root
+    mode: "0600"
+  loop:
+    - platform
+    - provisioner
+    - gateway
+    - postgres
+  notify: restart rootnode-platform
+
+- name: Render production docker-compose.yml
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../docker/platform/docker-compose.production.yml"
+    dest: /opt/rootnode/platform/docker-compose.yml
+    owner: root
+    group: root
+    mode: "0640"
+  notify: restart rootnode-platform
+
+- name: Render Caddyfile
+  ansible.builtin.template:
+    src: "{{ role_path }}/../../../docker/platform/Caddyfile.j2"
+    dest: /opt/rootnode/platform/Caddyfile
+    owner: root
+    group: root
+    mode: "0644"
+  notify: restart rootnode-platform
+
+- name: Deploy systemd unit
+  ansible.builtin.template:
+    src: rootnode-platform.service.j2
+    dest: /etc/systemd/system/rootnode-platform.service
+    mode: "0644"
+  notify: restart rootnode-platform
+
+# The `backup` role (applied after this one in site.yml) deploys the backup script and cron —
+# kept separate since it also has to run correctly on a host that ISN'T freshly deployed.
+
+- name: Enable and start rootnode-platform
+  ansible.builtin.systemd:
+    name: rootnode-platform
+    enabled: true
+    state: started
+    daemon_reload: true

infra/ansible/roles/platform/templates/gateway.env.j2

@@ -0,0 +1,13 @@
+# Rendered by Ansible — see platform.env.j2 for the pattern.
+PLATFORM_API_URL=http://api:4300
+PROVISIONER_URL=http://provisioner:4100
+PROVISIONER_SHARED_SECRET={{ vault_provisioner_shared_secret }}
+GATEWAY_PORT=4200
+# Phase 4.1 — mTLS client identity for the Lab Host Router (data plane). Only used when a
+# lab_host's routing_mode is 'router' (see platform/gateway/src/lab-endpoint.js); harmless if
+# every configured lab_host is still 'direct'. Paths are inside the container — mounted from
+# /etc/rootnode/lab-router-tls on the host by the `docker-tls` role, same pattern as the
+# Provisioner's Docker TLS client cert.
+LAB_ROUTER_TLS_CA=/tls/ca.pem
+LAB_ROUTER_TLS_CERT=/tls/gateway-client-cert.pem
+LAB_ROUTER_TLS_KEY=/tls/gateway-client-key.pem

infra/ansible/roles/platform/templates/platform.env.j2

@@ -0,0 +1,24 @@
+# Rendered by Ansible from infra/environments/{{ env }}/platform.env.template's shape —
+# real values come from group_vars/{{ env }}.yml + the encrypted vault file. Do not hand-edit
+# on the host; re-run the platform role instead.
+
+# service names, not localhost — production runs these on a named Docker bridge network
+# (rn-platform-internal), not network_mode: host like the dev compose file does.
+PLATFORM_DB_URL=postgres://rootnode_platform:{{ vault_postgres_password }}@postgres:5432/rootnode_platform
+PROVISIONER_URL=http://provisioner:4100
+PROVISIONER_SHARED_SECRET={{ vault_provisioner_shared_secret }}
+PLATFORM_API_PORT=4300
+
+STUDENT_SESSION_TTL_HOURS=8
+STUDENT_SESSION_IDLE_HOURS=2
+INSTRUCTOR_SESSION_TTL_HOURS=12
+INSTRUCTOR_SESSION_IDLE_HOURS=2
+
+COOKIE_SECURE=1
+TRUST_PROXY=1
+
+# Phase 4.1 §17 — instructor "needs attention" thresholds. Commented out (code defaults apply:
+# 3 failures, 7 stale days) — uncomment and override in group_vars/{{ env }}.yml only if a
+# cohort's real usage shows the defaults are wrong for it.
+# NEEDS_ATTENTION_FAIL_THRESHOLD=3
+# NEEDS_ATTENTION_STALE_DAYS=7

infra/ansible/roles/platform/templates/postgres.env.j2

@@ -0,0 +1,4 @@
+# Rendered by Ansible — see platform.env.j2 for the pattern.
+POSTGRES_USER=rootnode_platform
+POSTGRES_PASSWORD={{ vault_postgres_password }}
+POSTGRES_DB=rootnode_platform

infra/ansible/roles/platform/templates/provisioner.env.j2

@@ -0,0 +1,11 @@
+# Rendered by Ansible — see platform.env.j2 for the pattern.
+PLATFORM_DB_URL=postgres://rootnode_platform:{{ vault_postgres_password }}@postgres:5432/rootnode_platform
+PROVISIONER_SHARED_SECRET={{ vault_provisioner_shared_secret }}
+PROVISIONER_PORT=4100
+PROVISIONER_CONFIG_DIR=/app/config
+
+LAB_IMAGE_REF={{ registry }}/rootnode/lab:{{ image_tag }}
+
+EGRESS_PROXY_PORT=3128
+DNS_RESOLVER=127.0.0.11
+NO_PROXY_CIDRS=10.42.0.0/16,10.43.0.0/16

infra/ansible/roles/platform/templates/rootnode-platform.service.j2

@@ -0,0 +1,17 @@
+[Unit]
+Description=RootNode Academy Platform (docker compose)
+Requires=docker.service
+After=docker.service network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+WorkingDirectory=/opt/rootnode/platform
+ExecStart=/usr/bin/docker compose -f docker-compose.yml up -d --remove-orphans
+ExecStop=/usr/bin/docker compose -f docker-compose.yml down
+ExecReload=/usr/bin/docker compose -f docker-compose.yml up -d --remove-orphans
+TimeoutStartSec=300
+
+[Install]
+WantedBy=multi-user.target

infra/ansible/site.yml

@@ -0,0 +1,52 @@
+---
+# Top-level playbook — full production/staging deployment. Idempotent: re-running against an
+# already-deployed host converges it, doesn't rebuild from scratch (Phase 4 §20 requirement).
+#
+# Usage:
+#   ansible-playbook site.yml -i inventory/production.ini --vault-password-file .vault_pass
+#   ansible-playbook site.yml -i inventory/staging.ini -e env=staging
+#
+# Role order matters: firewall and docker-tls must land before platform/lab-host bring up
+# services that depend on the network policy / TLS material already being in place.
+
+- name: Base hardening — all RootNode hosts
+  hosts: rootnode
+  become: true
+  roles:
+    - common
+
+- name: Firewall policy — all RootNode hosts
+  hosts: rootnode
+  become: true
+  roles:
+    - firewall
+
+- name: Docker TLS — both hosts (server cert on lab_host, client cert on platform)
+  hosts: rootnode
+  become: true
+  roles:
+    - docker-tls
+
+- name: Deploy the Platform VM
+  hosts: platform
+  become: true
+  roles:
+    - platform
+
+- name: Deploy the Lab Host VM
+  hosts: lab_host
+  become: true
+  roles:
+    - lab-host
+
+- name: Deploy monitoring configuration — both hosts
+  hosts: rootnode
+  become: true
+  roles:
+    - monitoring-agent
+
+- name: Deploy backup automation — Platform VM only (that's where Postgres lives)
+  hosts: platform
+  become: true
+  roles:
+    - backup

infra/backup/backup-postgres.sh

@@ -0,0 +1,77 @@
+#!/usr/bin/env bash
+# Postgres backup — Tier 1 (irreplaceable) data per docs/BACKUP_DR.md. Runs via cron on
+# rn-platform-01 (deployed by the `backup` Ansible role), and locally testable against ANY
+# reachable Postgres (that's exactly what infra/backup/test-backup-restore.sh does).
+#
+# Produces one gzip-compressed pg_dump per run, encrypts it, prunes local copies past
+# BACKUP_RETENTION_DAYS, and (if BACKUP_OFFSITE_TARGET is set) rsyncs the encrypted copy
+# off-host. Writes a Prometheus textfile-collector metric so infra/monitoring/'s BackupFailed/
+# BackupAging alerts have real data to fire on.
+#
+# Required env: PLATFORM_DB_URL, BACKUP_DIR, BACKUP_ENCRYPTION_PASSPHRASE
+# Optional env: BACKUP_RETENTION_DAYS (default 30), BACKUP_OFFSITE_TARGET, METRICS_TEXTFILE_DIR
+set -euo pipefail
+
+: "${PLATFORM_DB_URL:?PLATFORM_DB_URL is required}"
+: "${BACKUP_DIR:?BACKUP_DIR is required}"
+: "${BACKUP_ENCRYPTION_PASSPHRASE:?BACKUP_ENCRYPTION_PASSPHRASE is required — see docs/BACKUP_DR.md}"
+BACKUP_RETENTION_DAYS="${BACKUP_RETENTION_DAYS:-30}"
+METRICS_TEXTFILE_DIR="${METRICS_TEXTFILE_DIR:-}"
+
+TS="$(date -u +%Y%m%dT%H%M%SZ)"
+mkdir -p "$BACKUP_DIR"
+DUMP_FILE="$BACKUP_DIR/rootnode-platform-$TS.sql.gz"
+ENC_FILE="$DUMP_FILE.gpg"
+
+write_metric() {
+  # $1 = metric line. Written atomically (tmp+rename) so Prometheus's node-exporter
+  # textfile-collector never scrapes a half-written file.
+  [ -n "$METRICS_TEXTFILE_DIR" ] || return 0
+  mkdir -p "$METRICS_TEXTFILE_DIR"
+  local tmp; tmp="$(mktemp "$METRICS_TEXTFILE_DIR/.rootnode_backup.XXXXXX")"
+  echo "$1" > "$tmp"
+  mv -f "$tmp" "$METRICS_TEXTFILE_DIR/rootnode_backup.prom"
+}
+
+fail() {
+  echo "BACKUP FAILED: $1" >&2
+  write_metric "rootnode_backup_last_run_success 0"
+  exit 1
+}
+trap 'fail "unexpected error at line $LINENO"' ERR
+
+echo "==> Dumping $PLATFORM_DB_URL to $DUMP_FILE"
+pg_dump --format=plain --no-owner --no-privileges "$PLATFORM_DB_URL" | gzip -9 > "$DUMP_FILE"
+
+# Fail loudly on a suspiciously small dump (empty/truncated) rather than "successfully"
+# backing up nothing — a real, if crude, sanity check.
+DUMP_SIZE=$(stat -c%s "$DUMP_FILE" 2>/dev/null || stat -f%z "$DUMP_FILE")
+[ "$DUMP_SIZE" -gt 1024 ] || fail "dump is suspiciously small ($DUMP_SIZE bytes) — refusing to treat this as a real backup"
+
+echo "==> Encrypting (AES-256, symmetric passphrase — see docs/BACKUP_DR.md for the recovery-key handling policy)"
+gpg --batch --yes --passphrase "$BACKUP_ENCRYPTION_PASSPHRASE" --symmetric --cipher-algo AES256 \
+  --output "$ENC_FILE" "$DUMP_FILE"
+rm -f "$DUMP_FILE"   # never leave the plaintext dump on disk once the encrypted copy exists
+
+echo "==> Local backup complete: $ENC_FILE ($(stat -c%s "$ENC_FILE" 2>/dev/null || stat -f%z "$ENC_FILE") bytes)"
+
+if [ -n "${BACKUP_OFFSITE_TARGET:-}" ]; then
+  echo "==> Copying off-host to $BACKUP_OFFSITE_TARGET"
+  # rsync over SSH works for both a Hetzner Storage Box (SSH/SFTP-native) and any other
+  # SSH-reachable target — the adapter point the Phase 4 spec asked for is this one variable,
+  # not a vendor-specific script. An S3-compatible target uses `aws s3 cp` / `rclone` instead;
+  # see docs/BACKUP_DR.md for both forms.
+  rsync -az --timeout=120 "$ENC_FILE" "$BACKUP_OFFSITE_TARGET/" \
+    || fail "off-host copy to $BACKUP_OFFSITE_TARGET failed"
+else
+  echo "==> BACKUP_OFFSITE_TARGET not set — local-only backup (fine for a local test run, NOT fine for production, see docs/BACKUP_DR.md)"
+fi
+
+echo "==> Pruning local backups older than $BACKUP_RETENTION_DAYS days"
+find "$BACKUP_DIR" -name 'rootnode-platform-*.sql.gz.gpg' -mtime "+$BACKUP_RETENTION_DAYS" -delete
+
+write_metric "rootnode_backup_last_run_success 1
+rootnode_backup_last_success_timestamp_seconds $(date +%s)
+rootnode_backup_size_bytes $(stat -c%s "$ENC_FILE" 2>/dev/null || stat -f%z "$ENC_FILE")"
+
+echo "==> Backup succeeded"

infra/backup/restore-postgres.sh

@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+# Restores an encrypted backup produced by backup-postgres.sh into a target database.
+# Deliberately requires an EXPLICIT target — never silently restores over the live production
+# database ("what would you do if this ran the wrong way" is a real disaster-recovery failure
+# mode, so making that impossible to trigger by accident matters more than convenience here).
+#
+# Usage: ./restore-postgres.sh <encrypted-backup-file> <target-db-url>
+#
+# For a REAL production restore, target-db-url should point at a FRESH database created for
+# this purpose (docs/RUNBOOK_POSTGRES_RESTORE.md walks through creating it) — restoring into a
+# database that already has data merges/conflicts rather than cleanly replacing it.
+set -euo pipefail
+
+ENC_FILE="${1:?usage: restore-postgres.sh <encrypted-backup-file> <target-db-url>}"
+TARGET_DB_URL="${2:?usage: restore-postgres.sh <encrypted-backup-file> <target-db-url>}"
+: "${BACKUP_ENCRYPTION_PASSPHRASE:?BACKUP_ENCRYPTION_PASSPHRASE is required to decrypt}"
+
+[ -f "$ENC_FILE" ] || { echo "No such file: $ENC_FILE" >&2; exit 1; }
+
+TMPDIR="$(mktemp -d)"
+trap 'rm -rf "$TMPDIR"' EXIT
+DUMP_FILE="$TMPDIR/restore.sql.gz"
+
+echo "==> Decrypting $ENC_FILE"
+gpg --batch --yes --passphrase "$BACKUP_ENCRYPTION_PASSPHRASE" --decrypt --output "$DUMP_FILE" "$ENC_FILE"
+
+echo "==> Confirming target is reachable"
+psql "$TARGET_DB_URL" -c "SELECT 1" > /dev/null
+
+echo "==> Target database currently has these tables (should be empty/unimportant before you continue):"
+psql "$TARGET_DB_URL" -c "\dt" || true
+read -r -p "Type RESTORE to proceed against $TARGET_DB_URL: " CONFIRM
+[ "$CONFIRM" = "RESTORE" ] || { echo "Aborted."; exit 1; }
+
+echo "==> Restoring"
+gunzip -c "$DUMP_FILE" | psql "$TARGET_DB_URL"
+
+echo "==> Restore complete. Verify with infra/backup/verify-restore.sh $TARGET_DB_URL"

infra/backup/seed-realistic-data.js

@@ -0,0 +1,94 @@
+#!/usr/bin/env node
+// Seeds realistic academy data for the Phase 4.1 §20 destructive restore drill — an owner, an
+// admin instructor, a cohort, 3 students, progress, attempts, an assessment definition +
+// session, and an achievement. Uses the REAL Platform API over HTTP for everything that
+// doesn't need a live Provisioner/k3s Lab (owner login, instructor/cohort/student/assessment
+// creation — genuinely exercises the real RBAC/creation code paths), and direct SQL inserts
+// only for progress/attempts/achievements (those need real scenario activity in the full app;
+// re-running the whole learning engine here would just re-prove what e2e-phase3b.js already
+// proves — this drill is about DATA DURABILITY, not re-testing scenario validation).
+//
+// Usage: PLATFORM_API_URL=http://localhost:PORT PLATFORM_DB_URL=postgres://... node seed-realistic-data.js
+
+const { Pool } = require('../../platform/api/node_modules/pg');
+
+const API = process.env.PLATFORM_API_URL || 'http://localhost:4390';
+const DB_URL = process.env.PLATFORM_DB_URL;
+if (!DB_URL) { console.error('Set PLATFORM_DB_URL'); process.exit(1); }
+
+function makeSession() {
+  const jar = new Map();
+  function cookieHeader() { return [...jar].map(([k, v]) => `${k}=${v}`).join('; '); }
+  function absorb(res) {
+    const cookies = typeof res.headers.getSetCookie === 'function' ? res.headers.getSetCookie() : [];
+    for (const c of cookies) {
+      const [pair] = c.split(';');
+      const eq = pair.indexOf('=');
+      if (eq === -1) continue;
+      jar.set(pair.slice(0, eq).trim(), pair.slice(eq + 1).trim());
+    }
+  }
+  return {
+    async req(method, path, body) {
+      const headers = { 'Content-Type': 'application/json' };
+      if (cookieHeader()) headers.Cookie = cookieHeader();
+      if (!['GET', 'HEAD'].includes(method)) headers['X-CSRF-Token'] = jar.get('rn_csrf_i') || '';
+      const res = await fetch(`${API}${path}`, { method, headers, body: body ? JSON.stringify(body) : undefined });
+      absorb(res);
+      const text = await res.text();
+      let json; try { json = JSON.parse(text); } catch { json = text; }
+      if (!res.ok) throw new Error(`${method} ${path} -> ${res.status}: ${JSON.stringify(json)}`);
+      return json;
+    },
+  };
+}
+
+async function main() {
+  const owner = makeSession();
+  await owner.req('POST', '/api/auth/instructor/login', { email: '[email protected]', password: 'DrillOwnerPass!2026' });
+  console.log('  owner login ok');
+
+  const admin = await owner.req('POST', '/api/admin/instructors', {
+    email: '[email protected]', display_name: 'Drill Admin', password: 'DrillAdminPass!2026', role: 'admin',
+  });
+  console.log('  admin instructor created:', admin.instructor_id);
+
+  const cohort = await owner.req('POST', '/api/admin/cohorts', { cohort_id: 'drill-cohort-01', name: 'Restore Drill Cohort' });
+  console.log('  cohort created');
+
+  const students = [];
+  for (const name of ['Drill Alice', 'Drill Bob', 'Drill Carlos']) {
+    const s = await owner.req('POST', '/api/admin/students', { display_name: name, cohort_id: 'drill-cohort-01' });
+    students.push(s.student_id);
+  }
+  console.log('  3 students rostered and enrolled:', students.join(', '));
+
+  const assessment = await owner.req('POST', '/api/admin/assessments', {
+    id: 'drill-exam-01', type: 'exam', title: 'Restore Drill Exam', duration_minutes: 30, hint_policy: 'disabled',
+    pool_spec: { required: ['pod-basics-mcq'], random: [] },
+  });
+  console.log('  assessment definition created:', assessment.assessment_definition_id || 'drill-exam-01');
+
+  // Direct SQL for progress/attempts/achievements — see file header for why.
+  const pool = new Pool({ connectionString: DB_URL });
+  for (const studentId of students) {
+    // environment_id is a real FK to the Provisioner's environments table (nullable) — left
+    // NULL here rather than faking a row that doesn't exist, same as how a genuinely old
+    // attempt whose environment has long since been cleaned up would look in real data.
+    await pool.query(
+      `INSERT INTO scenario_attempts (student_id, scenario_id, status, result, started_at, completed_at, validation_attempt_count, validation_pass_count)
+       VALUES ($1, 'pod-basics-mcq', 'completed', 'passed', now() - interval '1 hour', now(), 1, 1)`,
+      [studentId]
+    );
+    await pool.query(
+      `INSERT INTO achievements (student_id, achievement_type, stage_id, achieved_at) VALUES ($1, 'stage_completed', 'foundation', now())`,
+      [studentId]
+    );
+  }
+  await pool.end();
+  console.log('  progress/attempts/achievements seeded for all 3 students');
+
+  console.log('\nSeed complete.');
+}
+
+main().catch((e) => { console.error('SEED FAILED:', e.message); process.exit(1); });

infra/backup/test-backup-restore.sh

@@ -0,0 +1,133 @@
+#!/usr/bin/env bash
+# Local test (Phase 4 §29): a REAL end-to-end backup/restore cycle against disposable local
+# Postgres containers — not a mock. Applies the actual Platform API migration runner (the same
+# code path production uses, platform/api/src/db.js) so the schema under test is genuinely the
+# real one, inserts sample data, backs it up, restores into a SECOND empty Postgres, and
+# verifies the result. Proves the backup/restore MECHANISM works; does NOT prove anything about
+# the real off-host target (Storage Box/S3) — that needs real credentials, marked as
+# hardware/credential-dependent below.
+set -euo pipefail
+cd "$(dirname "$0")/../.."   # repo root
+
+SRC_CONTAINER=rootnode-backup-test-src
+DST_CONTAINER=rootnode-backup-test-dst
+SRC_PORT=55432
+DST_PORT=55433
+TESTDIR="$(mktemp -d)"
+export BACKUP_ENCRYPTION_PASSPHRASE="test-passphrase-not-for-production"
+
+cleanup() {
+  docker rm -f "$SRC_CONTAINER" "$DST_CONTAINER" >/dev/null 2>&1 || true
+  rm -rf "$TESTDIR"
+}
+trap cleanup EXIT
+
+# backup-postgres.sh/restore-postgres.sh assume real postgresql-client + gpg binaries — true on
+# a real deployment host (the `backup` Ansible role installs postgresql-client there). This dev
+# machine may not have psql/pg_dump installed at all, so for THIS TEST ONLY we shim them as
+# thin wrappers around `docker exec` into the two disposable test containers — the scripts
+# under test are otherwise completely unmodified from what production actually runs.
+if ! command -v psql >/dev/null 2>&1 || ! command -v pg_dump >/dev/null 2>&1; then
+  echo "==> psql/pg_dump not found locally — shimming both via docker exec for this test run"
+  SHIM_BIN="$TESTDIR/bin"
+  mkdir -p "$SHIM_BIN"
+  # Both wrappers pick a container from the target port (explicit -p, or embedded in a
+  # postgres://...:PORT/... URL), then re-exec psql/pg_dump INSIDE that container against ITS
+  # OWN localhost:5432 — the host-side port mapping (55432/55433) doesn't exist inside the
+  # container's network namespace, so the connection target itself must be replaced, not just
+  # which container runs the command.
+  cat > "$SHIM_BIN/psql" <<'EOF'
+#!/usr/bin/env bash
+args=(); port=""; want_port=0; skip_next=0
+for a in "$@"; do
+  if [ "$skip_next" = "1" ]; then
+    [ "$want_port" = "1" ] && port="$a"
+    skip_next=0; want_port=0; continue
+  fi
+  if [ "$a" = "-p" ]; then skip_next=1; want_port=1; continue; fi
+  if [ "$a" = "-h" ]; then skip_next=1; want_port=0; continue; fi
+  if [[ "$a" =~ ^postgres://[^:]+:[^@]+@[^:]+:([0-9]+)/ ]]; then
+    port="${BASH_REMATCH[1]}"
+    continue   # drop the URL — reconnect locally inside the container instead
+  fi
+  args+=("$a")
+done
+case "$port" in
+  55433) c="rootnode-backup-test-dst" ;;
+  *) c="rootnode-backup-test-src" ;;
+esac
+# Only forward local stdin (-i) for the one call that actually reads SQL from it (no -c/-tAc
+# in its args, i.e. `psql <target>` fed by a pipe) — every -c/-tAc call is self-contained and
+# must NOT hold docker exec's stdin open, or it silently steals bytes meant for a later `read`
+# elsewhere in the calling script (exactly what broke restore-postgres.sh's confirmation prompt
+# the first time this shim was written).
+needs_stdin=1
+for a in "${args[@]}"; do [ "$a" = "-c" ] || [ "$a" = "-tAc" ] && needs_stdin=0; done
+if [ "$needs_stdin" = "1" ]; then
+  exec docker exec -e PGPASSWORD=postgres -i "$c" psql -h localhost -U postgres "${args[@]}"
+else
+  exec docker exec -e PGPASSWORD=postgres "$c" psql -h localhost -U postgres "${args[@]}"
+fi
+EOF
+  cat > "$SHIM_BIN/pg_dump" <<'EOF'
+#!/usr/bin/env bash
+args=()
+for a in "$@"; do
+  if [[ "$a" =~ ^postgres:// ]]; then continue; fi   # source is always the src container here
+  args+=("$a")
+done
+exec docker exec -e PGPASSWORD=postgres -i rootnode-backup-test-src pg_dump -h localhost -U postgres "${args[@]}"
+EOF
+  chmod +x "$SHIM_BIN/psql" "$SHIM_BIN/pg_dump"
+  export PATH="$SHIM_BIN:$PATH"
+fi
+
+echo "==> Starting source Postgres (the one we'll back up)"
+docker run -d --name "$SRC_CONTAINER" -e POSTGRES_PASSWORD=postgres -p "$SRC_PORT:5432" postgres:16-alpine >/dev/null
+echo "==> Starting destination Postgres (the one we'll restore into)"
+docker run -d --name "$DST_CONTAINER" -e POSTGRES_PASSWORD=postgres -p "$DST_PORT:5432" postgres:16-alpine >/dev/null
+
+echo "==> Waiting for both to accept connections"
+for port in "$SRC_PORT" "$DST_PORT"; do
+  for i in $(seq 1 30); do
+    PGPASSWORD=postgres psql -h localhost -p "$port" -U postgres -c "SELECT 1" >/dev/null 2>&1 && break
+    sleep 1
+  done
+done
+
+SRC_DB_URL="postgres://postgres:postgres@localhost:$SRC_PORT/postgres"
+DST_DB_URL="postgres://postgres:postgres@localhost:$DST_PORT/postgres"
+
+echo "==> Applying the REAL Provisioner migrations first (Platform API's scenario_attempts has a real FK to Provisioner's environments table — documented cross-service ordering dependency, see docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md)"
+( cd platform/provisioner && [ -d node_modules ] || npm install --no-audit --no-fund >/dev/null
+  PLATFORM_DB_URL="$SRC_DB_URL" node -e "require('./src/db').runMigrations().then((m)=>console.log('Applied:', m.join(', ')))" )
+
+echo "==> Applying the REAL Platform API migrations against the source"
+( cd platform/api && [ -d node_modules ] || npm install --no-audit --no-fund >/dev/null
+  PLATFORM_DB_URL="$SRC_DB_URL" node -e "require('./src/db').runMigrations().then((m)=>console.log('Applied:', m.join(', ')))" )
+
+echo "==> Inserting sample data (a student, so the backup has something real to carry)"
+PGPASSWORD=postgres psql -h localhost -p "$SRC_PORT" -U postgres -c \
+  "INSERT INTO students (student_id, display_name) VALUES ('student-backup-test', 'Backup Test Student');"
+
+echo "==> Running backup-postgres.sh against the source"
+BACKUP_DIR="$TESTDIR/backups" PLATFORM_DB_URL="$SRC_DB_URL" ./infra/backup/backup-postgres.sh
+
+ENC_FILE=$(ls "$TESTDIR"/backups/*.sql.gz.gpg | head -1)
+[ -f "$ENC_FILE" ] || { echo "FAIL: no encrypted backup file produced"; exit 1; }
+echo "==> Backup produced: $ENC_FILE"
+
+echo "==> Restoring into the destination (non-interactive: piping the confirmation)"
+echo "RESTORE" | BACKUP_ENCRYPTION_PASSPHRASE="$BACKUP_ENCRYPTION_PASSPHRASE" ./infra/backup/restore-postgres.sh "$ENC_FILE" "$DST_DB_URL"
+
+echo "==> Verifying the restore"
+./infra/backup/verify-restore.sh "$DST_DB_URL"
+
+echo "==> Confirming the sample student actually round-tripped"
+RESTORED_NAME=$(PGPASSWORD=postgres psql -h localhost -p "$DST_PORT" -U postgres -tAc \
+  "SELECT display_name FROM students WHERE student_id = 'student-backup-test'")
+[ "$RESTORED_NAME" = "Backup Test Student" ] || { echo "FAIL: restored student data doesn't match"; exit 1; }
+
+echo
+echo "OK: full backup -> encrypt -> restore -> verify cycle succeeded against real local Postgres."
+echo "NOT TESTED HERE (needs real credentials/hardware): off-host copy to a real Storage Box/S3 target."

infra/backup/test-destructive-restore-drill.sh

@@ -0,0 +1,149 @@
+#!/usr/bin/env bash
+# Phase 4.1 §20 — the DESTRUCTIVE disaster-recovery drill, one level up from
+# test-backup-restore.sh's mechanism-only test: seeds realistic academy data through the REAL
+# Platform API (owner, admin instructor, cohort, 3 students, progress, attempts, an assessment
+# definition, achievements), backs it up, DESTROYS the disposable Postgres storage entirely
+# (docker rm — not just "restore into a second container," the actual source is gone),
+# recreates Postgres from nothing, restores, and boots the REAL Platform API again against the
+# restored database to prove the whole stack — not just the SQL — comes back.
+#
+# Never touches the developer's real persistent database — every Postgres here is a disposable
+# container on a throwaway port, created and destroyed entirely within this script.
+set -euo pipefail
+cd "$(dirname "$0")/../.."   # repo root
+
+CONTAINER=rootnode-drill-postgres
+PORT=55434
+API_PORT=4390
+TESTDIR="$(mktemp -d)"
+# curriculum.js defaults to platform/api/scenarios/{data,bundles} — only present inside the
+# Docker image build (COPY scenarios/ ./scenarios/). Running src/server.js directly from
+# source (as this drill does, to avoid needing a full container build) needs these pointed at
+# the repo's real scenario content instead.
+export SCENARIOS_DIR="$PWD/scenarios/data"
+export BUNDLES_DIR="$PWD/scenarios/bundles"
+export CURRICULUM_PATH_FILE="$PWD/scenarios/rootnode-path.json"
+export BACKUP_ENCRYPTION_PASSPHRASE="drill-passphrase-not-for-production"
+export PROVISIONER_SHARED_SECRET="drill-shared-secret-not-for-production"
+
+API_PID=""
+cleanup() {
+  [ -n "$API_PID" ] && kill "$API_PID" 2>/dev/null || true
+  docker rm -f "$CONTAINER" >/dev/null 2>&1 || true
+  rm -rf "$TESTDIR"
+}
+trap cleanup EXIT
+
+if ! command -v psql >/dev/null 2>&1 || ! command -v pg_dump >/dev/null 2>&1; then
+  echo "==> psql/pg_dump not found locally — reusing the same docker-exec shim as test-backup-restore.sh"
+  SHIM_BIN="$TESTDIR/bin"; mkdir -p "$SHIM_BIN"
+  # Same shim as test-backup-restore.sh (see that file's comment for the full "why") — the URL
+  # is DROPPED, not kept, since the host-side port it names doesn't exist inside the
+  # container's own network namespace; -tAc counts as a self-contained call (needs no stdin)
+  # the same way -c does, or the admin-verification queries below would hang waiting on stdin.
+  cat > "$SHIM_BIN/psql" <<'EOF'
+#!/usr/bin/env bash
+args=(); skip_next=0
+for a in "$@"; do
+  if [ "$skip_next" = "1" ]; then skip_next=0; continue; fi
+  if [ "$a" = "-p" ] || [ "$a" = "-h" ]; then skip_next=1; continue; fi
+  if [[ "$a" =~ ^postgres://[^:]+:[^@]+@[^:]+:([0-9]+)/ ]]; then continue; fi   # drop the URL — reconnect locally inside the container instead
+  args+=("$a")
+done
+container="rootnode-drill-postgres"
+needs_stdin=1
+for a in "${args[@]}"; do { [ "$a" = "-c" ] || [ "$a" = "-tAc" ]; } && needs_stdin=0; done
+if [ "$needs_stdin" = "1" ]; then
+  exec docker exec -e PGPASSWORD=postgres -i "$container" psql -h localhost -U postgres "${args[@]}"
+else
+  exec docker exec -e PGPASSWORD=postgres "$container" psql -h localhost -U postgres "${args[@]}"
+fi
+EOF
+  cat > "$SHIM_BIN/pg_dump" <<'EOF'
+#!/usr/bin/env bash
+args=()
+for a in "$@"; do
+  if [[ "$a" =~ ^postgres://[^:]+:[^@]+@[^:]+:([0-9]+)/(.*)$ ]]; then args+=("postgres://postgres:postgres@localhost:5432/${BASH_REMATCH[2]}"); continue; fi
+  args+=("$a")
+done
+exec docker exec -e PGPASSWORD=postgres rootnode-drill-postgres pg_dump "${args[@]}"
+EOF
+  chmod +x "$SHIM_BIN/psql" "$SHIM_BIN/pg_dump"
+  export PATH="$SHIM_BIN:$PATH"
+fi
+
+DB_URL="postgres://postgres:postgres@localhost:$PORT/postgres"
+
+echo "==> [1/9] Starting disposable Postgres"
+docker run -d --name "$CONTAINER" -e POSTGRES_PASSWORD=postgres -p "$PORT:5432" postgres:16-alpine >/dev/null
+for i in $(seq 1 30); do PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -c "SELECT 1" >/dev/null 2>&1 && break; sleep 1; done
+
+echo "==> [2/9] Applying real Provisioner + Platform API migrations"
+( cd platform/provisioner && [ -d node_modules ] || npm install --no-audit --no-fund >/dev/null
+  PLATFORM_DB_URL="$DB_URL" node -e "require('./src/db').runMigrations().then((m)=>console.log('  Provisioner:', m.join(', ')))" )
+( cd platform/api && [ -d node_modules ] || npm install --no-audit --no-fund >/dev/null
+  PLATFORM_DB_URL="$DB_URL" node -e "require('./src/db').runMigrations().then((m)=>console.log('  Platform API:', m.join(', ')))" )
+
+echo "==> [3/9] Bootstrapping the drill owner and starting a REAL Platform API"
+( cd platform/api && PLATFORM_DB_URL="$DB_URL" node bootstrap-owner.js [email protected] "Drill Owner" 'DrillOwnerPass!2026' )
+( cd platform/api && PLATFORM_DB_URL="$DB_URL" PROVISIONER_URL="http://localhost:1" PLATFORM_API_PORT="$API_PORT" node src/server.js > "$TESTDIR/api.log" 2>&1 & echo $! > "$TESTDIR/api.pid" )
+API_PID=$(cat "$TESTDIR/api.pid")
+for i in $(seq 1 30); do curl -sf "http://localhost:$API_PORT/healthz" >/dev/null 2>&1 && break; sleep 1; done
+curl -sf "http://localhost:$API_PORT/healthz" >/dev/null || { echo "FAIL: Platform API never came up"; cat "$TESTDIR/api.log"; exit 1; }
+echo "  Platform API up on :$API_PORT"
+
+echo "==> [4/9] Seeding realistic academy data through the real API (owner/admin/cohort/3 students/progress/attempts/assessment/achievements)"
+( cd infra/backup && PLATFORM_API_URL="http://localhost:$API_PORT" PLATFORM_DB_URL="$DB_URL" node seed-realistic-data.js )
+
+echo "==> [5/9] Backing up (real backup-postgres.sh)"
+BACKUP_DIR="$TESTDIR/backups" PLATFORM_DB_URL="$DB_URL" ./infra/backup/backup-postgres.sh
+ENC_FILE=$(ls "$TESTDIR"/backups/*.sql.gz.gpg | head -1)
+[ -f "$ENC_FILE" ] || { echo "FAIL: no encrypted backup produced"; exit 1; }
+
+echo "==> [6/9] DESTROYING the Postgres storage entirely (this is the destructive part — not restoring into a second container, the original is gone)"
+kill "$API_PID" 2>/dev/null || true; API_PID=""
+docker rm -f "$CONTAINER" >/dev/null
+echo "  source container + its storage are gone"
+
+echo "==> [7/9] Recreating Postgres from nothing"
+docker run -d --name "$CONTAINER" -e POSTGRES_PASSWORD=postgres -p "$PORT:5432" postgres:16-alpine >/dev/null
+for i in $(seq 1 30); do PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -c "SELECT 1" >/dev/null 2>&1 && break; sleep 1; done
+
+echo "==> [8/9] Restoring from the encrypted backup (schema + data both come from the dump — no separate migration step)"
+echo "RESTORE" | BACKUP_ENCRYPTION_PASSPHRASE="$BACKUP_ENCRYPTION_PASSPHRASE" ./infra/backup/restore-postgres.sh "$ENC_FILE" "$DB_URL"
+./infra/backup/verify-restore.sh "$DB_URL"
+
+echo "==> [9/9] Booting the REAL Platform API again against the RESTORED database"
+( cd platform/api && PLATFORM_DB_URL="$DB_URL" PROVISIONER_URL="http://localhost:1" PLATFORM_API_PORT="$API_PORT" node src/server.js > "$TESTDIR/api2.log" 2>&1 & echo $! > "$TESTDIR/api2.pid" )
+API_PID=$(cat "$TESTDIR/api2.pid")
+for i in $(seq 1 30); do curl -sf "http://localhost:$API_PORT/healthz" >/dev/null 2>&1 && break; sleep 1; done
+curl -sf "http://localhost:$API_PORT/healthz" >/dev/null || { echo "FAIL: Platform API did not come back up against the restored DB"; cat "$TESTDIR/api2.log"; exit 1; }
+echo "  Platform API is back up, against the RESTORED database — no fresh migration run needed"
+
+echo "==> Verifying accounts/RBAC/cohorts/students/progress/attempts/achievements survived"
+FAIL=0
+check() { local desc="$1" want="$2" got="$3"; if [ "$want" = "$got" ]; then echo "  ok — $desc"; else echo "  FAIL — $desc (want $want, got $got)"; FAIL=1; fi; }
+
+check "owner account (role) restored" "owner" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT role FROM instructors WHERE email='[email protected]'" | tr -d '[:space:]')"
+check "admin instructor restored" "admin" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT role FROM instructors WHERE email='[email protected]'" | tr -d '[:space:]')"
+check "cohort restored" "1" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM cohorts WHERE cohort_id='drill-cohort-01'" | tr -d '[:space:]')"
+check "3 students restored" "3" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM cohort_enrollments WHERE cohort_id='drill-cohort-01'" | tr -d '[:space:]')"
+check "3 completed attempts restored" "3" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM scenario_attempts WHERE result='passed' AND scenario_id='pod-basics-mcq'" | tr -d '[:space:]')"
+check "3 achievements restored" "3" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM achievements WHERE stage_id='foundation'" | tr -d '[:space:]')"
+check "assessment definition restored" "1" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM assessment_definitions WHERE assessment_definition_id='drill-exam-01'" | tr -d '[:space:]')"
+
+echo "==> Verifying RBAC still works against the restored DB (real login through the real API)"
+LOGIN_STATUS=$(curl -s -o /dev/null -w '%{http_code}' -X POST "http://localhost:$API_PORT/api/auth/instructor/login" \
+  -H 'Content-Type: application/json' -d '{"email":"[email protected]","password":"DrillAdminPass!2026"}')
+check "restored admin instructor can log in" "200" "$LOGIN_STATUS"
+
+if [ "$FAIL" = "1" ]; then
+  echo; echo "DRILL FAILED — see above."; exit 1
+fi
+
+echo
+echo "OK: full destructive restore drill passed — owner/admin/cohort/3 students/progress/attempts/"
+echo "assessment/achievements all survived a real Postgres storage loss + recreation + restore, and"
+echo "the real Platform API booted and authenticated against the restored database with no fresh"
+echo "migration run required."
+echo "NOT TESTED HERE (needs real credentials/hardware): the actual off-host Storage Box/S3 copy."

infra/backup/verify-restore.sh

@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+# Sanity-checks a restored database — confirms the expected Platform API schema landed and has
+# data, not just that psql exited 0 (a restore that silently imported zero rows would still
+# "succeed" by that measure). Used both by restore-postgres.sh's own suggestion and by
+# test-backup-restore.sh's automated local test.
+set -euo pipefail
+DB_URL="${1:?usage: verify-restore.sh <db-url>}"
+
+EXPECTED_TABLES=(students cohorts instructors sessions scenario_attempts assessment_definitions achievements audit_events)
+FAIL=0
+
+for t in "${EXPECTED_TABLES[@]}"; do
+  exists=$(psql "$DB_URL" -tAc "SELECT to_regclass('public.$t') IS NOT NULL")
+  if [ "$exists" != "t" ]; then
+    echo "MISSING TABLE: $t" >&2
+    FAIL=1
+  fi
+done
+
+if [ "$FAIL" = "1" ]; then
+  echo "FAIL: restored database is missing expected tables" >&2
+  exit 1
+fi
+
+MIGRATIONS_APPLIED=$(psql "$DB_URL" -tAc "SELECT count(*) FROM schema_migrations" 2>/dev/null || echo 0)
+echo "OK: all expected tables present. $MIGRATIONS_APPLIED migrations recorded in schema_migrations."

infra/ci/gitea-workflows/build-and-test.yml

@@ -0,0 +1,162 @@
+# Gitea Actions workflow — build/test on every push to main. Gitea Actions uses the same
+# workflow syntax as GitHub Actions (a compatible runner), so this is portable if the canonical
+# remote ever needs to move CI providers, without being GitHub Actions itself or requiring a
+# paid CI provider. Runs against the canonical Gitea repo only — see docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md
+# §18 for why this never touches the GitHub kubekosh fork.
+#
+# Deploy this file to .gitea/workflows/build-and-test.yml at the repo root to activate it (kept
+# under infra/ci/ in this PR so it's reviewed alongside the rest of the infra package first,
+# per the same "build now, activate deliberately" posture as everything else server-dependent
+# in this phase — enabling Gitea Actions on the instance is itself a one-time admin step that
+# may not be done yet).
+
+name: build-and-test
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  regression:
+    runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: postgres:16-alpine
+        env:
+          POSTGRES_PASSWORD: postgres
+        ports: ["5432:5432"]
+        options: >-
+          --health-cmd "pg_isready -U postgres"
+          --health-interval 10s --health-timeout 5s --health-retries 10
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Node
+        uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+
+      - name: Install dependencies (all first-party services)
+        run: |
+          for d in backend frontend platform/api platform/gateway platform/provisioner platform/lab-router; do
+            (cd "$d" && npm ci)
+          done
+
+      - name: Syntax-check every service
+        run: |
+          for d in backend frontend/src platform/api platform/gateway platform/provisioner platform/lab-router; do
+            find "$d" -name '*.js' -not -path '*/node_modules/*' -exec node --check {} \;
+          done
+
+      - name: Lab Host Router unit tests (Phase 4.1 — fast, no Docker daemon needed)
+        run: (cd platform/lab-router && npm test)
+
+      - name: Build frontend
+        run: (cd frontend && npm run build)
+
+      - name: Provisioner migrations against the service Postgres
+        run: |
+          cd platform/provisioner
+          PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5432/postgres \
+            node -e "require('./src/db').runMigrations().then(m=>console.log('Applied:',m.join(', ')))"
+
+      - name: Platform API migrations (depends on Provisioner's, same order as production)
+        run: |
+          cd platform/api
+          PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5432/postgres \
+            node -e "require('./src/db').runMigrations().then(m=>console.log('Applied:',m.join(', ')))"
+
+      - name: Validate infra configs (Phase 4 local tests — no hardware needed)
+        run: |
+          ./infra/scripts/validate-compose.sh
+          ./infra/firewall/validate-firewall.sh
+
+      # The real k3s-boot-dependent e2e suites (platform/test/e2e-phase*.js) need a full Docker
+      # Compose stack with --privileged Lab containers, which a shared CI runner typically
+      # can't grant safely — they stay a LOCAL/staging pre-promotion step
+      # (infra/scripts/staging-smoke-test.sh), not this job. See docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §18.
+
+  # Phase 4.1 §24 — free/open-source security scanning, no paid tooling, no signup required for
+  # any of these. Severity policy (docs/SECURITY_SCANNING.md): CRITICAL/HIGH findings fail the
+  # build; MEDIUM/LOW are reported but never block a merge — a build that's permanently red over
+  # an unrelated low-severity transitive dependency finding trains everyone to ignore red builds.
+  security-scan:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+
+      - name: npm audit (dependency vulnerabilities) — fails only on high/critical
+        run: |
+          FAIL=0
+          for d in backend frontend platform/api platform/gateway platform/provisioner platform/lab-router; do
+            echo "== $d =="
+            (cd "$d" && npm audit --audit-level=high) || FAIL=1
+          done
+          exit $FAIL
+
+      - name: Secret scanning (gitleaks — free, no account needed)
+        run: docker run --rm -v "$PWD:/repo" zricethezav/gitleaks:latest detect --source=/repo --no-git -v
+
+      - name: Dockerfile linting (hadolint — free)
+        run: |
+          FAIL=0
+          for f in Dockerfile platform/api/Dockerfile platform/gateway/Dockerfile platform/provisioner/Dockerfile platform/egress-proxy/Dockerfile platform/lab-router/Dockerfile; do
+            echo "== $f =="
+            docker run --rm -i hadolint/hadolint < "$f" || FAIL=1
+          done
+          exit $FAIL
+
+      # -S error, not warning — matches docs/SECURITY_SCANNING.md's severity policy. Real run
+      # against this tree at -S warning surfaces only style nits (unused loop vars, a missing
+      # `cd ... || exit` guard) on scripts that are otherwise correct; failing the build on
+      # those trains everyone to ignore the check.
+      - name: Shell linting (shellcheck — free, standard)
+        run: |
+          sudo apt-get update -qq && sudo apt-get install -y -qq shellcheck
+          find infra -name '*.sh' -exec shellcheck -S error {} \+
+
+      - name: Container image scanning (trivy — free, no account needed)
+        run: |
+          docker build -q -t rootnode-scan/platform-api -f platform/api/Dockerfile .
+          docker run --rm -v /var/run/docker.sock:/var/run/docker.sock aquasec/trivy:latest image \
+            --severity HIGH,CRITICAL --exit-code 1 rootnode-scan/platform-api
+
+  build-images:
+    needs: [regression, security-scan]
+    if: github.ref == 'refs/heads/main'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Build and tag images
+        run: ./infra/scripts/ci/tag-and-push.sh
+        env:
+          REGISTRY: ${{ vars.REGISTRY }}
+          REGISTRY_USER: ${{ secrets.REGISTRY_USER }}
+          REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
+
+  deploy-staging:
+    needs: build-images
+    if: github.ref == 'refs/heads/main'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Deploy to staging (automatic)
+        run: |
+          cd infra/ansible
+          ansible-playbook site.yml -i inventory/staging.ini \
+            -e "image_tag=staging-${GITHUB_SHA::12}" \
+            --vault-password-file <(echo "$VAULT_PASSWORD")
+        env:
+          VAULT_PASSWORD: ${{ secrets.ANSIBLE_VAULT_PASSWORD }}
+
+      - name: Staging smoke test
+        run: ./infra/scripts/staging-smoke-test.sh
+
+  # Production is NEVER automatic — a human runs infra/scripts/deploy-production.sh after
+  # reviewing the staging smoke test result, per the explicit "manual approval step" preference.
+  # No job here does that; see docs/RUNBOOK_NORMAL_DEPLOY.md for the actual command.

infra/docker/IMAGE_STRATEGY.md

@@ -0,0 +1,50 @@
+# Container image strategy
+
+## Images
+
+| Image | Built from | Registry path |
+|---|---|---|
+| `rootnode/platform-api` | `platform/api/Dockerfile` | `{{ registry }}/rootnode/platform-api` |
+| `rootnode/gateway` | `platform/gateway/Dockerfile` | `{{ registry }}/rootnode/gateway` |
+| `rootnode/provisioner` | `platform/provisioner/Dockerfile` | `{{ registry }}/rootnode/provisioner` |
+| `rootnode/egress-proxy` | `platform/egress-proxy/Dockerfile` | `{{ registry }}/rootnode/egress-proxy` |
+| `rootnode/lab-router` | `platform/lab-router/Dockerfile` (Phase 4.1) | `{{ registry }}/rootnode/lab-router` |
+| `rootnode/lab` | root `Dockerfile` (the student Lab image) | `{{ registry }}/rootnode/lab` |
+
+## Tagging
+
+**Never `:latest` in staging or production.** Every build is tagged with:
+
+- the full git SHA (`sha-<40 hex>`) — always, every build, the ground truth
+- `staging-<short-sha>` on every push to `main` that passes CI (moving pointer, staging only)
+- a semantic tag (`vX.Y.Z`) only on an explicit, human-triggered production release
+
+`infra/scripts/ci/tag-and-push.sh` implements this — see §18 in
+`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`.
+
+Deployment configs (`infra/environments/*/​*.env.template`, Ansible `group_vars`) reference the
+tag as a variable (`image_tag`), never a hardcoded value baked into a compose file — rolling
+back is changing one variable and re-running the `platform`/`lab-host` Ansible role, not editing
+a compose file by hand (see `docs/RUNBOOK_ROLLBACK.md`).
+
+## Registry
+
+No paid registry required. Two supported options, chosen at deploy time via `registry` in
+`infra/ansible/group_vars/production.yml`:
+
+1. **Gitea's own built-in container registry** (`git.synthossystems.com`) — zero new
+   infrastructure, already authenticated the same way the repo itself is. Preferred default.
+2. **A self-hosted registry** (e.g. a `registry:2` container on rn-platform-01 itself, or
+   elsewhere in the homelab) — a documented fallback if the Gitea registry isn't enabled on
+   this instance.
+
+Either way: private registry, credentials pulled the same way any other production secret is
+(`docs/PRODUCTION_DEPLOYMENT.md` §10), never a public Docker Hub push of anything beyond what's
+already public (this project doesn't publish anything to Docker Hub in Phase 4).
+
+## Rollback
+
+Every image is content-addressable by SHA tag, so "roll back" is always available as long as
+the previous tag hasn't been garbage-collected from the registry — see
+`docs/RUNBOOK_ROLLBACK.md` for the exact procedure (re-point `image_tag`, re-run the Ansible
+role, verify).

infra/docker/lab-host/docker-compose.production.yml

@@ -0,0 +1,91 @@
+# Production compose for rn-lab-host-01. Deployed by infra/ansible/roles/lab-host/ to
+# /opt/rootnode/lab-host/docker-compose.yml, run as a systemd unit
+# (rootnode-lab-host.service.j2). Runs the shared egress proxy every student Lab's outbound
+# traffic passes through (platform/egress-proxy/, unchanged since Phase 1) and, as of Phase
+# 4.1, the Lab Host Router (platform/lab-router/) that gives the Gateway on rn-platform-01 a
+# way to reach a running Lab without ever touching a Docker-internal address — see
+# docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3. Student Lab containers themselves are NOT
+# part of this compose file — they're created/destroyed individually by the Provisioner over
+# the remote Docker TLS endpoint (dockerd itself, not compose), same mechanism as dev, just
+# remote; the Router finds them by label, purely from local Docker state (see its own README).
+
+networks:
+  rn-egress:
+    driver: bridge
+
+services:
+  egress-proxy:
+    image: "{{ registry }}/rootnode/egress-proxy:{{ image_tag }}"
+    restart: unless-stopped
+    ports:
+      - "3128:3128"   # reachable from every per-Lab Docker network on this host — restricted
+                       # to those networks by the Provisioner's per-environment DOCKER-USER
+                       # chains, not by a published-port firewall rule (there is no sensible
+                       # single source IP to restrict this to; the egress proxy's OWN allowlist
+                       # is the actual control, see platform/egress-proxy/allowlist.txt)
+    networks: [rn-egress]
+
+  lab-router:
+    image: "{{ registry }}/rootnode/lab-router:{{ image_tag }}"
+    restart: unless-stopped
+    # Needs the host's own routing table to reach whichever per-Lab bridge network
+    # (rootnode-lab-net-<id>) the Provisioner created for a given container — those networks
+    # aren't known ahead of time and aren't attachable one-by-one, the same reason egress-proxy
+    # would need this too if it initiated connections outward instead of just accepting them.
+    # Restricted to platform_ip on port {{ lab_router_port | default('8443') }} at the host
+    # firewall (infra/firewall/rn-lab-host-01.rules) — this is the ONLY thing that makes host
+    # networking here safe, same defense-in-depth pattern as the Docker TLS port.
+    network_mode: host
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /etc/rootnode/lab-router-tls:/tls:ro
+    environment:
+      LAB_ROUTER_PORT: "{{ lab_router_port | default('8443') }}"
+      LAB_ROUTER_TLS_CA: /tls/ca.pem
+      LAB_ROUTER_TLS_CERT: /tls/router-server-cert.pem
+      LAB_ROUTER_TLS_KEY: /tls/router-server-key.pem
+
+  node-exporter:
+    image: prom/node-exporter:latest
+    restart: unless-stopped
+    network_mode: host
+    pid: host
+    volumes:
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/rootfs:ro
+    command:
+      - "--path.procfs=/host/proc"
+      - "--path.sysfs=/host/sys"
+      - "--path.rootfs=/rootfs"
+      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
+
+  cadvisor:
+    image: gcr.io/cadvisor/cadvisor:latest
+    restart: unless-stopped
+    privileged: true
+    devices:
+      - /dev/kmsg
+    volumes:
+      - /:/rootfs:ro
+      - /var/run:/var/run:ro
+      - /sys:/sys:ro
+      - /var/lib/docker/:/var/lib/docker:ro
+    ports:
+      - "8080:8080"   # scraped remotely by Prometheus on rn-platform-01 — restricted to that
+                       # one source IP by the host firewall (rn-lab-host-01.rules), not by
+                       # binding to localhost, since Prometheus isn't local on this host.
+
+  promtail:
+    image: grafana/promtail:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/lab-host/promtail-config.yml:/etc/promtail/config.yml:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /var/log/journal:/var/log/journal:ro
+      - /etc/machine-id:/etc/machine-id:ro
+    command: ["-config.file=/etc/promtail/config.yml"]
+    # host networking so promtail can reach Loki on rn-platform-01 over the network — this
+    # traffic is OUTBOUND from rn-lab-host-01 (log push, client-initiated), which the host
+    # firewall's default-ACCEPT OUTPUT chain already allows; nothing new needed there.
+    network_mode: host

infra/docker/platform/Caddyfile.j2

@@ -0,0 +1,38 @@
+# Caddy 2 — single stable RootNode Academy origin, automatic HTTPS (Let's Encrypt) unless
+# Cloudflare origin certs are used instead (see docs/PRODUCTION_NETWORK_SECURITY.md §8 for the
+# Cloudflare-vs-direct-LE decision). Rendered by the `platform` Ansible role from
+# infra/ansible/group_vars/production.yml's `academy_domain`.
+#
+# The student/instructor path proxies to the Gateway (platform/gateway) — the sole
+# student-facing entrypoint, unchanged architectural rule from Phase 2A onward. Nothing here
+# reaches the Provisioner, Postgres, or Docker TLS directly; those have no public route at all.
+# Grafana is reachable at /grafana/ but restricted to the management CIDR even though it has
+# its own login — internal observability tooling shouldn't be internet-facing at all, same
+# posture as everything else that isn't the student/instructor product surface.
+
+{{ academy_domain }} {
+	encode gzip zstd
+
+	handle_path /grafana/* {
+		@notmanagement not remote_ip {{ management_cidr }}
+		respond @notmanagement 403
+		reverse_proxy grafana:3000
+	}
+
+	# WebSocket terminal traffic (shell-ws) needs the connection upgraded, not buffered —
+	# Caddy's reverse_proxy handles this automatically for Upgrade: websocket requests, no
+	# special directive needed, but flush_interval -1 avoids buffering SSE/long-poll-shaped
+	# responses too. `gateway` is the Docker Compose service name (rn-platform-internal
+	# network), not localhost — Caddy no longer shares host networking with the Gateway in
+	# production, unlike the dev compose file.
+	reverse_proxy gateway:4200 {
+		flush_interval -1
+	}
+
+	log {
+		output file /var/log/caddy/access.log {
+			roll_size 100mb
+			roll_keep 7
+		}
+	}
+}

infra/docker/platform/docker-compose.production.yml

@@ -0,0 +1,221 @@
+# Production compose for rn-platform-01. Deployed by infra/ansible/roles/platform/ to
+# /opt/rootnode/platform/docker-compose.yml, run as a systemd-managed `docker compose` unit
+# (infra/ansible/roles/platform/templates/rootnode-platform.service.j2).
+#
+# Differs from platform/docker-compose.dev.yml in the ways that actually matter for
+# production, and nothing else:
+#   - images are pulled from a registry by DIGEST-PINNED TAG (never :latest, never built
+#     in-place) — see infra/docker/IMAGE_STRATEGY.md
+#   - Postgres data and backups live on a bind-mounted ZFS dataset, not an anonymous volume
+#   - a reverse proxy (Caddy) terminates TLS and is the only container with a port published to
+#     the internet; a few other services publish loopback-only or single-source-IP-restricted
+#     ports for specific operational needs (Postgres for the host's own backup cron, Loki for
+#     rn-lab-host-01's log shipper) — see each service's own comment
+#   - no `network_mode: host` on gateway/provisioner — production uses a named bridge network,
+#     since rn-platform-01 no longer needs to reach ephemeral container IPs directly (those live
+#     on rn-lab-host-01, reached over the network, not shared Docker networking)
+#   - the api/gateway/provisioner containers still don't reach the internet except through
+#     what's explicitly allowed by the host firewall (infra/firewall/), not Docker's own
+#     default bridge NAT behavior
+#
+# All image tags, ports, and paths below are Ansible Jinja2 variables — this file is a
+# template rendered by the `platform` role, not run directly. See
+# infra/ansible/group_vars/production.yml for the values.
+
+networks:
+  rn-platform-internal:
+    driver: bridge
+    internal: false   # postgres/api/gateway/provisioner need it false to reach rn-lab-host-01
+                       # over the network; the HOST firewall (not Docker) is what actually
+                       # restricts what that traffic can reach — see docs/PRODUCTION_NETWORK_SECURITY.md
+
+volumes:
+  postgres-data:
+    driver: local
+    driver_opts:
+      type: none
+      o: bind
+      device: "{{ rootnode_data_root }}/postgres"
+  caddy-data:
+  caddy-config:
+  prometheus-data:
+  alertmanager-data:
+  grafana-data:
+  loki-data:
+
+services:
+  postgres:
+    image: postgres:16-alpine
+    restart: unless-stopped
+    env_file: /etc/rootnode/postgres.env
+    volumes:
+      - postgres-data:/var/lib/postgresql/data
+      - /opt/rootnode/platform/postgresql.conf.d:/etc/postgresql/conf.d:ro
+    # include_dir appends our tuning files to the image's own auto-generated postgresql.conf —
+    # no need to replace or template the base config file at all.
+    command: ["postgres", "-c", "include_dir=/etc/postgresql/conf.d"]
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-rootnode_platform}"]
+      interval: 10s
+      timeout: 5s
+      retries: 10
+    networks: [rn-platform-internal]
+    ports:
+      - "127.0.0.1:5432:5432"   # loopback-only — reachable from other containers on
+                                 # rn-platform-internal AND from the host's own backup cron
+                                 # (infra/backup/), never from another host on the network, let
+                                 # alone the internet. This is NOT the same as the "Postgres
+                                 # never exposed on this VM's external interface" rule in
+                                 # rn-platform-01.rules — that rule (and the host firewall in
+                                 # general) still applies to this port; 127.0.0.1 binding is a
+                                 # second, independent reason nothing off-host can ever reach it.
+
+  provisioner:
+    image: "{{ registry }}/rootnode/provisioner:{{ image_tag }}"
+    restart: unless-stopped
+    env_file: /etc/rootnode/provisioner.env
+    volumes:
+      - /etc/rootnode/provisioner-config:/app/config:ro
+      - /etc/rootnode/docker-tls:/app/docker-tls:ro
+    depends_on:
+      postgres:
+        condition: service_healthy
+    networks: [rn-platform-internal]
+    cap_add: [NET_ADMIN, NET_RAW]   # only needed while a lab_host row is still a local Unix
+                                    # socket (dev); harmless no-op against a remote TLS endpoint
+
+  api:
+    image: "{{ registry }}/rootnode/platform-api:{{ image_tag }}"
+    restart: unless-stopped
+    env_file: /etc/rootnode/platform.env
+    environment:
+      # Phase 4.1 §25 — not a secret, just the deployed tag; set directly here rather than in
+      # the env template since it's the same value already used for the image tag above, not a
+      # separately-managed setting.
+      GIT_SHA: "{{ image_tag }}"
+    depends_on:
+      postgres:
+        condition: service_healthy
+      provisioner:
+        condition: service_started
+    networks: [rn-platform-internal]
+
+  gateway:
+    image: "{{ registry }}/rootnode/gateway:{{ image_tag }}"
+    restart: unless-stopped
+    env_file: /etc/rootnode/gateway.env
+    volumes:
+      # mTLS client identity for the Lab Host Router (data plane) — see gateway.env.j2's
+      # LAB_ROUTER_TLS_* vars. Read-only; the private key never leaves this one bind mount.
+      - /etc/rootnode/lab-router-tls:/tls:ro
+    depends_on: [provisioner, api]
+    networks: [rn-platform-internal]
+
+  reverse-proxy:
+    image: caddy:2-alpine
+    restart: unless-stopped
+    ports:
+      - "443:443"
+      - "80:80"   # ACME HTTP-01 challenge / plain-HTTP-to-HTTPS redirect only
+    volumes:
+      - /opt/rootnode/platform/Caddyfile:/etc/caddy/Caddyfile:ro
+      - caddy-data:/data
+      - caddy-config:/config
+    depends_on: [gateway]
+    networks: [rn-platform-internal]
+
+  # ── Observability agents (infra/monitoring/) ────────────────────────────────────────────
+  node-exporter:
+    image: prom/node-exporter:latest
+    restart: unless-stopped
+    network_mode: host   # needs host-level metrics; this is metrics-only, not app traffic
+    pid: host
+    volumes:
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/rootfs:ro
+      - /var/lib/node-exporter/textfile:/textfile:ro   # backup-postgres.sh writes its metric here
+    command:
+      - "--path.procfs=/host/proc"
+      - "--path.sysfs=/host/sys"
+      - "--path.rootfs=/rootfs"
+      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
+      - "--collector.textfile.directory=/textfile"
+
+  postgres-exporter:
+    image: prometheuscommunity/postgres-exporter:latest
+    restart: unless-stopped
+    environment:
+      DATA_SOURCE_NAME: "postgresql://{{ postgres_exporter_user }}:{{ vault_postgres_exporter_password }}@postgres:5432/rootnode_platform?sslmode=disable"
+    depends_on: [postgres]
+    networks: [rn-platform-internal]
+
+  blackbox-exporter:
+    image: prom/blackbox-exporter:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
+    networks: [rn-platform-internal]
+
+  prometheus:
+    image: prom/prometheus:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - /opt/rootnode/monitoring/rules:/etc/prometheus/rules:ro
+      - prometheus-data:/prometheus
+    command:
+      - "--config.file=/etc/prometheus/prometheus.yml"
+      - "--storage.tsdb.retention.time={{ prometheus_retention_days }}d"
+    networks: [rn-platform-internal]
+    ports:
+      - "127.0.0.1:9090:9090"   # bound to loopback only — access via SSH tunnel or Grafana, never public
+
+  alertmanager:
+    image: prom/alertmanager:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
+      - alertmanager-data:/alertmanager
+    networks: [rn-platform-internal]
+
+  grafana:
+    image: grafana/grafana-oss:latest
+    restart: unless-stopped
+    environment:
+      GF_SECURITY_ADMIN_PASSWORD: "{{ vault_grafana_admin_password }}"
+      GF_SERVER_ROOT_URL: "https://{{ academy_domain }}/grafana/"
+      GF_SERVER_SERVE_FROM_SUB_PATH: "true"
+      GF_USERS_ALLOW_SIGN_UP: "false"
+    volumes:
+      - grafana-data:/var/lib/grafana
+      - /opt/rootnode/monitoring/grafana/provisioning:/etc/grafana/provisioning:ro
+      - /opt/rootnode/monitoring/grafana/dashboards:/var/lib/grafana/dashboards:ro
+    networks: [rn-platform-internal]
+    # No published port — reached only via the reverse proxy's /grafana/ path (Caddyfile),
+    # never a direct public listener. See docs/PRODUCTION_NETWORK_SECURITY.md §8.
+
+  loki:
+    image: grafana/loki:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/loki-config.yml:/etc/loki/local-config.yaml:ro
+      - loki-data:/loki
+    command: ["-config.file=/etc/loki/local-config.yaml"]
+    networks: [rn-platform-internal]
+    ports:
+      - "3100:3100"   # reachable from rn-lab-host-01's promtail over the network — restricted
+                       # to that one source IP by the host firewall (rn-platform-01.rules), same
+                       # pattern as Docker TLS's 2376. Not published to the internet.
+
+  promtail:
+    image: grafana/promtail:latest
+    restart: unless-stopped
+    volumes:
+      - /opt/rootnode/monitoring/promtail-config.yml:/etc/promtail/config.yml:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /var/log/journal:/var/log/journal:ro
+      - /etc/machine-id:/etc/machine-id:ro
+    command: ["-config.file=/etc/promtail/config.yml"]
+    networks: [rn-platform-internal]
+    depends_on: [loki]

infra/docker/postgres/postgresql.conf.d/00-rootnode-baseline.conf

@@ -0,0 +1,30 @@
+# Conservative production tuning baseline — deliberately NOT hand-tuned to a specific RAM
+# allocation before rn-platform-01's real resources are known (Phase 4 instruction: do not
+# over-tune before actual hardware exists). Sized for a modest ~4GB Postgres allocation, safe
+# defaults everywhere else. Revisit once VM RAM is final — see docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §11.
+
+shared_buffers = 1GB
+effective_cache_size = 3GB
+maintenance_work_mem = 256MB
+work_mem = 16MB
+
+max_connections = 100
+
+# WAL — safe defaults for a single-instance setup with pg_dump/pg_basebackup-style backups
+# (infra/backup/). Not configured for streaming replication (out of scope this phase — see
+# docs/BACKUP_DR.md's DR model for why pg_dump + periodic pg_basebackup is the chosen approach,
+# not a hot standby).
+wal_level = replica
+max_wal_size = 2GB
+min_wal_size = 512MB
+checkpoint_completion_target = 0.9
+
+# Query logging — slow queries only, never full statement logging (would capture bind
+# parameters that can include session tokens' hashes / PII in edge cases). See docs/OBSERVABILITY.md.
+log_min_duration_statement = 500
+log_line_prefix = '%m [%p] %q%u@%d '
+log_checkpoints = on
+log_connections = on
+log_disconnections = on
+
+# Autovacuum stays on defaults — no reason to disable it for this workload size.

infra/docker/tls/README.md

@@ -0,0 +1,19 @@
+# Docker TLS material (generated, not committed)
+
+Generate with `infra/scripts/docker-tls/generate-docker-tls.sh`. Deploy with the `docker-tls`
+Ansible role. Verify with `infra/scripts/docker-tls/verify-docker-tls.sh`.
+
+**Nothing in this directory is ever committed** — see `infra/.gitignore`. This README (and only
+this README) is tracked, so the directory itself exists in the repo as documentation of what
+belongs here operationally.
+
+Files that will exist here on a real deployment host (never in git):
+
+```
+ca.pem                              # public — safe to distribute to both VMs
+ca-key.pem                          # PRIVATE — offline/secrets-vault only, never on either VM
+server-cert.pem / server-key.pem                    # rn-lab-host-01's dockerd
+provisioner-client-cert.pem / provisioner-client-key.pem   # rn-platform-01's Provisioner
+```
+
+Rotation procedure: `docs/RUNBOOK_DOCKER_TLS_ROTATION.md`.

infra/environments/README.md

@@ -0,0 +1,21 @@
+# Environment configuration
+
+Three environments, explicitly separated so production never silently inherits a dev default:
+
+| | dev | staging | production |
+|---|---|---|---|
+| Where it runs | any dev machine, `platform/docker-compose.dev.yml` | homelab or a resource-limited slice of the dedicated server once it exists | rn-platform-01 / rn-lab-host-01 on Proxmox |
+| `COOKIE_SECURE` | unset (plain HTTP) | `1` if served over HTTPS, else unset | `1` — required |
+| `TRUST_PROXY` | unset | `1` if fronted by a proxy | `1` — required, real reverse proxy in front |
+| `ADMIN`/owner bootstrap | `platform/api/bootstrap-owner.js` manually | same, against staging DB | same, against production DB, run once by a human, credential handled per `docs/PRODUCTION_DEPLOYMENT.md` §10 |
+| Secrets | `platform/.env` (gitignored, throwaway) | `infra/environments/staging/*.env` (gitignored, real but low-stakes) | `infra/environments/production/*.env` (gitignored, real, backed up per `docs/BACKUP_DR.md`) |
+| Lab image | built locally, `:test` tag | tagged build from CI, `:staging-<sha>` | tagged release, `:<sha>` or `:vX.Y.Z`, never `:latest` |
+
+Each environment directory holds `*.env.template` files (tracked) — copy to `*.env` (gitignored
+via `infra/.gitignore`) and fill in real values. **Never commit a filled `.env` file.** See
+`docs/PRODUCTION_DEPLOYMENT.md` §10 for how production secrets are actually generated, stored,
+and rotated (not just "don't commit them").
+
+`dev/` is intentionally thin — it just points back at the existing
+`platform/docker-compose.dev.yml` workflow (Phase 1–3B, unchanged); there was nothing new to
+build there for Phase 4.

infra/environments/dev/README.md

@@ -0,0 +1,7 @@
+# Dev environment
+
+Nothing new here for Phase 4 — dev still means `platform/docker-compose.dev.yml` +
+`platform/.env` (gitignored), exactly as built in Phases 1–3B. See `platform/README.md`.
+
+This directory exists only so `infra/environments/` visibly has all three tiers, not because
+dev needed new infrastructure code.

infra/environments/production/capacity.yml.example

@@ -0,0 +1,18 @@
+# Capacity planning inputs (Phase 4 §25). Copy to capacity.yml and fill in REAL measured values
+# once the actual Hetzner host and a real load test (infra/scripts/loadtest/) exist. The
+# numbers below are illustrative placeholders for a ~16c/32t/128GB host — do not treat them as
+# a real recommendation; run infra/scripts/capacity-model.js against real ones once measured.
+_placeholder: true
+
+host_cpu_cores: 32          # threads, if the host does hyperthreading — matches how Docker/cgroups count them
+host_mem_mb: 131072          # 128 GB
+
+platform_reserved_cpu_cores: 4    # rn-platform-01's VM allocation, reserved off the Lab Host's own pool
+platform_reserved_mem_mb: 16384
+
+proxmox_overhead_pct: 5      # hypervisor + host OS overhead — a conservative starting estimate
+
+measured_lab_cpu_cores: 2    # from infra/scripts/loadtest/sample-resources.sh peak CPU per Lab
+measured_lab_mem_mb: 2048    # from the same — peak memory per Lab
+
+safety_margin_pct: 20        # headroom kept unallocated even at the "recommended" threshold

infra/environments/production/gateway.env.template

@@ -0,0 +1,8 @@
+# Production environment template for rn-platform-01's `gateway` service.
+# `api`/`provisioner` are Docker Compose service names, not localhost — production runs these
+# on a named bridge network (rn-platform-internal), not network_mode: host like dev.
+
+PLATFORM_API_URL=http://api:4300
+PROVISIONER_URL=http://provisioner:4100
+PROVISIONER_SHARED_SECRET=__REPLACE_same_value_as_platform.env_and_provisioner.env__
+GATEWAY_PORT=4200

infra/environments/production/platform.env.template

@@ -0,0 +1,38 @@
+# Production environment template for rn-platform-01's `api` service.
+# Copy to platform.env OUTSIDE git (see infra/backup/README.md "secrets" and
+# docs/PRODUCTION_DEPLOYMENT.md §10), fill in real values, never commit the filled file.
+# infra/.gitignore already excludes *.env (only *.env.template is tracked).
+
+# ── Postgres ────────────────────────────────────────────────────────────────────────────
+# `postgres`/`provisioner` are Docker Compose service names, not localhost — production runs
+# these on a named bridge network (rn-platform-internal), not network_mode: host like dev.
+PLATFORM_DB_URL=postgres://rootnode_platform:__REPLACE_WITH_STRONG_PASSWORD__@postgres:5432/rootnode_platform
+
+# ── Provisioner ─────────────────────────────────────────────────────────────────────────
+PROVISIONER_URL=http://provisioner:4100
+PROVISIONER_SHARED_SECRET=__REPLACE_openssl_rand_hex_32__
+
+# ── Listen ──────────────────────────────────────────────────────────────────────────────
+PLATFORM_API_PORT=4300
+
+# ── Sessions (production values — do NOT inherit dev defaults) ────────────────────────────
+STUDENT_SESSION_TTL_HOURS=8
+STUDENT_SESSION_IDLE_HOURS=2
+INSTRUCTOR_SESSION_TTL_HOURS=12
+INSTRUCTOR_SESSION_IDLE_HOURS=2
+
+# ── Cookies / proxy — MUST be set for production, unlike dev ──────────────────────────────
+# Secure requires the origin to actually be HTTPS end-to-end (see docs/PRODUCTION_NETWORK_SECURITY.md §8).
+COOKIE_SECURE=1
+# Only set to 1 once a trusted reverse proxy (Cloudflare/nginx/Caddy, see infra/docker/reverse-proxy/)
+# is actually in front of the Gateway — otherwise req.ip becomes spoofable via X-Forwarded-For.
+TRUST_PROXY=1
+
+# ── Frontend (baked into the platform/api image at build time; listed here for reference) ─
+# FRONTEND_DIST_DIR=/frontend-dist
+
+# ── Future OIDC (Phase 3A adapter — not wired yet, see docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md §16) ─
+# OIDC_ISSUER_URL=
+# OIDC_CLIENT_ID=
+# OIDC_CLIENT_SECRET=__do_not_commit__
+# OIDC_REDIRECT_URL=https://academy.example.com/api/auth/oidc/callback

infra/environments/production/postgres.env.template

@@ -0,0 +1,14 @@
+# Production environment template for rn-platform-01's `postgres` service.
+# Postgres itself binds only to localhost/the platform-internal Docker network — never exposed
+# beyond rn-platform-01, see docs/PRODUCTION_NETWORK_SECURITY.md §6.
+
+POSTGRES_USER=rootnode_platform
+POSTGRES_PASSWORD=__REPLACE_WITH_STRONG_PASSWORD__
+POSTGRES_DB=rootnode_platform
+
+# Tuning baseline — conservative, meant to be revisited once rn-platform-01's real allocated
+# RAM is known (see docs/production-postgres section in docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md
+# §11). Applied via infra/docker/postgres/postgresql.conf.d/ — do not hand-edit the container.
+# POSTGRES_SHARED_BUFFERS=1GB
+# POSTGRES_EFFECTIVE_CACHE_SIZE=3GB
+# POSTGRES_MAX_CONNECTIONS=100

infra/environments/production/provisioner-config/README.md

@@ -0,0 +1,15 @@
+# Production Provisioner config overlay
+
+Deployed by the `platform` Ansible role to `/etc/rootnode/provisioner-config/` on
+rn-platform-01 (`PROVISIONER_CONFIG_DIR`), replacing the dev-only files shipped in
+`platform/provisioner/config/`.
+
+- `lab-hosts.yaml` — points at the real rn-lab-host-01 VM over TLS instead of the dev Unix
+  socket. TLS paths reference material generated by `infra/scripts/docker-tls/` and deployed by
+  the `docker-tls` Ansible role — **never checked into this repo** (see
+  `infra/docker/tls/.gitignore`).
+- `resource-policy.yaml` — deliberately still the Phase 1 dev defaults, unchanged, with a
+  loud comment explaining why. Per the explicit Phase 4 instruction: do not hard-code final
+  per-Lab CPU/RAM numbers before the real host is benchmarked (`docs/CAPACITY_PLANNING.md`).
+  Whoever runs the first production load test (`infra/scripts/loadtest/`) updates this file
+  with real measured numbers, not this deployment.