diff --git a/README.md b/README.md index 616d166..739a22d 100644 --- a/README.md +++ b/README.md @@ -196,8 +196,11 @@ Production topology and what's built vs. not yet: `docs/LAB_ARCHITECTURE.md`, The production infrastructure package (Proxmox/Ansible/Terraform/firewall/monitoring/backup/CI) that runs all of this on real hardware lives in `infra/` — see -`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`. Built and locally validated now; deployment is -pending purchase of the target Hetzner dedicated server. +`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` and `docs/PHASE4_1_PREPRODUCTION_HARDENING.md` (the +Gateway-to-Lab cross-VM routing fix, plus application hardening — real pagination, abandonment +handling, a Postgres-backed rate limiter, a structured assessment editor, admission control). +Built and locally validated now; deployment is pending purchase of the target Hetzner dedicated +server. --- diff --git a/docs/BACKUP_DR.md b/docs/BACKUP_DR.md index 192f88e..d5e8906 100644 --- a/docs/BACKUP_DR.md +++ b/docs/BACKUP_DR.md @@ -37,10 +37,19 @@ requires typing `RESTORE` to confirm, then decrypts and pipes into `psql`. **Genuinely tested**, not just written: `infra/backup/test-backup-restore.sh` runs the full cycle against two disposable local Postgres containers, with the actual production migrations -applied (all 17 — 3 Provisioner + 14 Platform API, in the correct cross-service order), a +applied (3 Provisioner + Platform API, in the correct cross-service order), a sample row inserted, backed up, restored into the second container, and verified byte-for-byte. Confirmed passing, exit 0. +**Phase 4.1's destructive drill goes one step further** +(`infra/backup/test-destructive-restore-drill.sh`): realistic data (owner, admin instructor, +cohort, 3 students, progress, attempts, an assessment definition, achievements) seeded through +the REAL Platform API, backed up, the source Postgres **container genuinely destroyed** +(`docker rm`, not just left running alongside a second one), a fresh Postgres created from +nothing, restored, and the real Platform API booted again against the restored database and +used to log in as the restored admin instructor over HTTP — proving the whole stack recovers, +not just the SQL. Never touches the developer's real persistent database. Confirmed passing. + ## Disaster recovery scenarios | Scenario | RPO | RTO (estimate) | What happens | diff --git a/docs/ENGINEERING_SOP.md b/docs/ENGINEERING_SOP.md index 0554392..56241c0 100644 --- a/docs/ENGINEERING_SOP.md +++ b/docs/ENGINEERING_SOP.md @@ -296,12 +296,40 @@ Compose, firewall, monitoring, backup, CI, load-test tooling). - **Where the boundary held:** the Platform VM / Lab Host VM security split from `docs/LAB_ARCHITECTURE.md` is codified in firewall rulesets and mutual-TLS Docker config, not weakened anywhere. -- **Known gap surfaced, not silently fixed:** the Gateway's Lab-reachability mechanism doesn't - survive the move to two real separate VMs — see `docs/PRODUCTION_NETWORK_SECURITY.md`'s - "Known gap" section for what's built (firewall policy) vs. what's an explicit, reviewed, - not-yet-built application follow-up (Provisioner/Gateway address handling). +- **Known gap surfaced, then resolved in Phase 4.1:** the Gateway's Lab-reachability mechanism + didn't survive the move to two real separate VMs — flagged explicitly rather than silently + fixed, then actually built and simulation-tested (a Lab Host Router, + `platform/lab-router/`) in Phase 4.1. See §13 below. - **Validated without the hardware:** real `tofu validate`, `ansible-playbook --syntax-check`, `docker compose config`, `caddy validate`, `promtool`/`amtool`, and a genuine end-to-end backup→encrypt→restore→verify cycle against disposable local Postgres with the real production migrations applied. None of this proves the real Hetzner network boundary — that needs `docs/PRODUCTION_VALIDATION_CHECKLIST.md` walked against the real server. + +## 13. Pre-Production Hardening (Phase 4.1) + +Full design: `docs/PHASE4_1_PREPRODUCTION_HARDENING.md`, plus +`docs/PRODUCTION_NETWORK_SECURITY.md` (routing resolution), `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`, +`docs/SECURITY_SCANNING.md`. + +- **What it is:** closes the Gateway-to-Lab cross-VM gap Phase 4 flagged (a Lab Host Router, + `platform/lab-router/`, plus a routing abstraction in the Gateway, + `platform/gateway/src/lab-endpoint.js`), then hardens the application layer found along the + way: abandoned-attempt handling (`scenario_attempts.status = 'abandoned'`, never a punitive + failure), a Postgres-backed login rate limiter (survives a restart, ready to be + shared/multi-instance), real pagination on audit/attempt/assessment-session history, a + structured assessment authoring UI (owner-only raw JSON remains as a fallback), a minimal + `role_in_cohort` enforcement (`teacher`/`viewer`), a `/api/version` endpoint, and admission + control (a clean 503 "capacity full" instead of a 500 once a configured cap is reached). +- **Tested, not just designed:** a real two-Docker-network routing simulation + (`infra/scripts/simulate-two-host/`, 13/13), a destructive Postgres restore drill + (`infra/backup/test-destructive-restore-drill.sh`), a Lab Host loss/rebuild drill + (`infra/scripts/drills/test-lab-host-loss.js`), a deployment rollback drill, a CI-fails-closed + drill (which found and fixed a real pre-existing `set -e` bug in both smoke-test scripts — + see that drill's own output), and a full fake-cohort dress rehearsal + (`infra/scripts/drills/test-fake-cohort-rehearsal.js`) — all passing, all in + `infra/scripts/drills/` or `infra/backup/`. +- **Known limitation carried forward:** real remote/self-paced delivery needs actual student + authentication (today's `student_id`-only login relies on instructors controlling who gets a + valid id) — see `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md` for the full accounting of what + already works vs. what's genuinely Phase 5 scope. diff --git a/docs/LAB_ARCHITECTURE.md b/docs/LAB_ARCHITECTURE.md index db65aa3..6d8bc08 100644 --- a/docs/LAB_ARCHITECTURE.md +++ b/docs/LAB_ARCHITECTURE.md @@ -491,8 +491,10 @@ networking, egress, resource policy, reaper) against these decisions. Phase 4 (`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`) codifies the Platform/Lab-Host split this document designed as two real, separately-firewalled Proxmox VMs (`rn-platform-01`, -`rn-lab-host-01`) connected only by mutual-TLS Docker and a tightly scoped firewall allowance — -see `docs/PRODUCTION_NETWORK_SECURITY.md`. That document also records a gap Phase 4 surfaced but -did not resolve: the Gateway's `env.internal_ip` mechanism for reaching a Lab container assumes -Gateway and Lab share a Docker daemon, which breaks once they're genuinely on separate VMs. Read -that document's "Known gap" section before treating two-VM production as fully wired. +`rn-lab-host-01`) connected by two separate mTLS channels (Provisioner↔dockerd CONTROL plane, +Gateway↔Lab Host Router DATA plane) and a tightly scoped firewall allowance — see +`docs/PRODUCTION_NETWORK_SECURITY.md`. Phase 4 surfaced a gap in the Gateway's `env.internal_ip` +mechanism for reaching a Lab container (it assumes Gateway and Lab share a Docker daemon, which +breaks once they're genuinely on separate VMs); Phase 4.1 +(`docs/PHASE4_1_PREPRODUCTION_HARDENING.md`) built and simulation-tested the fix, a Lab Host +Router (`platform/lab-router/`). diff --git a/docs/PHASE4_1_PREPRODUCTION_HARDENING.md b/docs/PHASE4_1_PREPRODUCTION_HARDENING.md new file mode 100644 index 0000000..cf58a8c --- /dev/null +++ b/docs/PHASE4_1_PREPRODUCTION_HARDENING.md @@ -0,0 +1,254 @@ +# Phase 4.1 — Pre-Production Hardening + +**Status: PHASE 4.1 COMPLETE. PRE-PRODUCTION HARDENING COMPLETE. READY FOR HARDWARE DEPLOYMENT.** + +Phase 4 built the production infrastructure package with the real Hetzner/Proxmox hardware not +yet purchased, and flagged one genuine architectural gap: the Gateway's mechanism for reaching +a student Lab assumed Gateway and Lab shared a Docker daemon, which is false the moment Labs run +on a genuinely separate `rn-lab-host-01` VM. Phase 4.1's job was to close that gap and harden +everything else discovered along the way, still without the real hardware. + +## 1. The core fix: a Lab Host Router + +**Before:** Gateway → `http://${env.internal_ip}:${env.internal_port}` directly — a Docker +bridge-local address, unreachable across a real VM boundary. + +**After:** Gateway → (mode-aware) → either `direct` (unchanged, dev/single-host) or `router` +(production): a small new service, `platform/lab-router/`, runs on the Lab Host itself, +authenticates the Gateway via mTLS (the same private CA the Docker TLS control plane already +used, extended with two new leaf certs), resolves an opaque routing token to a local container +purely from `docker ps` label state (no Postgres access, no call back to the Provisioner), and +reverse-proxies HTTP and WebSocket traffic to it. + +The decision of which mode a given `lab_host` uses is data-driven +(`lab_hosts.routing_mode`/`router_endpoint`, `platform/provisioner/migrations/004_lab_hosts_routing.sql`), +resolved once per request in `platform/gateway/src/lab-endpoint.js`'s `labTarget()` — the only +place in the Gateway that knows this distinction exists. Every proxy/fetch call site in +`platform/gateway/src/server.js` goes through it. + +**Exact `env.internal_ip` replacement:** `labTarget(env, token)` returns `{ url, headers, agent, +mode }`. In `direct` mode, `url` is exactly the old `http://internal_ip:internal_port`. In +`router` mode, `url` is the lab_host's `router_endpoint`, `headers` carries +`X-RootNode-Route-Token: ` (computed server-side from the Provisioner-verified +environment row — never trusted from the browser), and `agent` is an `https.Agent` configured +with the Gateway's mTLS client identity. + +## 2. No per-student host port + +Deliberately not solved by publishing a host port per Lab (the originally-flagged, rejected +approach). The Router resolves entirely from Docker's own label state on its own host — no +port-per-student anything, no dynamic firewall rule per Lab. + +## 3. Lab Host Router — design + +See `platform/lab-router/src/server.js`'s header comment for the full "narrow by design" list: +one route (proxy-by-token), requires mTLS (no bearer-token fallback), never exposes Docker API/ +container IDs/network info to its caller, never provides student-to-student routing (each +request resolves to exactly the one container matching its own token). + +## 4. Multi-Lab-Host ready + +Routing is resolved per `lab_host_id` — adding `rn-lab-host-02` later is one new `lab_hosts` row +(its own `routing_mode`/`router_endpoint`/TLS material), no Gateway code change. No scheduler +was built (explicitly out of scope) — the Provisioner still picks "the one enabled lab_host" +(`selectLabHost`), unchanged from Phase 1. + +## 5. Local development compatibility + +`dev-local` (the Unix-socket lab_host used by `docker-compose.dev.yml`) stays `routing_mode: +direct` — zero behavior change for the T480/any local dev setup, confirmed by the full existing +regression suite passing unmodified (§29 below) and by every new drill in this phase running +against the same dev stack. The standalone single-container Lab image build/boot is unaffected +(it never touches the Gateway or Provisioner at all). + +## 6-9. Simulation, HTTP/WS validation, security tests, routing token security + +`infra/scripts/simulate-two-host/` — a real two-Docker-network simulation (not mocks): Gateway +attached ONLY to an isolated bridge with no route to any Lab's bridge network, Lab Host Router +host-networked (mirrors `egress-proxy`'s existing pattern) so it can reach whatever per-Lab +bridge the Provisioner creates. 13/13 real assertions pass, including: + +- Gateway genuinely cannot open a TCP connection to a Lab's Docker-internal IP (proven from + inside the Gateway container's own network namespace via a raw connect attempt). +- Gateway reaches the Lab correctly through the Router — HTTP AND a real WebSocket terminal + session running a genuine `kubectl get nodes` inside the Lab. +- Alice/Bob isolation (own lab ALLOW, other's DENY — HTTP and WebSocket both), garbage tokens, + anonymous requests, and destroyed-environment tokens are all rejected through the real code + path. +- The Router refuses any connection with no client certificate — mTLS is the real + authentication mechanism. + +Routing token security reviewed: already CSPRNG since Phase 1 (`generateRoutingToken`), never +logged in plaintext (the Router's `tokenFingerprint` truncates), redacted in every API response +except the two narrow internal reads that legitimately need it, and — critically — the header +the Router trusts is computed server-side from the Provisioner-verified environment row, never +read from the browser's own request, so a browser cannot forge routing by supplying its own +`X-RootNode-Route-Token`. + +## 10. Control plane vs data plane + +Kept strictly separate — see `docs/PRODUCTION_NETWORK_SECURITY.md`'s dedicated section. Student +browser traffic is never proxied through the Docker API at any point. + +## 11. Firewall update + +`infra/firewall/rn-lab-host-01.rules`'s old speculative wide port-range allowance replaced with +one narrow rule: `platform_ip` → the Router's port only (`8443` default, configurable). + +## 12. Self-paced / remote delivery architecture check + +`docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`. Short version: student-driven provisioning, +cohort-less students, and "idle reaper destroys → progress survives → student returns → fresh +Lab → curriculum continues" all already worked before this phase and are now proven end-to-end +by two new drills. The one real gap for actual remote/self-paced delivery is student +authentication (today's bare `student_id` login assumes an instructor-controlled roster) — not +built here, explicitly Phase 5 scope. + +## 13. Abandoned attempt cleanup + +`scenario_attempts.status = 'abandoned'` (already a valid value since Phase 3B, never +previously set) now gets set on: environment destroy/reset (student- or instructor-triggered), +logout, session expiry (piggybacked on the existing hourly session sweeper — no new background +job), and switching to a different scenario while one is still open. Never a punitive marker — +`result` stays NULL, history is preserved, instructors see an honest "abandoned (reason)" +instead of a phantom "still in progress." + +## 14. Gateway session cache + +Reviewed — already a tight 2-second TTL since Phase 3A, specifically to bound the +revocation-propagation window. Decision: **keep it, don't add push-based invalidation** — the +added coupling (Platform API would need to know how to reach the Gateway) isn't justified by +shrinking an already-2-second window further. Documented reasoning, not a silently-accepted gap. + +## 15. Rate limiting + +Login rate limiting moved from an in-memory `Map` (reset on every restart, not shareable across +instances) to Postgres (`login_attempts` table, `migrations/016_login_attempts.sql`) — +`platform/api/src/security.js`'s `tooManyAttempts()`. No Redis added. Restart behavior +confirmed for real: `platform/test/e2e-phase3a.js`'s rate-limiting test passes after a genuine +Platform API restart in the same run. + +## 16. Pagination + +Real server-side pagination (`platform/api/src/pagination.js`) on: the audit log (was a flat +`LIMIT 200`), student attempt history (was `LIMIT 100`), and assessment session history (was +**fully unbounded** — the one genuine growth risk found). Scenario analytics was deliberately +left as-is — grouped by scenario id, naturally bounded around ~100 rows, no real growth risk to +paginate against. Frontend (`AuditView.jsx`, `AssessmentsView.jsx`, `StudentsView.jsx`) updated +to match; `frontend` build confirmed clean. + +## 17. Needs-attention configuration + +`NEEDS_ATTENTION_FAIL_THRESHOLD`/`NEEDS_ATTENTION_STALE_DAYS` moved from hardcoded constants to +env-var-configurable (matching the existing session-TTL pattern), sensible defaults unchanged. + +## 18. Assessment authoring UX + +Raw-JSON `pool_spec` textarea replaced with a structured editor +(`frontend/src/admin/AssessmentsView.jsx`'s `PoolSpecEditor`): required scenarios and random +pools picked from a real scenario catalog (`GET /api/admin/scenario-catalog`, new), grouped by +curriculum stage, via native multi-select — no new UI framework. An owner-only "advanced" raw +JSON view remains for anything the structured editor doesn't cover. + +## 19. Cohort role-in-cohort + +`cohort_instructors.role_in_cohort` existed since Phase 2B but was never read. Implemented +minimally: `teacher` (unchanged full capability) vs `viewer` (read-only — dashboards/analytics +still visible, but Prepare/Destroy Labs and provisioning a student's environment are now 403'd +server-side, `rbac.js`'s `requireCohortWriteAccess`/`requireStudentCohortWriteAccess`). + +## 20. Backup / restore destructive drill + +`infra/backup/test-destructive-restore-drill.sh` — real owner/admin/cohort/3-students/progress/ +attempts/assessment/achievements seeded through the real Platform API, encrypted-backed-up, the +disposable Postgres container **genuinely destroyed** (not just restored into a second +container), a fresh Postgres created from nothing, restored, and the real Platform API booted +again against the restored database and used to log in as the restored admin over HTTP. All +data verified present; never touches the developer's real database. + +## 21. Lab Host rebuild drill + +`infra/scripts/drills/test-lab-host-loss.js` — a student's Lab container is destroyed out from +under the Provisioner (no clean destroy call, simulating total Lab Host loss). Proves: Platform +stays up throughout, the stale environment is never presented as healthy once the Provisioner's +real `reconcileOnStartup` reconciliation runs, academic progress is untouched, and the student +can launch a genuinely new Lab and continue. + +## 22. Deployment rollback test + +`infra/scripts/drills/test-deployment-rollback.sh` — builds a known-good and a deliberately +broken gateway image, "deploys" each via the real deterministic-tag mechanism, proves the smoke +test catches the regression, computes the rollback target with the exact same +`deploy-history.log` logic `rollback-production.sh` uses, and proves the rollback recovers. +Documents explicitly that image rollback never implies a DB migration rollback. + +## 23. CI/CD failure tests + +`infra/scripts/drills/test-ci-fails-closed.sh` — proves all six required failure modes (test/ +syntax failure, frontend build failure, image build failure, config validation failure, +migration preflight failure, staging smoke test failure) genuinely exit nonzero. **Found and +fixed two real pre-existing bugs**: `staging-smoke-test.sh` and `production-smoke-test.sh` both +had a classic `set -e` gotcha (`curl ... && echo "OK"` as one statement — a failing command +that isn't the last in an `&&` chain doesn't trigger `-e`) that let an unreachable/broken +endpoint fall through silently instead of failing the smoke test. Fixed in both files. + +## 24. Security scanning + +`infra/ci/gitea-workflows/build-and-test.yml`'s new `security-scan` job — npm audit, gitleaks +(secrets), hadolint (Dockerfiles), shellcheck (shell scripts), trivy (container image CVEs). +All free/open-source. Severity policy documented in `docs/SECURITY_SCANNING.md`: CRITICAL/HIGH +blocks the build, MEDIUM/LOW is reported only. `security-scan` gates `build-images` the same +way `regression` does. + +## 25. Version / release visibility + +`GET /api/version` (Platform API) — app version (`package.json`), deployed Git SHA +(`GIT_SHA` env var, set to the same value as the image tag in the production compose file), and +the most recently applied migration as a schema-version stand-in. Unauthenticated, minimal, no +infrastructure detail. + +## 26. Admission control failure mode + +`resource-policy.yaml`'s new (disabled-by-default) `admission_control.max_concurrent_labs` — +when set, the Provisioner checks it BEFORE touching Docker at all, so a full host fails cleanly +(503, `CAPACITY_EXCEEDED`) instead of failing mid-provision. Relayed by the Platform API as a +clean "Lab capacity is temporarily full. Try again shortly." — never a raw 500. A new +`GET /capacity` Provisioner endpoint feeds the instructor dashboard's capacity display. + +## 27. Fake cohort dress rehearsal + +`infra/scripts/drills/test-fake-cohort-rehearsal.js` — one continuous narrative: instructor +login, cohort + 4 students, curriculum assignment, Prepare Labs, students working different +scenarios, hints, a deliberate failed validation, instructor analytics, a Lab reset, a Lab +reaped (same consequence as the real idle reaper — see the drill's own comment on why the real +multi-minute timer isn't waited out), the student returning and continuing, an exam +assigned/taken, and end-of-class Lab destruction — progress verified intact after every +destructive step. 14/14 passing. + +## 28. Documentation + +This file, `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`, `docs/SECURITY_SCANNING.md` (new). +Updated: `docs/LAB_ARCHITECTURE.md`, `docs/PRODUCTION_NETWORK_SECURITY.md`, +`docs/PRODUCTION_DEPLOYMENT.md`, `docs/PRODUCTION_VALIDATION_CHECKLIST.md`, `docs/BACKUP_DR.md`, +`docs/ENGINEERING_SOP.md`, `README.md`, `platform/README.md`. + +## 29. Regression + +Re-ran the full existing suite after every application-code change in this phase: +`e2e-phase2a.js` (10/10), `e2e-phase2b.js` (12/12), `e2e-phase3a.js` (21/21 — including a real +audit-log pagination-shape fix and confirming rate-limit persistence across a genuine restart), +`e2e-phase3b.js` (24/24). Frontend production build clean. Real Lab image builds/boots (both +dev-compose and standalone single-container). `tofu validate`, `ansible-playbook +--syntax-check`, production Compose config (both platform and lab-host), `caddy validate`, +`promtool`/`amtool`, and the backup/restore mechanism all re-validated clean after the +infrastructure changes in this phase (new `lab-router` service, new firewall rule, new Ansible +tasks). No knowingly-unexplained regression was left in this codebase at the end of this phase. + +## What Phase 4.1 deliberately did NOT build + +Production deployment itself, Hetzner/Proxmox installation, public course signup, payments, +subscriptions, parent portal, marketing site, full LMS, video hosting, email automation, +certificates, course marketplace, mobile app, a multi-host scheduler, an AI tutor, invasive +proctoring, held scenario content, student email/password authentication (documented as the one +real gap for self-paced delivery, not built). All explicit non-goals per this phase's +authorization. diff --git a/docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md b/docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md index a265b72..1695b72 100644 --- a/docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md +++ b/docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md @@ -170,15 +170,18 @@ unchanged. No parent portal, billing, held scenarios, or production OIDC provide RootNode production infrastructure is dedicated to RootNode Academy — never folded into existing Synthos production servers, and canonical source stays Gitea, never GitHub `kubekosh`. -## Known gap surfaced during this phase (not silently papered over) +## Known gap surfaced during this phase (not silently papered over) — RESOLVED in Phase 4.1 Building the real production Compose/network config surfaced that the Gateway's existing mechanism for reaching a Lab container (`env.internal_ip`, a Docker-bridge-local address) only works when Gateway and Lab share a Docker daemon — true in every dev/single-host setup so far, -**not** true once Labs run on a genuinely separate `rn-lab-host-01`. The network-policy half is -built (a firewall allowance for Gateway→Lab-Host on a published port range, -`infra/firewall/rn-lab-host-01.rules`); the application half (the Provisioner recording a -reachable external address for each Lab, and the Gateway using it) is an explicit, reviewed, -NOT-YET-BUILT follow-up — see `docs/PRODUCTION_NETWORK_SECURITY.md` for the two candidate -designs and why neither was silently chosen here. This must land before real two-VM go-live; -flagging it clearly was judged better than guessing at an architecture change mid-infra-phase. +**not** true once Labs run on a genuinely separate `rn-lab-host-01`. At the time this phase +closed, the network-policy half was built but the application half was flagged as an explicit, +not-yet-built follow-up rather than guessed at mid-infra-phase. + +**Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1) built and proved the application +half** — a Lab Host Router (`platform/lab-router/`) plus a `labTarget()` routing abstraction in +the Gateway, real-tested against a genuine two-network Docker simulation +(`infra/scripts/simulate-two-host/`), not just designed. See +`docs/PRODUCTION_NETWORK_SECURITY.md`'s "Gateway-to-Lab reachability" section for the full +resolution. This item no longer blocks real two-VM go-live. diff --git a/docs/PRODUCTION_DEPLOYMENT.md b/docs/PRODUCTION_DEPLOYMENT.md index 9815f1e..c1a618b 100644 --- a/docs/PRODUCTION_DEPLOYMENT.md +++ b/docs/PRODUCTION_DEPLOYMENT.md @@ -31,7 +31,7 @@ tofu apply Creates `rn-platform-01` and `rn-lab-host-01` from the cloud-init template. Confirm both boot and are SSH-reachable from the management network before continuing. -## 2. Generate Docker TLS certificates (offline, once) +## 2. Generate TLS certificates (offline, once) ```bash export ROOTNODE_TLS_BUNDLE_DIR=/secure/offline/path # never inside the repo @@ -39,8 +39,14 @@ export ROOTNODE_TLS_BUNDLE_DIR=/secure/offline/path # never inside the repo ./infra/scripts/docker-tls/verify-docker-tls.sh ``` -The CA private key stays in `ROOTNODE_TLS_BUNDLE_DIR` and never touches either VM — only the CA -cert (public) and the relevant leaf cert+key go to each host. +Generates FOUR leaf certs under one CA (Phase 4.1 extended this from two): the CONTROL-plane +pair (`server-*`/`provisioner-client-*`, for Provisioner↔dockerd) and the DATA-plane pair +(`router-server-*`/`gateway-client-*`, for Gateway↔Lab Host Router — see +`docs/PRODUCTION_NETWORK_SECURITY.md`'s "control plane vs data plane" section). The CA private +key stays in `ROOTNODE_TLS_BUNDLE_DIR` and never touches either VM — only the CA cert (public) +and each host's own leaf cert+key go to each host, both pairs deployed by the same `docker-tls` +Ansible role into two separate directories (`/etc/rootnode/docker-tls/` and +`/etc/rootnode/lab-router-tls/`). ## 3. Configure both hosts diff --git a/docs/PRODUCTION_NETWORK_SECURITY.md b/docs/PRODUCTION_NETWORK_SECURITY.md index b5a1a49..2017f14 100644 --- a/docs/PRODUCTION_NETWORK_SECURITY.md +++ b/docs/PRODUCTION_NETWORK_SECURITY.md @@ -12,7 +12,8 @@ management_cidr ──SSH──▶ rn-platform-01 │ management_cidr ──SSH──▶ rn-lab-host-01 │ proxmox_mgmt_cidr ──22/8006──▶ Proxmox host │ │ -rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd) +rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd) [CONTROL plane] +rn-platform-01 (Gateway) ──mTLS:8443──▶ rn-lab-host-01 (Lab Host Router) [DATA plane] rn-lab-host-01 (promtail) ──3100──▶ rn-platform-01 (Loki) rn-lab-host-01 (node-exporter/cadvisor) ◀──scrape── rn-platform-01 (Prometheus) ``` @@ -37,6 +38,25 @@ Firewall source of truth: `infra/firewall/{proxmox-host,rn-platform-01,rn-lab-ho (Grafana is reachable only via Caddy's `/grafana/*` path, itself restricted to `management_cidr` by a `caddy` matcher — see `infra/docker/platform/Caddyfile.j2`). +## Control plane vs data plane (Phase 4.1 §10) + +Two separate, never-conflated mTLS connections between the two VMs, deliberately kept apart +both conceptually and operationally: + +| | Control plane | Data plane | +|---|---|---| +| What | Provisioner manages Lab containers | Gateway serves student browser traffic | +| Path | Provisioner → dockerd (port 2376) | Gateway → Lab Host Router (port 8443) | +| Who initiates | Provisioner only | Gateway only | +| What it can do | Create/start/stop/remove containers, full Docker API | Reverse-proxy one HTTP/WS request to one already-running container by opaque token — nothing else | +| Certs | `provisioner-client-*` / `server-*` | `gateway-client-*` / `router-server-*` | +| Exposes Docker API to caller? | Yes (that's its job — Provisioner IS a Docker client) | **Never** — the Router never exposes Docker API, container IDs, or network info to its caller | + +The student browser's traffic is proxied at the Gateway/Router layer and never touches the +Docker API at any point — the Router deliberately has no route that echoes back anything +Docker-internal (see `platform/lab-router/src/server.js`'s header comment for the full "narrow +by design" list). + ## Docker TLS (Provisioner ↔ Lab Host) Mutual TLS on port 2376, restricted at the firewall to `platform_ip` only. Application support @@ -57,34 +77,51 @@ front this with Cloudflare or expose the origin directly is left as a deployment being able to see plaintext unless "Full (strict)" mode is used) — not decided here since it doesn't affect anything else in this document. -## Known gap: Gateway-to-Lab reachability across two real VMs +## Gateway-to-Lab reachability across two real VMs — RESOLVED in Phase 4.1 -**This is the most important item in this document.** +Phase 4 flagged this as the single most important open item; Phase 4.1 +(`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1) closed it. Kept here as the historical record of +what the gap was and how it was closed, since the reasoning matters for anyone extending this +later (e.g. adding a second Lab Host). -The Gateway currently reaches a running Lab container via `env.internal_ip` — a Docker -bridge-local address recorded by the Provisioner when it creates the container -(`platform/gateway/src/server.js`, `platform/provisioner`, built in Phase 2A). This works when -Gateway and the Lab share a Docker daemon, which has been true in every setup so far, including -dev and any single-host deployment. **It stops being true once Labs run on a genuinely separate -`rn-lab-host-01`** — a Docker bridge-internal IP on one host is not routable from a different -host by default. +**The gap, as originally found:** the Gateway used to reach a running Lab container via +`env.internal_ip` — a Docker bridge-local address recorded by the Provisioner when it creates +the container. That only works when Gateway and the Lab share a Docker daemon, true in dev and +any single-host setup, false the moment Labs run on a genuinely separate `rn-lab-host-01`. -What's built now (network-policy half only): a firewall allowance in -`infra/firewall/rn-lab-host-01.rules` for `platform_ip` → `lab_gateway_port_range` -(`30000:32767` by default, configurable), so that once Lab containers publish ports on the Lab -Host's own interface, the Gateway VM is permitted to reach them. +**The fix — a Lab Host Router (`platform/lab-router/`):** a small, narrow service that runs on +rn-lab-host-01 (host-networked, like `egress-proxy`, so it can reach whatever per-Lab bridge +network the Provisioner creates) and does exactly one thing: given an opaque routing token over +mTLS, find the ONE local container labeled with that token (`docker ps --filter +label=rootnode.routing-token=` — resolved purely from local Docker state, no Postgres, +no call back to the Provisioner) and reverse-proxy the request to it, HTTP and WebSocket both. -What's **not yet built** (application half — explicitly flagged, not silently decided here): -the Provisioner needs to either (a) publish each Lab container's relevant port(s) on the Lab -Host's interface and record a reachable `external_ip`/`external_port` alongside (or instead of) -`internal_ip`/`internal_port`, or (b) route Gateway→Lab traffic through a proxy/tunnel that -lives on the Lab Host and forwards to the container's internal address. Option (a) is simpler -and consistent with how the egress proxy and cadvisor are already handled in this phase's -compose files; option (b) avoids publishing a wide port range but adds a moving part. This is a -small, reviewable, application-level change — not a network-policy change — and should land as -its own reviewed commit before real two-VM production traffic depends on it. Until it lands, -Gateway↔Lab terminal/scenario traffic will not work correctly across two separate VMs, even -though every other production networking, firewall, and TLS boundary is in place and correct. +The Gateway now decides `direct` vs `router` per `lab_host` (`lab_hosts.routing_mode` + +`router_endpoint`, `platform/provisioner/migrations/004_lab_hosts_routing.sql`) via +`platform/gateway/src/lab-endpoint.js` — a single function, `labTarget(env, token)`, that every +proxy/fetch call site in `server.js` goes through instead of building +`http://${env.internal_ip}:${env.internal_port}` itself. `dev-local` stays `direct` (T480/local +dev unaffected, zero behavior change there); a real `rn-lab-host-01` is configured `router`. + +**Firewall (`infra/firewall/rn-lab-host-01.rules`):** the old speculative +`lab_gateway_port_range` allowance (which assumed option (a), publishing per-Lab ports) was +replaced with a single, narrow rule: `platform_ip` → port `{{ lab_router_port | default('8443') +}}` only. No wide port range, no per-Lab port publishing at all — the Router is the only thing +the Gateway ever talks to on rn-lab-host-01 for Lab traffic. + +**mTLS:** the Gateway↔Router pair uses the SAME private CA as the Provisioner↔dockerd pair +(`infra/scripts/docker-tls/generate-docker-tls.sh`, extended in Phase 4.1 to also issue +`router-server`/`gateway-client` leaf certs), deployed to a separate directory +(`/etc/rootnode/lab-router-tls/`) by the same `docker-tls` Ansible role. + +**Proven, not just designed:** `infra/scripts/simulate-two-host/` builds a real two-network +Docker simulation (Gateway genuinely isolated on its own bridge, no route to any Lab's bridge) +and runs 13 real assertions against it — Gateway cannot reach a Lab's Docker-internal IP +directly, but reaches it correctly through the Router (HTTP and a real WebSocket +`kubectl get nodes`), Alice/Bob isolation holds, garbage/destroyed tokens are rejected, and the +Router itself refuses any connection without the Gateway's client certificate. See that +directory's README for exactly what is and isn't proven by a same-daemon simulation versus real +separate hardware. ## Cookie / auth production settings diff --git a/docs/PRODUCTION_VALIDATION_CHECKLIST.md b/docs/PRODUCTION_VALIDATION_CHECKLIST.md index 11ba11d..1947dda 100644 --- a/docs/PRODUCTION_VALIDATION_CHECKLIST.md +++ b/docs/PRODUCTION_VALIDATION_CHECKLIST.md @@ -19,8 +19,9 @@ can prove in advance. ## Lab Host VM - [ ] Docker daemon TLS verified: `docker --tlsverify --tlscacert=... -H tcp://:2376 info` succeeds from the Platform VM, fails from anywhere else +- [ ] Lab Host Router mTLS verified: a request to `https://:8443` with no client cert is rejected; with the Gateway's real client cert it isn't (mirrors `infra/scripts/simulate-two-host/test-remote-routing.js`'s "direct public access: NOT AVAILABLE" case, now against the real host) - [ ] Provisioner successfully creates a real Lab against the real Lab Host -- [ ] Gateway can actually reach that Lab's terminal — **this depends on the not-yet-built application change documented in `docs/PRODUCTION_NETWORK_SECURITY.md`'s "Known gap" section; do not mark this item done until that change has landed and been verified for real** +- [ ] Gateway can actually reach that Lab's terminal (HTTP UI load + WebSocket `kubectl` command) — application mechanism built and simulation-tested in Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1/§6/§7); this line is the real-hardware confirmation that the same result holds across the actual two VMs, not two Docker networks on one daemon - [ ] Egress proxy correctly restricts a Lab's outbound traffic (Provisioner's per-Lab `DOCKER-USER` chain) ## Security @@ -38,7 +39,7 @@ can prove in advance. ## Backup - [ ] Manual `backup-postgres.sh` run succeeds against real Postgres, real off-host target reachable -- [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness +- [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness — Phase 4.1's `infra/backup/test-destructive-restore-drill.sh` proved the full mechanism (realistic data, real storage destruction, restore, real Platform API boot) locally; this line is the real-hardware confirmation - [ ] Cron job confirmed scheduled (`crontab -l`) ## Observability @@ -46,5 +47,12 @@ can prove in advance. - [ ] Trigger one alert deliberately (e.g. stop a service) and confirm it fires and reaches the webhook - [ ] Loki has logs flowing from both VMs +## Pre-production hardening (Phase 4.1 — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md) +- [ ] `resource-policy.yaml`'s `admission_control.max_concurrent_labs` set from a real measured + `infra/scripts/capacity-model.js` run against the real Lab Host, not left disabled +- [ ] A deliberate over-capacity attempt returns the clean 503 "Lab capacity is temporarily + full" message, not a 500 — confirms the real host's admission control matches the local test +- [ ] `docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md`'s steps still make sense once capacity numbers are real + Only once every box above is checked against the real server does the deployment graduate from "infrastructure package built" to "actually in production." diff --git a/docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md b/docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md new file mode 100644 index 0000000..d3cdfc5 --- /dev/null +++ b/docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md @@ -0,0 +1,85 @@ +# Remote & Self-Paced Delivery Architecture + +Phase 4.1 §12. This is an architecture review, not a new product — no self-registration, +payments, or public enrollment is built here (explicit non-goal). The question this document +answers: does the infrastructure and application assume "an instructor always provisions the +Lab," and if so, where, and what would need to change for self-paced delivery to work. + +## The short answer: less of that assumption exists than expected + +RootNode Academy's Lab lifecycle was built student-driven from Phase 1 — `POST +/api/environment/launch` is called by the STUDENT'S OWN session, not by an instructor action. +"Prepare Labs" (`POST /api/admin/cohorts/:id/prepare-labs`) is a classroom **convenience** that +front-loads several students' launches at once before a synchronous class session, not the only +path to a running Lab. This was true before Phase 4.1 and confirmed still true by this phase's +own regression suites (`platform/test/e2e-phase2a.js` roster's two students with `cohort_id: +null` and takes them through login → curriculum → launch → scenario work with zero cohort +involvement) and by two new Phase 4.1 drills built specifically to exercise the remote/returning +-student narrative: + +- `infra/scripts/drills/test-lab-host-loss.js` — student launches, makes progress, their Lab is + destroyed out from under them (standing in for an idle-reaper destroy), Platform stays up, + progress is untouched, they launch again unprompted and continue. +- `infra/scripts/drills/test-fake-cohort-rehearsal.js` — the same "returns later, fresh Lab, + curriculum continues" narrative as part of a fuller classroom rehearsal. + +Both pass today, with zero application code changes needed for the self-paced-relevant parts of +this flow specifically — see the Phase 4.1 report for the full run results. + +## What already works, unmodified + +- **Student-initiated provisioning** — no instructor step is required between "student wants to + work" and "Lab exists." (`platform/api/src/server.js`'s `/api/environment/launch`.) +- **Cohort-less students** — `cohort_id` is nullable on student creation + (`POST /api/admin/students`); a student with no cohort logs in, sees the curriculum, launches + Labs, and makes progress identically to a cohorted one. Confirmed by inspection AND by the + existing regression suite, not just by schema reading. +- **Idle reaper → academic state survives → return → continue** — the exact lifecycle this + section describes as the target self-paced experience is what Phase 1's reaper + + Phase 2B's Postgres-backed academic data (separate from the disposable Lab) were already + designed to produce. Phase 4.1 proved it end-to-end, twice (see above). +- **Curriculum content itself is not cohort-gated** — `GET /api/curriculum/path` reads the one + curriculum path file directly (`curriculum.loadCurriculumPath()`); it never checks + `curriculum_assignments` or cohort membership at all. A self-paced student sees the full + curriculum with no instructor action required to "unlock" it. + +## What's genuinely missing for a real self-paced product (Phase 5, not built here) + +1. **No self-registration / public enrollment.** A student row is created only by an instructor + (`POST /api/admin/students`, admin/owner RBAC-gated). There is no public "sign up" surface. + This is almost certainly the right call to leave alone for now — it's explicitly a Phase 5+ + product/business decision (course commerce, is it free, who's allowed to enroll), not an + infrastructure gap. + +2. **No real student authentication.** A student session is created from a bare `student_id` + (`POST /api/login`, no password) — deliberately, since today every `student_id` is issued and + controlled by an instructor via the roster, not chosen or known by the public. A genuinely + remote/self-paced product where students access their account without an instructor in the + loop needs real credentials (email+password, or a magic link) — reusing the same + `password_hash`/`scrypt` machinery `instructors` already has (`src/auth/password.js`) would + be the natural extension, not a new auth system. **This is the one item on this list that + would actually block self-paced delivery today** — everything else already works or is a + product decision, not an infrastructure one. + +3. **`curriculum_assignments.student_id` is schema-ready but unwired.** The table has supported + a direct per-student curriculum assignment (independent of any cohort) since Phase 2B + (`CONSTRAINT curriculum_assignment_target CHECK (cohort_id IS NOT NULL OR student_id IS NOT + NULL)`) — but nothing in the application ever reads or writes that column today; there's only + one curriculum path, so nothing has needed to differentiate yet. Not a gap that blocks + self-paced delivery now (every student already sees the one path with no gate), but the + moment a second curriculum path exists, this is where per-individual assignment lands — the + schema was designed for exactly this and doesn't need a migration to support it. + +4. **No delivery-mode concept anywhere.** In-person, remote, and self-paced sessions aren't + distinguished in any data model today, and nothing currently needs them to be — capacity + planning, the reaper, and academic records all behave identically regardless of how a student + is attending. Worth being aware this is unmodeled, not worth building a taxonomy for before a + real product reason exists (YAGNI). + +## What Phase 4.1 deliberately did NOT do here + +No signup flow, no student password/email auth implementation, no per-student curriculum +assignment route, no payments, no public course catalog, no held/premium scenario content. All +explicit non-goals per this phase's authorization. This document exists so Phase 5 planning +starts from an accurate picture of the gap (one real item: student authentication) instead of a +guess. diff --git a/infra/.gitignore b/infra/.gitignore index da291ed..d5d1b36 100644 --- a/infra/.gitignore +++ b/infra/.gitignore @@ -4,26 +4,31 @@ # Docker TLS material — CA/server/client certs and keys, generated by # infra/scripts/docker-tls/generate-ca.sh and friends. Templates/READMEs stay tracked. -infra/docker/tls/*.pem -infra/docker/tls/*.key -infra/docker/tls/*.crt -infra/docker/tls/*.srl -!infra/docker/tls/**/README.md +docker/tls/*.pem +docker/tls/*.key +docker/tls/*.crt +docker/tls/*.srl +!docker/tls/**/README.md # Ansible Vault password files and any decrypted vars -infra/ansible/.vault_pass -infra/ansible/**/vault_plain.yml +ansible/.vault_pass +ansible/**/vault_plain.yml # Terraform/OpenTofu state and generated vars (may contain resource IDs/IPs) -infra/terraform/**/*.tfstate -infra/terraform/**/*.tfstate.* -infra/terraform/**/.terraform/ -infra/terraform/**/terraform.tfvars -!infra/terraform/**/terraform.tfvars.example +# Phase 4.1: these patterns were previously written repo-root-relative (e.g. +# "infra/terraform/...") inside THIS file, which git resolves relative to the .gitignore's own +# directory (infra/) — so they silently matched nothing at all. Found via `git status` showing +# a real .terraform/ provider cache as untracked when it should have been ignored. Fixed by +# dropping the redundant leading "infra/" from every pattern below. +terraform/**/*.tfstate +terraform/**/*.tfstate.* +terraform/**/.terraform/ +terraform/**/terraform.tfvars +!terraform/**/terraform.tfvars.example # Backup output and restore scratch space -infra/backup/output/ -infra/backup/restore-test/ +backup/output/ +backup/restore-test/ # Load test results (local artifacts, not source) -infra/scripts/loadtest/results/ +scripts/loadtest/results/ diff --git a/platform/README.md b/platform/README.md index e9448f7..5a3a0f8 100644 --- a/platform/README.md +++ b/platform/README.md @@ -23,6 +23,10 @@ entrypoint/Dockerfile unmodified; frontend bundle gained Phase 3B learning UX, s `docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md` (routing), `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md` (persistent sessions, CSRF), `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` §3 (hint gating). - `egress-proxy/` — shared allowlisting proxy every Lab's outbound traffic passes through. +- `lab-router/` — Phase 4.1 addition. Runs on the Lab Host VM (or `dev-local`'s stand-in in + local dev, though `direct` mode there never exercises it); resolves an opaque routing token + to a local Lab container purely from Docker state, over mTLS, so the Gateway never needs a + Docker-internal address. Design: `docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1-3. - `docker-compose.dev.yml` — the full stack (Postgres, egress proxy, Provisioner, Platform API, Gateway) on one machine. This is the Phase 1/2A/2B **local dev exception** — production is the two-VM Proxmox split described in `docs/LAB_ARCHITECTURE.md` §1. @@ -77,13 +81,20 @@ those need a browser-equivalent client and aren't automated here. See `provisioner/README.md` (Phase 1), `docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md` §9 (Phase 2A), `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md` (Phase 3A identity limitations), and `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` §20 (Phase 3B learning/assessment limitations) for the -full, honest lists. +full, honest lists. Phase 3B's "no automatic abandoned-attempt detection" and "no per-cohort +needs-attention threshold configuration" limitations were both closed in Phase 4.1 (§13/§17 of +`docs/PHASE4_1_PREPRODUCTION_HARDENING.md`); its raw-JSON pool_spec limitation was replaced with +a structured editor (§18) with an owner-only raw-JSON fallback still available. Phase 4.1's own +known limitation: real remote/self-paced delivery needs student email/password authentication, +not built — see `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`. ## Running this in production `../infra/` has the full production infrastructure package (Ansible, Terraform, firewall, monitoring, backup, CI/CD) for deploying the Platform API/Gateway/Provisioner/Postgres onto a -dedicated `rn-platform-01` VM, with student Labs on a separate `rn-lab-host-01` VM. See +dedicated `rn-platform-01` VM, with student Labs — and a new `lab-router` service +(`../platform/lab-router/`) — on a separate `rn-lab-host-01` VM. See `../docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` for the overview and -`../docs/PRODUCTION_NETWORK_SECURITY.md` for a known gap in Gateway-to-Lab reachability across -two real VMs that must be closed before production go-live. +`../docs/PHASE4_1_PREPRODUCTION_HARDENING.md` for how Gateway-to-Lab reachability across two +real VMs was closed (a Lab Host Router + mTLS data plane, simulation-tested in +`../infra/scripts/simulate-two-host/`) and for the rest of Phase 4.1's pre-production hardening.