Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5bc0743b08 | ||
|
|
f3c356951c | ||
|
|
f5110d3c1b | ||
|
|
a8e543234c | ||
|
|
addf474b70 | ||
|
|
6dda13b837 | ||
|
|
eae8e48d43 |
@@ -196,8 +196,11 @@ Production topology and what's built vs. not yet: `docs/LAB_ARCHITECTURE.md`,
|
||||
|
||||
The production infrastructure package (Proxmox/Ansible/Terraform/firewall/monitoring/backup/CI)
|
||||
that runs all of this on real hardware lives in `infra/` — see
|
||||
`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`. Built and locally validated now; deployment is
|
||||
pending purchase of the target Hetzner dedicated server.
|
||||
`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` and `docs/PHASE4_1_PREPRODUCTION_HARDENING.md` (the
|
||||
Gateway-to-Lab cross-VM routing fix, plus application hardening — real pagination, abandonment
|
||||
handling, a Postgres-backed rate limiter, a structured assessment editor, admission control).
|
||||
Built and locally validated now; deployment is pending purchase of the target Hetzner dedicated
|
||||
server.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+10
-1
@@ -37,10 +37,19 @@ requires typing `RESTORE` to confirm, then decrypts and pipes into `psql`.
|
||||
|
||||
**Genuinely tested**, not just written: `infra/backup/test-backup-restore.sh` runs the full
|
||||
cycle against two disposable local Postgres containers, with the actual production migrations
|
||||
applied (all 17 — 3 Provisioner + 14 Platform API, in the correct cross-service order), a
|
||||
applied (3 Provisioner + Platform API, in the correct cross-service order), a
|
||||
sample row inserted, backed up, restored into the second container, and verified byte-for-byte.
|
||||
Confirmed passing, exit 0.
|
||||
|
||||
**Phase 4.1's destructive drill goes one step further**
|
||||
(`infra/backup/test-destructive-restore-drill.sh`): realistic data (owner, admin instructor,
|
||||
cohort, 3 students, progress, attempts, an assessment definition, achievements) seeded through
|
||||
the REAL Platform API, backed up, the source Postgres **container genuinely destroyed**
|
||||
(`docker rm`, not just left running alongside a second one), a fresh Postgres created from
|
||||
nothing, restored, and the real Platform API booted again against the restored database and
|
||||
used to log in as the restored admin instructor over HTTP — proving the whole stack recovers,
|
||||
not just the SQL. Never touches the developer's real persistent database. Confirmed passing.
|
||||
|
||||
## Disaster recovery scenarios
|
||||
|
||||
| Scenario | RPO | RTO (estimate) | What happens |
|
||||
|
||||
+32
-4
@@ -296,12 +296,40 @@ Compose, firewall, monitoring, backup, CI, load-test tooling).
|
||||
- **Where the boundary held:** the Platform VM / Lab Host VM security split from
|
||||
`docs/LAB_ARCHITECTURE.md` is codified in firewall rulesets and mutual-TLS Docker config, not
|
||||
weakened anywhere.
|
||||
- **Known gap surfaced, not silently fixed:** the Gateway's Lab-reachability mechanism doesn't
|
||||
survive the move to two real separate VMs — see `docs/PRODUCTION_NETWORK_SECURITY.md`'s
|
||||
"Known gap" section for what's built (firewall policy) vs. what's an explicit, reviewed,
|
||||
not-yet-built application follow-up (Provisioner/Gateway address handling).
|
||||
- **Known gap surfaced, then resolved in Phase 4.1:** the Gateway's Lab-reachability mechanism
|
||||
didn't survive the move to two real separate VMs — flagged explicitly rather than silently
|
||||
fixed, then actually built and simulation-tested (a Lab Host Router,
|
||||
`platform/lab-router/`) in Phase 4.1. See §13 below.
|
||||
- **Validated without the hardware:** real `tofu validate`, `ansible-playbook --syntax-check`,
|
||||
`docker compose config`, `caddy validate`, `promtool`/`amtool`, and a genuine end-to-end
|
||||
backup→encrypt→restore→verify cycle against disposable local Postgres with the real
|
||||
production migrations applied. None of this proves the real Hetzner network boundary — that
|
||||
needs `docs/PRODUCTION_VALIDATION_CHECKLIST.md` walked against the real server.
|
||||
|
||||
## 13. Pre-Production Hardening (Phase 4.1)
|
||||
|
||||
Full design: `docs/PHASE4_1_PREPRODUCTION_HARDENING.md`, plus
|
||||
`docs/PRODUCTION_NETWORK_SECURITY.md` (routing resolution), `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`,
|
||||
`docs/SECURITY_SCANNING.md`.
|
||||
|
||||
- **What it is:** closes the Gateway-to-Lab cross-VM gap Phase 4 flagged (a Lab Host Router,
|
||||
`platform/lab-router/`, plus a routing abstraction in the Gateway,
|
||||
`platform/gateway/src/lab-endpoint.js`), then hardens the application layer found along the
|
||||
way: abandoned-attempt handling (`scenario_attempts.status = 'abandoned'`, never a punitive
|
||||
failure), a Postgres-backed login rate limiter (survives a restart, ready to be
|
||||
shared/multi-instance), real pagination on audit/attempt/assessment-session history, a
|
||||
structured assessment authoring UI (owner-only raw JSON remains as a fallback), a minimal
|
||||
`role_in_cohort` enforcement (`teacher`/`viewer`), a `/api/version` endpoint, and admission
|
||||
control (a clean 503 "capacity full" instead of a 500 once a configured cap is reached).
|
||||
- **Tested, not just designed:** a real two-Docker-network routing simulation
|
||||
(`infra/scripts/simulate-two-host/`, 13/13), a destructive Postgres restore drill
|
||||
(`infra/backup/test-destructive-restore-drill.sh`), a Lab Host loss/rebuild drill
|
||||
(`infra/scripts/drills/test-lab-host-loss.js`), a deployment rollback drill, a CI-fails-closed
|
||||
drill (which found and fixed a real pre-existing `set -e` bug in both smoke-test scripts —
|
||||
see that drill's own output), and a full fake-cohort dress rehearsal
|
||||
(`infra/scripts/drills/test-fake-cohort-rehearsal.js`) — all passing, all in
|
||||
`infra/scripts/drills/` or `infra/backup/`.
|
||||
- **Known limitation carried forward:** real remote/self-paced delivery needs actual student
|
||||
authentication (today's `student_id`-only login relies on instructors controlling who gets a
|
||||
valid id) — see `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md` for the full accounting of what
|
||||
already works vs. what's genuinely Phase 5 scope.
|
||||
|
||||
@@ -491,8 +491,10 @@ networking, egress, resource policy, reaper) against these decisions.
|
||||
|
||||
Phase 4 (`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`) codifies the Platform/Lab-Host split this
|
||||
document designed as two real, separately-firewalled Proxmox VMs (`rn-platform-01`,
|
||||
`rn-lab-host-01`) connected only by mutual-TLS Docker and a tightly scoped firewall allowance —
|
||||
see `docs/PRODUCTION_NETWORK_SECURITY.md`. That document also records a gap Phase 4 surfaced but
|
||||
did not resolve: the Gateway's `env.internal_ip` mechanism for reaching a Lab container assumes
|
||||
Gateway and Lab share a Docker daemon, which breaks once they're genuinely on separate VMs. Read
|
||||
that document's "Known gap" section before treating two-VM production as fully wired.
|
||||
`rn-lab-host-01`) connected by two separate mTLS channels (Provisioner↔dockerd CONTROL plane,
|
||||
Gateway↔Lab Host Router DATA plane) and a tightly scoped firewall allowance — see
|
||||
`docs/PRODUCTION_NETWORK_SECURITY.md`. Phase 4 surfaced a gap in the Gateway's `env.internal_ip`
|
||||
mechanism for reaching a Lab container (it assumes Gateway and Lab share a Docker daemon, which
|
||||
breaks once they're genuinely on separate VMs); Phase 4.1
|
||||
(`docs/PHASE4_1_PREPRODUCTION_HARDENING.md`) built and simulation-tested the fix, a Lab Host
|
||||
Router (`platform/lab-router/`).
|
||||
|
||||
@@ -0,0 +1,254 @@
|
||||
# Phase 4.1 — Pre-Production Hardening
|
||||
|
||||
**Status: PHASE 4.1 COMPLETE. PRE-PRODUCTION HARDENING COMPLETE. READY FOR HARDWARE DEPLOYMENT.**
|
||||
|
||||
Phase 4 built the production infrastructure package with the real Hetzner/Proxmox hardware not
|
||||
yet purchased, and flagged one genuine architectural gap: the Gateway's mechanism for reaching
|
||||
a student Lab assumed Gateway and Lab shared a Docker daemon, which is false the moment Labs run
|
||||
on a genuinely separate `rn-lab-host-01` VM. Phase 4.1's job was to close that gap and harden
|
||||
everything else discovered along the way, still without the real hardware.
|
||||
|
||||
## 1. The core fix: a Lab Host Router
|
||||
|
||||
**Before:** Gateway → `http://${env.internal_ip}:${env.internal_port}` directly — a Docker
|
||||
bridge-local address, unreachable across a real VM boundary.
|
||||
|
||||
**After:** Gateway → (mode-aware) → either `direct` (unchanged, dev/single-host) or `router`
|
||||
(production): a small new service, `platform/lab-router/`, runs on the Lab Host itself,
|
||||
authenticates the Gateway via mTLS (the same private CA the Docker TLS control plane already
|
||||
used, extended with two new leaf certs), resolves an opaque routing token to a local container
|
||||
purely from `docker ps` label state (no Postgres access, no call back to the Provisioner), and
|
||||
reverse-proxies HTTP and WebSocket traffic to it.
|
||||
|
||||
The decision of which mode a given `lab_host` uses is data-driven
|
||||
(`lab_hosts.routing_mode`/`router_endpoint`, `platform/provisioner/migrations/004_lab_hosts_routing.sql`),
|
||||
resolved once per request in `platform/gateway/src/lab-endpoint.js`'s `labTarget()` — the only
|
||||
place in the Gateway that knows this distinction exists. Every proxy/fetch call site in
|
||||
`platform/gateway/src/server.js` goes through it.
|
||||
|
||||
**Exact `env.internal_ip` replacement:** `labTarget(env, token)` returns `{ url, headers, agent,
|
||||
mode }`. In `direct` mode, `url` is exactly the old `http://internal_ip:internal_port`. In
|
||||
`router` mode, `url` is the lab_host's `router_endpoint`, `headers` carries
|
||||
`X-RootNode-Route-Token: <token>` (computed server-side from the Provisioner-verified
|
||||
environment row — never trusted from the browser), and `agent` is an `https.Agent` configured
|
||||
with the Gateway's mTLS client identity.
|
||||
|
||||
## 2. No per-student host port
|
||||
|
||||
Deliberately not solved by publishing a host port per Lab (the originally-flagged, rejected
|
||||
approach). The Router resolves entirely from Docker's own label state on its own host — no
|
||||
port-per-student anything, no dynamic firewall rule per Lab.
|
||||
|
||||
## 3. Lab Host Router — design
|
||||
|
||||
See `platform/lab-router/src/server.js`'s header comment for the full "narrow by design" list:
|
||||
one route (proxy-by-token), requires mTLS (no bearer-token fallback), never exposes Docker API/
|
||||
container IDs/network info to its caller, never provides student-to-student routing (each
|
||||
request resolves to exactly the one container matching its own token).
|
||||
|
||||
## 4. Multi-Lab-Host ready
|
||||
|
||||
Routing is resolved per `lab_host_id` — adding `rn-lab-host-02` later is one new `lab_hosts` row
|
||||
(its own `routing_mode`/`router_endpoint`/TLS material), no Gateway code change. No scheduler
|
||||
was built (explicitly out of scope) — the Provisioner still picks "the one enabled lab_host"
|
||||
(`selectLabHost`), unchanged from Phase 1.
|
||||
|
||||
## 5. Local development compatibility
|
||||
|
||||
`dev-local` (the Unix-socket lab_host used by `docker-compose.dev.yml`) stays `routing_mode:
|
||||
direct` — zero behavior change for the T480/any local dev setup, confirmed by the full existing
|
||||
regression suite passing unmodified (§29 below) and by every new drill in this phase running
|
||||
against the same dev stack. The standalone single-container Lab image build/boot is unaffected
|
||||
(it never touches the Gateway or Provisioner at all).
|
||||
|
||||
## 6-9. Simulation, HTTP/WS validation, security tests, routing token security
|
||||
|
||||
`infra/scripts/simulate-two-host/` — a real two-Docker-network simulation (not mocks): Gateway
|
||||
attached ONLY to an isolated bridge with no route to any Lab's bridge network, Lab Host Router
|
||||
host-networked (mirrors `egress-proxy`'s existing pattern) so it can reach whatever per-Lab
|
||||
bridge the Provisioner creates. 13/13 real assertions pass, including:
|
||||
|
||||
- Gateway genuinely cannot open a TCP connection to a Lab's Docker-internal IP (proven from
|
||||
inside the Gateway container's own network namespace via a raw connect attempt).
|
||||
- Gateway reaches the Lab correctly through the Router — HTTP AND a real WebSocket terminal
|
||||
session running a genuine `kubectl get nodes` inside the Lab.
|
||||
- Alice/Bob isolation (own lab ALLOW, other's DENY — HTTP and WebSocket both), garbage tokens,
|
||||
anonymous requests, and destroyed-environment tokens are all rejected through the real code
|
||||
path.
|
||||
- The Router refuses any connection with no client certificate — mTLS is the real
|
||||
authentication mechanism.
|
||||
|
||||
Routing token security reviewed: already CSPRNG since Phase 1 (`generateRoutingToken`), never
|
||||
logged in plaintext (the Router's `tokenFingerprint` truncates), redacted in every API response
|
||||
except the two narrow internal reads that legitimately need it, and — critically — the header
|
||||
the Router trusts is computed server-side from the Provisioner-verified environment row, never
|
||||
read from the browser's own request, so a browser cannot forge routing by supplying its own
|
||||
`X-RootNode-Route-Token`.
|
||||
|
||||
## 10. Control plane vs data plane
|
||||
|
||||
Kept strictly separate — see `docs/PRODUCTION_NETWORK_SECURITY.md`'s dedicated section. Student
|
||||
browser traffic is never proxied through the Docker API at any point.
|
||||
|
||||
## 11. Firewall update
|
||||
|
||||
`infra/firewall/rn-lab-host-01.rules`'s old speculative wide port-range allowance replaced with
|
||||
one narrow rule: `platform_ip` → the Router's port only (`8443` default, configurable).
|
||||
|
||||
## 12. Self-paced / remote delivery architecture check
|
||||
|
||||
`docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`. Short version: student-driven provisioning,
|
||||
cohort-less students, and "idle reaper destroys → progress survives → student returns → fresh
|
||||
Lab → curriculum continues" all already worked before this phase and are now proven end-to-end
|
||||
by two new drills. The one real gap for actual remote/self-paced delivery is student
|
||||
authentication (today's bare `student_id` login assumes an instructor-controlled roster) — not
|
||||
built here, explicitly Phase 5 scope.
|
||||
|
||||
## 13. Abandoned attempt cleanup
|
||||
|
||||
`scenario_attempts.status = 'abandoned'` (already a valid value since Phase 3B, never
|
||||
previously set) now gets set on: environment destroy/reset (student- or instructor-triggered),
|
||||
logout, session expiry (piggybacked on the existing hourly session sweeper — no new background
|
||||
job), and switching to a different scenario while one is still open. Never a punitive marker —
|
||||
`result` stays NULL, history is preserved, instructors see an honest "abandoned (reason)"
|
||||
instead of a phantom "still in progress."
|
||||
|
||||
## 14. Gateway session cache
|
||||
|
||||
Reviewed — already a tight 2-second TTL since Phase 3A, specifically to bound the
|
||||
revocation-propagation window. Decision: **keep it, don't add push-based invalidation** — the
|
||||
added coupling (Platform API would need to know how to reach the Gateway) isn't justified by
|
||||
shrinking an already-2-second window further. Documented reasoning, not a silently-accepted gap.
|
||||
|
||||
## 15. Rate limiting
|
||||
|
||||
Login rate limiting moved from an in-memory `Map` (reset on every restart, not shareable across
|
||||
instances) to Postgres (`login_attempts` table, `migrations/016_login_attempts.sql`) —
|
||||
`platform/api/src/security.js`'s `tooManyAttempts()`. No Redis added. Restart behavior
|
||||
confirmed for real: `platform/test/e2e-phase3a.js`'s rate-limiting test passes after a genuine
|
||||
Platform API restart in the same run.
|
||||
|
||||
## 16. Pagination
|
||||
|
||||
Real server-side pagination (`platform/api/src/pagination.js`) on: the audit log (was a flat
|
||||
`LIMIT 200`), student attempt history (was `LIMIT 100`), and assessment session history (was
|
||||
**fully unbounded** — the one genuine growth risk found). Scenario analytics was deliberately
|
||||
left as-is — grouped by scenario id, naturally bounded around ~100 rows, no real growth risk to
|
||||
paginate against. Frontend (`AuditView.jsx`, `AssessmentsView.jsx`, `StudentsView.jsx`) updated
|
||||
to match; `frontend` build confirmed clean.
|
||||
|
||||
## 17. Needs-attention configuration
|
||||
|
||||
`NEEDS_ATTENTION_FAIL_THRESHOLD`/`NEEDS_ATTENTION_STALE_DAYS` moved from hardcoded constants to
|
||||
env-var-configurable (matching the existing session-TTL pattern), sensible defaults unchanged.
|
||||
|
||||
## 18. Assessment authoring UX
|
||||
|
||||
Raw-JSON `pool_spec` textarea replaced with a structured editor
|
||||
(`frontend/src/admin/AssessmentsView.jsx`'s `PoolSpecEditor`): required scenarios and random
|
||||
pools picked from a real scenario catalog (`GET /api/admin/scenario-catalog`, new), grouped by
|
||||
curriculum stage, via native multi-select — no new UI framework. An owner-only "advanced" raw
|
||||
JSON view remains for anything the structured editor doesn't cover.
|
||||
|
||||
## 19. Cohort role-in-cohort
|
||||
|
||||
`cohort_instructors.role_in_cohort` existed since Phase 2B but was never read. Implemented
|
||||
minimally: `teacher` (unchanged full capability) vs `viewer` (read-only — dashboards/analytics
|
||||
still visible, but Prepare/Destroy Labs and provisioning a student's environment are now 403'd
|
||||
server-side, `rbac.js`'s `requireCohortWriteAccess`/`requireStudentCohortWriteAccess`).
|
||||
|
||||
## 20. Backup / restore destructive drill
|
||||
|
||||
`infra/backup/test-destructive-restore-drill.sh` — real owner/admin/cohort/3-students/progress/
|
||||
attempts/assessment/achievements seeded through the real Platform API, encrypted-backed-up, the
|
||||
disposable Postgres container **genuinely destroyed** (not just restored into a second
|
||||
container), a fresh Postgres created from nothing, restored, and the real Platform API booted
|
||||
again against the restored database and used to log in as the restored admin over HTTP. All
|
||||
data verified present; never touches the developer's real database.
|
||||
|
||||
## 21. Lab Host rebuild drill
|
||||
|
||||
`infra/scripts/drills/test-lab-host-loss.js` — a student's Lab container is destroyed out from
|
||||
under the Provisioner (no clean destroy call, simulating total Lab Host loss). Proves: Platform
|
||||
stays up throughout, the stale environment is never presented as healthy once the Provisioner's
|
||||
real `reconcileOnStartup` reconciliation runs, academic progress is untouched, and the student
|
||||
can launch a genuinely new Lab and continue.
|
||||
|
||||
## 22. Deployment rollback test
|
||||
|
||||
`infra/scripts/drills/test-deployment-rollback.sh` — builds a known-good and a deliberately
|
||||
broken gateway image, "deploys" each via the real deterministic-tag mechanism, proves the smoke
|
||||
test catches the regression, computes the rollback target with the exact same
|
||||
`deploy-history.log` logic `rollback-production.sh` uses, and proves the rollback recovers.
|
||||
Documents explicitly that image rollback never implies a DB migration rollback.
|
||||
|
||||
## 23. CI/CD failure tests
|
||||
|
||||
`infra/scripts/drills/test-ci-fails-closed.sh` — proves all six required failure modes (test/
|
||||
syntax failure, frontend build failure, image build failure, config validation failure,
|
||||
migration preflight failure, staging smoke test failure) genuinely exit nonzero. **Found and
|
||||
fixed two real pre-existing bugs**: `staging-smoke-test.sh` and `production-smoke-test.sh` both
|
||||
had a classic `set -e` gotcha (`curl ... && echo "OK"` as one statement — a failing command
|
||||
that isn't the last in an `&&` chain doesn't trigger `-e`) that let an unreachable/broken
|
||||
endpoint fall through silently instead of failing the smoke test. Fixed in both files.
|
||||
|
||||
## 24. Security scanning
|
||||
|
||||
`infra/ci/gitea-workflows/build-and-test.yml`'s new `security-scan` job — npm audit, gitleaks
|
||||
(secrets), hadolint (Dockerfiles), shellcheck (shell scripts), trivy (container image CVEs).
|
||||
All free/open-source. Severity policy documented in `docs/SECURITY_SCANNING.md`: CRITICAL/HIGH
|
||||
blocks the build, MEDIUM/LOW is reported only. `security-scan` gates `build-images` the same
|
||||
way `regression` does.
|
||||
|
||||
## 25. Version / release visibility
|
||||
|
||||
`GET /api/version` (Platform API) — app version (`package.json`), deployed Git SHA
|
||||
(`GIT_SHA` env var, set to the same value as the image tag in the production compose file), and
|
||||
the most recently applied migration as a schema-version stand-in. Unauthenticated, minimal, no
|
||||
infrastructure detail.
|
||||
|
||||
## 26. Admission control failure mode
|
||||
|
||||
`resource-policy.yaml`'s new (disabled-by-default) `admission_control.max_concurrent_labs` —
|
||||
when set, the Provisioner checks it BEFORE touching Docker at all, so a full host fails cleanly
|
||||
(503, `CAPACITY_EXCEEDED`) instead of failing mid-provision. Relayed by the Platform API as a
|
||||
clean "Lab capacity is temporarily full. Try again shortly." — never a raw 500. A new
|
||||
`GET /capacity` Provisioner endpoint feeds the instructor dashboard's capacity display.
|
||||
|
||||
## 27. Fake cohort dress rehearsal
|
||||
|
||||
`infra/scripts/drills/test-fake-cohort-rehearsal.js` — one continuous narrative: instructor
|
||||
login, cohort + 4 students, curriculum assignment, Prepare Labs, students working different
|
||||
scenarios, hints, a deliberate failed validation, instructor analytics, a Lab reset, a Lab
|
||||
reaped (same consequence as the real idle reaper — see the drill's own comment on why the real
|
||||
multi-minute timer isn't waited out), the student returning and continuing, an exam
|
||||
assigned/taken, and end-of-class Lab destruction — progress verified intact after every
|
||||
destructive step. 14/14 passing.
|
||||
|
||||
## 28. Documentation
|
||||
|
||||
This file, `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`, `docs/SECURITY_SCANNING.md` (new).
|
||||
Updated: `docs/LAB_ARCHITECTURE.md`, `docs/PRODUCTION_NETWORK_SECURITY.md`,
|
||||
`docs/PRODUCTION_DEPLOYMENT.md`, `docs/PRODUCTION_VALIDATION_CHECKLIST.md`, `docs/BACKUP_DR.md`,
|
||||
`docs/ENGINEERING_SOP.md`, `README.md`, `platform/README.md`.
|
||||
|
||||
## 29. Regression
|
||||
|
||||
Re-ran the full existing suite after every application-code change in this phase:
|
||||
`e2e-phase2a.js` (10/10), `e2e-phase2b.js` (12/12), `e2e-phase3a.js` (21/21 — including a real
|
||||
audit-log pagination-shape fix and confirming rate-limit persistence across a genuine restart),
|
||||
`e2e-phase3b.js` (24/24). Frontend production build clean. Real Lab image builds/boots (both
|
||||
dev-compose and standalone single-container). `tofu validate`, `ansible-playbook
|
||||
--syntax-check`, production Compose config (both platform and lab-host), `caddy validate`,
|
||||
`promtool`/`amtool`, and the backup/restore mechanism all re-validated clean after the
|
||||
infrastructure changes in this phase (new `lab-router` service, new firewall rule, new Ansible
|
||||
tasks). No knowingly-unexplained regression was left in this codebase at the end of this phase.
|
||||
|
||||
## What Phase 4.1 deliberately did NOT build
|
||||
|
||||
Production deployment itself, Hetzner/Proxmox installation, public course signup, payments,
|
||||
subscriptions, parent portal, marketing site, full LMS, video hosting, email automation,
|
||||
certificates, course marketplace, mobile app, a multi-host scheduler, an AI tutor, invasive
|
||||
proctoring, held scenario content, student email/password authentication (documented as the one
|
||||
real gap for self-paced delivery, not built). All explicit non-goals per this phase's
|
||||
authorization.
|
||||
@@ -170,15 +170,18 @@ unchanged. No parent portal, billing, held scenarios, or production OIDC provide
|
||||
RootNode production infrastructure is dedicated to RootNode Academy — never folded into
|
||||
existing Synthos production servers, and canonical source stays Gitea, never GitHub `kubekosh`.
|
||||
|
||||
## Known gap surfaced during this phase (not silently papered over)
|
||||
## Known gap surfaced during this phase (not silently papered over) — RESOLVED in Phase 4.1
|
||||
|
||||
Building the real production Compose/network config surfaced that the Gateway's existing
|
||||
mechanism for reaching a Lab container (`env.internal_ip`, a Docker-bridge-local address) only
|
||||
works when Gateway and Lab share a Docker daemon — true in every dev/single-host setup so far,
|
||||
**not** true once Labs run on a genuinely separate `rn-lab-host-01`. The network-policy half is
|
||||
built (a firewall allowance for Gateway→Lab-Host on a published port range,
|
||||
`infra/firewall/rn-lab-host-01.rules`); the application half (the Provisioner recording a
|
||||
reachable external address for each Lab, and the Gateway using it) is an explicit, reviewed,
|
||||
NOT-YET-BUILT follow-up — see `docs/PRODUCTION_NETWORK_SECURITY.md` for the two candidate
|
||||
designs and why neither was silently chosen here. This must land before real two-VM go-live;
|
||||
flagging it clearly was judged better than guessing at an architecture change mid-infra-phase.
|
||||
**not** true once Labs run on a genuinely separate `rn-lab-host-01`. At the time this phase
|
||||
closed, the network-policy half was built but the application half was flagged as an explicit,
|
||||
not-yet-built follow-up rather than guessed at mid-infra-phase.
|
||||
|
||||
**Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1) built and proved the application
|
||||
half** — a Lab Host Router (`platform/lab-router/`) plus a `labTarget()` routing abstraction in
|
||||
the Gateway, real-tested against a genuine two-network Docker simulation
|
||||
(`infra/scripts/simulate-two-host/`), not just designed. See
|
||||
`docs/PRODUCTION_NETWORK_SECURITY.md`'s "Gateway-to-Lab reachability" section for the full
|
||||
resolution. This item no longer blocks real two-VM go-live.
|
||||
|
||||
@@ -31,7 +31,7 @@ tofu apply
|
||||
Creates `rn-platform-01` and `rn-lab-host-01` from the cloud-init template. Confirm both boot
|
||||
and are SSH-reachable from the management network before continuing.
|
||||
|
||||
## 2. Generate Docker TLS certificates (offline, once)
|
||||
## 2. Generate TLS certificates (offline, once)
|
||||
|
||||
```bash
|
||||
export ROOTNODE_TLS_BUNDLE_DIR=/secure/offline/path # never inside the repo
|
||||
@@ -39,8 +39,14 @@ export ROOTNODE_TLS_BUNDLE_DIR=/secure/offline/path # never inside the repo
|
||||
./infra/scripts/docker-tls/verify-docker-tls.sh
|
||||
```
|
||||
|
||||
The CA private key stays in `ROOTNODE_TLS_BUNDLE_DIR` and never touches either VM — only the CA
|
||||
cert (public) and the relevant leaf cert+key go to each host.
|
||||
Generates FOUR leaf certs under one CA (Phase 4.1 extended this from two): the CONTROL-plane
|
||||
pair (`server-*`/`provisioner-client-*`, for Provisioner↔dockerd) and the DATA-plane pair
|
||||
(`router-server-*`/`gateway-client-*`, for Gateway↔Lab Host Router — see
|
||||
`docs/PRODUCTION_NETWORK_SECURITY.md`'s "control plane vs data plane" section). The CA private
|
||||
key stays in `ROOTNODE_TLS_BUNDLE_DIR` and never touches either VM — only the CA cert (public)
|
||||
and each host's own leaf cert+key go to each host, both pairs deployed by the same `docker-tls`
|
||||
Ansible role into two separate directories (`/etc/rootnode/docker-tls/` and
|
||||
`/etc/rootnode/lab-router-tls/`).
|
||||
|
||||
## 3. Configure both hosts
|
||||
|
||||
|
||||
@@ -12,7 +12,8 @@ management_cidr ──SSH──▶ rn-platform-01 │
|
||||
management_cidr ──SSH──▶ rn-lab-host-01 │
|
||||
proxmox_mgmt_cidr ──22/8006──▶ Proxmox host │
|
||||
│
|
||||
rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd)
|
||||
rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd) [CONTROL plane]
|
||||
rn-platform-01 (Gateway) ──mTLS:8443──▶ rn-lab-host-01 (Lab Host Router) [DATA plane]
|
||||
rn-lab-host-01 (promtail) ──3100──▶ rn-platform-01 (Loki)
|
||||
rn-lab-host-01 (node-exporter/cadvisor) ◀──scrape── rn-platform-01 (Prometheus)
|
||||
```
|
||||
@@ -37,6 +38,25 @@ Firewall source of truth: `infra/firewall/{proxmox-host,rn-platform-01,rn-lab-ho
|
||||
(Grafana is reachable only via Caddy's `/grafana/*` path, itself restricted to
|
||||
`management_cidr` by a `caddy` matcher — see `infra/docker/platform/Caddyfile.j2`).
|
||||
|
||||
## Control plane vs data plane (Phase 4.1 §10)
|
||||
|
||||
Two separate, never-conflated mTLS connections between the two VMs, deliberately kept apart
|
||||
both conceptually and operationally:
|
||||
|
||||
| | Control plane | Data plane |
|
||||
|---|---|---|
|
||||
| What | Provisioner manages Lab containers | Gateway serves student browser traffic |
|
||||
| Path | Provisioner → dockerd (port 2376) | Gateway → Lab Host Router (port 8443) |
|
||||
| Who initiates | Provisioner only | Gateway only |
|
||||
| What it can do | Create/start/stop/remove containers, full Docker API | Reverse-proxy one HTTP/WS request to one already-running container by opaque token — nothing else |
|
||||
| Certs | `provisioner-client-*` / `server-*` | `gateway-client-*` / `router-server-*` |
|
||||
| Exposes Docker API to caller? | Yes (that's its job — Provisioner IS a Docker client) | **Never** — the Router never exposes Docker API, container IDs, or network info to its caller |
|
||||
|
||||
The student browser's traffic is proxied at the Gateway/Router layer and never touches the
|
||||
Docker API at any point — the Router deliberately has no route that echoes back anything
|
||||
Docker-internal (see `platform/lab-router/src/server.js`'s header comment for the full "narrow
|
||||
by design" list).
|
||||
|
||||
## Docker TLS (Provisioner ↔ Lab Host)
|
||||
|
||||
Mutual TLS on port 2376, restricted at the firewall to `platform_ip` only. Application support
|
||||
@@ -57,34 +77,51 @@ front this with Cloudflare or expose the origin directly is left as a deployment
|
||||
being able to see plaintext unless "Full (strict)" mode is used) — not decided here since it
|
||||
doesn't affect anything else in this document.
|
||||
|
||||
## Known gap: Gateway-to-Lab reachability across two real VMs
|
||||
## Gateway-to-Lab reachability across two real VMs — RESOLVED in Phase 4.1
|
||||
|
||||
**This is the most important item in this document.**
|
||||
Phase 4 flagged this as the single most important open item; Phase 4.1
|
||||
(`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1) closed it. Kept here as the historical record of
|
||||
what the gap was and how it was closed, since the reasoning matters for anyone extending this
|
||||
later (e.g. adding a second Lab Host).
|
||||
|
||||
The Gateway currently reaches a running Lab container via `env.internal_ip` — a Docker
|
||||
bridge-local address recorded by the Provisioner when it creates the container
|
||||
(`platform/gateway/src/server.js`, `platform/provisioner`, built in Phase 2A). This works when
|
||||
Gateway and the Lab share a Docker daemon, which has been true in every setup so far, including
|
||||
dev and any single-host deployment. **It stops being true once Labs run on a genuinely separate
|
||||
`rn-lab-host-01`** — a Docker bridge-internal IP on one host is not routable from a different
|
||||
host by default.
|
||||
**The gap, as originally found:** the Gateway used to reach a running Lab container via
|
||||
`env.internal_ip` — a Docker bridge-local address recorded by the Provisioner when it creates
|
||||
the container. That only works when Gateway and the Lab share a Docker daemon, true in dev and
|
||||
any single-host setup, false the moment Labs run on a genuinely separate `rn-lab-host-01`.
|
||||
|
||||
What's built now (network-policy half only): a firewall allowance in
|
||||
`infra/firewall/rn-lab-host-01.rules` for `platform_ip` → `lab_gateway_port_range`
|
||||
(`30000:32767` by default, configurable), so that once Lab containers publish ports on the Lab
|
||||
Host's own interface, the Gateway VM is permitted to reach them.
|
||||
**The fix — a Lab Host Router (`platform/lab-router/`):** a small, narrow service that runs on
|
||||
rn-lab-host-01 (host-networked, like `egress-proxy`, so it can reach whatever per-Lab bridge
|
||||
network the Provisioner creates) and does exactly one thing: given an opaque routing token over
|
||||
mTLS, find the ONE local container labeled with that token (`docker ps --filter
|
||||
label=rootnode.routing-token=<token>` — resolved purely from local Docker state, no Postgres,
|
||||
no call back to the Provisioner) and reverse-proxy the request to it, HTTP and WebSocket both.
|
||||
|
||||
What's **not yet built** (application half — explicitly flagged, not silently decided here):
|
||||
the Provisioner needs to either (a) publish each Lab container's relevant port(s) on the Lab
|
||||
Host's interface and record a reachable `external_ip`/`external_port` alongside (or instead of)
|
||||
`internal_ip`/`internal_port`, or (b) route Gateway→Lab traffic through a proxy/tunnel that
|
||||
lives on the Lab Host and forwards to the container's internal address. Option (a) is simpler
|
||||
and consistent with how the egress proxy and cadvisor are already handled in this phase's
|
||||
compose files; option (b) avoids publishing a wide port range but adds a moving part. This is a
|
||||
small, reviewable, application-level change — not a network-policy change — and should land as
|
||||
its own reviewed commit before real two-VM production traffic depends on it. Until it lands,
|
||||
Gateway↔Lab terminal/scenario traffic will not work correctly across two separate VMs, even
|
||||
though every other production networking, firewall, and TLS boundary is in place and correct.
|
||||
The Gateway now decides `direct` vs `router` per `lab_host` (`lab_hosts.routing_mode` +
|
||||
`router_endpoint`, `platform/provisioner/migrations/004_lab_hosts_routing.sql`) via
|
||||
`platform/gateway/src/lab-endpoint.js` — a single function, `labTarget(env, token)`, that every
|
||||
proxy/fetch call site in `server.js` goes through instead of building
|
||||
`http://${env.internal_ip}:${env.internal_port}` itself. `dev-local` stays `direct` (T480/local
|
||||
dev unaffected, zero behavior change there); a real `rn-lab-host-01` is configured `router`.
|
||||
|
||||
**Firewall (`infra/firewall/rn-lab-host-01.rules`):** the old speculative
|
||||
`lab_gateway_port_range` allowance (which assumed option (a), publishing per-Lab ports) was
|
||||
replaced with a single, narrow rule: `platform_ip` → port `{{ lab_router_port | default('8443')
|
||||
}}` only. No wide port range, no per-Lab port publishing at all — the Router is the only thing
|
||||
the Gateway ever talks to on rn-lab-host-01 for Lab traffic.
|
||||
|
||||
**mTLS:** the Gateway↔Router pair uses the SAME private CA as the Provisioner↔dockerd pair
|
||||
(`infra/scripts/docker-tls/generate-docker-tls.sh`, extended in Phase 4.1 to also issue
|
||||
`router-server`/`gateway-client` leaf certs), deployed to a separate directory
|
||||
(`/etc/rootnode/lab-router-tls/`) by the same `docker-tls` Ansible role.
|
||||
|
||||
**Proven, not just designed:** `infra/scripts/simulate-two-host/` builds a real two-network
|
||||
Docker simulation (Gateway genuinely isolated on its own bridge, no route to any Lab's bridge)
|
||||
and runs 13 real assertions against it — Gateway cannot reach a Lab's Docker-internal IP
|
||||
directly, but reaches it correctly through the Router (HTTP and a real WebSocket
|
||||
`kubectl get nodes`), Alice/Bob isolation holds, garbage/destroyed tokens are rejected, and the
|
||||
Router itself refuses any connection without the Gateway's client certificate. See that
|
||||
directory's README for exactly what is and isn't proven by a same-daemon simulation versus real
|
||||
separate hardware.
|
||||
|
||||
## Cookie / auth production settings
|
||||
|
||||
|
||||
@@ -19,8 +19,9 @@ can prove in advance.
|
||||
|
||||
## Lab Host VM
|
||||
- [ ] Docker daemon TLS verified: `docker --tlsverify --tlscacert=... -H tcp://<lab_host_ip>:2376 info` succeeds from the Platform VM, fails from anywhere else
|
||||
- [ ] Lab Host Router mTLS verified: a request to `https://<lab_host_ip>:8443` with no client cert is rejected; with the Gateway's real client cert it isn't (mirrors `infra/scripts/simulate-two-host/test-remote-routing.js`'s "direct public access: NOT AVAILABLE" case, now against the real host)
|
||||
- [ ] Provisioner successfully creates a real Lab against the real Lab Host
|
||||
- [ ] Gateway can actually reach that Lab's terminal — **this depends on the not-yet-built application change documented in `docs/PRODUCTION_NETWORK_SECURITY.md`'s "Known gap" section; do not mark this item done until that change has landed and been verified for real**
|
||||
- [ ] Gateway can actually reach that Lab's terminal (HTTP UI load + WebSocket `kubectl` command) — application mechanism built and simulation-tested in Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1/§6/§7); this line is the real-hardware confirmation that the same result holds across the actual two VMs, not two Docker networks on one daemon
|
||||
- [ ] Egress proxy correctly restricts a Lab's outbound traffic (Provisioner's per-Lab `DOCKER-USER` chain)
|
||||
|
||||
## Security
|
||||
@@ -38,7 +39,7 @@ can prove in advance.
|
||||
|
||||
## Backup
|
||||
- [ ] Manual `backup-postgres.sh` run succeeds against real Postgres, real off-host target reachable
|
||||
- [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness
|
||||
- [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness — Phase 4.1's `infra/backup/test-destructive-restore-drill.sh` proved the full mechanism (realistic data, real storage destruction, restore, real Platform API boot) locally; this line is the real-hardware confirmation
|
||||
- [ ] Cron job confirmed scheduled (`crontab -l`)
|
||||
|
||||
## Observability
|
||||
@@ -46,5 +47,12 @@ can prove in advance.
|
||||
- [ ] Trigger one alert deliberately (e.g. stop a service) and confirm it fires and reaches the webhook
|
||||
- [ ] Loki has logs flowing from both VMs
|
||||
|
||||
## Pre-production hardening (Phase 4.1 — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md)
|
||||
- [ ] `resource-policy.yaml`'s `admission_control.max_concurrent_labs` set from a real measured
|
||||
`infra/scripts/capacity-model.js` run against the real Lab Host, not left disabled
|
||||
- [ ] A deliberate over-capacity attempt returns the clean 503 "Lab capacity is temporarily
|
||||
full" message, not a 500 — confirms the real host's admission control matches the local test
|
||||
- [ ] `docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md`'s steps still make sense once capacity numbers are real
|
||||
|
||||
Only once every box above is checked against the real server does the deployment graduate from
|
||||
"infrastructure package built" to "actually in production."
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
# Remote & Self-Paced Delivery Architecture
|
||||
|
||||
Phase 4.1 §12. This is an architecture review, not a new product — no self-registration,
|
||||
payments, or public enrollment is built here (explicit non-goal). The question this document
|
||||
answers: does the infrastructure and application assume "an instructor always provisions the
|
||||
Lab," and if so, where, and what would need to change for self-paced delivery to work.
|
||||
|
||||
## The short answer: less of that assumption exists than expected
|
||||
|
||||
RootNode Academy's Lab lifecycle was built student-driven from Phase 1 — `POST
|
||||
/api/environment/launch` is called by the STUDENT'S OWN session, not by an instructor action.
|
||||
"Prepare Labs" (`POST /api/admin/cohorts/:id/prepare-labs`) is a classroom **convenience** that
|
||||
front-loads several students' launches at once before a synchronous class session, not the only
|
||||
path to a running Lab. This was true before Phase 4.1 and confirmed still true by this phase's
|
||||
own regression suites (`platform/test/e2e-phase2a.js` roster's two students with `cohort_id:
|
||||
null` and takes them through login → curriculum → launch → scenario work with zero cohort
|
||||
involvement) and by two new Phase 4.1 drills built specifically to exercise the remote/returning
|
||||
-student narrative:
|
||||
|
||||
- `infra/scripts/drills/test-lab-host-loss.js` — student launches, makes progress, their Lab is
|
||||
destroyed out from under them (standing in for an idle-reaper destroy), Platform stays up,
|
||||
progress is untouched, they launch again unprompted and continue.
|
||||
- `infra/scripts/drills/test-fake-cohort-rehearsal.js` — the same "returns later, fresh Lab,
|
||||
curriculum continues" narrative as part of a fuller classroom rehearsal.
|
||||
|
||||
Both pass today, with zero application code changes needed for the self-paced-relevant parts of
|
||||
this flow specifically — see the Phase 4.1 report for the full run results.
|
||||
|
||||
## What already works, unmodified
|
||||
|
||||
- **Student-initiated provisioning** — no instructor step is required between "student wants to
|
||||
work" and "Lab exists." (`platform/api/src/server.js`'s `/api/environment/launch`.)
|
||||
- **Cohort-less students** — `cohort_id` is nullable on student creation
|
||||
(`POST /api/admin/students`); a student with no cohort logs in, sees the curriculum, launches
|
||||
Labs, and makes progress identically to a cohorted one. Confirmed by inspection AND by the
|
||||
existing regression suite, not just by schema reading.
|
||||
- **Idle reaper → academic state survives → return → continue** — the exact lifecycle this
|
||||
section describes as the target self-paced experience is what Phase 1's reaper +
|
||||
Phase 2B's Postgres-backed academic data (separate from the disposable Lab) were already
|
||||
designed to produce. Phase 4.1 proved it end-to-end, twice (see above).
|
||||
- **Curriculum content itself is not cohort-gated** — `GET /api/curriculum/path` reads the one
|
||||
curriculum path file directly (`curriculum.loadCurriculumPath()`); it never checks
|
||||
`curriculum_assignments` or cohort membership at all. A self-paced student sees the full
|
||||
curriculum with no instructor action required to "unlock" it.
|
||||
|
||||
## What's genuinely missing for a real self-paced product (Phase 5, not built here)
|
||||
|
||||
1. **No self-registration / public enrollment.** A student row is created only by an instructor
|
||||
(`POST /api/admin/students`, admin/owner RBAC-gated). There is no public "sign up" surface.
|
||||
This is almost certainly the right call to leave alone for now — it's explicitly a Phase 5+
|
||||
product/business decision (course commerce, is it free, who's allowed to enroll), not an
|
||||
infrastructure gap.
|
||||
|
||||
2. **No real student authentication.** A student session is created from a bare `student_id`
|
||||
(`POST /api/login`, no password) — deliberately, since today every `student_id` is issued and
|
||||
controlled by an instructor via the roster, not chosen or known by the public. A genuinely
|
||||
remote/self-paced product where students access their account without an instructor in the
|
||||
loop needs real credentials (email+password, or a magic link) — reusing the same
|
||||
`password_hash`/`scrypt` machinery `instructors` already has (`src/auth/password.js`) would
|
||||
be the natural extension, not a new auth system. **This is the one item on this list that
|
||||
would actually block self-paced delivery today** — everything else already works or is a
|
||||
product decision, not an infrastructure one.
|
||||
|
||||
3. **`curriculum_assignments.student_id` is schema-ready but unwired.** The table has supported
|
||||
a direct per-student curriculum assignment (independent of any cohort) since Phase 2B
|
||||
(`CONSTRAINT curriculum_assignment_target CHECK (cohort_id IS NOT NULL OR student_id IS NOT
|
||||
NULL)`) — but nothing in the application ever reads or writes that column today; there's only
|
||||
one curriculum path, so nothing has needed to differentiate yet. Not a gap that blocks
|
||||
self-paced delivery now (every student already sees the one path with no gate), but the
|
||||
moment a second curriculum path exists, this is where per-individual assignment lands — the
|
||||
schema was designed for exactly this and doesn't need a migration to support it.
|
||||
|
||||
4. **No delivery-mode concept anywhere.** In-person, remote, and self-paced sessions aren't
|
||||
distinguished in any data model today, and nothing currently needs them to be — capacity
|
||||
planning, the reaper, and academic records all behave identically regardless of how a student
|
||||
is attending. Worth being aware this is unmodeled, not worth building a taxonomy for before a
|
||||
real product reason exists (YAGNI).
|
||||
|
||||
## What Phase 4.1 deliberately did NOT do here
|
||||
|
||||
No signup flow, no student password/email auth implementation, no per-student curriculum
|
||||
assignment route, no payments, no public course catalog, no held/premium scenario content. All
|
||||
explicit non-goals per this phase's authorization. This document exists so Phase 5 planning
|
||||
starts from an accurate picture of the gap (one real item: student authentication) instead of a
|
||||
guess.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Security Scanning (Phase 4.1 §24)
|
||||
|
||||
`infra/ci/gitea-workflows/build-and-test.yml`'s `security-scan` job. All free, open-source,
|
||||
no account/signup required for any of them — no paid scanner was added.
|
||||
|
||||
## What's scanned
|
||||
|
||||
| Tool | Checks | Scope |
|
||||
|---|---|---|
|
||||
| `npm audit --audit-level=high` | Known-vulnerable dependencies | Every first-party Node service |
|
||||
| [gitleaks](https://github.com/gitleaks/gitleaks) | Committed secrets (keys, tokens, credentials) | Whole repo |
|
||||
| [hadolint](https://github.com/hadolint/hadolint) | Dockerfile best-practice/security lint | Every first-party Dockerfile |
|
||||
| [shellcheck](https://www.shellcheck.net/) | Shell script correctness/security | `infra/**/*.sh` |
|
||||
| [trivy](https://github.com/aquasecurity/trivy) | OS/library CVEs in a built container image | `platform-api` image (representative — same base/layers as the other Node services) |
|
||||
|
||||
## Severity policy
|
||||
|
||||
- **CRITICAL / HIGH** — fails the build (`security-scan` blocks `build-images`, which blocks
|
||||
`deploy-staging`, which blocks production promotion — see the job dependency chain in the
|
||||
workflow file).
|
||||
- **MEDIUM / LOW** — reported in the job output, never blocks. A build that's permanently red
|
||||
over a low-severity transitive dependency finding nobody can immediately fix trains everyone
|
||||
to ignore red builds, which is worse than not scanning at all.
|
||||
|
||||
`npm audit --audit-level=high` and trivy's `--severity HIGH,CRITICAL` both implement this
|
||||
directly at the tool level, not via manual triage.
|
||||
|
||||
## What's NOT in the automated job (and why)
|
||||
|
||||
- **Dedicated IaC security linting** (checkov/tflint) — not added. `infra/scripts/validate-compose.sh`,
|
||||
`infra/firewall/validate-firewall.sh`, `ansible-playbook --syntax-check`, and `tofu validate`
|
||||
(all in the `regression` job) already catch structural/syntax problems in the infra code;
|
||||
a dedicated security-policy linter for Terraform/Ansible is a reasonable Phase 5 addition once
|
||||
there's a real cloud footprint to misconfigure, not before.
|
||||
- **DAST/penetration testing** — out of scope for CI; this is a build-time gate, not a running-system
|
||||
test. Covered separately by `docs/PRODUCTION_VALIDATION_CHECKLIST.md`'s security section once
|
||||
real hardware exists.
|
||||
- **SAST beyond hadolint/shellcheck** — the codebase is small enough (a handful of first-party
|
||||
Node services) that `npm audit` + secret scanning + Dockerfile/shell lint cover the realistic
|
||||
risk surface; a dedicated SAST tool (Semgrep, CodeQL) is a reasonable addition once the
|
||||
codebase or team grows enough to justify the added CI time.
|
||||
|
||||
## Local reproduction
|
||||
|
||||
Every check above can be run locally with the exact same free tools — no CI-only dependency.
|
||||
See the `security-scan` job's steps in `infra/ci/gitea-workflows/build-and-test.yml` for the
|
||||
exact commands (they're copy-pasteable).
|
||||
@@ -57,3 +57,6 @@ tr:hover td { background: var(--surface2); }
|
||||
.linkBtn { background: none; border: none; color: var(--blue); cursor: pointer; font-size: 0.85rem; padding: 0; text-decoration: underline; }
|
||||
.errorBox { background: var(--red-dim); color: var(--red); border: 1px solid var(--red); border-radius: var(--radius); padding: 0.6rem 0.8rem; font-size: 0.85rem; margin-bottom: 1rem; }
|
||||
.muted { color: var(--text-2); font-size: 0.85rem; }
|
||||
|
||||
/* Phase 4.1 §16 — shared pagination control (AuditView, StudentsView attempts). */
|
||||
.pager { display: flex; align-items: center; gap: 0.75rem; margin-top: 0.75rem; }
|
||||
|
||||
@@ -29,7 +29,7 @@ export default function AssessmentsView({ role }) {
|
||||
{canAuthor && (
|
||||
<div className={styles.card}>
|
||||
<h2>Create assessment</h2>
|
||||
<CreateAssessmentForm onCreated={reload} />
|
||||
<CreateAssessmentForm onCreated={reload} role={role} />
|
||||
</div>
|
||||
)}
|
||||
<div className={styles.card}>
|
||||
@@ -56,20 +56,86 @@ export default function AssessmentsView({ role }) {
|
||||
)
|
||||
}
|
||||
|
||||
function CreateAssessmentForm({ onCreated }) {
|
||||
const [form, setForm] = useState({ id: '', type: 'exam', title: '', duration_minutes: 60, hint_policy: 'disabled', pool_spec: '' })
|
||||
// Phase 4.1 §18 — pool_spec is built from structured picks (required scenarios + named random
|
||||
// pools), not hand-typed JSON. `catalog` ({ stages: [{ id, name, scenarios: [{id,title}] }] })
|
||||
// comes from GET /api/admin/scenario-catalog. An owner can still drop into a raw-JSON view for
|
||||
// anything the structured editor doesn't cover — see the toggle below.
|
||||
function PoolSpecEditor({ catalog, required, setRequired, pools, setPools }) {
|
||||
const allScenarios = (catalog?.stages || []).flatMap((s) => s.scenarios.map((sc) => ({ ...sc, stageName: s.name })))
|
||||
const titleFor = (id) => allScenarios.find((s) => s.id === id)?.title || id
|
||||
|
||||
const addPool = () => setPools([...pools, { count: 1, from: [] }])
|
||||
const removePool = (i) => setPools(pools.filter((_, idx) => idx !== i))
|
||||
const updatePool = (i, patch) => setPools(pools.map((p, idx) => (idx === i ? { ...p, ...patch } : p)))
|
||||
|
||||
return (
|
||||
<div>
|
||||
<label className={styles.label}>Required scenarios (always included)</label>
|
||||
<select className={styles.select} multiple size={6} style={{ width: '100%' }}
|
||||
value={required} onChange={(e) => setRequired([...e.target.selectedOptions].map((o) => o.value))}>
|
||||
{catalog?.stages.map((stage) => (
|
||||
<optgroup key={stage.id} label={stage.name}>
|
||||
{stage.scenarios.map((sc) => <option key={sc.id} value={sc.id}>{sc.title}</option>)}
|
||||
</optgroup>
|
||||
))}
|
||||
</select>
|
||||
<p className={styles.muted}>{required.length ? `${required.length} selected: ${required.map(titleFor).join(', ')}` : 'None selected — cmd/ctrl-click to pick multiple.'}</p>
|
||||
|
||||
<label className={styles.label} style={{ marginTop: '1rem', display: 'block' }}>Random pools (draw N at random from a set, every attempt)</label>
|
||||
{pools.map((pool, i) => (
|
||||
<div key={i} className={styles.card} style={{ marginBottom: '0.5rem' }}>
|
||||
<div className={styles.formRow}>
|
||||
<div style={{ maxWidth: '100px' }}><label className={styles.label}>Draw count</label>
|
||||
<input className={styles.input} type="number" min="1" value={pool.count}
|
||||
onChange={(e) => updatePool(i, { count: Number(e.target.value) })} /></div>
|
||||
<div><button type="button" className={styles.linkBtn} onClick={() => removePool(i)}>Remove pool</button></div>
|
||||
</div>
|
||||
<label className={styles.label}>Draw from</label>
|
||||
<select className={styles.select} multiple size={5} style={{ width: '100%' }}
|
||||
value={pool.from} onChange={(e) => updatePool(i, { from: [...e.target.selectedOptions].map((o) => o.value) })}>
|
||||
{catalog?.stages.map((stage) => (
|
||||
<optgroup key={stage.id} label={stage.name}>
|
||||
{stage.scenarios.map((sc) => <option key={sc.id} value={sc.id}>{sc.title}</option>)}
|
||||
</optgroup>
|
||||
))}
|
||||
</select>
|
||||
{pool.count > pool.from.length && <p className={styles.muted} style={{ color: 'var(--amber, #b58900)' }}>Draw count ({pool.count}) exceeds the pool size ({pool.from.length}) — every draw will use every scenario, not a random subset.</p>}
|
||||
</div>
|
||||
))}
|
||||
<button type="button" className={styles.linkBtn} onClick={addPool}>+ Add random pool</button>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
function CreateAssessmentForm({ onCreated, role }) {
|
||||
const [form, setForm] = useState({ id: '', type: 'exam', title: '', duration_minutes: 60, hint_policy: 'disabled' })
|
||||
const [catalog, setCatalog] = useState(null)
|
||||
const [required, setRequired] = useState([])
|
||||
const [pools, setPools] = useState([])
|
||||
const [advanced, setAdvanced] = useState(false)
|
||||
const [rawPoolSpec, setRawPoolSpec] = useState('')
|
||||
const [error, setError] = useState(null)
|
||||
const [busy, setBusy] = useState(false)
|
||||
|
||||
useEffect(() => { adminApi.scenarioCatalog().then(setCatalog).catch((e) => setError(e.message)) }, [])
|
||||
|
||||
const submit = async (e) => {
|
||||
e.preventDefault()
|
||||
setBusy(true); setError(null)
|
||||
try {
|
||||
const pool_spec = JSON.parse(form.pool_spec || '{"required":[],"random":[]}')
|
||||
// Owner's raw-JSON view, when open, is the source of truth — otherwise built from the
|
||||
// structured pickers above.
|
||||
const pool_spec = advanced
|
||||
? JSON.parse(rawPoolSpec || '{"required":[],"random":[]}')
|
||||
: { required, random: pools.filter((p) => p.count > 0 && p.from.length > 0) };
|
||||
if (!advanced && !required.length && !pool_spec.random.length) {
|
||||
throw new Error('Pick at least one required scenario or one random pool.')
|
||||
}
|
||||
await adminApi.createAssessment({ ...form, duration_minutes: Number(form.duration_minutes), pool_spec })
|
||||
setForm({ id: '', type: 'exam', title: '', duration_minutes: 60, hint_policy: 'disabled', pool_spec: '' })
|
||||
setForm({ id: '', type: 'exam', title: '', duration_minutes: 60, hint_policy: 'disabled' })
|
||||
setRequired([]); setPools([]); setRawPoolSpec('')
|
||||
onCreated()
|
||||
} catch (e) { setError(e.message?.includes('JSON') ? 'pool_spec must be valid JSON' : e.message) } finally { setBusy(false) }
|
||||
} catch (e) { setError(e.message?.includes('JSON') ? 'Raw pool_spec must be valid JSON' : e.message) } finally { setBusy(false) }
|
||||
}
|
||||
|
||||
return (
|
||||
@@ -91,26 +157,47 @@ function CreateAssessmentForm({ onCreated }) {
|
||||
<option value="disabled">disabled</option><option value="flagged">flagged</option><option value="free">free</option>
|
||||
</select></div>
|
||||
</div>
|
||||
<label className={styles.label}>Pool spec (JSON)</label>
|
||||
<textarea className={styles.input} style={{ width: '100%', fontFamily: 'var(--mono)', minHeight: '4rem' }}
|
||||
placeholder='{"required":["scenario-id"],"random":[{"count":1,"from":["a","b"]}]}'
|
||||
value={form.pool_spec} onChange={(e) => setForm({ ...form, pool_spec: e.target.value })} required />
|
||||
|
||||
{role === 'owner' && (
|
||||
<p className={styles.muted}>
|
||||
<button type="button" className={styles.linkBtn} onClick={() => setAdvanced(!advanced)}>
|
||||
{advanced ? '← Back to the structured editor' : 'Advanced: edit pool_spec as raw JSON →'}
|
||||
</button>
|
||||
</p>
|
||||
)}
|
||||
|
||||
{advanced ? (
|
||||
<>
|
||||
<label className={styles.label}>Pool spec (JSON)</label>
|
||||
<textarea className={styles.input} style={{ width: '100%', fontFamily: 'var(--mono)', minHeight: '4rem' }}
|
||||
placeholder='{"required":["scenario-id"],"random":[{"count":1,"from":["a","b"]}]}'
|
||||
value={rawPoolSpec} onChange={(e) => setRawPoolSpec(e.target.value)} required />
|
||||
</>
|
||||
) : !catalog ? <p className={styles.muted}>Loading scenario catalog…</p> : (
|
||||
<PoolSpecEditor catalog={catalog} required={required} setRequired={setRequired} pools={pools} setPools={setPools} />
|
||||
)}
|
||||
|
||||
<button className={styles.btn} style={{ marginTop: '0.75rem' }} disabled={busy}>{busy ? 'Creating…' : 'Create'}</button>
|
||||
</form>
|
||||
)
|
||||
}
|
||||
|
||||
function AssessmentDetail({ definition, cohorts, onBack }) {
|
||||
const [sessions, setSessions] = useState(null)
|
||||
// Phase 4.1 §16 — assessmentSessions is now paginated (was unbounded); `result` holds the
|
||||
// full { items, page, total, total_pages } envelope, `sessions` stays the plain array for
|
||||
// the table below so its render logic doesn't need to change.
|
||||
const [result, setResult] = useState(null)
|
||||
const [page, setPage] = useState(1)
|
||||
const [error, setError] = useState(null)
|
||||
const [notice, setNotice] = useState(null)
|
||||
const [cohortId, setCohortId] = useState(cohorts?.[0]?.cohort_id || '')
|
||||
const [busy, setBusy] = useState(false)
|
||||
|
||||
const reload = useCallback(() => {
|
||||
adminApi.assessmentSessions(definition.assessment_definition_id).then(setSessions).catch((e) => setError(e.message))
|
||||
}, [definition])
|
||||
adminApi.assessmentSessions(definition.assessment_definition_id, page).then(setResult).catch((e) => setError(e.message))
|
||||
}, [definition, page])
|
||||
useEffect(reload, [reload])
|
||||
const sessions = result?.items || null
|
||||
|
||||
const assign = async () => {
|
||||
setBusy(true); setError(null); setNotice(null)
|
||||
@@ -158,6 +245,13 @@ function AssessmentDetail({ definition, cohorts, onBack }) {
|
||||
</tbody>
|
||||
</table>
|
||||
)}
|
||||
{result && result.total_pages > 1 && (
|
||||
<div className={styles.pager}>
|
||||
<button disabled={page <= 1} onClick={() => setPage((p) => p - 1)}>Previous</button>
|
||||
<span className={styles.muted}>Page {result.page} of {result.total_pages} ({result.total} total)</span>
|
||||
<button disabled={page >= result.total_pages} onClick={() => setPage((p) => p + 1)}>Next</button>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</>
|
||||
)
|
||||
|
||||
@@ -2,36 +2,47 @@ import { useState, useEffect } from 'react'
|
||||
import { adminApi } from './api'
|
||||
import styles from './Admin.module.css'
|
||||
|
||||
// admin/owner only (enforced server-side too). Latest 200 events, no pagination UI — this is a
|
||||
// pilot-scale audit log, not a SIEM; add pagination if that ever stops being true.
|
||||
// admin/owner only (enforced server-side too). Phase 4.1 §16 — real server-side pagination
|
||||
// (platform/api/src/pagination.js) replaces the old flat "latest 200, nothing older" cap.
|
||||
export default function AuditView() {
|
||||
const [events, setEvents] = useState(null)
|
||||
const [page, setPage] = useState(1)
|
||||
const [result, setResult] = useState(null)
|
||||
const [error, setError] = useState(null)
|
||||
|
||||
useEffect(() => {
|
||||
adminApi.auditEvents().then(setEvents).catch((e) => setError(e.message))
|
||||
}, [])
|
||||
setResult(null)
|
||||
adminApi.auditEvents(page).then(setResult).catch((e) => setError(e.message))
|
||||
}, [page])
|
||||
|
||||
const events = result?.items || []
|
||||
|
||||
return (
|
||||
<div className={styles.card}>
|
||||
<h2>Audit log</h2>
|
||||
<p className={styles.muted}>Most recent 200 security/admin events.</p>
|
||||
<p className={styles.muted}>{result ? `${result.total} total events` : 'Security/admin events, most recent first.'}</p>
|
||||
{error && <div className={styles.errorBox}>{error}</div>}
|
||||
{!events ? <p className={styles.muted}>Loading…</p> : (
|
||||
<table>
|
||||
<thead><tr><th>When</th><th>Actor</th><th>Action</th><th>Resource</th><th>Result</th></tr></thead>
|
||||
<tbody>
|
||||
{events.map((e) => (
|
||||
<tr key={e.audit_event_id}>
|
||||
<td>{new Date(e.created_at).toLocaleString()}</td>
|
||||
<td>{e.actor_type}{e.actor_id ? ` · ${e.actor_id.slice(0, 8)}` : ''}</td>
|
||||
<td>{e.action}</td>
|
||||
<td>{e.resource_type ? `${e.resource_type}${e.resource_id ? ` (${e.resource_id})` : ''}` : '—'}</td>
|
||||
<td><span className={`${styles.pill} ${e.result === 'success' ? styles.pillOk : styles.pillErr}`}>{e.result}</span></td>
|
||||
</tr>
|
||||
))}
|
||||
</tbody>
|
||||
</table>
|
||||
{!result ? <p className={styles.muted}>Loading…</p> : (
|
||||
<>
|
||||
<table>
|
||||
<thead><tr><th>When</th><th>Actor</th><th>Action</th><th>Resource</th><th>Result</th></tr></thead>
|
||||
<tbody>
|
||||
{events.map((e) => (
|
||||
<tr key={e.audit_event_id}>
|
||||
<td>{new Date(e.created_at).toLocaleString()}</td>
|
||||
<td>{e.actor_type}{e.actor_id ? ` · ${e.actor_id.slice(0, 8)}` : ''}</td>
|
||||
<td>{e.action}</td>
|
||||
<td>{e.resource_type ? `${e.resource_type}${e.resource_id ? ` (${e.resource_id})` : ''}` : '—'}</td>
|
||||
<td><span className={`${styles.pill} ${e.result === 'success' ? styles.pillOk : styles.pillErr}`}>{e.result}</span></td>
|
||||
</tr>
|
||||
))}
|
||||
</tbody>
|
||||
</table>
|
||||
<div className={styles.pager}>
|
||||
<button disabled={page <= 1} onClick={() => setPage((p) => p - 1)}>Previous</button>
|
||||
<span className={styles.muted}>Page {result.page} of {result.total_pages}</span>
|
||||
<button disabled={page >= result.total_pages} onClick={() => setPage((p) => p + 1)}>Next</button>
|
||||
</div>
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
|
||||
@@ -49,7 +49,10 @@ function StudentDetail({ studentId, onBack }) {
|
||||
|
||||
const reload = useCallback(() => {
|
||||
adminApi.studentProgress(studentId).then(setProgress).catch(() => {})
|
||||
adminApi.studentAttempts(studentId).then(setAttempts).catch(() => {})
|
||||
// Phase 4.1 §16 — studentAttempts is now paginated; this panel only ever shows the most
|
||||
// recent page (labeled "Recent attempts," not a full history browser — see
|
||||
// /api/admin/students/:id/attempts's own pagination for the underlying capability).
|
||||
adminApi.studentAttempts(studentId).then((r) => setAttempts(r.items)).catch(() => {})
|
||||
adminApi.studentEnvironment(studentId).then((rows) => setEnv(rows?.find((e) => e.status !== 'destroyed') || null)).catch(() => {})
|
||||
}, [studentId])
|
||||
useEffect(reload, [reload])
|
||||
|
||||
@@ -52,7 +52,7 @@ export const adminApi = {
|
||||
createStudent: (display_name, cohort_id) => post('/api/admin/students', { display_name, cohort_id }),
|
||||
updateStudent: (id, fields) => patch(`/api/admin/students/${id}`, fields),
|
||||
studentProgress: (id) => get(`/api/admin/students/${id}/progress`),
|
||||
studentAttempts: (id) => get(`/api/admin/students/${id}/attempts`),
|
||||
studentAttempts: (id, page = 1) => get(`/api/admin/students/${id}/attempts?page=${page}`), // Phase 4.1 §16 — paginated, see pagination.js
|
||||
studentEnvironment: (id) => get(`/api/admin/students/${id}/environment`),
|
||||
provisionStudentLab: (id) => post(`/api/admin/students/${id}/environment/provision`),
|
||||
assignStudentCohort: (id, cohort_id) => post(`/api/admin/students/${id}/cohort`, { cohort_id }),
|
||||
@@ -79,19 +79,20 @@ export const adminApi = {
|
||||
revokeInstructorSessions: (id) => post(`/api/admin/instructors/${id}/revoke-sessions`),
|
||||
resetInstructorPassword: (id, new_password) => post(`/api/admin/instructors/${id}/reset-password`, { new_password }),
|
||||
|
||||
auditEvents: () => get('/api/admin/audit-events'),
|
||||
auditEvents: (page = 1) => get(`/api/admin/audit-events?page=${page}`), // Phase 4.1 §16 — paginated, see pagination.js
|
||||
|
||||
// ── Phase 3B: learning analytics ────────────────────────────────────────────────────────
|
||||
cohortLearning: (id) => get(`/api/admin/cohorts/${id}/learning`),
|
||||
studentLearning: (id) => get(`/api/admin/students/${id}/learning`),
|
||||
scenarioAnalytics: (cohortId) => get(`/api/admin/scenario-analytics${cohortId ? `?cohort_id=${cohortId}` : ''}`),
|
||||
needsAttention: (cohortId) => get(`/api/admin/cohorts/${cohortId}/needs-attention`),
|
||||
scenarioCatalog: () => get('/api/admin/scenario-catalog'), // Phase 4.1 §18 — structured assessment editor
|
||||
|
||||
// ── Phase 3B: assessment (exam + capstone) management ───────────────────────────────────
|
||||
assessments: () => get('/api/admin/assessments'),
|
||||
createAssessment: (fields) => post('/api/admin/assessments', fields),
|
||||
updateAssessment: (id, fields) => patch(`/api/admin/assessments/${id}`, fields),
|
||||
assignAssessment: (id, cohort_id) => post(`/api/admin/assessments/${id}/assign`, { cohort_id }),
|
||||
assessmentSessions: (id) => get(`/api/admin/assessments/${id}/sessions`),
|
||||
assessmentSessions: (id, page = 1) => get(`/api/admin/assessments/${id}/sessions?page=${page}`), // Phase 4.1 §16 — paginated, see pagination.js
|
||||
cancelAssessmentSession: (sessionId) => post(`/api/admin/assessment-sessions/${sessionId}/cancel`),
|
||||
}
|
||||
|
||||
+20
-15
@@ -4,26 +4,31 @@
|
||||
|
||||
# Docker TLS material — CA/server/client certs and keys, generated by
|
||||
# infra/scripts/docker-tls/generate-ca.sh and friends. Templates/READMEs stay tracked.
|
||||
infra/docker/tls/*.pem
|
||||
infra/docker/tls/*.key
|
||||
infra/docker/tls/*.crt
|
||||
infra/docker/tls/*.srl
|
||||
!infra/docker/tls/**/README.md
|
||||
docker/tls/*.pem
|
||||
docker/tls/*.key
|
||||
docker/tls/*.crt
|
||||
docker/tls/*.srl
|
||||
!docker/tls/**/README.md
|
||||
|
||||
# Ansible Vault password files and any decrypted vars
|
||||
infra/ansible/.vault_pass
|
||||
infra/ansible/**/vault_plain.yml
|
||||
ansible/.vault_pass
|
||||
ansible/**/vault_plain.yml
|
||||
|
||||
# Terraform/OpenTofu state and generated vars (may contain resource IDs/IPs)
|
||||
infra/terraform/**/*.tfstate
|
||||
infra/terraform/**/*.tfstate.*
|
||||
infra/terraform/**/.terraform/
|
||||
infra/terraform/**/terraform.tfvars
|
||||
!infra/terraform/**/terraform.tfvars.example
|
||||
# Phase 4.1: these patterns were previously written repo-root-relative (e.g.
|
||||
# "infra/terraform/...") inside THIS file, which git resolves relative to the .gitignore's own
|
||||
# directory (infra/) — so they silently matched nothing at all. Found via `git status` showing
|
||||
# a real .terraform/ provider cache as untracked when it should have been ignored. Fixed by
|
||||
# dropping the redundant leading "infra/" from every pattern below.
|
||||
terraform/**/*.tfstate
|
||||
terraform/**/*.tfstate.*
|
||||
terraform/**/.terraform/
|
||||
terraform/**/terraform.tfvars
|
||||
!terraform/**/terraform.tfvars.example
|
||||
|
||||
# Backup output and restore scratch space
|
||||
infra/backup/output/
|
||||
infra/backup/restore-test/
|
||||
backup/output/
|
||||
backup/restore-test/
|
||||
|
||||
# Load test results (local artifacts, not source)
|
||||
infra/scripts/loadtest/results/
|
||||
scripts/loadtest/results/
|
||||
|
||||
@@ -13,7 +13,7 @@ lab_host_ip: __LAB_HOST_VM_IP__
|
||||
management_cidr: __YOUR_ADMIN_ACCESS_CIDR__ # e.g. a VPN/WireGuard subnet, never 0.0.0.0/0
|
||||
proxmox_mgmt_cidr: __PROXMOX_HOST_MGMT_CIDR__
|
||||
lab_subnet_range: "10.42.0.0/16" # matches NO_PROXY_CIDRS already used by the app
|
||||
lab_gateway_port_range: "30000:32767" # see docs/PRODUCTION_NETWORK_SECURITY.md — Gateway-to-Lab reachability follow-up
|
||||
lab_router_port: "8443" # Gateway -> Lab Host Router (data plane) — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3
|
||||
|
||||
# ── Storage (Phase 4 §4 — a decision framework, not a final layout; see docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §4)
|
||||
rootnode_data_root: /srv/rootnode-data # expected to be a ZFS dataset mountpoint on the real host
|
||||
|
||||
@@ -4,3 +4,19 @@
|
||||
name: docker
|
||||
state: restarted
|
||||
daemon_reload: true
|
||||
|
||||
# Phase 4.1 — these restart the whole per-host compose wrapper unit (no per-container systemd
|
||||
# units exist in this project's convention, see the `platform`/`lab-host` roles), which is
|
||||
# heavier than a surgical single-container restart but consistent with how every other
|
||||
# config-changed-so-restart-the-stack case is already handled here.
|
||||
- name: restart lab-router
|
||||
ansible.builtin.systemd:
|
||||
name: rootnode-lab-host.service
|
||||
state: restarted
|
||||
when: inventory_hostname in groups['lab_host']
|
||||
|
||||
- name: restart gateway
|
||||
ansible.builtin.systemd:
|
||||
name: rootnode-platform.service
|
||||
state: restarted
|
||||
when: inventory_hostname in groups['platform']
|
||||
|
||||
@@ -87,3 +87,50 @@
|
||||
mode: "0644"
|
||||
when: inventory_hostname in groups['lab_host']
|
||||
notify: restart docker with tls
|
||||
|
||||
# ── Phase 4.1: the DATA-plane mTLS pair (Gateway <-> Lab Host Router). Same CA as the
|
||||
# CONTROL-plane pair above, separate directory (/etc/rootnode/lab-router-tls) so the two
|
||||
# concerns stay bind-mount-separable — the `lab-router`/`gateway` containers only ever need
|
||||
# their own pair mounted in, never the Docker TLS one. See
|
||||
# docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3.
|
||||
- name: Create /etc/rootnode/lab-router-tls
|
||||
ansible.builtin.file:
|
||||
path: /etc/rootnode/lab-router-tls
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0700"
|
||||
|
||||
- name: Deploy CA cert for the data-plane pair (public, both hosts)
|
||||
ansible.builtin.copy:
|
||||
src: "{{ docker_tls_local_bundle_dir }}/ca.pem"
|
||||
dest: /etc/rootnode/lab-router-tls/ca.pem
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0444"
|
||||
|
||||
- name: Deploy Lab Host Router server cert/key (rn-lab-host-01 only)
|
||||
ansible.builtin.copy:
|
||||
src: "{{ docker_tls_local_bundle_dir }}/{{ item.src }}"
|
||||
dest: "/etc/rootnode/lab-router-tls/{{ item.dest }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "{{ item.mode }}"
|
||||
loop:
|
||||
- { src: router-server-cert.pem, dest: router-server-cert.pem, mode: "0444" }
|
||||
- { src: router-server-key.pem, dest: router-server-key.pem, mode: "0400" }
|
||||
when: inventory_hostname in groups['lab_host']
|
||||
notify: restart lab-router
|
||||
|
||||
- name: Deploy Gateway client cert/key (rn-platform-01 only)
|
||||
ansible.builtin.copy:
|
||||
src: "{{ docker_tls_local_bundle_dir }}/{{ item.src }}"
|
||||
dest: "/etc/rootnode/lab-router-tls/{{ item.dest }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "{{ item.mode }}"
|
||||
loop:
|
||||
- { src: gateway-client-cert.pem, dest: gateway-client-cert.pem, mode: "0444" }
|
||||
- { src: gateway-client-key.pem, dest: gateway-client-key.pem, mode: "0400" }
|
||||
when: inventory_hostname in groups['platform']
|
||||
notify: restart gateway
|
||||
|
||||
@@ -3,3 +3,11 @@ PLATFORM_API_URL=http://api:4300
|
||||
PROVISIONER_URL=http://provisioner:4100
|
||||
PROVISIONER_SHARED_SECRET={{ vault_provisioner_shared_secret }}
|
||||
GATEWAY_PORT=4200
|
||||
# Phase 4.1 — mTLS client identity for the Lab Host Router (data plane). Only used when a
|
||||
# lab_host's routing_mode is 'router' (see platform/gateway/src/lab-endpoint.js); harmless if
|
||||
# every configured lab_host is still 'direct'. Paths are inside the container — mounted from
|
||||
# /etc/rootnode/lab-router-tls on the host by the `docker-tls` role, same pattern as the
|
||||
# Provisioner's Docker TLS client cert.
|
||||
LAB_ROUTER_TLS_CA=/tls/ca.pem
|
||||
LAB_ROUTER_TLS_CERT=/tls/gateway-client-cert.pem
|
||||
LAB_ROUTER_TLS_KEY=/tls/gateway-client-key.pem
|
||||
|
||||
@@ -16,3 +16,9 @@ INSTRUCTOR_SESSION_IDLE_HOURS=2
|
||||
|
||||
COOKIE_SECURE=1
|
||||
TRUST_PROXY=1
|
||||
|
||||
# Phase 4.1 §17 — instructor "needs attention" thresholds. Commented out (code defaults apply:
|
||||
# 3 failures, 7 stale days) — uncomment and override in group_vars/{{ env }}.yml only if a
|
||||
# cohort's real usage shows the defaults are wrong for it.
|
||||
# NEEDS_ATTENTION_FAIL_THRESHOLD=3
|
||||
# NEEDS_ATTENTION_STALE_DAYS=7
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
#!/usr/bin/env node
|
||||
// Seeds realistic academy data for the Phase 4.1 §20 destructive restore drill — an owner, an
|
||||
// admin instructor, a cohort, 3 students, progress, attempts, an assessment definition +
|
||||
// session, and an achievement. Uses the REAL Platform API over HTTP for everything that
|
||||
// doesn't need a live Provisioner/k3s Lab (owner login, instructor/cohort/student/assessment
|
||||
// creation — genuinely exercises the real RBAC/creation code paths), and direct SQL inserts
|
||||
// only for progress/attempts/achievements (those need real scenario activity in the full app;
|
||||
// re-running the whole learning engine here would just re-prove what e2e-phase3b.js already
|
||||
// proves — this drill is about DATA DURABILITY, not re-testing scenario validation).
|
||||
//
|
||||
// Usage: PLATFORM_API_URL=http://localhost:PORT PLATFORM_DB_URL=postgres://... node seed-realistic-data.js
|
||||
|
||||
const { Pool } = require('../../platform/api/node_modules/pg');
|
||||
|
||||
const API = process.env.PLATFORM_API_URL || 'http://localhost:4390';
|
||||
const DB_URL = process.env.PLATFORM_DB_URL;
|
||||
if (!DB_URL) { console.error('Set PLATFORM_DB_URL'); process.exit(1); }
|
||||
|
||||
function makeSession() {
|
||||
const jar = new Map();
|
||||
function cookieHeader() { return [...jar].map(([k, v]) => `${k}=${v}`).join('; '); }
|
||||
function absorb(res) {
|
||||
const cookies = typeof res.headers.getSetCookie === 'function' ? res.headers.getSetCookie() : [];
|
||||
for (const c of cookies) {
|
||||
const [pair] = c.split(';');
|
||||
const eq = pair.indexOf('=');
|
||||
if (eq === -1) continue;
|
||||
jar.set(pair.slice(0, eq).trim(), pair.slice(eq + 1).trim());
|
||||
}
|
||||
}
|
||||
return {
|
||||
async req(method, path, body) {
|
||||
const headers = { 'Content-Type': 'application/json' };
|
||||
if (cookieHeader()) headers.Cookie = cookieHeader();
|
||||
if (!['GET', 'HEAD'].includes(method)) headers['X-CSRF-Token'] = jar.get('rn_csrf_i') || '';
|
||||
const res = await fetch(`${API}${path}`, { method, headers, body: body ? JSON.stringify(body) : undefined });
|
||||
absorb(res);
|
||||
const text = await res.text();
|
||||
let json; try { json = JSON.parse(text); } catch { json = text; }
|
||||
if (!res.ok) throw new Error(`${method} ${path} -> ${res.status}: ${JSON.stringify(json)}`);
|
||||
return json;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const owner = makeSession();
|
||||
await owner.req('POST', '/api/auth/instructor/login', { email: '[email protected]', password: 'DrillOwnerPass!2026' });
|
||||
console.log(' owner login ok');
|
||||
|
||||
const admin = await owner.req('POST', '/api/admin/instructors', {
|
||||
email: '[email protected]', display_name: 'Drill Admin', password: 'DrillAdminPass!2026', role: 'admin',
|
||||
});
|
||||
console.log(' admin instructor created:', admin.instructor_id);
|
||||
|
||||
const cohort = await owner.req('POST', '/api/admin/cohorts', { cohort_id: 'drill-cohort-01', name: 'Restore Drill Cohort' });
|
||||
console.log(' cohort created');
|
||||
|
||||
const students = [];
|
||||
for (const name of ['Drill Alice', 'Drill Bob', 'Drill Carlos']) {
|
||||
const s = await owner.req('POST', '/api/admin/students', { display_name: name, cohort_id: 'drill-cohort-01' });
|
||||
students.push(s.student_id);
|
||||
}
|
||||
console.log(' 3 students rostered and enrolled:', students.join(', '));
|
||||
|
||||
const assessment = await owner.req('POST', '/api/admin/assessments', {
|
||||
id: 'drill-exam-01', type: 'exam', title: 'Restore Drill Exam', duration_minutes: 30, hint_policy: 'disabled',
|
||||
pool_spec: { required: ['pod-basics-mcq'], random: [] },
|
||||
});
|
||||
console.log(' assessment definition created:', assessment.assessment_definition_id || 'drill-exam-01');
|
||||
|
||||
// Direct SQL for progress/attempts/achievements — see file header for why.
|
||||
const pool = new Pool({ connectionString: DB_URL });
|
||||
for (const studentId of students) {
|
||||
// environment_id is a real FK to the Provisioner's environments table (nullable) — left
|
||||
// NULL here rather than faking a row that doesn't exist, same as how a genuinely old
|
||||
// attempt whose environment has long since been cleaned up would look in real data.
|
||||
await pool.query(
|
||||
`INSERT INTO scenario_attempts (student_id, scenario_id, status, result, started_at, completed_at, validation_attempt_count, validation_pass_count)
|
||||
VALUES ($1, 'pod-basics-mcq', 'completed', 'passed', now() - interval '1 hour', now(), 1, 1)`,
|
||||
[studentId]
|
||||
);
|
||||
await pool.query(
|
||||
`INSERT INTO achievements (student_id, achievement_type, stage_id, achieved_at) VALUES ($1, 'stage_completed', 'foundation', now())`,
|
||||
[studentId]
|
||||
);
|
||||
}
|
||||
await pool.end();
|
||||
console.log(' progress/attempts/achievements seeded for all 3 students');
|
||||
|
||||
console.log('\nSeed complete.');
|
||||
}
|
||||
|
||||
main().catch((e) => { console.error('SEED FAILED:', e.message); process.exit(1); });
|
||||
+149
@@ -0,0 +1,149 @@
|
||||
#!/usr/bin/env bash
|
||||
# Phase 4.1 §20 — the DESTRUCTIVE disaster-recovery drill, one level up from
|
||||
# test-backup-restore.sh's mechanism-only test: seeds realistic academy data through the REAL
|
||||
# Platform API (owner, admin instructor, cohort, 3 students, progress, attempts, an assessment
|
||||
# definition, achievements), backs it up, DESTROYS the disposable Postgres storage entirely
|
||||
# (docker rm — not just "restore into a second container," the actual source is gone),
|
||||
# recreates Postgres from nothing, restores, and boots the REAL Platform API again against the
|
||||
# restored database to prove the whole stack — not just the SQL — comes back.
|
||||
#
|
||||
# Never touches the developer's real persistent database — every Postgres here is a disposable
|
||||
# container on a throwaway port, created and destroyed entirely within this script.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/../.." # repo root
|
||||
|
||||
CONTAINER=rootnode-drill-postgres
|
||||
PORT=55434
|
||||
API_PORT=4390
|
||||
TESTDIR="$(mktemp -d)"
|
||||
# curriculum.js defaults to platform/api/scenarios/{data,bundles} — only present inside the
|
||||
# Docker image build (COPY scenarios/ ./scenarios/). Running src/server.js directly from
|
||||
# source (as this drill does, to avoid needing a full container build) needs these pointed at
|
||||
# the repo's real scenario content instead.
|
||||
export SCENARIOS_DIR="$PWD/scenarios/data"
|
||||
export BUNDLES_DIR="$PWD/scenarios/bundles"
|
||||
export CURRICULUM_PATH_FILE="$PWD/scenarios/rootnode-path.json"
|
||||
export BACKUP_ENCRYPTION_PASSPHRASE="drill-passphrase-not-for-production"
|
||||
export PROVISIONER_SHARED_SECRET="drill-shared-secret-not-for-production"
|
||||
|
||||
API_PID=""
|
||||
cleanup() {
|
||||
[ -n "$API_PID" ] && kill "$API_PID" 2>/dev/null || true
|
||||
docker rm -f "$CONTAINER" >/dev/null 2>&1 || true
|
||||
rm -rf "$TESTDIR"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
if ! command -v psql >/dev/null 2>&1 || ! command -v pg_dump >/dev/null 2>&1; then
|
||||
echo "==> psql/pg_dump not found locally — reusing the same docker-exec shim as test-backup-restore.sh"
|
||||
SHIM_BIN="$TESTDIR/bin"; mkdir -p "$SHIM_BIN"
|
||||
# Same shim as test-backup-restore.sh (see that file's comment for the full "why") — the URL
|
||||
# is DROPPED, not kept, since the host-side port it names doesn't exist inside the
|
||||
# container's own network namespace; -tAc counts as a self-contained call (needs no stdin)
|
||||
# the same way -c does, or the admin-verification queries below would hang waiting on stdin.
|
||||
cat > "$SHIM_BIN/psql" <<'EOF'
|
||||
#!/usr/bin/env bash
|
||||
args=(); skip_next=0
|
||||
for a in "$@"; do
|
||||
if [ "$skip_next" = "1" ]; then skip_next=0; continue; fi
|
||||
if [ "$a" = "-p" ] || [ "$a" = "-h" ]; then skip_next=1; continue; fi
|
||||
if [[ "$a" =~ ^postgres://[^:]+:[^@]+@[^:]+:([0-9]+)/ ]]; then continue; fi # drop the URL — reconnect locally inside the container instead
|
||||
args+=("$a")
|
||||
done
|
||||
container="rootnode-drill-postgres"
|
||||
needs_stdin=1
|
||||
for a in "${args[@]}"; do { [ "$a" = "-c" ] || [ "$a" = "-tAc" ]; } && needs_stdin=0; done
|
||||
if [ "$needs_stdin" = "1" ]; then
|
||||
exec docker exec -e PGPASSWORD=postgres -i "$container" psql -h localhost -U postgres "${args[@]}"
|
||||
else
|
||||
exec docker exec -e PGPASSWORD=postgres "$container" psql -h localhost -U postgres "${args[@]}"
|
||||
fi
|
||||
EOF
|
||||
cat > "$SHIM_BIN/pg_dump" <<'EOF'
|
||||
#!/usr/bin/env bash
|
||||
args=()
|
||||
for a in "$@"; do
|
||||
if [[ "$a" =~ ^postgres://[^:]+:[^@]+@[^:]+:([0-9]+)/(.*)$ ]]; then args+=("postgres://postgres:postgres@localhost:5432/${BASH_REMATCH[2]}"); continue; fi
|
||||
args+=("$a")
|
||||
done
|
||||
exec docker exec -e PGPASSWORD=postgres rootnode-drill-postgres pg_dump "${args[@]}"
|
||||
EOF
|
||||
chmod +x "$SHIM_BIN/psql" "$SHIM_BIN/pg_dump"
|
||||
export PATH="$SHIM_BIN:$PATH"
|
||||
fi
|
||||
|
||||
DB_URL="postgres://postgres:postgres@localhost:$PORT/postgres"
|
||||
|
||||
echo "==> [1/9] Starting disposable Postgres"
|
||||
docker run -d --name "$CONTAINER" -e POSTGRES_PASSWORD=postgres -p "$PORT:5432" postgres:16-alpine >/dev/null
|
||||
for i in $(seq 1 30); do PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -c "SELECT 1" >/dev/null 2>&1 && break; sleep 1; done
|
||||
|
||||
echo "==> [2/9] Applying real Provisioner + Platform API migrations"
|
||||
( cd platform/provisioner && [ -d node_modules ] || npm install --no-audit --no-fund >/dev/null
|
||||
PLATFORM_DB_URL="$DB_URL" node -e "require('./src/db').runMigrations().then((m)=>console.log(' Provisioner:', m.join(', ')))" )
|
||||
( cd platform/api && [ -d node_modules ] || npm install --no-audit --no-fund >/dev/null
|
||||
PLATFORM_DB_URL="$DB_URL" node -e "require('./src/db').runMigrations().then((m)=>console.log(' Platform API:', m.join(', ')))" )
|
||||
|
||||
echo "==> [3/9] Bootstrapping the drill owner and starting a REAL Platform API"
|
||||
( cd platform/api && PLATFORM_DB_URL="$DB_URL" node bootstrap-owner.js [email protected] "Drill Owner" 'DrillOwnerPass!2026' )
|
||||
( cd platform/api && PLATFORM_DB_URL="$DB_URL" PROVISIONER_URL="http://localhost:1" PLATFORM_API_PORT="$API_PORT" node src/server.js > "$TESTDIR/api.log" 2>&1 & echo $! > "$TESTDIR/api.pid" )
|
||||
API_PID=$(cat "$TESTDIR/api.pid")
|
||||
for i in $(seq 1 30); do curl -sf "http://localhost:$API_PORT/healthz" >/dev/null 2>&1 && break; sleep 1; done
|
||||
curl -sf "http://localhost:$API_PORT/healthz" >/dev/null || { echo "FAIL: Platform API never came up"; cat "$TESTDIR/api.log"; exit 1; }
|
||||
echo " Platform API up on :$API_PORT"
|
||||
|
||||
echo "==> [4/9] Seeding realistic academy data through the real API (owner/admin/cohort/3 students/progress/attempts/assessment/achievements)"
|
||||
( cd infra/backup && PLATFORM_API_URL="http://localhost:$API_PORT" PLATFORM_DB_URL="$DB_URL" node seed-realistic-data.js )
|
||||
|
||||
echo "==> [5/9] Backing up (real backup-postgres.sh)"
|
||||
BACKUP_DIR="$TESTDIR/backups" PLATFORM_DB_URL="$DB_URL" ./infra/backup/backup-postgres.sh
|
||||
ENC_FILE=$(ls "$TESTDIR"/backups/*.sql.gz.gpg | head -1)
|
||||
[ -f "$ENC_FILE" ] || { echo "FAIL: no encrypted backup produced"; exit 1; }
|
||||
|
||||
echo "==> [6/9] DESTROYING the Postgres storage entirely (this is the destructive part — not restoring into a second container, the original is gone)"
|
||||
kill "$API_PID" 2>/dev/null || true; API_PID=""
|
||||
docker rm -f "$CONTAINER" >/dev/null
|
||||
echo " source container + its storage are gone"
|
||||
|
||||
echo "==> [7/9] Recreating Postgres from nothing"
|
||||
docker run -d --name "$CONTAINER" -e POSTGRES_PASSWORD=postgres -p "$PORT:5432" postgres:16-alpine >/dev/null
|
||||
for i in $(seq 1 30); do PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -c "SELECT 1" >/dev/null 2>&1 && break; sleep 1; done
|
||||
|
||||
echo "==> [8/9] Restoring from the encrypted backup (schema + data both come from the dump — no separate migration step)"
|
||||
echo "RESTORE" | BACKUP_ENCRYPTION_PASSPHRASE="$BACKUP_ENCRYPTION_PASSPHRASE" ./infra/backup/restore-postgres.sh "$ENC_FILE" "$DB_URL"
|
||||
./infra/backup/verify-restore.sh "$DB_URL"
|
||||
|
||||
echo "==> [9/9] Booting the REAL Platform API again against the RESTORED database"
|
||||
( cd platform/api && PLATFORM_DB_URL="$DB_URL" PROVISIONER_URL="http://localhost:1" PLATFORM_API_PORT="$API_PORT" node src/server.js > "$TESTDIR/api2.log" 2>&1 & echo $! > "$TESTDIR/api2.pid" )
|
||||
API_PID=$(cat "$TESTDIR/api2.pid")
|
||||
for i in $(seq 1 30); do curl -sf "http://localhost:$API_PORT/healthz" >/dev/null 2>&1 && break; sleep 1; done
|
||||
curl -sf "http://localhost:$API_PORT/healthz" >/dev/null || { echo "FAIL: Platform API did not come back up against the restored DB"; cat "$TESTDIR/api2.log"; exit 1; }
|
||||
echo " Platform API is back up, against the RESTORED database — no fresh migration run needed"
|
||||
|
||||
echo "==> Verifying accounts/RBAC/cohorts/students/progress/attempts/achievements survived"
|
||||
FAIL=0
|
||||
check() { local desc="$1" want="$2" got="$3"; if [ "$want" = "$got" ]; then echo " ok — $desc"; else echo " FAIL — $desc (want $want, got $got)"; FAIL=1; fi; }
|
||||
|
||||
check "owner account (role) restored" "owner" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT role FROM instructors WHERE email='[email protected]'" | tr -d '[:space:]')"
|
||||
check "admin instructor restored" "admin" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT role FROM instructors WHERE email='[email protected]'" | tr -d '[:space:]')"
|
||||
check "cohort restored" "1" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM cohorts WHERE cohort_id='drill-cohort-01'" | tr -d '[:space:]')"
|
||||
check "3 students restored" "3" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM cohort_enrollments WHERE cohort_id='drill-cohort-01'" | tr -d '[:space:]')"
|
||||
check "3 completed attempts restored" "3" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM scenario_attempts WHERE result='passed' AND scenario_id='pod-basics-mcq'" | tr -d '[:space:]')"
|
||||
check "3 achievements restored" "3" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM achievements WHERE stage_id='foundation'" | tr -d '[:space:]')"
|
||||
check "assessment definition restored" "1" "$(PGPASSWORD=postgres psql -h localhost -p "$PORT" -U postgres -tAc "SELECT count(*) FROM assessment_definitions WHERE assessment_definition_id='drill-exam-01'" | tr -d '[:space:]')"
|
||||
|
||||
echo "==> Verifying RBAC still works against the restored DB (real login through the real API)"
|
||||
LOGIN_STATUS=$(curl -s -o /dev/null -w '%{http_code}' -X POST "http://localhost:$API_PORT/api/auth/instructor/login" \
|
||||
-H 'Content-Type: application/json' -d '{"email":"[email protected]","password":"DrillAdminPass!2026"}')
|
||||
check "restored admin instructor can log in" "200" "$LOGIN_STATUS"
|
||||
|
||||
if [ "$FAIL" = "1" ]; then
|
||||
echo; echo "DRILL FAILED — see above."; exit 1
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "OK: full destructive restore drill passed — owner/admin/cohort/3 students/progress/attempts/"
|
||||
echo "assessment/achievements all survived a real Postgres storage loss + recreation + restore, and"
|
||||
echo "the real Platform API booted and authenticated against the restored database with no fresh"
|
||||
echo "migration run required."
|
||||
echo "NOT TESTED HERE (needs real credentials/hardware): the actual off-host Storage Box/S3 copy."
|
||||
@@ -40,16 +40,19 @@ jobs:
|
||||
|
||||
- name: Install dependencies (all first-party services)
|
||||
run: |
|
||||
for d in backend frontend platform/api platform/gateway platform/provisioner; do
|
||||
for d in backend frontend platform/api platform/gateway platform/provisioner platform/lab-router; do
|
||||
(cd "$d" && npm ci)
|
||||
done
|
||||
|
||||
- name: Syntax-check every service
|
||||
run: |
|
||||
for d in backend frontend/src platform/api platform/gateway platform/provisioner; do
|
||||
for d in backend frontend/src platform/api platform/gateway platform/provisioner platform/lab-router; do
|
||||
find "$d" -name '*.js' -not -path '*/node_modules/*' -exec node --check {} \;
|
||||
done
|
||||
|
||||
- name: Lab Host Router unit tests (Phase 4.1 — fast, no Docker daemon needed)
|
||||
run: (cd platform/lab-router && npm test)
|
||||
|
||||
- name: Build frontend
|
||||
run: (cd frontend && npm run build)
|
||||
|
||||
@@ -75,8 +78,56 @@ jobs:
|
||||
# can't grant safely — they stay a LOCAL/staging pre-promotion step
|
||||
# (infra/scripts/staging-smoke-test.sh), not this job. See docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §18.
|
||||
|
||||
# Phase 4.1 §24 — free/open-source security scanning, no paid tooling, no signup required for
|
||||
# any of these. Severity policy (docs/SECURITY_SCANNING.md): CRITICAL/HIGH findings fail the
|
||||
# build; MEDIUM/LOW are reported but never block a merge — a build that's permanently red over
|
||||
# an unrelated low-severity transitive dependency finding trains everyone to ignore red builds.
|
||||
security-scan:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: "20"
|
||||
|
||||
- name: npm audit (dependency vulnerabilities) — fails only on high/critical
|
||||
run: |
|
||||
FAIL=0
|
||||
for d in backend frontend platform/api platform/gateway platform/provisioner platform/lab-router; do
|
||||
echo "== $d =="
|
||||
(cd "$d" && npm audit --audit-level=high) || FAIL=1
|
||||
done
|
||||
exit $FAIL
|
||||
|
||||
- name: Secret scanning (gitleaks — free, no account needed)
|
||||
run: docker run --rm -v "$PWD:/repo" zricethezav/gitleaks:latest detect --source=/repo --no-git -v
|
||||
|
||||
- name: Dockerfile linting (hadolint — free)
|
||||
run: |
|
||||
FAIL=0
|
||||
for f in Dockerfile platform/api/Dockerfile platform/gateway/Dockerfile platform/provisioner/Dockerfile platform/egress-proxy/Dockerfile platform/lab-router/Dockerfile; do
|
||||
echo "== $f =="
|
||||
docker run --rm -i hadolint/hadolint < "$f" || FAIL=1
|
||||
done
|
||||
exit $FAIL
|
||||
|
||||
# -S error, not warning — matches docs/SECURITY_SCANNING.md's severity policy. Real run
|
||||
# against this tree at -S warning surfaces only style nits (unused loop vars, a missing
|
||||
# `cd ... || exit` guard) on scripts that are otherwise correct; failing the build on
|
||||
# those trains everyone to ignore the check.
|
||||
- name: Shell linting (shellcheck — free, standard)
|
||||
run: |
|
||||
sudo apt-get update -qq && sudo apt-get install -y -qq shellcheck
|
||||
find infra -name '*.sh' -exec shellcheck -S error {} \+
|
||||
|
||||
- name: Container image scanning (trivy — free, no account needed)
|
||||
run: |
|
||||
docker build -q -t rootnode-scan/platform-api -f platform/api/Dockerfile .
|
||||
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock aquasec/trivy:latest image \
|
||||
--severity HIGH,CRITICAL --exit-code 1 rootnode-scan/platform-api
|
||||
|
||||
build-images:
|
||||
needs: regression
|
||||
needs: [regression, security-scan]
|
||||
if: github.ref == 'refs/heads/main'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
|
||||
@@ -8,6 +8,7 @@
|
||||
| `rootnode/gateway` | `platform/gateway/Dockerfile` | `{{ registry }}/rootnode/gateway` |
|
||||
| `rootnode/provisioner` | `platform/provisioner/Dockerfile` | `{{ registry }}/rootnode/provisioner` |
|
||||
| `rootnode/egress-proxy` | `platform/egress-proxy/Dockerfile` | `{{ registry }}/rootnode/egress-proxy` |
|
||||
| `rootnode/lab-router` | `platform/lab-router/Dockerfile` (Phase 4.1) | `{{ registry }}/rootnode/lab-router` |
|
||||
| `rootnode/lab` | root `Dockerfile` (the student Lab image) | `{{ registry }}/rootnode/lab` |
|
||||
|
||||
## Tagging
|
||||
|
||||
@@ -1,10 +1,13 @@
|
||||
# Production compose for rn-lab-host-01. Deployed by infra/ansible/roles/lab-host/ to
|
||||
# /opt/rootnode/lab-host/docker-compose.yml, run as a systemd unit
|
||||
# (rootnode-lab-host.service.j2). Runs exactly one long-lived service: the shared egress proxy
|
||||
# every student Lab's outbound traffic passes through (platform/egress-proxy/, unchanged since
|
||||
# Phase 1). Student Lab containers themselves are NOT part of this compose file — they're
|
||||
# created/destroyed individually by the Provisioner over the remote Docker TLS endpoint
|
||||
# (dockerd itself, not compose), same mechanism as dev, just remote.
|
||||
# (rootnode-lab-host.service.j2). Runs the shared egress proxy every student Lab's outbound
|
||||
# traffic passes through (platform/egress-proxy/, unchanged since Phase 1) and, as of Phase
|
||||
# 4.1, the Lab Host Router (platform/lab-router/) that gives the Gateway on rn-platform-01 a
|
||||
# way to reach a running Lab without ever touching a Docker-internal address — see
|
||||
# docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3. Student Lab containers themselves are NOT
|
||||
# part of this compose file — they're created/destroyed individually by the Provisioner over
|
||||
# the remote Docker TLS endpoint (dockerd itself, not compose), same mechanism as dev, just
|
||||
# remote; the Router finds them by label, purely from local Docker state (see its own README).
|
||||
|
||||
networks:
|
||||
rn-egress:
|
||||
@@ -22,6 +25,26 @@ services:
|
||||
# is the actual control, see platform/egress-proxy/allowlist.txt)
|
||||
networks: [rn-egress]
|
||||
|
||||
lab-router:
|
||||
image: "{{ registry }}/rootnode/lab-router:{{ image_tag }}"
|
||||
restart: unless-stopped
|
||||
# Needs the host's own routing table to reach whichever per-Lab bridge network
|
||||
# (rootnode-lab-net-<id>) the Provisioner created for a given container — those networks
|
||||
# aren't known ahead of time and aren't attachable one-by-one, the same reason egress-proxy
|
||||
# would need this too if it initiated connections outward instead of just accepting them.
|
||||
# Restricted to platform_ip on port {{ lab_router_port | default('8443') }} at the host
|
||||
# firewall (infra/firewall/rn-lab-host-01.rules) — this is the ONLY thing that makes host
|
||||
# networking here safe, same defense-in-depth pattern as the Docker TLS port.
|
||||
network_mode: host
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
- /etc/rootnode/lab-router-tls:/tls:ro
|
||||
environment:
|
||||
LAB_ROUTER_PORT: "{{ lab_router_port | default('8443') }}"
|
||||
LAB_ROUTER_TLS_CA: /tls/ca.pem
|
||||
LAB_ROUTER_TLS_CERT: /tls/router-server-cert.pem
|
||||
LAB_ROUTER_TLS_KEY: /tls/router-server-key.pem
|
||||
|
||||
node-exporter:
|
||||
image: prom/node-exporter:latest
|
||||
restart: unless-stopped
|
||||
|
||||
@@ -88,6 +88,11 @@ services:
|
||||
image: "{{ registry }}/rootnode/platform-api:{{ image_tag }}"
|
||||
restart: unless-stopped
|
||||
env_file: /etc/rootnode/platform.env
|
||||
environment:
|
||||
# Phase 4.1 §25 — not a secret, just the deployed tag; set directly here rather than in
|
||||
# the env template since it's the same value already used for the image tag above, not a
|
||||
# separately-managed setting.
|
||||
GIT_SHA: "{{ image_tag }}"
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
@@ -99,6 +104,10 @@ services:
|
||||
image: "{{ registry }}/rootnode/gateway:{{ image_tag }}"
|
||||
restart: unless-stopped
|
||||
env_file: /etc/rootnode/gateway.env
|
||||
volumes:
|
||||
# mTLS client identity for the Lab Host Router (data plane) — see gateway.env.j2's
|
||||
# LAB_ROUTER_TLS_* vars. Read-only; the private key never leaves this one bind mount.
|
||||
- /etc/rootnode/lab-router-tls:/tls:ro
|
||||
depends_on: [provisioner, api]
|
||||
networks: [rn-platform-internal]
|
||||
|
||||
|
||||
@@ -1,7 +1,12 @@
|
||||
# Production lab_hosts seed — points at the real rn-lab-host-01 VM instead of Phase 1's dev
|
||||
# Unix socket. Same mechanism the Provisioner has had since Phase 1
|
||||
# (platform/provisioner/config/lab-hosts.seed.yaml), just pointed at a real remote endpoint —
|
||||
# no application code changes required for this, see docs/PRODUCTION_NETWORK_SECURITY.md §7.
|
||||
# (platform/provisioner/config/lab-hosts.seed.yaml), just pointed at real remote endpoints.
|
||||
#
|
||||
# Phase 4.1 correction: routing_mode/router_endpoint is REQUIRED here — Phase 4's original
|
||||
# claim that "no application code changes required" was true for the Provisioner's CONTROL
|
||||
# plane (Docker mTLS) but false for the Gateway's DATA plane, which used to assume
|
||||
# env.internal_ip was directly reachable. It is not, across two real VMs. See
|
||||
# docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1 and docs/PRODUCTION_NETWORK_SECURITY.md.
|
||||
#
|
||||
# {{ lab_host_ip }} is an Ansible template variable — see infra/ansible/group_vars/production.yml.
|
||||
# Rendered by the `platform` role; the raw file here keeps the placeholder for readability.
|
||||
@@ -9,8 +14,14 @@
|
||||
lab_hosts:
|
||||
- lab_host_id: labhost-01
|
||||
display_name: "RootNode Lab Host VM 01 (production)"
|
||||
# CONTROL plane: Provisioner -> dockerd, mTLS, port 2376.
|
||||
endpoint: "tcp://{{ lab_host_ip }}:2376"
|
||||
tls_ca_path: /etc/rootnode/docker-tls/ca.pem
|
||||
tls_cert_path: /etc/rootnode/docker-tls/provisioner-client-cert.pem
|
||||
tls_key_path: /etc/rootnode/docker-tls/provisioner-client-key.pem
|
||||
enabled: true
|
||||
# DATA plane: Gateway -> Lab Host Router, mTLS, router's own port (8443 by default —
|
||||
# infra/docker/lab-host/docker-compose.production.yml). Gateway never touches a
|
||||
# Docker-internal address once routing_mode is 'router'.
|
||||
routing_mode: router
|
||||
router_endpoint: "https://{{ lab_host_ip }}:8443"
|
||||
|
||||
@@ -1,16 +1,17 @@
|
||||
# Production lab_hosts seed — points at the real rn-lab-host-01 VM instead of Phase 1's dev
|
||||
# Unix socket. Same mechanism the Provisioner has had since Phase 1
|
||||
# (platform/provisioner/config/lab-hosts.seed.yaml), just pointed at a real remote endpoint —
|
||||
# no application code changes required for this, see docs/PRODUCTION_NETWORK_SECURITY.md §7.
|
||||
#
|
||||
# {{ lab_host_ip }} is an Ansible template variable — see infra/ansible/group_vars/production.yml.
|
||||
# Rendered by the `platform` role; the raw file here keeps the placeholder for readability.
|
||||
# Staging lab_hosts seed — see infra/environments/production/provisioner-config/lab-hosts.yaml
|
||||
# for the full explanation; this is the same shape, lower stakes. Staging's `lab_host_ip` is
|
||||
# distinct from production's own (infra/ansible/group_vars/staging.yml), so it needs the same
|
||||
# router routing_mode as production once staging genuinely runs on two hosts. If staging is
|
||||
# ever collapsed onto a single host instead, change routing_mode to `direct` and drop
|
||||
# router_endpoint — that's the only edit needed either way.
|
||||
|
||||
lab_hosts:
|
||||
- lab_host_id: labhost-01
|
||||
display_name: "RootNode Lab Host VM 01 (production)"
|
||||
display_name: "RootNode Lab Host VM 01 (staging)"
|
||||
endpoint: "tcp://{{ lab_host_ip }}:2376"
|
||||
tls_ca_path: /etc/rootnode/docker-tls/ca.pem
|
||||
tls_cert_path: /etc/rootnode/docker-tls/provisioner-client-cert.pem
|
||||
tls_key_path: /etc/rootnode/docker-tls/provisioner-client-key.pem
|
||||
enabled: true
|
||||
routing_mode: router
|
||||
router_endpoint: "https://{{ lab_host_ip }}:8443"
|
||||
|
||||
@@ -38,13 +38,14 @@
|
||||
# endpoint: ALLOW" example in the Phase 4 spec, encoded literally here.
|
||||
-A INPUT -p tcp --dport 2376 -s {{ platform_ip }} -m conntrack --ctstate NEW -j ACCEPT
|
||||
|
||||
# Gateway -> per-Lab published port range, rn-platform-01 only. REQUIRED so the Gateway can
|
||||
# reach each student's Lab UI/terminal over the network once Labs run on a genuinely separate
|
||||
# host — see docs/PRODUCTION_NETWORK_SECURITY.md §"Gateway-to-Lab cross-VM reachability" for
|
||||
# why this exists and the small Provisioner/Gateway code change it depends on (NOT YET BUILT —
|
||||
# this firewall rule is the network-policy half; the port-publishing half is an explicit,
|
||||
# reviewed application-code follow-up, tracked there rather than silently assumed here).
|
||||
-A INPUT -p tcp --dport {{ lab_gateway_port_range | default('30000:32767') }} -s {{ platform_ip }} -m conntrack --ctstate NEW -j ACCEPT
|
||||
# Lab Host Router (8443) — rn-platform-01 (the Gateway) only. This is the DATA-plane
|
||||
# equivalent of the Docker TLS rule above: the Gateway never reaches a Lab's Docker-internal
|
||||
# address directly (no per-Lab port publishing, no wide port range needed) — it reaches this
|
||||
# one mTLS-authenticated Router instead, which resolves the request to a local container on
|
||||
# ITS OWN host. See docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3 and
|
||||
# docs/PRODUCTION_NETWORK_SECURITY.md — this closes the "Gateway-to-Lab cross-VM reachability"
|
||||
# gap Phase 4 flagged as a required follow-up; it is no longer open.
|
||||
-A INPUT -p tcp --dport {{ lab_router_port | default('8443') }} -s {{ platform_ip }} -m conntrack --ctstate NEW -j ACCEPT
|
||||
|
||||
# Prometheus (rn-platform-01) scraping this host's node-exporter/cadvisor — metrics only,
|
||||
# read-only exposition endpoints, still restricted to the one legitimate scraper.
|
||||
|
||||
@@ -21,7 +21,7 @@ render() {
|
||||
-e 's/{{ lab_host_ip }}/10.10.10.20/g' \
|
||||
-e 's/{{ lab_subnet_range }}/10.42.0.0\/16/g' \
|
||||
-e 's/{{ proxmox_mgmt_cidr }}/203.0.113.0\/24/g' \
|
||||
-e "s/{{ lab_gateway_port_range | default('30000:32767') }}/30000:32767/g" \
|
||||
-e "s/{{ lab_router_port | default('8443') }}/8443/g" \
|
||||
"$1" > "$2"
|
||||
}
|
||||
|
||||
|
||||
@@ -22,6 +22,7 @@ declare -A IMAGES=(
|
||||
[rootnode/gateway]="platform/gateway/Dockerfile platform/gateway"
|
||||
[rootnode/provisioner]="platform/provisioner/Dockerfile platform/provisioner"
|
||||
[rootnode/egress-proxy]="platform/egress-proxy/Dockerfile platform/egress-proxy"
|
||||
[rootnode/lab-router]="platform/lab-router/Dockerfile platform/lab-router"
|
||||
[rootnode/lab]="Dockerfile ."
|
||||
)
|
||||
|
||||
|
||||
@@ -1,14 +1,21 @@
|
||||
#!/usr/bin/env bash
|
||||
# Generates a private CA and a mutual-TLS cert/key pair for the Provisioner <-> rn-lab-host-01
|
||||
# Docker Engine connection (docs/PRODUCTION_NETWORK_SECURITY.md §7). The application already
|
||||
# supports this (platform/provisioner/src/docker.js buildClient() has required mTLS since
|
||||
# Phase 1) — this script only ever needed to exist for the CERTIFICATES, not new app code.
|
||||
# Generates a private CA and TWO mutual-TLS cert/key pairs, both under that one CA:
|
||||
# 1. Provisioner <-> rn-lab-host-01 Docker Engine (CONTROL plane, port 2376). The application
|
||||
# already supports this (platform/provisioner/src/docker.js buildClient() has required
|
||||
# mTLS since Phase 1) — this script only ever needed to exist for the CERTIFICATES.
|
||||
# 2. Gateway <-> rn-lab-host-01 Lab Host Router (DATA plane, port 8443) — Phase 4.1, see
|
||||
# docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3 and docs/PRODUCTION_NETWORK_SECURITY.md.
|
||||
# One CA for both keeps the PKI surface small (one thing to protect, one rotation runbook) —
|
||||
# see docs/RUNBOOK_DOCKER_TLS_ROTATION.md, which now covers all four leaf certs, not just two.
|
||||
#
|
||||
# Output goes to $OUT_DIR (default ./out, gitignored) — never into a tracked path. Deploy the
|
||||
# resulting files with the `docker-tls` Ansible role, which places:
|
||||
# rn-lab-host-01: ca.pem, server-cert.pem, server-key.pem -> dockerd's --tlsverify config
|
||||
# rn-platform-01: ca.pem, provisioner-client-cert.pem, provisioner-client-key.pem
|
||||
# rn-lab-host-01: ca.pem, server-cert.pem, server-key.pem -> dockerd --tlsverify
|
||||
# ca.pem, router-server-cert.pem, router-server-key.pem -> lab-router (data plane)
|
||||
# rn-platform-01: ca.pem, provisioner-client-cert.pem, provisioner-client-key.pem
|
||||
# -> /etc/rootnode/docker-tls/ (referenced by lab-hosts.yaml)
|
||||
# ca.pem, gateway-client-cert.pem, gateway-client-key.pem
|
||||
# -> /etc/rootnode/lab-router-tls/ (referenced by gateway's LAB_ROUTER_TLS_*)
|
||||
#
|
||||
# Usage:
|
||||
# ./generate-docker-tls.sh [--out-dir DIR] [--lab-host-ip IP] [--lab-host-dns NAME] [--days N]
|
||||
@@ -61,13 +68,37 @@ EOF
|
||||
openssl x509 -req -days "$DAYS" -sha256 -in client.csr -CA ca.pem -CAkey ca-key.pem \
|
||||
-CAcreateserial -out provisioner-client-cert.pem -extfile client-extfile.cnf
|
||||
|
||||
rm -f server.csr client.csr server-extfile.cnf client-extfile.cnf
|
||||
chmod 0400 ca-key.pem server-key.pem provisioner-client-key.pem
|
||||
chmod 0444 ca.pem server-cert.pem provisioner-client-cert.pem
|
||||
echo "==> Generating server cert for rn-lab-host-01 (Lab Host Router, data plane)"
|
||||
openssl genrsa -out router-server-key.pem 4096
|
||||
openssl req -subj "/CN=$LAB_HOST_DNS" -sha256 -new -key router-server-key.pem -out router-server.csr
|
||||
|
||||
cat > router-server-extfile.cnf <<EOF
|
||||
subjectAltName = DNS:$LAB_HOST_DNS,IP:$LAB_HOST_IP
|
||||
extendedKeyUsage = serverAuth
|
||||
EOF
|
||||
|
||||
openssl x509 -req -days "$DAYS" -sha256 -in router-server.csr -CA ca.pem -CAkey ca-key.pem \
|
||||
-CAcreateserial -out router-server-cert.pem -extfile router-server-extfile.cnf
|
||||
|
||||
echo "==> Generating client cert for the Gateway (data plane)"
|
||||
openssl genrsa -out gateway-client-key.pem 4096
|
||||
openssl req -subj "/CN=rootnode-gateway" -new -key gateway-client-key.pem -out gateway-client.csr
|
||||
|
||||
openssl x509 -req -days "$DAYS" -sha256 -in gateway-client.csr -CA ca.pem -CAkey ca-key.pem \
|
||||
-CAcreateserial -out gateway-client-cert.pem -extfile client-extfile.cnf
|
||||
|
||||
rm -f server.csr client.csr router-server.csr gateway-client.csr server-extfile.cnf client-extfile.cnf router-server-extfile.cnf
|
||||
chmod 0400 ca-key.pem server-key.pem provisioner-client-key.pem router-server-key.pem gateway-client-key.pem
|
||||
chmod 0444 ca.pem server-cert.pem provisioner-client-cert.pem router-server-cert.pem gateway-client-cert.pem
|
||||
|
||||
echo "==> Done. Generated in $OUT_DIR:"
|
||||
ls -la
|
||||
echo
|
||||
echo "Deploy server-{cert,key}.pem + ca.pem to rn-lab-host-01 (docker-tls Ansible role)."
|
||||
echo "Deploy provisioner-client-{cert,key}.pem + ca.pem to rn-platform-01 (same role)."
|
||||
echo "Verify with: ../verify-docker-tls.sh $OUT_DIR <lab-host-ip>"
|
||||
echo "CONTROL plane (Provisioner -> dockerd, port 2376):"
|
||||
echo " Deploy server-{cert,key}.pem + ca.pem to rn-lab-host-01 (docker-tls Ansible role)."
|
||||
echo " Deploy provisioner-client-{cert,key}.pem + ca.pem to rn-platform-01 (same role)."
|
||||
echo "DATA plane (Gateway -> Lab Host Router, port 8443):"
|
||||
echo " Deploy router-server-{cert,key}.pem + ca.pem to rn-lab-host-01 (lab-router Ansible role)."
|
||||
echo " Deploy gateway-client-{cert,key}.pem + ca.pem to rn-platform-01 (platform Ansible role)."
|
||||
echo
|
||||
echo "Verify Docker control-plane TLS with: ../verify-docker-tls.sh $OUT_DIR <lab-host-ip>"
|
||||
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env bash
|
||||
# Orchestrates the Phase 4.1 §27 fake cohort dress rehearsal: brings up the real dev stack,
|
||||
# bootstraps an owner, runs test-fake-cohort-rehearsal.js, tears down.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "${BASH_SOURCE[0]}")/../../../platform"
|
||||
|
||||
export PROVISIONER_SHARED_SECRET="${PROVISIONER_SHARED_SECRET:-rehearsal-shared-secret-not-for-production}"
|
||||
|
||||
docker compose -f docker-compose.dev.yml up -d --build
|
||||
|
||||
cleanup() {
|
||||
echo "==> Tearing down"
|
||||
docker ps -aq --filter "name=rootnode-lab-" | xargs -r docker rm -f >/dev/null 2>&1 || true
|
||||
docker network ls --filter "name=rootnode-lab-net-" --format "{{.Name}}" | xargs -r -n1 docker network rm >/dev/null 2>&1 || true
|
||||
docker compose -f docker-compose.dev.yml down >/dev/null 2>&1 || true
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
echo "==> Waiting for the stack"
|
||||
for _ in $(seq 1 60); do
|
||||
curl -sf http://localhost:4300/healthz >/dev/null 2>&1 && curl -sf http://localhost:4200/healthz >/dev/null 2>&1 && break
|
||||
sleep 2
|
||||
done
|
||||
|
||||
echo "==> Bootstrapping the rehearsal owner"
|
||||
( cd api && PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5433/rootnode_platform \
|
||||
node bootstrap-owner.js [email protected] "Rehearsal Owner" 'RehearsalPass!2026' )
|
||||
|
||||
echo "==> Running the rehearsal"
|
||||
OWNER_EMAIL=[email protected] OWNER_PASSWORD='RehearsalPass!2026' \
|
||||
node ../infra/scripts/drills/test-fake-cohort-rehearsal.js
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
#!/usr/bin/env bash
|
||||
# Orchestrates the Phase 4.1 §21 Lab Host loss/rebuild drill: brings up the real dev stack,
|
||||
# bootstraps an owner, runs test-lab-host-loss.js, tears down. See that file's header for what
|
||||
# it actually proves.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "${BASH_SOURCE[0]}")/../../../platform"
|
||||
|
||||
export PROVISIONER_SHARED_SECRET="${PROVISIONER_SHARED_SECRET:-drill-shared-secret-not-for-production}"
|
||||
|
||||
docker compose -f docker-compose.dev.yml up -d --build
|
||||
|
||||
cleanup() {
|
||||
echo "==> Tearing down"
|
||||
docker ps -aq --filter "name=rootnode-lab-" | xargs -r docker rm -f >/dev/null 2>&1 || true
|
||||
docker network ls --filter "name=rootnode-lab-net-" --format "{{.Name}}" | xargs -r -n1 docker network rm >/dev/null 2>&1 || true
|
||||
docker compose -f docker-compose.dev.yml down >/dev/null 2>&1 || true
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
echo "==> Waiting for the stack"
|
||||
for i in $(seq 1 60); do
|
||||
curl -sf http://localhost:4300/healthz >/dev/null 2>&1 && curl -sf http://localhost:4200/healthz >/dev/null 2>&1 && break
|
||||
sleep 2
|
||||
done
|
||||
|
||||
echo "==> Bootstrapping the drill owner"
|
||||
( cd api && PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5433/rootnode_platform \
|
||||
node bootstrap-owner.js [email protected] "LabHost Drill Owner" 'LabHostDrillPass!2026' )
|
||||
|
||||
echo "==> Running the drill"
|
||||
OWNER_EMAIL=[email protected] OWNER_PASSWORD='LabHostDrillPass!2026' \
|
||||
node ../infra/scripts/drills/test-lab-host-loss.js
|
||||
Executable
+77
@@ -0,0 +1,77 @@
|
||||
#!/usr/bin/env bash
|
||||
# Phase 4.1 §23 — proves the CI pipeline (infra/ci/gitea-workflows/build-and-test.yml) fails
|
||||
# CLOSED on every listed failure mode, i.e. the underlying command genuinely exits nonzero on
|
||||
# broken input, which is what makes a Gitea/GitHub Actions step (and therefore the job, and
|
||||
# therefore `needs: regression` gating build-images/deploy-staging) fail. Doesn't need a real
|
||||
# CI run — checks the actual commands the workflow calls, directly, the same way Actions would
|
||||
# invoke them.
|
||||
set -uo pipefail # NOT -e — this script's whole point is deliberately triggering failures and checking their exit codes
|
||||
cd "$(dirname "${BASH_SOURCE[0]}")/../../.." # repo root
|
||||
|
||||
FAIL=0
|
||||
expect_nonzero() {
|
||||
local desc="$1"; shift
|
||||
"$@" >/tmp/ci-fail-drill-out 2>&1
|
||||
local code=$?
|
||||
if [ "$code" -eq 0 ]; then
|
||||
echo " FAIL — $desc (command succeeded, but it should have failed)"; cat /tmp/ci-fail-drill-out; FAIL=1
|
||||
else
|
||||
echo " ok — $desc correctly failed (exit $code)"
|
||||
fi
|
||||
}
|
||||
|
||||
echo "Phase 4.1 CI-fails-closed drill"
|
||||
echo
|
||||
|
||||
echo "== 1. Test/syntax-check failure =="
|
||||
BAD_JS="$(mktemp).js"
|
||||
echo 'function broken( {' > "$BAD_JS"
|
||||
expect_nonzero "node --check on syntactically broken JS" node --check "$BAD_JS"
|
||||
rm -f "$BAD_JS"
|
||||
|
||||
echo "== 2. Frontend build failure =="
|
||||
cp frontend/src/App.jsx /tmp/App.jsx.bak
|
||||
echo 'this is not valid jsx <<<' >> frontend/src/App.jsx
|
||||
(cd frontend && expect_nonzero "npm run build with a broken component" npm run build)
|
||||
cp /tmp/App.jsx.bak frontend/src/App.jsx; rm -f /tmp/App.jsx.bak
|
||||
|
||||
echo "== 3. Image build failure =="
|
||||
TMPDOCKERFILE="$(mktemp -d)/Dockerfile"
|
||||
cat > "$TMPDOCKERFILE" <<'EOF'
|
||||
FROM node:20-slim
|
||||
COPY this-file-does-not-exist.js /app/
|
||||
EOF
|
||||
expect_nonzero "docker build with a COPY of a nonexistent file" docker build -q -f "$TMPDOCKERFILE" "$(dirname "$TMPDOCKERFILE")"
|
||||
|
||||
echo "== 4. Config validation failure =="
|
||||
BAD_COMPOSE="$(mktemp -d)/docker-compose.yml"
|
||||
cat > "$BAD_COMPOSE" <<'EOF'
|
||||
services:
|
||||
a:
|
||||
image: alpine
|
||||
a:
|
||||
image: alpine
|
||||
EOF
|
||||
expect_nonzero "docker compose config on a file with a duplicate service key" docker compose -f "$BAD_COMPOSE" config
|
||||
|
||||
echo "== 5. Migration preflight failure =="
|
||||
( cd platform/api && expect_nonzero "runMigrations() against an unreachable Postgres" \
|
||||
env PLATFORM_DB_URL="postgres://nobody:[email protected]:1/nope" \
|
||||
node -e "require('./src/db').runMigrations().then(()=>process.exit(0)).catch((e)=>{console.error(e.message);process.exit(1)})" )
|
||||
|
||||
echo "== 6. Staging smoke test failure =="
|
||||
expect_nonzero "staging-smoke-test.sh with no STAGING_GATEWAY_URL set" \
|
||||
env -u STAGING_GATEWAY_URL bash infra/scripts/staging-smoke-test.sh
|
||||
expect_nonzero "staging-smoke-test.sh against an unreachable staging URL" \
|
||||
env STAGING_GATEWAY_URL="http://127.0.0.1:1" bash infra/scripts/staging-smoke-test.sh
|
||||
|
||||
echo
|
||||
if [ "$FAIL" = "1" ]; then
|
||||
echo "DRILL FAILED — at least one failure mode did NOT fail closed. See above."
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: every listed failure mode genuinely exits nonzero — CI fails closed on all of them."
|
||||
echo "NOT TESTED HERE: an actual Gitea Actions run (needs Gitea Actions enabled on the real"
|
||||
echo "instance and this workflow file activated at .gitea/workflows/ — see build-and-test.yml's"
|
||||
echo "own header comment). This proves the underlying commands, which is what actually"
|
||||
echo "determines pass/fail once a real run happens."
|
||||
+74
@@ -0,0 +1,74 @@
|
||||
#!/usr/bin/env bash
|
||||
# Phase 4.1 §22 — deployment rollback drill. Builds two real gateway image versions (a known
|
||||
# good build, and a deliberately broken one), "deploys" each in turn to a local container,
|
||||
# proves the smoke test genuinely catches the regression, then rolls back using the EXACT same
|
||||
# decision logic rollback-production.sh uses against deploy-history.log (tail -2 | head -1 |
|
||||
# awk '{print $2}' — the tag before the current one), and proves the rollback recovers.
|
||||
#
|
||||
# What this proves: the deterministic-tag + deploy-history.log rollback MECHANISM is real and
|
||||
# correct. What it does NOT prove: the actual `ansible-playbook site.yml -e image_tag=...` step
|
||||
# rollback-production.sh runs against a real host — that needs the real hardware
|
||||
# (docs/PRODUCTION_VALIDATION_CHECKLIST.md). This drill swaps a local Docker container's image
|
||||
# tag directly instead, as the closest local equivalent.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "${BASH_SOURCE[0]}")/../../../platform/gateway"
|
||||
|
||||
CONTAINER=rootnode-rollback-drill
|
||||
PORT=4295
|
||||
DEPLOY_LOG="$(mktemp)"
|
||||
trap 'docker rm -f "$CONTAINER" >/dev/null 2>&1 || true; rm -f "$DEPLOY_LOG"; rm -rf /tmp/rootnode-rollback-drill-broken' EXIT
|
||||
|
||||
smoke_test() {
|
||||
curl -sf "http://localhost:$PORT/healthz" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
echo "==> [1/6] Building the known-good gateway image (current source)"
|
||||
docker build -q -t rootnode-gateway-drill:known-good -f Dockerfile . >/dev/null
|
||||
|
||||
echo "==> [2/6] Building a deliberately broken version (healthz returns 500)"
|
||||
rm -rf /tmp/rootnode-rollback-drill-broken
|
||||
cp -r . /tmp/rootnode-rollback-drill-broken
|
||||
sed -i "s#app.get('/healthz', (req, res) => res.json({ ok: true }));#app.get('/healthz', (req, res) => res.status(500).json({ ok: false, broken: 'DELIBERATE_DRILL_REGRESSION' }));#" \
|
||||
/tmp/rootnode-rollback-drill-broken/src/server.js
|
||||
grep -q DELIBERATE_DRILL_REGRESSION /tmp/rootnode-rollback-drill-broken/src/server.js || { echo "FAIL: sed didn't apply — healthz line text must have changed upstream"; exit 1; }
|
||||
docker build -q -t rootnode-gateway-drill:broken -f /tmp/rootnode-rollback-drill-broken/Dockerfile /tmp/rootnode-rollback-drill-broken >/dev/null
|
||||
|
||||
deploy() {
|
||||
local tag="$1"
|
||||
docker rm -f "$CONTAINER" >/dev/null 2>&1 || true
|
||||
docker run -d --name "$CONTAINER" -p "$PORT:4200" -e GATEWAY_PORT=4200 -e PROVISIONER_URL=http://127.0.0.1:1 \
|
||||
-e PROVISIONER_SHARED_SECRET=drill -e PLATFORM_API_URL=http://127.0.0.1:1 \
|
||||
"rootnode-gateway-drill:$tag" >/dev/null
|
||||
sleep 2
|
||||
echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) $tag" >> "$DEPLOY_LOG"
|
||||
}
|
||||
|
||||
echo "==> [3/6] Deploying known-good (deploy #1)"
|
||||
deploy known-good
|
||||
smoke_test && echo " smoke test: PASS (as expected)" || { echo "FAIL: known-good should have passed"; exit 1; }
|
||||
|
||||
echo "==> [4/6] Deploying the broken version (deploy #2 — the 'bad release' this drill catches)"
|
||||
deploy broken
|
||||
if smoke_test; then echo "FAIL: the broken deploy's regression was NOT caught by the smoke test — that defeats the whole point"; exit 1; fi
|
||||
echo " smoke test: FAIL (correctly caught — this is exactly what should stop a real promotion, see docs/RUNBOOK_ROLLBACK.md)"
|
||||
|
||||
echo "==> [5/6] Computing the rollback target — same logic as infra/scripts/rollback-production.sh"
|
||||
TARGET_TAG=$(tail -2 "$DEPLOY_LOG" | head -1 | awk '{print $2}')
|
||||
[ "$TARGET_TAG" = "known-good" ] || { echo "FAIL: expected rollback target 'known-good', computed '$TARGET_TAG'"; exit 1; }
|
||||
echo " rollback target: $TARGET_TAG (correctly identified as the tag before the broken one)"
|
||||
|
||||
echo "==> [6/6] Rolling back"
|
||||
deploy "$TARGET_TAG"
|
||||
smoke_test && echo " smoke test: PASS — rollback recovered service" || { echo "FAIL: rollback did not recover"; exit 1; }
|
||||
|
||||
echo
|
||||
echo "OK: deployment rollback drill passed. Deterministic image tags + deploy-history.log"
|
||||
echo "correctly identify and recover from a bad release."
|
||||
echo
|
||||
echo "On migration compatibility (docs/RUNBOOK_FAILED_MIGRATION.md, docs/RUNBOOK_ROLLBACK.md):"
|
||||
echo "this script rolls back APPLICATION IMAGES ONLY. There is no automatic down-migration —"
|
||||
echo "if the bad release also ran a schema-changing migration, rolling the image back does NOT"
|
||||
echo "undo that migration, and old code may not run correctly against the new schema. That"
|
||||
echo "case requires restoring from the pre-migration backup (docs/RUNBOOK_POSTGRES_RESTORE.md),"
|
||||
echo "not a rollback, which is exactly why migrations in this project are check-before-alter"
|
||||
echo "and additive-only (docs/ENGINEERING_SOP.md §6) — to make image-only rollback usually safe."
|
||||
@@ -0,0 +1,150 @@
|
||||
#!/usr/bin/env node
|
||||
// Phase 4.1 §27 — the fake cohort dress rehearsal. One continuous narrative through the real
|
||||
// application (not synthetic unit assertions): instructor login, cohort + 4 students, assign
|
||||
// curriculum, Prepare Labs, students launch, work different scenarios, use hints, fail a
|
||||
// validation, complete scenarios, instructor checks analytics, an exam is assigned/taken,
|
||||
// a Lab is reset, a Lab is reaped (destroyed out-of-band, the same consequence the real idle
|
||||
// reaper produces — see the note below on why this doesn't wait out the real multi-minute
|
||||
// reaper timer), the student returns and continues, capstone state is checked, and the cohort's
|
||||
// Labs are destroyed at the end.
|
||||
//
|
||||
// Run via run-fake-cohort-rehearsal.sh, which brings up platform/docker-compose.dev.yml first.
|
||||
// No framework, matching project convention. Problems found are printed, not silently passed —
|
||||
// see the Phase 4.1 report for anything this surfaced.
|
||||
|
||||
const assert = require('assert');
|
||||
const { makeSession } = require('../../../platform/test/session-helper');
|
||||
|
||||
const GW = process.env.GATEWAY_URL || 'http://localhost:4200';
|
||||
const OWNER_EMAIL = process.env.OWNER_EMAIL;
|
||||
const OWNER_PASSWORD = process.env.OWNER_PASSWORD;
|
||||
if (!OWNER_EMAIL || !OWNER_PASSWORD) { console.error('Set OWNER_EMAIL and OWNER_PASSWORD'); process.exit(1); }
|
||||
|
||||
let passed = 0, failed = 0;
|
||||
async function step(name, fn) {
|
||||
try { await fn(); console.log(` ok — ${name}`); passed++; }
|
||||
catch (e) { console.error(` FAIL — ${name}\n ${e.stack || e.message}`); failed++; }
|
||||
}
|
||||
|
||||
async function main() {
|
||||
console.log('Phase 4.1 fake cohort dress rehearsal — slow, real k3s boots\n');
|
||||
|
||||
const instructor = makeSession(GW);
|
||||
await instructor.request('POST', '/api/auth/instructor/login', { email: OWNER_EMAIL, password: OWNER_PASSWORD });
|
||||
|
||||
const names = ['Rehearsal-Ada', 'Rehearsal-Bo', 'Rehearsal-Cy', 'Rehearsal-Di'];
|
||||
const students = {};
|
||||
const sessions = {};
|
||||
|
||||
await step('instructor: create a cohort and roster 4 students', async () => {
|
||||
await instructor.request('POST', '/api/admin/cohorts', { cohort_id: 'rehearsal-cohort', name: 'Rehearsal Cohort' });
|
||||
for (const name of names) {
|
||||
const { body } = await instructor.request('POST', '/api/admin/students', { display_name: name, cohort_id: 'rehearsal-cohort' });
|
||||
students[name] = body.student_id;
|
||||
sessions[name] = makeSession(GW);
|
||||
await sessions[name].request('POST', '/api/login', { student_id: body.student_id });
|
||||
}
|
||||
});
|
||||
|
||||
await step('instructor: assign curriculum to the cohort', async () => {
|
||||
const { status } = await instructor.request('POST', '/api/admin/cohorts/rehearsal-cohort/curriculum', { curriculum_path: 'rootnode-path' });
|
||||
assert.ok([200, 201].includes(status));
|
||||
});
|
||||
|
||||
await step('instructor: Prepare Labs for the whole cohort (concurrent)', async () => {
|
||||
const { status, body } = await instructor.request('POST', '/api/admin/cohorts/rehearsal-cohort/prepare-labs');
|
||||
assert.equal(status, 200);
|
||||
assert.ok(body.results.every((r) => r.status === 'running'), `expected every Lab running, got: ${JSON.stringify(body.results)}`);
|
||||
});
|
||||
|
||||
const launches = {};
|
||||
await step('students: everyone\'s prepared Lab is already running (no re-launch wait)', async () => {
|
||||
for (const name of names) {
|
||||
const { body } = await sessions[name].request('POST', '/api/environment/launch');
|
||||
assert.equal(body.status, 'running');
|
||||
launches[name] = body;
|
||||
}
|
||||
});
|
||||
|
||||
await step('students: Ada and Bo work DIFFERENT scenarios', async () => {
|
||||
await sessions['Rehearsal-Ada'].request('POST', `${launches['Rehearsal-Ada'].launch_path}api/scenarios/pod-basics-mcq/answer`, { selected: 'b' });
|
||||
await sessions['Rehearsal-Bo'].request('POST', `${launches['Rehearsal-Bo'].launch_path}api/scenarios/pod-basics-mcq/start`);
|
||||
});
|
||||
|
||||
await step('student: Cy uses a hint on a scenario', async () => {
|
||||
const res = await sessions['Rehearsal-Cy'].request('POST', `${launches['Rehearsal-Cy'].launch_path}api/scenarios/pod-basics-mcq/hint`);
|
||||
assert.equal(res.status, 200);
|
||||
});
|
||||
|
||||
await step('student: Di fails a validation on purpose', async () => {
|
||||
const res = await sessions['Rehearsal-Di'].request('POST', `${launches['Rehearsal-Di'].launch_path}api/scenarios/pod-basics-mcq/answer`, { selected: 'a' });
|
||||
assert.equal(res.status, 200);
|
||||
assert.equal(res.body.correct, false, 'expected a deliberately wrong answer to be marked incorrect');
|
||||
});
|
||||
|
||||
await step('instructor: cohort learning summary reflects real activity', async () => {
|
||||
const { body } = await instructor.request('GET', '/api/admin/cohorts/rehearsal-cohort/learning');
|
||||
assert.equal(body.student_count, 4);
|
||||
});
|
||||
|
||||
await step('instructor: needs-attention returns real signals for the struggling students, no punitive language', async () => {
|
||||
const { body } = await instructor.request('GET', '/api/admin/cohorts/rehearsal-cohort/needs-attention');
|
||||
const text = JSON.stringify(body).toLowerCase();
|
||||
assert.ok(!text.includes('fail') || text.includes('failed validations'), 'the word "fail" should only appear in a factual "N failed validations" context, never as a standalone judgment');
|
||||
});
|
||||
|
||||
await step('instructor: reset Ada\'s Lab — academic progress is untouched', async () => {
|
||||
const before = await sessions['Rehearsal-Ada'].request('GET', '/api/progress');
|
||||
// Admin environment routes need the real environment_id, which publicEnvView doesn't echo
|
||||
// back to the student directly — resolved via the student environment lookup instead.
|
||||
const envRes = await instructor.request('GET', `/api/admin/students/${students['Rehearsal-Ada']}/environment`);
|
||||
const env = envRes.body.find((e) => e.status !== 'destroyed');
|
||||
await instructor.request('POST', `/api/admin/environments/${env.environment_id}/reset`);
|
||||
const after = await sessions['Rehearsal-Ada'].request('GET', '/api/progress');
|
||||
assert.deepEqual(before.body.details, after.body.details, 'progress must be identical before/after a Lab reset');
|
||||
});
|
||||
|
||||
await step('§27 — a Lab is reaped: same consequence as the real idle reaper (real multi-minute timer not waited out here — see reaper.js/resource-policy.yaml for the real thresholds; this exercises the destroy-and-preserve-progress consequence directly, already proven mechanistically by infra/scripts/drills/test-lab-host-loss.js)', async () => {
|
||||
const before = await sessions['Rehearsal-Bo'].request('GET', '/api/progress');
|
||||
const envRes = await instructor.request('GET', `/api/admin/students/${students['Rehearsal-Bo']}/environment`);
|
||||
const env = envRes.body.find((e) => e.status !== 'destroyed');
|
||||
await instructor.request('DELETE', `/api/admin/environments/${env.environment_id}`);
|
||||
const after = await sessions['Rehearsal-Bo'].request('GET', '/api/progress');
|
||||
assert.deepEqual(before.body.details, after.body.details, 'progress must survive a reaped Lab unchanged');
|
||||
});
|
||||
|
||||
await step('student: Bo returns later — a fresh Lab is created and curriculum continues', async () => {
|
||||
const { status, body } = await sessions['Rehearsal-Bo'].request('POST', '/api/environment/launch');
|
||||
assert.equal(status, 200); assert.equal(body.status, 'running');
|
||||
const curriculum = await sessions['Rehearsal-Bo'].request('GET', '/api/curriculum/path');
|
||||
assert.ok(curriculum.body.stages.length > 0, 'curriculum must still be there for the returning student');
|
||||
});
|
||||
|
||||
await step('instructor: assign an exam to the cohort and Ada takes it', async () => {
|
||||
const create = await instructor.request('POST', '/api/admin/assessments', {
|
||||
id: 'rehearsal-exam', type: 'exam', title: 'Rehearsal Exam', duration_minutes: 30, hint_policy: 'disabled',
|
||||
pool_spec: { required: ['pod-basics-mcq'], random: [] },
|
||||
});
|
||||
assert.ok([200, 201, 409].includes(create.status)); // 409 if a prior run already created it — fine, idempotent enough for a rehearsal
|
||||
const assign = await instructor.request('POST', '/api/admin/assessments/rehearsal-exam/assign', { cohort_id: 'rehearsal-cohort' });
|
||||
assert.ok([200, 201].includes(assign.status));
|
||||
const start = await sessions['Rehearsal-Ada'].request('POST', '/api/assessments/rehearsal-exam/start');
|
||||
assert.equal(start.status, 201);
|
||||
assert.ok(start.body.scenarios.length > 0);
|
||||
});
|
||||
|
||||
await step('instructor: destroy all cohort Labs after class — progress remains intact for everyone', async () => {
|
||||
const before = {};
|
||||
for (const name of names) before[name] = (await sessions[name].request('GET', '/api/progress')).body.details;
|
||||
await instructor.request('POST', '/api/admin/cohorts/rehearsal-cohort/destroy-labs');
|
||||
for (const name of names) {
|
||||
const after = (await sessions[name].request('GET', '/api/progress')).body.details;
|
||||
assert.deepEqual(before[name], after, `${name}'s progress must survive end-of-class Lab destruction`);
|
||||
}
|
||||
});
|
||||
|
||||
console.log(`\n${passed} passed, ${failed} failed`);
|
||||
if (failed > 0) process.exit(1);
|
||||
}
|
||||
|
||||
main().catch((e) => { console.error(e); process.exit(1); });
|
||||
@@ -0,0 +1,104 @@
|
||||
#!/usr/bin/env node
|
||||
// Phase 4.1 §21 — Lab Host loss/rebuild drill. Proves disposable infrastructure is genuinely
|
||||
// disposable: a student's Lab container is destroyed OUT FROM UNDER the Provisioner (no clean
|
||||
// destroy call — simulating the whole rn-lab-host-01 VM disappearing), and the system must:
|
||||
// - keep the Platform (API/Gateway) up throughout — it never depended on that one container
|
||||
// - never present the now-gone environment as healthy/running once reconciliation runs
|
||||
// - leave academic data (progress/attempts) completely untouched
|
||||
// - let the student launch a brand-new Lab afterward and continue where they left off
|
||||
//
|
||||
// Run via run-lab-host-loss-drill.sh, which brings up platform/docker-compose.dev.yml first.
|
||||
// No framework, matching project convention.
|
||||
|
||||
const assert = require('assert');
|
||||
const { execFileSync } = require('child_process');
|
||||
const { makeSession } = require('../../../platform/test/session-helper');
|
||||
|
||||
const GW = process.env.GATEWAY_URL || 'http://localhost:4200';
|
||||
const API = process.env.PLATFORM_API_URL || 'http://localhost:4300';
|
||||
const OWNER_EMAIL = process.env.OWNER_EMAIL;
|
||||
const OWNER_PASSWORD = process.env.OWNER_PASSWORD;
|
||||
if (!OWNER_EMAIL || !OWNER_PASSWORD) { console.error('Set OWNER_EMAIL and OWNER_PASSWORD'); process.exit(1); }
|
||||
|
||||
let passed = 0, failed = 0;
|
||||
async function test(name, fn) {
|
||||
try { await fn(); console.log(` ok — ${name}`); passed++; }
|
||||
catch (e) { console.error(` FAIL — ${name}\n ${e.stack || e.message}`); failed++; }
|
||||
}
|
||||
|
||||
async function main() {
|
||||
console.log('Phase 4.1 Lab Host loss/rebuild drill — slow, real k3s boots\n');
|
||||
|
||||
const admin = makeSession(GW);
|
||||
const { status: loginStatus } = await admin.request('POST', '/api/auth/instructor/login', { email: OWNER_EMAIL, password: OWNER_PASSWORD });
|
||||
assert.equal(loginStatus, 200, 'owner login must succeed');
|
||||
|
||||
let studentId;
|
||||
await test('roster: add a student', async () => {
|
||||
const { body } = await admin.request('POST', '/api/admin/students', { display_name: 'Drill-LabHostLoss', cohort_id: null });
|
||||
studentId = body.student_id;
|
||||
});
|
||||
|
||||
const session = makeSession(GW);
|
||||
await session.request('POST', '/api/login', { student_id: studentId });
|
||||
|
||||
let launch;
|
||||
await test('launch: student gets a running Lab (real k3s boot)', async () => {
|
||||
const { status, body } = await session.request('POST', '/api/environment/launch');
|
||||
assert.equal(status, 200); assert.equal(body.status, 'running');
|
||||
launch = body;
|
||||
});
|
||||
|
||||
await test('progress: student completes a real scenario before the Lab Host is lost', async () => {
|
||||
const res = await session.request('POST', `${launch.launch_path}api/scenarios/pod-basics-mcq/answer`, { selected: 'b' });
|
||||
assert.equal(res.status, 200);
|
||||
const progress = await session.request('GET', '/api/progress');
|
||||
assert.equal(progress.body.details['pod-basics-mcq']?.status, 'completed', 'progress must show completed before we destroy anything');
|
||||
});
|
||||
|
||||
const token = launch.launch_path.split('/')[2];
|
||||
|
||||
await test('§21 — simulate total Lab Host loss: kill the Lab container directly, bypassing the Provisioner entirely', async () => {
|
||||
const containers = execFileSync('docker', ['ps', '--filter', 'name=rootnode-lab-', '--format', '{{.Names}}']).toString().trim().split('\n').filter(Boolean);
|
||||
assert.ok(containers.length >= 1, 'expected at least one running Lab container to destroy');
|
||||
for (const c of containers) execFileSync('docker', ['rm', '-f', c]);
|
||||
console.log(` destroyed ${containers.length} Lab container(s) out from under the Provisioner`);
|
||||
});
|
||||
|
||||
await test('§21 — Platform stays available throughout (it never depended on that container)', async () => {
|
||||
const api = await fetch(`${API}/healthz`);
|
||||
const gw = await fetch(`${GW}/healthz`);
|
||||
assert.equal(api.status, 200);
|
||||
assert.equal(gw.status, 200);
|
||||
});
|
||||
|
||||
await test('§21 — academic progress is completely untouched by the infrastructure loss', async () => {
|
||||
const progress = await session.request('GET', '/api/progress');
|
||||
assert.equal(progress.body.details['pod-basics-mcq']?.status, 'completed', 'progress must survive the Lab Host loss unchanged');
|
||||
});
|
||||
|
||||
await test('§21 — the now-gone environment is NEVER presented as healthy once reconciliation runs (Provisioner restart, its real startup path)', async () => {
|
||||
// cwd is already platform/ — run-lab-host-loss-drill.sh cd's there before invoking this script.
|
||||
execFileSync('docker', ['compose', '-f', 'docker-compose.dev.yml', 'restart', 'provisioner'], { stdio: 'inherit' });
|
||||
// Give the Provisioner a moment to finish its real reconcileOnStartup pass.
|
||||
await new Promise((r) => setTimeout(r, 5000));
|
||||
const res = await session.rawFetch(`/lab/${token}/`);
|
||||
assert.ok([503, 404].includes(res.status), `expected the stale environment to be rejected (503/404), got ${res.status} — a healthy-looking response here would mean a destroyed environment is being silently presented as available`);
|
||||
});
|
||||
|
||||
await test('§21 — a replacement Lab can be launched and boots successfully (the "replacement Lab Host" case, minus needing a literal second VM)', async () => {
|
||||
const { status, body } = await session.request('POST', '/api/environment/launch');
|
||||
assert.equal(status, 200); assert.equal(body.status, 'running');
|
||||
assert.notEqual(body.launch_path, launch.launch_path, 'the replacement environment must be a genuinely new one, not the destroyed one still being served');
|
||||
});
|
||||
|
||||
await test('§21 — previous academic progress is still there after the replacement Lab boots', async () => {
|
||||
const progress = await session.request('GET', '/api/progress');
|
||||
assert.equal(progress.body.details['pod-basics-mcq']?.status, 'completed');
|
||||
});
|
||||
|
||||
console.log(`\n${passed} passed, ${failed} failed`);
|
||||
if (failed > 0) process.exit(1);
|
||||
}
|
||||
|
||||
main().catch((e) => { console.error(e); process.exit(1); });
|
||||
@@ -6,18 +6,26 @@ set -euo pipefail
|
||||
|
||||
PROD_URL="${PRODUCTION_URL:?set PRODUCTION_URL, e.g. https://academy.example.com}"
|
||||
|
||||
# Phase 4.1 §23: every `curl ... && echo "OK"` below used to be one statement — under `set -e`,
|
||||
# a failing command that isn't the LAST one in an && list does not trigger the shell to exit (a
|
||||
# well-known bash gotcha), so an unreachable/broken production endpoint used to fall through
|
||||
# silently instead of failing this script. Split into two statements each, so the real exit
|
||||
# status is what `set -e` sees. Found by infra/scripts/drills/test-ci-fails-closed.sh.
|
||||
echo "==> Gateway health"
|
||||
curl -sf "$PROD_URL/healthz" > /dev/null && echo "OK"
|
||||
curl -sf "$PROD_URL/healthz" > /dev/null
|
||||
echo "OK"
|
||||
|
||||
echo "==> TLS certificate is valid and not expiring soon"
|
||||
EXPIRY=$(echo | openssl s_client -connect "$(echo "$PROD_URL" | sed -e 's#https://##')":443 -servername "$(echo "$PROD_URL" | sed -e 's#https://##')" 2>/dev/null | openssl x509 -noout -enddate | cut -d= -f2)
|
||||
echo "Certificate expires: $EXPIRY"
|
||||
|
||||
echo "==> Roster endpoint responds (public, unauthenticated, read-only)"
|
||||
curl -sf "$PROD_URL/api/roster/cohorts" > /dev/null && echo "OK"
|
||||
curl -sf "$PROD_URL/api/roster/cohorts" > /dev/null
|
||||
echo "OK"
|
||||
|
||||
echo "==> Admin login page loads (does not attempt to log in)"
|
||||
curl -sf "$PROD_URL/admin" -o /dev/null -w '%{http_code}\n' | grep -q 200 && echo "OK"
|
||||
curl -sf "$PROD_URL/admin" -o /dev/null -w '%{http_code}\n' | grep -q 200
|
||||
echo "OK"
|
||||
|
||||
echo
|
||||
echo "OK: production smoke test passed. For a full functional check (owner/instructor login,"
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
# Two-Host Routing Simulation
|
||||
|
||||
Proves, with real Docker network isolation (not mocks), that the Gateway-to-Lab data path
|
||||
described in `docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1 actually works before the real
|
||||
Hetzner/Proxmox hardware exists. See that document for the full design; this is the test harness.
|
||||
|
||||
## Run it
|
||||
|
||||
```bash
|
||||
cd infra/scripts/simulate-two-host
|
||||
PROVISIONER_SHARED_SECRET=... ./run-simulation.sh # builds, runs, tears down
|
||||
PROVISIONER_SHARED_SECRET=... ./run-simulation.sh --keep-up # leaves the stack up for debugging
|
||||
```
|
||||
|
||||
Confirmed passing, 13/13, as of this phase's implementation — see the Phase 4.1 report for the
|
||||
full run transcript.
|
||||
|
||||
## What this genuinely proves (TESTED LOCALLY, with real Docker networking)
|
||||
|
||||
- The `gateway` container, attached ONLY to an isolated bridge network with no route to any
|
||||
Lab's per-environment bridge network, **cannot** open a TCP connection to a Lab container's
|
||||
Docker-internal IP — verified via a raw connect attempt run from inside the actual gateway
|
||||
container's own network namespace, not asserted from the test driver's (irrelevant) vantage
|
||||
point.
|
||||
- The Gateway **does** reach the Lab correctly through the Lab Host Router, over real mTLS,
|
||||
for both a plain HTTP request and a full WebSocket terminal session — including a genuine
|
||||
`kubectl get nodes` executed inside the Lab and its real output read back.
|
||||
- Alice/Bob isolation (own lab ALLOW, other's lab DENY, including over WebSocket), garbage
|
||||
tokens, anonymous requests, and destroyed-environment tokens are all rejected through the
|
||||
real code path — the same `checkOwnership`/`resolveTarget` logic production will run.
|
||||
- The Lab Host Router refuses a connection with no client certificate at all — mTLS is the
|
||||
actual authentication mechanism, not a documented intention.
|
||||
|
||||
## What this does NOT prove (SIMULATED PRODUCTION BOUNDARY, not identical to real hardware)
|
||||
|
||||
- Real cross-VM latency, MTU, or anything about two genuinely separate kernels — this is one
|
||||
Docker daemon with two isolated networks, a faithful stand-in for network *reachability*
|
||||
rules but not a physical network.
|
||||
- The actual Hetzner/Proxmox firewall enforcement (`infra/firewall/*.rules`) — this harness
|
||||
doesn't touch iptables at all; Docker's own network isolation is what's under test here.
|
||||
- Provisioner/API/Postgres reachability across the VM boundary — deliberately kept
|
||||
host-networked and unchanged from `docker-compose.dev.yml` in this harness, because that
|
||||
control-plane path was already proven separately (Phase 4's real mTLS handshake test against
|
||||
the Docker Engine TLS port) and isn't what changed this phase. Only the Gateway's isolation
|
||||
is new here.
|
||||
|
||||
## REQUIRES REAL HARDWARE (tracked in docs/PRODUCTION_VALIDATION_CHECKLIST.md, not here)
|
||||
|
||||
Running this exact flow against the real `rn-platform-01`/`rn-lab-host-01` VMs, through the
|
||||
real firewall, with real network latency between two physical/virtual hosts.
|
||||
@@ -0,0 +1,136 @@
|
||||
# Phase 4.1 two-host routing simulation — docs/PHASE4_1_PREPRODUCTION_HARDENING.md §6.
|
||||
#
|
||||
# Proves the ONE thing that matters using REAL Docker network isolation, not mocks: a Gateway
|
||||
# container with no route to a Lab container's bridge network genuinely cannot reach it, and
|
||||
# can only reach it through the Lab Host Router.
|
||||
#
|
||||
# What's genuinely simulated (real Docker networking facts, not faked):
|
||||
# - `gateway` is attached ONLY to `sim-platform-net`. It has no network_mode: host, no
|
||||
# attachment to any per-Lab bridge network (rootnode-lab-net-<id>, created dynamically by
|
||||
# the Provisioner) — its network namespace simply has no route to one. This is the same
|
||||
# kind of isolation two separate Proxmox VMs would have, achieved here with two Docker
|
||||
# networks on one daemon instead of two physical hosts.
|
||||
# - `lab-router` runs with network_mode: host — exactly like `egress-proxy` already does in
|
||||
# docker-compose.dev.yml (see that file's comment on why: a host-networked process can
|
||||
# reach every bridge network's containers via the host's own routing table, which is
|
||||
# exactly what a real Lab Host Router process running directly on rn-lab-host-01 needs).
|
||||
# It's also reachable FROM `sim-platform-net` via that network's own gateway IP — the same
|
||||
# mechanism the egress-proxy already relies on for the reverse direction.
|
||||
# - `provisioner`/`api`/`postgres` stay host-networked, unchanged from docker-compose.dev.yml
|
||||
# — this phase's new proof target is the Gateway-to-Lab DATA plane, not the CONTROL plane
|
||||
# (already proven with a real mTLS handshake in Phase 4). Keeping them as-is is a
|
||||
# deliberate scope choice, not an oversight — see the README in this directory.
|
||||
#
|
||||
# What this does NOT prove (needs the real hardware — docs/PRODUCTION_VALIDATION_CHECKLIST.md):
|
||||
# real cross-VM network latency/MTU, the actual Hetzner/Proxmox firewall enforcement, or
|
||||
# anything about two genuinely separate kernels.
|
||||
#
|
||||
# Usage: see README.md in this directory.
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
environment:
|
||||
POSTGRES_USER: postgres
|
||||
POSTGRES_PASSWORD: postgres
|
||||
POSTGRES_DB: rootnode_platform_sim
|
||||
ports:
|
||||
- "5434:5432"
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U postgres"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
|
||||
egress-proxy:
|
||||
build: ../../../platform/egress-proxy
|
||||
network_mode: host
|
||||
|
||||
provisioner:
|
||||
build: ../../../platform/provisioner
|
||||
network_mode: host
|
||||
cap_add:
|
||||
- NET_ADMIN
|
||||
- NET_RAW
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
# run-simulation.sh generates this with the real routing_mode: router / router_endpoint
|
||||
# once it knows sim-platform-net's actual gateway IP — overrides just this one file, the
|
||||
# image's own resource-policy.yaml underneath is untouched.
|
||||
- ${SIM_LAB_HOSTS_SEED:?set by run-simulation.sh}:/app/config/lab-hosts.seed.yaml:ro
|
||||
environment:
|
||||
PLATFORM_DB_URL: postgres://postgres:postgres@localhost:5434/rootnode_platform_sim
|
||||
PROVISIONER_SHARED_SECRET: ${PROVISIONER_SHARED_SECRET:?set a real secret, do not use a default}
|
||||
PROVISIONER_PORT: "4101"
|
||||
LAB_IMAGE_REF: ${LAB_IMAGE_REF:-rootnode-academy:test}
|
||||
EGRESS_PROXY_PORT: "3128"
|
||||
DNS_RESOLVER: "127.0.0.11"
|
||||
NO_PROXY_CIDRS: "10.42.0.0/16,10.43.0.0/16"
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
|
||||
api:
|
||||
build:
|
||||
context: ../../..
|
||||
dockerfile: platform/api/Dockerfile
|
||||
network_mode: host
|
||||
environment:
|
||||
PLATFORM_DB_URL: postgres://postgres:postgres@localhost:5434/rootnode_platform_sim
|
||||
PROVISIONER_URL: "http://localhost:4101"
|
||||
PROVISIONER_SHARED_SECRET: ${PROVISIONER_SHARED_SECRET:?set a real secret, do not use a default}
|
||||
PLATFORM_API_PORT: "4301"
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
provisioner:
|
||||
condition: service_started
|
||||
|
||||
# Simulates rn-platform-01's Gateway — deliberately NOT host-networked, deliberately not
|
||||
# attached to any Lab bridge network. This is the isolation under test.
|
||||
gateway:
|
||||
build: ../../../platform/gateway
|
||||
networks:
|
||||
- sim-platform-net
|
||||
environment:
|
||||
GATEWAY_PORT: "4200"
|
||||
# Reaches host-networked Provisioner/API via sim-platform-net's own gateway IP — the
|
||||
# same "reachable via the bridge's gateway interface" mechanism egress-proxy already
|
||||
# relies on (see docker-compose.dev.yml). run-simulation.sh fills these in.
|
||||
PROVISIONER_URL: ${SIM_PROVISIONER_URL:?set by run-simulation.sh}
|
||||
PROVISIONER_SHARED_SECRET: ${PROVISIONER_SHARED_SECRET:?set a real secret, do not use a default}
|
||||
PLATFORM_API_URL: ${SIM_PLATFORM_API_URL:?set by run-simulation.sh}
|
||||
LAB_ROUTER_TLS_CA: /tls/ca.pem
|
||||
LAB_ROUTER_TLS_CERT: /tls/gateway-client-cert.pem
|
||||
LAB_ROUTER_TLS_KEY: /tls/gateway-client-key.pem
|
||||
volumes:
|
||||
- ${SIM_TLS_DIR:?set by run-simulation.sh}:/tls:ro
|
||||
ports:
|
||||
- "4210:4200" # exposed to the test driver script only, not "public" in any real sense
|
||||
depends_on:
|
||||
- provisioner
|
||||
- api
|
||||
|
||||
# Simulates rn-lab-host-01's Lab Host Router — host-networked like egress-proxy, for the same
|
||||
# reason (needs to reach whatever per-Lab bridge networks the Provisioner creates, which are
|
||||
# not known/attachable ahead of time).
|
||||
lab-router:
|
||||
build: ../../../platform/lab-router
|
||||
network_mode: host
|
||||
environment:
|
||||
LAB_ROUTER_PORT: "8443"
|
||||
LAB_ROUTER_TLS_CA: /tls/ca.pem
|
||||
LAB_ROUTER_TLS_CERT: /tls/router-server-cert.pem
|
||||
LAB_ROUTER_TLS_KEY: /tls/router-server-key.pem
|
||||
DOCKER_SOCKET: /var/run/docker.sock
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
- ${SIM_TLS_DIR:?set by run-simulation.sh}:/tls:ro
|
||||
|
||||
networks:
|
||||
# Pre-created by run-simulation.sh (so it can inspect the real gateway IP BEFORE any
|
||||
# container starts and hand it to provisioner/gateway/lab-router as config) — Compose is
|
||||
# told to use that existing network rather than create its own.
|
||||
sim-platform-net:
|
||||
external: true
|
||||
name: ${COMPOSE_NET_NAME:?set by run-simulation.sh}
|
||||
+76
@@ -0,0 +1,76 @@
|
||||
#!/usr/bin/env bash
|
||||
# Orchestrates the Phase 4.1 two-host routing simulation end-to-end: generates a real TLS
|
||||
# bundle, computes the sim-platform-net gateway IP, renders the lab_hosts seed with it, brings
|
||||
# the stack up, runs test-remote-routing.js, tears everything down. See README.md in this
|
||||
# directory and docs/PHASE4_1_PREPRODUCTION_HARDENING.md §6.
|
||||
#
|
||||
# Usage: ./run-simulation.sh [--keep-up] (--keep-up skips teardown, for debugging)
|
||||
set -euo pipefail
|
||||
cd "$(dirname "${BASH_SOURCE[0]}")"
|
||||
|
||||
KEEP_UP=false
|
||||
[ "${1:-}" = "--keep-up" ] && KEEP_UP=true
|
||||
|
||||
WORK_DIR="$(mktemp -d)"
|
||||
trap 'rm -rf "$WORK_DIR"' EXIT
|
||||
|
||||
export PROVISIONER_SHARED_SECRET="${PROVISIONER_SHARED_SECRET:-sim-shared-secret-not-for-real-use}"
|
||||
export LAB_IMAGE_REF="${LAB_IMAGE_REF:-rootnode-academy:test}"
|
||||
export SIM_TLS_DIR="$WORK_DIR/tls"
|
||||
export SIM_LAB_HOSTS_SEED="$WORK_DIR/lab-hosts.seed.yaml"
|
||||
|
||||
echo "==> [1/6] Creating sim-platform-net so we know its real gateway IP before generating certs"
|
||||
NET_NAME="rootnode-sim-platform-net"
|
||||
docker network inspect "$NET_NAME" >/dev/null 2>&1 || docker network create "$NET_NAME" >/dev/null
|
||||
GW_IP="$(docker network inspect "$NET_NAME" --format '{{(index .IPAM.Config 0).Gateway}}')"
|
||||
echo " sim-platform-net gateway IP: $GW_IP (lab-router and provisioner/api, all host-networked, are reachable from Gateway's isolated network through this address)"
|
||||
|
||||
echo "==> [2/6] Generating a real mTLS bundle for this simulation run (SAN covers the real gateway IP above)"
|
||||
../docker-tls/generate-docker-tls.sh --out-dir "$SIM_TLS_DIR" --lab-host-ip "$GW_IP" --lab-host-dns lab-router-sim >/dev/null
|
||||
|
||||
# Rewrite the compose file's network name to match what we just created/inspected, and point
|
||||
# the sim gateway/router services at the shared network by that exact name.
|
||||
export COMPOSE_NET_NAME="$NET_NAME"
|
||||
|
||||
echo "==> [3/6] Rendering lab-hosts.seed.yaml with routing_mode: router at the real gateway IP"
|
||||
cat > "$SIM_LAB_HOSTS_SEED" <<EOF
|
||||
lab_hosts:
|
||||
- lab_host_id: sim-lab-host-01
|
||||
display_name: "Simulated rn-lab-host-01"
|
||||
endpoint: "unix:///var/run/docker.sock"
|
||||
enabled: true
|
||||
routing_mode: router
|
||||
router_endpoint: "https://${GW_IP}:8443"
|
||||
EOF
|
||||
|
||||
export SIM_PROVISIONER_URL="http://${GW_IP}:4101"
|
||||
export SIM_PLATFORM_API_URL="http://${GW_IP}:4301"
|
||||
|
||||
echo "==> [4/6] Building and starting the simulated two-host stack"
|
||||
docker compose -f docker-compose.simulate.yml -p rootnode-sim up -d --build
|
||||
|
||||
cleanup() {
|
||||
if [ "$KEEP_UP" = true ]; then
|
||||
echo "==> --keep-up set, leaving the stack running (docker compose -f docker-compose.simulate.yml -p rootnode-sim down to clean up)"
|
||||
return
|
||||
fi
|
||||
echo "==> [6/6] Tearing down the simulation stack"
|
||||
docker compose -f docker-compose.simulate.yml -p rootnode-sim down -v >/dev/null 2>&1 || true
|
||||
docker network rm "$NET_NAME" >/dev/null 2>&1 || true
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
echo "==> Waiting for services to become healthy"
|
||||
for i in $(seq 1 60); do
|
||||
if curl -sf http://localhost:4301/healthz >/dev/null 2>&1 && curl -sf http://localhost:4210/healthz >/dev/null 2>&1; then
|
||||
break
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
|
||||
echo "==> [5/6] Running the routing test"
|
||||
GATEWAY_URL=http://localhost:4210 \
|
||||
PROVISIONER_URL=http://localhost:4101 \
|
||||
PROVISIONER_SHARED_SECRET="$PROVISIONER_SHARED_SECRET" \
|
||||
GATEWAY_CONTAINER=rootnode-sim-gateway-1 \
|
||||
node test-remote-routing.js
|
||||
@@ -0,0 +1,212 @@
|
||||
#!/usr/bin/env node
|
||||
// Phase 4.1 §6-9 — the real, unmocked proof that remote Lab routing works. Run via
|
||||
// run-simulation.sh, which brings up docker-compose.simulate.yml first. No framework, per
|
||||
// project convention (see platform/provisioner/test/run.js).
|
||||
//
|
||||
// Proves, against the real simulated stack:
|
||||
// §6/7 Gateway cannot reach a Lab's Docker-internal IP directly, but reaches it correctly
|
||||
// through the Lab Host Router — both HTTP and WebSocket terminal traffic, including a
|
||||
// real `kubectl get nodes` inside the Lab.
|
||||
// §8/9 Alice/Bob isolation, garbage tokens, destroyed environments, anonymous requests, and
|
||||
// that the Router itself is unreachable without the Gateway's mTLS client identity —
|
||||
// all through the real path, not asserted against mocks.
|
||||
|
||||
const assert = require('assert');
|
||||
const https = require('https');
|
||||
const { execFileSync } = require('child_process');
|
||||
const { makeSession } = require('../../../platform/test/session-helper');
|
||||
|
||||
const GW = process.env.GATEWAY_URL || 'http://localhost:4210';
|
||||
const PROVISIONER_URL = process.env.PROVISIONER_URL || 'http://localhost:4101';
|
||||
const PROVISIONER_KEY = process.env.PROVISIONER_SHARED_SECRET;
|
||||
const GATEWAY_CONTAINER = process.env.GATEWAY_CONTAINER || 'rootnode-sim-gateway-1';
|
||||
const WS_URL = GW.replace(/^http/, 'ws');
|
||||
|
||||
let passed = 0, failed = 0;
|
||||
async function test(name, fn) {
|
||||
try { await fn(); console.log(` ok — ${name}`); passed++; }
|
||||
catch (e) { console.error(` FAIL — ${name}\n ${e.stack || e.message}`); failed++; }
|
||||
}
|
||||
|
||||
async function provisionerCall(method, path) {
|
||||
const res = await fetch(`${PROVISIONER_URL}${path}`, { method, headers: { 'X-Provisioner-Key': PROVISIONER_KEY } });
|
||||
const body = await res.json().catch(() => null);
|
||||
return { status: res.status, body };
|
||||
}
|
||||
|
||||
// The "browser never learns internal_ip" invariant holds everywhere else in the app — this
|
||||
// test is the one deliberate, documented exception: it needs the real Docker-internal address
|
||||
// as ground truth to prove the Gateway genuinely CANNOT reach it, which requires calling the
|
||||
// Provisioner's internal API directly (the same privilege level an operator/CI job has, never
|
||||
// a student's).
|
||||
async function internalAddressForToken(token) {
|
||||
const { body } = await provisionerCall('GET', `/environments/by-token/${token}`);
|
||||
return body;
|
||||
}
|
||||
|
||||
// bootstrap-owner.js isn't copied into the api image (it's an operator-run one-off script, not
|
||||
// a runtime dependency — same reason platform/test regression runs hit this before, see
|
||||
// docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md's regression notes). Run it from the host against
|
||||
// the sim Postgres's published port instead of `docker exec`ing into the container.
|
||||
function bootstrapOwner(email, name, password) {
|
||||
execFileSync('node', ['bootstrap-owner.js', email, name, password], {
|
||||
cwd: require('path').join(__dirname, '../../../platform/api'),
|
||||
env: { ...process.env, PLATFORM_DB_URL: 'postgres://postgres:postgres@localhost:5434/rootnode_platform_sim' },
|
||||
stdio: 'inherit',
|
||||
});
|
||||
}
|
||||
|
||||
// Raw TCP connect attempt from WITHIN the gateway container's own network namespace — proves
|
||||
// isolation from the gateway's actual point of view, not the test driver's (which runs on the
|
||||
// host and has a completely different, irrelevant reachability picture).
|
||||
function gatewayCanReach(ip, port, timeoutMs = 2500) {
|
||||
const script = `
|
||||
const net = require('net');
|
||||
const s = net.createConnection({ host: '${ip}', port: ${port}, timeout: ${timeoutMs} });
|
||||
s.on('connect', () => { console.log('REACHABLE'); s.destroy(); process.exit(0); });
|
||||
s.on('timeout', () => { console.log('UNREACHABLE(timeout)'); process.exit(0); });
|
||||
s.on('error', (e) => { console.log('UNREACHABLE(' + e.code + ')'); process.exit(0); });
|
||||
`;
|
||||
const out = execFileSync('docker', ['exec', GATEWAY_CONTAINER, 'node', '-e', script]).toString().trim();
|
||||
return out.startsWith('REACHABLE');
|
||||
}
|
||||
|
||||
function connectRouterWithoutClientCert(host, port, timeoutMs = 2500) {
|
||||
return new Promise((resolve) => {
|
||||
const req = https.request({ host, port, method: 'GET', rejectUnauthorized: false, timeout: timeoutMs }, (res) => {
|
||||
resolve({ ok: true, status: res.statusCode });
|
||||
});
|
||||
req.on('error', (e) => resolve({ ok: false, error: e.code || e.message }));
|
||||
req.on('timeout', () => { req.destroy(); resolve({ ok: false, error: 'timeout' }); });
|
||||
req.end();
|
||||
});
|
||||
}
|
||||
|
||||
function openTerminal(launchPath, cookieHeader) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const ws = new WebSocket(`${WS_URL}${launchPath}shell-ws`, { headers: { Cookie: cookieHeader } });
|
||||
let buffer = '';
|
||||
const timer = setTimeout(() => { ws.close(); reject(new Error(`terminal timed out, buffer so far: ${buffer.slice(-400)}`)); }, 30000);
|
||||
ws.addEventListener('open', () => ws.send('kubectl get nodes\n'));
|
||||
ws.addEventListener('message', (ev) => {
|
||||
buffer += ev.data.toString();
|
||||
if (/Ready\s/.test(buffer) || /node\.?\d*\s+Ready/i.test(buffer)) {
|
||||
clearTimeout(timer);
|
||||
ws.close();
|
||||
resolve(buffer);
|
||||
}
|
||||
});
|
||||
ws.addEventListener('error', (e) => { clearTimeout(timer); reject(new Error(`ws error: ${e.message || e}`)); });
|
||||
});
|
||||
}
|
||||
|
||||
function expectWsRejected(launchPath, cookieHeader, timeoutMs = 5000) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const ws = new WebSocket(`${WS_URL}${launchPath}shell-ws`, { headers: { Cookie: cookieHeader } });
|
||||
const timer = setTimeout(() => { ws.close(); reject(new Error('expected the WS to be rejected, but it stayed open')); }, timeoutMs);
|
||||
ws.addEventListener('open', () => { clearTimeout(timer); ws.close(); reject(new Error('expected the WS upgrade to be rejected, but it succeeded')); });
|
||||
ws.addEventListener('close', () => { clearTimeout(timer); resolve(); });
|
||||
ws.addEventListener('error', () => { clearTimeout(timer); resolve(); });
|
||||
});
|
||||
}
|
||||
|
||||
async function main() {
|
||||
console.log('Phase 4.1 two-host remote-routing simulation — slow, real k3s boots\n');
|
||||
|
||||
bootstrapOwner('[email protected]', 'Sim Owner', 'SimOwnerPass!2026');
|
||||
const admin = makeSession(GW);
|
||||
const { status: loginStatus } = await admin.request('POST', '/api/auth/instructor/login', { email: '[email protected]', password: 'SimOwnerPass!2026' });
|
||||
assert.equal(loginStatus, 200, 'owner login must succeed through the Gateway');
|
||||
|
||||
let alice, bob;
|
||||
await test('roster: add Alice and Bob via admin (through the Gateway)', async () => {
|
||||
const a = await admin.request('POST', '/api/admin/students', { display_name: 'Sim-Alice', cohort_id: null });
|
||||
const b = await admin.request('POST', '/api/admin/students', { display_name: 'Sim-Bob', cohort_id: null });
|
||||
alice = a.body.student_id; bob = b.body.student_id;
|
||||
assert.notEqual(alice, bob);
|
||||
});
|
||||
|
||||
const aliceSession = makeSession(GW);
|
||||
const bobSession = makeSession(GW);
|
||||
await aliceSession.request('POST', '/api/login', { student_id: alice });
|
||||
await bobSession.request('POST', '/api/login', { student_id: bob });
|
||||
|
||||
let aliceLaunch, bobLaunch, aliceEnv;
|
||||
await test('launch: Alice and Bob each get a running environment on the simulated remote lab host', async () => {
|
||||
const a = await aliceSession.request('POST', '/api/environment/launch');
|
||||
const b = await bobSession.request('POST', '/api/environment/launch');
|
||||
assert.equal(a.status, 200); assert.equal(a.body.status, 'running');
|
||||
assert.equal(b.status, 200); assert.equal(b.body.status, 'running');
|
||||
aliceLaunch = a.body; bobLaunch = b.body;
|
||||
const aliceToken = aliceLaunch.launch_path.split('/')[2];
|
||||
aliceEnv = await internalAddressForToken(aliceToken);
|
||||
assert.equal(aliceEnv.lab_host_routing_mode, 'router', 'sim-lab-host-01 must be configured as routing_mode: router for this test to mean anything');
|
||||
});
|
||||
|
||||
await test('§6/7 — the Gateway CANNOT reach the Lab\'s Docker-internal IP directly', async () => {
|
||||
const reachable = gatewayCanReach(aliceEnv.internal_ip, aliceEnv.internal_port);
|
||||
assert.equal(reachable, false, `gateway container reached ${aliceEnv.internal_ip}:${aliceEnv.internal_port} directly — the network isolation this whole phase depends on is not real`);
|
||||
});
|
||||
|
||||
await test('§6/7 — but the Gateway DOES reach the Lab through the Lab Host Router (HTTP)', async () => {
|
||||
const res = await aliceSession.rawFetch(aliceLaunch.launch_path);
|
||||
assert.equal(res.status, 200);
|
||||
const text = await res.text();
|
||||
assert.ok(text.length > 0, 'expected real HTML/asset content back from the Lab, routed through the Router');
|
||||
});
|
||||
|
||||
await test('§8 — Alice -> her own Lab: ALLOW', async () => {
|
||||
const res = await aliceSession.rawFetch(aliceLaunch.launch_path);
|
||||
assert.equal(res.status, 200);
|
||||
});
|
||||
|
||||
await test('§8 — Bob -> his own Lab: ALLOW', async () => {
|
||||
const res = await bobSession.rawFetch(bobLaunch.launch_path);
|
||||
assert.equal(res.status, 200);
|
||||
});
|
||||
|
||||
await test('§8 — Alice -> Bob\'s Lab: DENY', async () => {
|
||||
const res = await aliceSession.rawFetch(bobLaunch.launch_path);
|
||||
assert.equal(res.status, 403);
|
||||
});
|
||||
|
||||
await test('§8 — garbage token: DENY', async () => {
|
||||
const res = await aliceSession.rawFetch('/lab/not-a-real-token-at-all/');
|
||||
assert.equal(res.status, 404);
|
||||
});
|
||||
|
||||
await test('§8 — anonymous Gateway request (no session at all): DENY', async () => {
|
||||
const res = await fetch(`${GW}${aliceLaunch.launch_path}`);
|
||||
assert.equal(res.status, 401);
|
||||
});
|
||||
|
||||
await test('§8/9 — direct public access to the Lab Host Router: NOT AVAILABLE (mTLS required, no client cert presented)', async () => {
|
||||
const routerUrl = new URL(aliceEnv.lab_host_router_endpoint);
|
||||
const result = await connectRouterWithoutClientCert(routerUrl.hostname, routerUrl.port);
|
||||
assert.equal(result.ok, false, 'the Router accepted a connection with no client certificate at all — mTLS is not actually being enforced');
|
||||
});
|
||||
|
||||
await test('§7 — WebSocket terminal connects and a real `kubectl get nodes` succeeds, through the Router', async () => {
|
||||
const output = await openTerminal(bobLaunch.launch_path, `rn_session=${bobSession.getCookie('rn_session')}`);
|
||||
assert.ok(/Ready/i.test(output), `expected a Ready node in kubectl output, got: ${output.slice(-300)}`);
|
||||
});
|
||||
|
||||
await test('§8 — Alice cannot open Bob\'s terminal WebSocket even with a valid session (her own)', async () => {
|
||||
await expectWsRejected(bobLaunch.launch_path, `rn_session=${aliceSession.getCookie('rn_session')}`);
|
||||
});
|
||||
|
||||
await test('§8/9 — destroyed environment: routing token stops working, both at the Gateway and at the Router directly', async () => {
|
||||
const aliceToken = aliceLaunch.launch_path.split('/')[2];
|
||||
await provisionerCall('DELETE', `/environments/${aliceEnv.environment_id}`);
|
||||
// Give the Gateway's short-TTL env-cache a moment to expire so this proves real
|
||||
// destroyed-environment handling, not just a stale cache hit.
|
||||
await new Promise((r) => setTimeout(r, 6000));
|
||||
const res = await aliceSession.rawFetch(`/lab/${aliceToken}/`);
|
||||
assert.ok([404, 503].includes(res.status), `expected the destroyed environment to be rejected, got ${res.status}`);
|
||||
});
|
||||
|
||||
console.log(`\n${passed} passed, ${failed} failed`);
|
||||
if (failed > 0) process.exit(1);
|
||||
}
|
||||
|
||||
main().catch((e) => { console.error(e); process.exit(1); });
|
||||
@@ -9,7 +9,13 @@ set -euo pipefail
|
||||
STAGING_URL="${STAGING_GATEWAY_URL:?set STAGING_GATEWAY_URL, e.g. https://staging.academy.example.com}"
|
||||
|
||||
echo "==> Health checks"
|
||||
curl -sf "$STAGING_URL/healthz" > /dev/null && echo "OK: Gateway /healthz"
|
||||
# Phase 4.1 §23: this was `curl ... && echo ...` as one statement — under `set -e`, a failing
|
||||
# command that ISN'T the last one in an && list does not trigger the shell to exit (a
|
||||
# well-known bash gotcha), so a genuinely unreachable staging Gateway used to fall through
|
||||
# silently instead of failing this script. Split into two statements so curl's own exit status
|
||||
# is what `set -e` actually sees. Found by infra/scripts/drills/test-ci-fails-closed.sh.
|
||||
curl -sf "$STAGING_URL/healthz" > /dev/null
|
||||
echo "OK: Gateway /healthz"
|
||||
|
||||
if [ -n "${OWNER_EMAIL:-}" ] && [ -n "${OWNER_PASSWORD:-}" ]; then
|
||||
echo "==> Full acceptance run (platform/test/e2e-phase3b.js against staging)"
|
||||
|
||||
@@ -36,3 +36,17 @@ sed -e 's/{{ academy_domain }}/academy.example.com/g' -e 's/{{ management_cidr }
|
||||
chmod a+r "$TMPDIR/Caddyfile"
|
||||
docker run --rm -v "$TMPDIR:/etc/caddy:ro" caddy:2-alpine caddy validate --config /etc/caddy/Caddyfile 2>&1 | grep -q "Valid configuration" \
|
||||
&& echo "OK: Caddyfile is valid" || { echo "FAIL: Caddyfile did not validate"; exit 1; }
|
||||
|
||||
# Phase 4.1: rn-lab-host-01's compose file (egress-proxy, lab-router, observability agents)
|
||||
# was never covered here before — added when the lab-router service landed.
|
||||
echo
|
||||
echo "--- lab-host production compose ---"
|
||||
LABHOST_TMPDIR="$(mktemp -d)"
|
||||
sed \
|
||||
-e 's/{{ registry }}/registry.example.com/g' \
|
||||
-e 's/{{ image_tag }}/test/g' \
|
||||
-e "s/{{ lab_router_port | default('8443') }}/8443/g" \
|
||||
../lab-host/docker-compose.production.yml > "$LABHOST_TMPDIR/docker-compose.yml"
|
||||
docker compose -f "$LABHOST_TMPDIR/docker-compose.yml" config --quiet
|
||||
echo "OK: lab-host production compose file is well-formed YAML and passes docker compose's own schema validation"
|
||||
rm -rf "$LABHOST_TMPDIR"
|
||||
|
||||
+15
-4
@@ -23,6 +23,10 @@ entrypoint/Dockerfile unmodified; frontend bundle gained Phase 3B learning UX, s
|
||||
`docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md` (routing), `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md`
|
||||
(persistent sessions, CSRF), `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` §3 (hint gating).
|
||||
- `egress-proxy/` — shared allowlisting proxy every Lab's outbound traffic passes through.
|
||||
- `lab-router/` — Phase 4.1 addition. Runs on the Lab Host VM (or `dev-local`'s stand-in in
|
||||
local dev, though `direct` mode there never exercises it); resolves an opaque routing token
|
||||
to a local Lab container purely from Docker state, over mTLS, so the Gateway never needs a
|
||||
Docker-internal address. Design: `docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1-3.
|
||||
- `docker-compose.dev.yml` — the full stack (Postgres, egress proxy, Provisioner, Platform API,
|
||||
Gateway) on one machine. This is the Phase 1/2A/2B **local dev exception** — production is
|
||||
the two-VM Proxmox split described in `docs/LAB_ARCHITECTURE.md` §1.
|
||||
@@ -77,13 +81,20 @@ those need a browser-equivalent client and aren't automated here.
|
||||
See `provisioner/README.md` (Phase 1), `docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md` §9 (Phase
|
||||
2A), `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md` (Phase 3A identity limitations), and
|
||||
`docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` §20 (Phase 3B learning/assessment limitations) for the
|
||||
full, honest lists.
|
||||
full, honest lists. Phase 3B's "no automatic abandoned-attempt detection" and "no per-cohort
|
||||
needs-attention threshold configuration" limitations were both closed in Phase 4.1 (§13/§17 of
|
||||
`docs/PHASE4_1_PREPRODUCTION_HARDENING.md`); its raw-JSON pool_spec limitation was replaced with
|
||||
a structured editor (§18) with an owner-only raw-JSON fallback still available. Phase 4.1's own
|
||||
known limitation: real remote/self-paced delivery needs student email/password authentication,
|
||||
not built — see `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`.
|
||||
|
||||
## Running this in production
|
||||
|
||||
`../infra/` has the full production infrastructure package (Ansible, Terraform, firewall,
|
||||
monitoring, backup, CI/CD) for deploying the Platform API/Gateway/Provisioner/Postgres onto a
|
||||
dedicated `rn-platform-01` VM, with student Labs on a separate `rn-lab-host-01` VM. See
|
||||
dedicated `rn-platform-01` VM, with student Labs — and a new `lab-router` service
|
||||
(`../platform/lab-router/`) — on a separate `rn-lab-host-01` VM. See
|
||||
`../docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` for the overview and
|
||||
`../docs/PRODUCTION_NETWORK_SECURITY.md` for a known gap in Gateway-to-Lab reachability across
|
||||
two real VMs that must be closed before production go-live.
|
||||
`../docs/PHASE4_1_PREPRODUCTION_HARDENING.md` for how Gateway-to-Lab reachability across two
|
||||
real VMs was closed (a Lab Host Router + mTLS data plane, simulation-tested in
|
||||
`../infra/scripts/simulate-two-host/`) and for the rest of Phase 4.1's pre-production hardening.
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
-- Abandoned/interrupted attempt handling — Phase 4.1, docs/PHASE4_1_PREPRODUCTION_HARDENING.md
|
||||
-- §13. 'abandoned' was already a valid scenario_attempts.status value (migration 012's CHECK
|
||||
-- constraint) but nothing ever set it — an attempt left open when a Lab was reaped, an
|
||||
-- environment destroyed, a session ended, or a student moved on to a different scenario just
|
||||
-- stayed 'in_progress' forever, an honest but incomplete signal. This is NOT an academic
|
||||
-- failure marker — an abandoned attempt has no `result`, doesn't count against the student,
|
||||
-- and the row is never deleted (see src/attempts.js abandonAttempt for the enforcement of
|
||||
-- that). It exists so instructor views and needs-attention logic can tell "still working" from
|
||||
-- "stopped, for a reason we know."
|
||||
|
||||
ALTER TABLE scenario_attempts ADD COLUMN IF NOT EXISTS abandoned_at TIMESTAMPTZ;
|
||||
ALTER TABLE scenario_attempts ADD COLUMN IF NOT EXISTS abandoned_reason TEXT;
|
||||
|
||||
DO $$ BEGIN
|
||||
ALTER TABLE scenario_attempts ADD CONSTRAINT scenario_attempts_abandoned_reason_check
|
||||
CHECK (abandoned_reason IS NULL OR abandoned_reason IN
|
||||
('environment_destroyed', 'session_ended', 'scenario_switched'));
|
||||
EXCEPTION
|
||||
WHEN duplicate_object THEN NULL;
|
||||
END $$;
|
||||
@@ -0,0 +1,15 @@
|
||||
-- Postgres-backed login rate limiting — Phase 4.1 §15. Replaces the in-memory sliding-window
|
||||
-- Map in security.js, which reset on every Platform API restart (an attacker could always just
|
||||
-- wait for a deploy) and would silently stop being shared the moment there's more than one
|
||||
-- Platform API instance. One small table, one cheap query per login attempt (this is a login
|
||||
-- form, not a hot path) — no Redis, no new infrastructure dependency.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS login_attempts (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
rate_key TEXT NOT NULL, -- "ip:email", same composite key security.js always used
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
-- The hot query is "count attempts for this key within the last N minutes" — this index makes
|
||||
-- that a range scan on one key instead of a table scan.
|
||||
CREATE INDEX IF NOT EXISTS idx_login_attempts_key_time ON login_attempts(rate_key, created_at);
|
||||
@@ -7,9 +7,13 @@
|
||||
// can do, per the Phase 3B RBAC matrix.
|
||||
|
||||
const { computeCurriculumState } = require('./curriculum-rules');
|
||||
const { parsePagination, paginatedResponse } = require('./pagination'); // Phase 4.1 §16
|
||||
|
||||
const NEEDS_ATTENTION_FAIL_THRESHOLD = 3; // repeated validation failures on one scenario
|
||||
const NEEDS_ATTENTION_STALE_DAYS = 7; // no progress in this many days
|
||||
// Phase 4.1 §17 — configurable via env (environment-backed policy, matching the pattern
|
||||
// sessions.js already uses for TTLs), not a database-backed settings table — that's more
|
||||
// machinery than two numbers need. Sensible defaults if unset.
|
||||
const NEEDS_ATTENTION_FAIL_THRESHOLD = Number(process.env.NEEDS_ATTENTION_FAIL_THRESHOLD || 3); // repeated validation failures on one scenario
|
||||
const NEEDS_ATTENTION_STALE_DAYS = Number(process.env.NEEDS_ATTENTION_STALE_DAYS || 7); // no progress in this many days
|
||||
|
||||
module.exports = function registerAdminLearningRoutes(app, deps) {
|
||||
const { instructorSession, rbac, curriculum, getPool, loadProgress, recordAudit } = deps;
|
||||
@@ -70,9 +74,13 @@ module.exports = function registerAdminLearningRoutes(app, deps) {
|
||||
FROM scenario_attempts WHERE student_id = $1 ORDER BY started_at DESC NULLS LAST LIMIT 100`,
|
||||
[req.params.id]
|
||||
);
|
||||
// Phase 4.1 §16 — this is a per-student summary embedded in a wider dashboard view, not a
|
||||
// standalone history list (that's /api/admin/assessments/:id/sessions, which got real
|
||||
// pagination above) — a bounded LIMIT is the appropriate fix here, not a second paginator
|
||||
// inside an already-composite response.
|
||||
const { rows: assessmentSessions } = await pool.query(
|
||||
`SELECT assessment_session_id, assessment_definition_id, type, title, status, started_at, submitted_at, result
|
||||
FROM assessment_sessions WHERE student_id = $1 ORDER BY created_at DESC`,
|
||||
FROM assessment_sessions WHERE student_id = $1 ORDER BY created_at DESC LIMIT 50`,
|
||||
[req.params.id]
|
||||
);
|
||||
res.json({
|
||||
@@ -152,6 +160,21 @@ module.exports = function registerAdminLearningRoutes(app, deps) {
|
||||
res.json(signals);
|
||||
});
|
||||
|
||||
// Phase 4.1 §18 — the scenario catalog (stage -> scenario id/title, no student-specific
|
||||
// status) the structured assessment editor picks from, instead of an instructor hand-typing
|
||||
// scenario ids into raw JSON. Same source data as the per-student progress view
|
||||
// (admin-routes.js's /api/admin/students/:id/progress), just without the progress overlay.
|
||||
admin.get('/api/admin/scenario-catalog', requireAnyInstructor, async (req, res) => {
|
||||
const path = curriculum.loadCurriculumPath();
|
||||
const scenarios = curriculum.loadScenarios();
|
||||
if (!path) return res.json({ stages: [] });
|
||||
const stages = path.stages.map((stage) => ({
|
||||
id: stage.id, name: stage.name,
|
||||
scenarios: stage.scenario_ids.map((sid) => ({ id: sid, title: scenarios.find((s) => s.id === sid)?.title || sid })),
|
||||
}));
|
||||
res.json({ stages });
|
||||
});
|
||||
|
||||
// ── Assessment definitions ───────────────────────────────────────────────────────────────
|
||||
admin.get('/api/admin/assessments', requireAnyInstructor, async (req, res) => {
|
||||
const { rows } = await getPool().query(`SELECT * FROM assessment_definitions ORDER BY type, title`);
|
||||
@@ -231,21 +254,32 @@ module.exports = function registerAdminLearningRoutes(app, deps) {
|
||||
res.json({ ok: true });
|
||||
});
|
||||
|
||||
// Phase 4.1 §16 — was fully unbounded (no LIMIT at all — the one endpoint here that could
|
||||
// genuinely grow into thousands of rows over a semester of exam sessions). Now real
|
||||
// pagination, same shape as audit-events/student-attempts.
|
||||
admin.get('/api/admin/assessments/:id/sessions', requireAnyInstructor, async (req, res) => {
|
||||
const pool = getPool();
|
||||
const pg = parsePagination(req);
|
||||
let scopeFilter = '';
|
||||
const params = [req.params.id];
|
||||
if (req.instructor.role === 'instructor') {
|
||||
scopeFilter = `AND s.cohort_id IN (SELECT cohort_id FROM cohort_instructors WHERE instructor_id = $2)`;
|
||||
params.push(req.instructor.instructor_id);
|
||||
}
|
||||
const { rows } = await pool.query(
|
||||
`SELECT s.assessment_session_id, s.student_id, st.display_name, s.status, s.started_at, s.submitted_at, s.expires_at, s.result
|
||||
FROM assessment_sessions s JOIN students st ON st.student_id = s.student_id
|
||||
WHERE s.assessment_definition_id = $1 ${scopeFilter} ORDER BY s.created_at DESC`,
|
||||
params
|
||||
);
|
||||
res.json(rows);
|
||||
const pageParams = [...params, pg.pageSize, pg.offset];
|
||||
const [{ rows }, { rows: countRows }] = await Promise.all([
|
||||
pool.query(
|
||||
`SELECT s.assessment_session_id, s.student_id, st.display_name, s.status, s.started_at, s.submitted_at, s.expires_at, s.result
|
||||
FROM assessment_sessions s JOIN students st ON st.student_id = s.student_id
|
||||
WHERE s.assessment_definition_id = $1 ${scopeFilter} ORDER BY s.created_at DESC LIMIT $${params.length + 1} OFFSET $${params.length + 2}`,
|
||||
pageParams
|
||||
),
|
||||
pool.query(
|
||||
`SELECT count(*)::int AS n FROM assessment_sessions s WHERE s.assessment_definition_id = $1 ${scopeFilter}`,
|
||||
params
|
||||
),
|
||||
]);
|
||||
res.json(paginatedResponse(rows, countRows[0].n, pg));
|
||||
});
|
||||
|
||||
admin.post('/api/admin/assessment-sessions/:sessionId/cancel', requireAnyInstructor, async (req, res) => {
|
||||
|
||||
@@ -14,10 +14,15 @@
|
||||
// destroy, events) — the Admin UI never talks to the Provisioner directly.
|
||||
|
||||
const { listInstructors, createInstructor, assertNotLastOwner } = require('./instructors');
|
||||
const attemptsLib = require('./attempts'); // Phase 4.1 §13 — abandonment on instructor-driven reset/destroy
|
||||
const { parsePagination, paginatedResponse } = require('./pagination'); // Phase 4.1 §16
|
||||
|
||||
module.exports = function registerAdminRoutes(app, deps) {
|
||||
const { instructorSession, rbac, provisioner, curriculum, getPool, loadProgress, recordAudit } = deps;
|
||||
const { requireOwner, requireAdminOrOwner, requireAnyInstructor, requireCohortAccess, requireStudentCohortAccess } = rbac;
|
||||
const {
|
||||
requireOwner, requireAdminOrOwner, requireAnyInstructor, requireCohortAccess, requireStudentCohortAccess,
|
||||
requireCohortWriteAccess, requireStudentCohortWriteAccess,
|
||||
} = rbac;
|
||||
|
||||
// Tiny helper: every admin.<verb> call goes through instructorSession first, then whatever
|
||||
// role/scope middleware is passed, then the handler (last function in the list — auto-wrapped
|
||||
@@ -71,6 +76,9 @@ module.exports = function registerAdminRoutes(app, deps) {
|
||||
`SELECT student_id, lab_host_id, status, error_reason, started_at
|
||||
FROM environments WHERE status = 'error' ORDER BY started_at DESC NULLS LAST LIMIT 10`
|
||||
);
|
||||
// Phase 4.1 §26 — best-effort: a Provisioner hiccup here shouldn't break the whole
|
||||
// dashboard, capacity is just omitted (null) rather than the request failing.
|
||||
const capacity = await provisioner.capacity().then((r) => r.status === 200 ? r.body : null).catch(() => null);
|
||||
res.json({
|
||||
active_cohorts: cohorts.rows[0].n,
|
||||
total_students: students.rows[0].n,
|
||||
@@ -80,6 +88,7 @@ module.exports = function registerAdminRoutes(app, deps) {
|
||||
error: envCounts.error || 0,
|
||||
idle_or_destroyed: envCounts.destroyed || 0,
|
||||
},
|
||||
capacity, // { current, max } — max is null when admission control is unconfigured (see resource-policy.yaml)
|
||||
recent_completions: recentCompletions.rows,
|
||||
recent_failures: scoped ? [] : recentFailures.rows, // env-level failures aren't cohort-scoped data; keep them admin/owner-only
|
||||
});
|
||||
@@ -143,11 +152,15 @@ module.exports = function registerAdminRoutes(app, deps) {
|
||||
res.json({ stages });
|
||||
});
|
||||
|
||||
// Phase 4.1 §16 — was a flat LIMIT 100; now real pagination (see pagination.js).
|
||||
admin.get('/api/admin/students/:id/attempts', requireAnyInstructor, requireStudentCohortAccess('id'), async (req, res) => {
|
||||
const { rows } = await getPool().query(
|
||||
`SELECT * FROM scenario_attempts WHERE student_id = $1 ORDER BY created_at DESC LIMIT 100`, [req.params.id]
|
||||
);
|
||||
res.json(rows);
|
||||
const pool = getPool();
|
||||
const pg = parsePagination(req);
|
||||
const [{ rows }, { rows: countRows }] = await Promise.all([
|
||||
pool.query(`SELECT * FROM scenario_attempts WHERE student_id = $1 ORDER BY created_at DESC LIMIT $2 OFFSET $3`, [req.params.id, pg.pageSize, pg.offset]),
|
||||
pool.query(`SELECT count(*)::int AS n FROM scenario_attempts WHERE student_id = $1`, [req.params.id]),
|
||||
]);
|
||||
res.json(paginatedResponse(rows, countRows[0].n, pg));
|
||||
});
|
||||
|
||||
admin.get('/api/admin/students/:id/environment', requireAnyInstructor, requireStudentCohortAccess('id'), async (req, res) => {
|
||||
@@ -156,7 +169,7 @@ module.exports = function registerAdminRoutes(app, deps) {
|
||||
res.json(body);
|
||||
});
|
||||
|
||||
admin.post('/api/admin/students/:id/environment/provision', requireAnyInstructor, requireStudentCohortAccess('id'), async (req, res) => {
|
||||
admin.post('/api/admin/students/:id/environment/provision', requireAnyInstructor, requireStudentCohortWriteAccess('id'), async (req, res) => {
|
||||
const { status, body } = await provisioner.provision(req.params.id);
|
||||
res.status(status === 409 ? 200 : status).json(body);
|
||||
});
|
||||
@@ -236,6 +249,12 @@ module.exports = function registerAdminRoutes(app, deps) {
|
||||
admin.post('/api/admin/cohorts/:id/instructors', requireAdminOrOwner, async (req, res) => {
|
||||
const { instructor_id, role_in_cohort } = req.body || {};
|
||||
if (!instructor_id) return res.status(400).json({ error: 'instructor_id required' });
|
||||
// Phase 4.1 §19 — role_in_cohort is now actually enforced (rbac.js's
|
||||
// requireCohortWriteAccess/requireStudentCohortWriteAccess), so it's validated here rather
|
||||
// than accepting any string that would silently never match either check.
|
||||
if (role_in_cohort && !['teacher', 'viewer'].includes(role_in_cohort)) {
|
||||
return res.status(400).json({ error: 'role_in_cohort must be teacher or viewer' });
|
||||
}
|
||||
await getPool().query(
|
||||
`INSERT INTO cohort_instructors (cohort_id, instructor_id, role_in_cohort) VALUES ($1, $2, $3)
|
||||
ON CONFLICT (cohort_id, instructor_id) DO UPDATE SET role_in_cohort = EXCLUDED.role_in_cohort`,
|
||||
@@ -286,7 +305,7 @@ module.exports = function registerAdminRoutes(app, deps) {
|
||||
});
|
||||
|
||||
// ── Classroom prep: Prepare Labs / Destroy Cohort Labs ──────────────────────────────────
|
||||
admin.post('/api/admin/cohorts/:id/prepare-labs', requireAnyInstructor, requireCohortAccess('id'), async (req, res) => {
|
||||
admin.post('/api/admin/cohorts/:id/prepare-labs', requireAnyInstructor, requireCohortWriteAccess('id'), async (req, res) => {
|
||||
const { rows: students } = await getPool().query(
|
||||
`SELECT s.student_id, s.display_name FROM cohort_enrollments e JOIN students s ON s.student_id = e.student_id
|
||||
WHERE e.cohort_id = $1 AND e.enrollment_status = 'enrolled' AND s.status = 'active'`,
|
||||
@@ -306,7 +325,7 @@ module.exports = function registerAdminRoutes(app, deps) {
|
||||
});
|
||||
});
|
||||
|
||||
admin.post('/api/admin/cohorts/:id/destroy-labs', requireAnyInstructor, requireCohortAccess('id'), async (req, res) => {
|
||||
admin.post('/api/admin/cohorts/:id/destroy-labs', requireAnyInstructor, requireCohortWriteAccess('id'), async (req, res) => {
|
||||
const { rows: students } = await getPool().query(
|
||||
`SELECT student_id FROM cohort_enrollments WHERE cohort_id = $1 AND enrollment_status = 'enrolled'`,
|
||||
[req.params.id]
|
||||
@@ -315,6 +334,9 @@ module.exports = function registerAdminRoutes(app, deps) {
|
||||
const { body } = await provisioner.listForStudent(s.student_id);
|
||||
const active = body?.find((e) => e.status !== 'destroyed');
|
||||
if (!active) return { student_id: s.student_id, status: 'none' };
|
||||
// Phase 4.1 §13 — before the destroy, so any still-open attempt is honestly abandoned
|
||||
// rather than left dangling once its environment no longer exists.
|
||||
await attemptsLib.abandonOpenAttemptsForEnvironment(getPool(), active.environment_id);
|
||||
await provisioner.destroy(active.environment_id);
|
||||
return { student_id: s.student_id, status: 'destroyed' };
|
||||
}));
|
||||
@@ -328,12 +350,14 @@ module.exports = function registerAdminRoutes(app, deps) {
|
||||
// resolving a bare environment id back to a cohort would need an extra Provisioner round
|
||||
// trip on every call — a documented simplification, see docs/PHASE3A...) ───────────────────
|
||||
admin.post('/api/admin/environments/:id/reset', requireAdminOrOwner, async (req, res) => {
|
||||
await attemptsLib.abandonOpenAttemptsForEnvironment(getPool(), req.params.id); // Phase 4.1 §13
|
||||
const { status, body } = await provisioner.reset(req.params.id);
|
||||
await recordAudit(getPool(), { actorType: 'instructor', actorId: req.instructor.instructor_id, action: 'lab_reset', resourceType: 'environment', resourceId: req.params.id, result: status === 200 ? 'success' : 'failure', sourceIp: req.ip });
|
||||
res.status(status).json(body);
|
||||
});
|
||||
|
||||
admin.delete('/api/admin/environments/:id', requireAdminOrOwner, async (req, res) => {
|
||||
await attemptsLib.abandonOpenAttemptsForEnvironment(getPool(), req.params.id); // Phase 4.1 §13
|
||||
const { status, body } = await provisioner.destroy(req.params.id);
|
||||
await recordAudit(getPool(), { actorType: 'instructor', actorId: req.instructor.instructor_id, action: 'lab_destroy', resourceType: 'environment', resourceId: req.params.id, result: status === 200 ? 'success' : 'failure', sourceIp: req.ip });
|
||||
res.status(status).json(body);
|
||||
@@ -423,8 +447,16 @@ module.exports = function registerAdminRoutes(app, deps) {
|
||||
});
|
||||
|
||||
// ── Audit log — admin/owner ──────────────────────────────────────────────────────────────
|
||||
// Phase 4.1 §16 — was a flat LIMIT 200 with no way to see anything older; now real
|
||||
// server-side pagination (see pagination.js). Default page size (50) intentionally smaller
|
||||
// than the old flat cap since this is now genuinely browsable instead of "the last 200 ever."
|
||||
admin.get('/api/admin/audit-events', requireAdminOrOwner, async (req, res) => {
|
||||
const { rows } = await getPool().query(`SELECT * FROM audit_events ORDER BY created_at DESC LIMIT 200`);
|
||||
res.json(rows);
|
||||
const pool = getPool();
|
||||
const pg = parsePagination(req);
|
||||
const [{ rows }, { rows: countRows }] = await Promise.all([
|
||||
pool.query(`SELECT * FROM audit_events ORDER BY created_at DESC LIMIT $1 OFFSET $2`, [pg.pageSize, pg.offset]),
|
||||
pool.query(`SELECT count(*)::int AS n FROM audit_events`),
|
||||
]);
|
||||
res.json(paginatedResponse(rows, countRows[0].n, pg));
|
||||
});
|
||||
};
|
||||
|
||||
@@ -2,11 +2,55 @@
|
||||
// session on a scenario (not per validation click, see migrations/012). Every mutation here
|
||||
// updates that one row in place; nothing is ever inserted twice for the same open attempt.
|
||||
//
|
||||
// Deliberately does NOT try to detect "abandoned" attempts automatically (e.g. on Lab destroy)
|
||||
// — that would need a reaper-style background job wired to Provisioner environment_events,
|
||||
// which is more machinery than the instructional-visibility goal here needs. An attempt just
|
||||
// stays 'in_progress' if the student never returns to finish it; instructors can already see
|
||||
// "in_progress, N failed validations" as a real signal. Documented as a known limitation.
|
||||
// Phase 4.1 (docs/PHASE4_1_PREPRODUCTION_HARDENING.md §13) added real abandonment handling —
|
||||
// see abandonAttempt/abandonOpenAttemptsFor* below and their call sites (environment
|
||||
// destroy/reset in server.js and admin-routes.js, logout in server.js, the session sweeper in
|
||||
// sessions.js, and the "different scenario" transition inline in getOrCreateOpenAttempt just
|
||||
// below). Still deliberately NOT a new standalone background job — every trigger either rides
|
||||
// an existing request handler or the session sweeper that already runs periodically.
|
||||
|
||||
// An attempt is never silently deleted or marked as a failure by abandonment — result stays
|
||||
// NULL, the row stays queryable, instructors see "abandoned (reason)" as a real, honest status
|
||||
// rather than a punitive one. No-op (and does not overwrite an existing abandoned_reason) if
|
||||
// the attempt is already terminal.
|
||||
async function abandonAttempt(pool, attemptId, reason) {
|
||||
const { rows } = await pool.query(
|
||||
`UPDATE scenario_attempts SET status = 'abandoned', abandoned_at = now(), abandoned_reason = $2
|
||||
WHERE attempt_id = $1 AND status = 'in_progress' RETURNING *`,
|
||||
[attemptId, reason]
|
||||
);
|
||||
return rows[0] || null;
|
||||
}
|
||||
|
||||
// Trigger: environment destroyed (idle reaper, student reset, admin reset/destroy — see
|
||||
// server.js's student reset route and admin-routes.js's reset/destroy routes, all of which
|
||||
// call this right after telling the Provisioner to tear the environment down). A reset creates
|
||||
// a fresh environment_id in some flows and reuses the same one in others (see
|
||||
// provisioner lifecycle.js's reset — same environment_id, fresh container) — either way, any
|
||||
// attempt still open against the OLD container state is honestly "not going to be resumed as
|
||||
// the same working session," so it's abandoned rather than left open indefinitely.
|
||||
async function abandonOpenAttemptsForEnvironment(pool, environmentId) {
|
||||
const { rows } = await pool.query(
|
||||
`UPDATE scenario_attempts SET status = 'abandoned', abandoned_at = now(), abandoned_reason = 'environment_destroyed'
|
||||
WHERE environment_id = $1 AND status = 'in_progress' RETURNING attempt_id`,
|
||||
[environmentId]
|
||||
);
|
||||
return rows.length;
|
||||
}
|
||||
|
||||
// Trigger: logout (server.js POST /api/logout) and the session sweeper's periodic pass for
|
||||
// students whose sessions have all expired (sessions.js). Deliberately student-scoped, not
|
||||
// environment-scoped — a student can log out while their Lab keeps running (it isn't
|
||||
// destroyed until reaped or explicitly ended), but the WORKING SESSION on whatever scenario
|
||||
// they had open is over the moment they're no longer authenticated to record more progress on it.
|
||||
async function abandonOpenAttemptsForStudent(pool, studentId) {
|
||||
const { rows } = await pool.query(
|
||||
`UPDATE scenario_attempts SET status = 'abandoned', abandoned_at = now(), abandoned_reason = 'session_ended'
|
||||
WHERE student_id = $1 AND status = 'in_progress' RETURNING attempt_id`,
|
||||
[studentId]
|
||||
);
|
||||
return rows.length;
|
||||
}
|
||||
|
||||
async function getOrCreateOpenAttempt(pool, { studentId, scenarioId, environmentId, cohortId, curriculumStage, assessmentSessionId }) {
|
||||
const { rows: existing } = await pool.query(
|
||||
@@ -17,6 +61,17 @@ async function getOrCreateOpenAttempt(pool, { studentId, scenarioId, environment
|
||||
);
|
||||
if (existing[0]) return existing[0];
|
||||
|
||||
// Trigger: student starts a different scenario while a previous one is still open (started
|
||||
// via /start or implicitly via /validate, never finished). Only one scenario is "the one
|
||||
// currently being worked on" per student+environment at a time from the app's own UX, so an
|
||||
// open attempt on ANY OTHER scenario in this same environment is abandoned before the new
|
||||
// one is created — never silently left as a phantom "still in progress" row.
|
||||
await pool.query(
|
||||
`UPDATE scenario_attempts SET status = 'abandoned', abandoned_at = now(), abandoned_reason = 'scenario_switched'
|
||||
WHERE student_id = $1 AND environment_id = $2 AND scenario_id != $3 AND status = 'in_progress'`,
|
||||
[studentId, environmentId, scenarioId]
|
||||
);
|
||||
|
||||
const { rows } = await pool.query(
|
||||
`INSERT INTO scenario_attempts
|
||||
(student_id, scenario_id, environment_id, cohort_id, curriculum_stage, started_at, status, assessment_session_id)
|
||||
@@ -80,4 +135,7 @@ async function getAttemptSummary(pool, { studentId, scenarioId, environmentId })
|
||||
return rows[0] || null;
|
||||
}
|
||||
|
||||
module.exports = { getOrCreateOpenAttempt, recordValidation, recordHintReveal, getAttemptSummary };
|
||||
module.exports = {
|
||||
getOrCreateOpenAttempt, recordValidation, recordHintReveal, getAttemptSummary,
|
||||
abandonAttempt, abandonOpenAttemptsForEnvironment, abandonOpenAttemptsForStudent,
|
||||
};
|
||||
|
||||
@@ -18,7 +18,7 @@ module.exports = function registerInstructorAuthRoutes(app, { pool, sessions, in
|
||||
app.post('/api/auth/instructor/login', async (req, res) => {
|
||||
const { email, password } = req.body || {};
|
||||
const rateKey = `${req.ip}:${String(email || '').toLowerCase()}`;
|
||||
if (tooManyAttempts(rateKey, LOGIN_MAX_ATTEMPTS, LOGIN_WINDOW_MS)) {
|
||||
if (await tooManyAttempts(pool, rateKey, LOGIN_MAX_ATTEMPTS, LOGIN_WINDOW_MS)) {
|
||||
return res.status(429).json({ error: 'too_many_attempts', message: 'Too many login attempts. Try again in a few minutes.' });
|
||||
}
|
||||
if (!email || !password) return res.status(400).json({ error: 'email and password are required' });
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
// Stable server-side pagination — Phase 4.1 §16. Shared by every list endpoint that could
|
||||
// otherwise grow unbounded (audit log, attempt histories, assessment session history, scenario
|
||||
// analytics). Simple offset/limit, not cursor-based — this admin/instructor scale (hundreds to
|
||||
// low thousands of rows, not a high-frequency infinite-scroll feed) doesn't need more than
|
||||
// that, and offset pagination is what every other query in this codebase already reads like.
|
||||
|
||||
const DEFAULT_PAGE_SIZE = 50;
|
||||
const MAX_PAGE_SIZE = 200;
|
||||
|
||||
// Reads ?page=&page_size= from a request, clamped to sane bounds — never trusts the client for
|
||||
// an unbounded page_size (that would defeat the entire point).
|
||||
function parsePagination(req) {
|
||||
const page = Math.max(1, parseInt(req.query.page, 10) || 1);
|
||||
const pageSize = Math.min(MAX_PAGE_SIZE, Math.max(1, parseInt(req.query.page_size, 10) || DEFAULT_PAGE_SIZE));
|
||||
return { page, pageSize, offset: (page - 1) * pageSize };
|
||||
}
|
||||
|
||||
function paginatedResponse(rows, total, { page, pageSize }) {
|
||||
return { items: rows, page, page_size: pageSize, total, total_pages: Math.max(1, Math.ceil(total / pageSize)) };
|
||||
}
|
||||
|
||||
module.exports = { parsePagination, paginatedResponse, DEFAULT_PAGE_SIZE, MAX_PAGE_SIZE };
|
||||
@@ -33,6 +33,7 @@ const provisioner = {
|
||||
reset: (id) => call('POST', `/environments/${encodeURIComponent(id)}/reset`),
|
||||
destroy: (id) => call('DELETE', `/environments/${encodeURIComponent(id)}`),
|
||||
events: (id) => call('GET', `/environments/${encodeURIComponent(id)}/events`),
|
||||
capacity: () => call('GET', '/capacity'), // Phase 4.1 §26 — instructor dashboard capacity state
|
||||
};
|
||||
|
||||
module.exports = { provisioner };
|
||||
|
||||
@@ -78,5 +78,61 @@ module.exports = function createRbac(pool) {
|
||||
};
|
||||
}
|
||||
|
||||
return { requireRole, requireOwner, requireAdminOrOwner, requireAnyInstructor, requireCohortAccess, requireStudentCohortAccess };
|
||||
// Phase 4.1 §19 — cohort_instructors.role_in_cohort existed since Phase 2B but was never
|
||||
// read anywhere (a misleading field: it looked authoritative but did nothing). Minimal
|
||||
// useful model: 'teacher' (default, existing full instructor capability within the cohort)
|
||||
// vs 'viewer' (read-only — sees dashboards/progress/analytics via requireCohortAccess above,
|
||||
// but cannot Prepare/Destroy Labs or provision a student's environment). owner/admin still
|
||||
// bypass entirely, same as every other cohort-scoped check.
|
||||
function requireCohortWriteAccess(cohortIdParam = 'id') {
|
||||
return async (req, res, next) => {
|
||||
const instructor = req.instructor;
|
||||
if (['owner', 'admin'].includes(instructor.role)) return next();
|
||||
const cohortId = req.params[cohortIdParam];
|
||||
const { rows } = await pool.query(
|
||||
`SELECT role_in_cohort FROM cohort_instructors WHERE cohort_id = $1 AND instructor_id = $2`,
|
||||
[cohortId, instructor.instructor_id]
|
||||
);
|
||||
if (!rows.length || rows[0].role_in_cohort === 'viewer') {
|
||||
await recordAudit(pool, {
|
||||
actorType: 'instructor', actorId: instructor.instructor_id, action: 'authz_denied',
|
||||
resourceType: 'cohort', resourceId: cohortId, result: 'failure', sourceIp: req.ip,
|
||||
metadata: rows.length ? { reason: 'viewer_role' } : undefined,
|
||||
});
|
||||
return res.status(403).json({ error: 'forbidden' });
|
||||
}
|
||||
next();
|
||||
};
|
||||
}
|
||||
|
||||
// Same, keyed by student_id — resolves through cohort_enrollments the same way
|
||||
// requireStudentCohortAccess does, then applies the same viewer check.
|
||||
function requireStudentCohortWriteAccess(studentIdParam = 'id') {
|
||||
return async (req, res, next) => {
|
||||
const instructor = req.instructor;
|
||||
if (['owner', 'admin'].includes(instructor.role)) return next();
|
||||
const studentId = req.params[studentIdParam];
|
||||
const { rows } = await pool.query(
|
||||
`SELECT ci.role_in_cohort FROM cohort_enrollments e
|
||||
JOIN cohort_instructors ci ON ci.cohort_id = e.cohort_id
|
||||
WHERE e.student_id = $1 AND e.enrollment_status = 'enrolled' AND ci.instructor_id = $2`,
|
||||
[studentId, instructor.instructor_id]
|
||||
);
|
||||
if (!rows.length || rows.every((r) => r.role_in_cohort === 'viewer')) {
|
||||
await recordAudit(pool, {
|
||||
actorType: 'instructor', actorId: instructor.instructor_id, action: 'authz_denied',
|
||||
resourceType: 'student', resourceId: studentId, result: 'failure', sourceIp: req.ip,
|
||||
metadata: rows.length ? { reason: 'viewer_role' } : undefined,
|
||||
});
|
||||
return res.status(403).json({ error: 'forbidden' });
|
||||
}
|
||||
next();
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
requireRole, requireOwner, requireAdminOrOwner, requireAnyInstructor,
|
||||
requireCohortAccess, requireStudentCohortAccess,
|
||||
requireCohortWriteAccess, requireStudentCohortWriteAccess,
|
||||
};
|
||||
};
|
||||
|
||||
@@ -33,25 +33,34 @@ function checkCsrf(req, cookieName) {
|
||||
}
|
||||
|
||||
// ── Login rate limiting ─────────────────────────────────────────────────────────────────
|
||||
// In-memory sliding window, keyed by "ip:identifier" (e.g. ip:email). Fine for a single-process
|
||||
// Platform API — a shared store (Postgres/Redis) would be needed for multi-instance, not a
|
||||
// concern at this scale (see docs/PHASE3A... known limitations).
|
||||
const attempts = new Map();
|
||||
|
||||
function tooManyAttempts(key, max, windowMs) {
|
||||
const now = Date.now();
|
||||
const arr = (attempts.get(key) || []).filter((t) => now - t < windowMs);
|
||||
arr.push(now);
|
||||
attempts.set(key, arr);
|
||||
return arr.length > max;
|
||||
// Phase 4.1 §15: Postgres-backed (migrations/016_login_attempts.sql), not in-memory — the
|
||||
// original in-memory sliding-window Map reset on every Platform API restart (an attacker could
|
||||
// just wait for a deploy) and would silently stop being shared the moment there's more than
|
||||
// one Platform API instance. This table is the fix for both, at essentially no cost: login is
|
||||
// not a hot path, and the query below is a single indexed range scan.
|
||||
//
|
||||
// Deliberately still just a table + two queries, not a new rate-limiting service or Redis —
|
||||
// the volume here (failed login attempts) never justifies that, and a Postgres row is already
|
||||
// exactly as "shared/persistent" as this needs to be.
|
||||
async function tooManyAttempts(pool, key, max, windowMs) {
|
||||
// Insert THEN count (including the just-inserted row), matching the original in-memory
|
||||
// version's semantics exactly: this call itself counts as one of the attempts.
|
||||
await pool.query(`INSERT INTO login_attempts (rate_key) VALUES ($1)`, [key]);
|
||||
const { rows } = await pool.query(
|
||||
`SELECT count(*)::int AS n FROM login_attempts WHERE rate_key = $1 AND created_at > now() - ($2 || ' milliseconds')::interval`,
|
||||
[key, windowMs]
|
||||
);
|
||||
return rows[0].n > max;
|
||||
}
|
||||
|
||||
setInterval(() => {
|
||||
const cutoff = Date.now() - 60 * 60 * 1000;
|
||||
for (const [key, arr] of attempts) {
|
||||
const kept = arr.filter((t) => t > cutoff);
|
||||
if (kept.length) attempts.set(key, kept); else attempts.delete(key);
|
||||
}
|
||||
}, 15 * 60 * 1000).unref();
|
||||
// Periodic cleanup so the table doesn't grow forever — cosmetic/storage only, mirrors
|
||||
// sessions.js's own sweeper pattern rather than inventing a new one.
|
||||
function startLoginAttemptSweeper(pool) {
|
||||
setInterval(() => {
|
||||
pool.query(`DELETE FROM login_attempts WHERE created_at < now() - interval '24 hours'`).catch((e) => {
|
||||
console.error('[login attempt sweep] failed:', e.message);
|
||||
});
|
||||
}, 60 * 60 * 1000).unref();
|
||||
}
|
||||
|
||||
module.exports = { makeCsrfToken, parseCookies, checkCsrf, tooManyAttempts };
|
||||
module.exports = { makeCsrfToken, parseCookies, checkCsrf, tooManyAttempts, startLoginAttemptSweeper };
|
||||
|
||||
@@ -16,7 +16,7 @@ const { provisioner } = require('./provisioner-client');
|
||||
const curriculum = require('./curriculum');
|
||||
const { loadProgress, saveOneProgress, touchStudent } = require('./progress');
|
||||
const { recordAudit } = require('./audit');
|
||||
const { parseCookies, checkCsrf, makeCsrfToken } = require('./security');
|
||||
const { parseCookies, checkCsrf, makeCsrfToken, startLoginAttemptSweeper } = require('./security');
|
||||
const { buildCookie, clearCookie } = require('./cookies');
|
||||
const createRbac = require('./rbac');
|
||||
const { INSTRUCTOR_COOKIE, CSRF_COOKIE: INSTRUCTOR_CSRF_COOKIE } = require('./instructor-auth-routes');
|
||||
@@ -88,6 +88,26 @@ app.get('/healthz', async (req, res) => {
|
||||
}
|
||||
});
|
||||
|
||||
// GET /api/version — Phase 4.1 §25. Unauthenticated like /healthz, and deliberately just as
|
||||
// minimal: an app version, the deployed Git SHA (or 'dev' locally — see
|
||||
// infra/docker/platform/docker-compose.production.yml's GIT_SHA env var, set to the same
|
||||
// value used for the image tag), and the most recently applied migration as a stand-in for
|
||||
// schema version. No infrastructure detail, no secrets — this is "what build is this," not a
|
||||
// diagnostics endpoint.
|
||||
const PACKAGE_VERSION = require('../package.json').version;
|
||||
app.get('/api/version', async (req, res) => {
|
||||
let schemaVersion = null;
|
||||
try {
|
||||
const { rows } = await getPool().query('SELECT version FROM schema_migrations ORDER BY applied_at DESC LIMIT 1');
|
||||
schemaVersion = rows[0]?.version || null;
|
||||
} catch (e) { /* best-effort — an unreachable DB shouldn't make /api/version itself fail */ }
|
||||
res.json({
|
||||
version: PACKAGE_VERSION,
|
||||
git_sha: process.env.GIT_SHA || 'dev',
|
||||
schema_version: schemaVersion,
|
||||
});
|
||||
});
|
||||
|
||||
// ═══════════════════════════════════════════════════════════════════════════════════════
|
||||
// STUDENT-FACING: roster / login / session
|
||||
// ═══════════════════════════════════════════════════════════════════════════════════════
|
||||
@@ -136,7 +156,12 @@ app.post('/api/login', async (req, res) => {
|
||||
app.post('/api/logout', async (req, res) => {
|
||||
if (!checkCsrf(req, STUDENT_CSRF_COOKIE)) return res.status(403).json({ error: 'csrf' });
|
||||
const token = parseCookies(req)[STUDENT_COOKIE];
|
||||
// Resolved BEFORE destroying, purely to know who to abandon open attempts for (Phase 4.1
|
||||
// §13) — deliberately not behind studentSession middleware so logout still succeeds
|
||||
// gracefully against an already-invalid/expired token, same as before this phase.
|
||||
const studentId = await sessions.resolveStudentSession(getPool(), token);
|
||||
await sessions.destroyStudentSession(getPool(), token);
|
||||
if (studentId) await attemptsLib.abandonOpenAttemptsForStudent(getPool(), studentId);
|
||||
res.append('Set-Cookie', clearCookie(STUDENT_COOKIE));
|
||||
res.append('Set-Cookie', clearCookie(STUDENT_CSRF_COOKIE));
|
||||
res.json({ ok: true });
|
||||
@@ -268,6 +293,12 @@ app.post('/api/environment/launch', wrap(studentSession), async (req, res) => {
|
||||
const retry = await provisioner.listForStudent(req.studentId);
|
||||
return res.json(await publicEnvView(retry.body?.[0]));
|
||||
}
|
||||
// Phase 4.1 §26 — relayed as its own clean status/message, not folded into the generic
|
||||
// provision_failed 502 below. The student sees "temporarily full, try again shortly," never
|
||||
// a raw 500/502 — this is expected operational backpressure, not an application error.
|
||||
if (status === 503 && body?.code === 'CAPACITY_EXCEEDED') {
|
||||
return res.status(503).json({ error: 'capacity_exceeded', message: 'Lab capacity is temporarily full. Try again shortly.' });
|
||||
}
|
||||
if (status !== 201) return res.status(502).json({ error: 'provision_failed', message: body?.error });
|
||||
res.json(await publicEnvView({ environment_id: body.environment_id, status: body.status, routing_token: body.routing_token }));
|
||||
});
|
||||
@@ -276,6 +307,11 @@ app.post('/api/environment/reset', wrap(studentSession), async (req, res) => {
|
||||
const existing = await provisioner.listForStudent(req.studentId);
|
||||
const env = existing.body?.find((e) => e.status !== 'destroyed');
|
||||
if (!env) return res.status(404).json({ error: 'no_environment' });
|
||||
// Before tearing the old container down — Phase 4.1 §13, see attempts.js's
|
||||
// abandonOpenAttemptsForEnvironment. The reset reuses the same environment_id (fresh
|
||||
// container, same DB row — provisioner lifecycle.js), so this must run BEFORE reset, not
|
||||
// after, or the abandonment would incorrectly apply to attempts against the NEW container.
|
||||
await attemptsLib.abandonOpenAttemptsForEnvironment(getPool(), env.environment_id);
|
||||
const { status, body } = await provisioner.reset(env.environment_id);
|
||||
if (status !== 200) return res.status(502).json({ error: 'reset_failed', message: body?.error });
|
||||
res.json(await publicEnvView(body));
|
||||
@@ -322,6 +358,7 @@ async function main() {
|
||||
const applied = await runMigrations();
|
||||
if (applied.length) console.log('Applied migrations:', applied.join(', '));
|
||||
sessions.startSessionSweeper(getPool());
|
||||
startLoginAttemptSweeper(getPool()); // Phase 4.1 §15
|
||||
app.listen(PORT, () => console.log(`Platform API listening on :${PORT}`));
|
||||
}
|
||||
|
||||
|
||||
@@ -104,11 +104,28 @@ async function listInstructorSessions(pool, instructorId) {
|
||||
|
||||
// Periodic cleanup of long-dead rows so the table doesn't grow forever — cosmetic/storage only,
|
||||
// not a security control (expired/revoked rows already fail resolveSession's WHERE clause).
|
||||
//
|
||||
// Phase 4.1 §13 also piggybacks the "session expires" abandonment trigger onto this same
|
||||
// existing interval, rather than adding a new background job: any scenario_attempts row still
|
||||
// 'in_progress' for a student with NO live session left (every session expired, idle-timed-out,
|
||||
// or revoked, not just deleted by the sweep above) gets abandoned. Deliberately does the
|
||||
// attempts update in the SAME pool.query call — one round trip, no separate job to keep in sync.
|
||||
function startSessionSweeper(pool) {
|
||||
setInterval(() => {
|
||||
pool.query(`DELETE FROM sessions WHERE expires_at < now() - interval '7 days'`).catch((e) => {
|
||||
console.error('[session sweep] failed:', e.message);
|
||||
});
|
||||
pool.query(`
|
||||
UPDATE scenario_attempts SET status = 'abandoned', abandoned_at = now(), abandoned_reason = 'session_ended'
|
||||
WHERE status = 'in_progress'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM sessions
|
||||
WHERE sessions.student_id = scenario_attempts.student_id
|
||||
AND sessions.revoked_at IS NULL AND sessions.expires_at > now() AND sessions.idle_expires_at > now()
|
||||
)
|
||||
`).catch((e) => {
|
||||
console.error('[session sweep] attempt abandonment failed:', e.message);
|
||||
});
|
||||
}, 60 * 60 * 1000).unref();
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
// Phase 4.1 — the ONE place the Gateway decides how to reach a student's Lab. Everything else
|
||||
// in server.js just calls labTarget(env, token) and gets back a plain { url, headers, agent }
|
||||
// it can hand to http-proxy or fetch, with zero knowledge of Docker, VMs, or lab_hosts. See
|
||||
// docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1 and docs/PRODUCTION_NETWORK_SECURITY.md.
|
||||
//
|
||||
// env.lab_host_routing_mode / env.lab_host_router_endpoint come from the Provisioner's
|
||||
// by-token response (a join against lab_hosts — see provisioner lifecycle.js
|
||||
// getEnvironmentByTokenWithRouting). Two modes:
|
||||
//
|
||||
// 'direct' (default, dev/single-host) — unchanged from every prior phase: the Gateway
|
||||
// reaches env.internal_ip:internal_port itself, because Gateway and the Lab share a Docker
|
||||
// daemon/network namespace.
|
||||
//
|
||||
// 'router' (production, two real VMs) — the Gateway never touches a Docker-internal
|
||||
// address. It reaches that lab_host's Lab Host Router (platform/lab-router) instead, over
|
||||
// mTLS, carrying the environment's routing_token in a header so the Router — which lives on
|
||||
// the SAME host as the Lab and can resolve the token from local `docker ps` alone — knows
|
||||
// which local container to proxy to.
|
||||
|
||||
const fs = require('fs');
|
||||
const http = require('http');
|
||||
const https = require('https');
|
||||
const { URL } = require('url');
|
||||
|
||||
const ROUTE_TOKEN_HEADER = 'x-rootnode-route-token';
|
||||
|
||||
// mTLS client identity the Gateway presents to a Lab Host Router. One shared cert/key/ca is
|
||||
// enough for Phase 4.1's single lab_host; if a second lab_host ever needs a distinct client
|
||||
// identity, this becomes a map keyed by lab_host_id — not needed yet (see docs
|
||||
// "multi-host-ready" note, no scheduler built here).
|
||||
const TLS_CA = process.env.LAB_ROUTER_TLS_CA;
|
||||
const TLS_CERT = process.env.LAB_ROUTER_TLS_CERT;
|
||||
const TLS_KEY = process.env.LAB_ROUTER_TLS_KEY;
|
||||
|
||||
let _agent = null;
|
||||
function routerAgent() {
|
||||
if (_agent) return _agent;
|
||||
if (!TLS_CA || !TLS_CERT || !TLS_KEY) {
|
||||
throw new Error(
|
||||
'A lab_host is configured with routing_mode=router but LAB_ROUTER_TLS_CA/CERT/KEY are not set — refusing to connect to the Lab Host Router unauthenticated.'
|
||||
);
|
||||
}
|
||||
_agent = new https.Agent({
|
||||
ca: fs.readFileSync(TLS_CA),
|
||||
cert: fs.readFileSync(TLS_CERT),
|
||||
key: fs.readFileSync(TLS_KEY),
|
||||
// The Router's server cert is issued by the same private CA (infra/scripts/docker-tls/
|
||||
// generate-docker-tls.sh) — never fall back to the public CA trust store for this.
|
||||
});
|
||||
return _agent;
|
||||
}
|
||||
|
||||
// env: the (possibly-cached) environment row from the Provisioner's by-token response.
|
||||
// token: the routing token the browser's URL already carried (Gateway never needs the
|
||||
// Provisioner to hand it back — see provisioner server.js redactRow's comment on why
|
||||
// routing_token is redacted everywhere except the two routes that legitimately need it,
|
||||
// neither of which is this one).
|
||||
function labTarget(env, token) {
|
||||
if (env.lab_host_routing_mode === 'router') {
|
||||
if (!env.lab_host_router_endpoint) {
|
||||
throw new Error(`lab_host '${env.lab_host_id}' has routing_mode=router but no router_endpoint`);
|
||||
}
|
||||
return {
|
||||
url: env.lab_host_router_endpoint,
|
||||
headers: { [ROUTE_TOKEN_HEADER]: token },
|
||||
agent: routerAgent(),
|
||||
mode: 'router',
|
||||
// The browser's original Host header (e.g. "localhost:4210" or the real public origin)
|
||||
// has nothing to do with the Router's own TLS identity — without this, http-proxy's
|
||||
// outgoing HTTPS connection inherits that unrelated Host header for SNI/hostname
|
||||
// verification and the Router's cert (correctly issued for ITS OWN name) fails to
|
||||
// match. changeOrigin makes http-proxy rewrite the outgoing Host header (and therefore
|
||||
// the TLS servername) to match the actual target — found by testing against the real
|
||||
// Phase 4.1 simulation harness, not theoretical.
|
||||
changeOrigin: true,
|
||||
};
|
||||
}
|
||||
// direct — unchanged behavior from every phase before this one.
|
||||
return { url: `http://${env.internal_ip}:${env.internal_port}`, headers: {}, agent: undefined, mode: 'direct', changeOrigin: false };
|
||||
}
|
||||
|
||||
// Used by the two Gateway call sites that need to make a one-shot request to the Lab and read
|
||||
// its JSON body (progress/answer/validate relay, hint-count stripping) — everywhere else
|
||||
// (the actual proxy paths) uses http-proxy directly with the same target/agent/headers.
|
||||
//
|
||||
// Deliberately built on node:http/node:https rather than global fetch: fetch's undici
|
||||
// dispatcher isn't a stable, dependency-free way to attach a custom mTLS agent across Node
|
||||
// versions, and http.request/https.request support client certs natively with zero new
|
||||
// dependencies (stdlib covers it — see the ladder in CLAUDE.md's coding-style guidance).
|
||||
// Mimics just enough of fetch's response shape ({ status, json() }) that call sites barely
|
||||
// change.
|
||||
function labRequest(env, token, path, { method = 'GET', headers = {}, body } = {}) {
|
||||
const target = labTarget(env, token);
|
||||
const url = new URL(path, target.url);
|
||||
const transport = url.protocol === 'https:' ? https : http;
|
||||
const payload = body === undefined ? undefined : (typeof body === 'string' ? body : JSON.stringify(body));
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
const req = transport.request(
|
||||
url,
|
||||
{
|
||||
method,
|
||||
agent: target.agent,
|
||||
headers: {
|
||||
...headers,
|
||||
...target.headers,
|
||||
...(payload !== undefined ? { 'Content-Length': Buffer.byteLength(payload) } : {}),
|
||||
},
|
||||
},
|
||||
(res) => {
|
||||
const chunks = [];
|
||||
res.on('data', (c) => chunks.push(c));
|
||||
res.on('end', () => {
|
||||
const text = Buffer.concat(chunks).toString('utf8');
|
||||
resolve({
|
||||
status: res.statusCode,
|
||||
json: async () => JSON.parse(text),
|
||||
});
|
||||
});
|
||||
}
|
||||
);
|
||||
req.on('error', reject);
|
||||
if (payload !== undefined) req.write(payload);
|
||||
req.end();
|
||||
});
|
||||
}
|
||||
|
||||
module.exports = { labTarget, labRequest, ROUTE_TOKEN_HEADER };
|
||||
@@ -19,6 +19,11 @@ const express = require('express');
|
||||
const httpProxy = require('http-proxy');
|
||||
const { provisioner } = require('./provisioner-client'); // still used ONLY for the token->environment lookup a Lab request needs; all mutating calls (provision/reset/destroy) now live in the Platform API.
|
||||
const envCache = require('./env-cache');
|
||||
// Phase 4.1 — the ONE place that knows how to reach a Lab (direct Docker-internal address, or
|
||||
// via that lab_host's Lab Host Router). Every proxy.web/proxy.ws/labRequest call below uses
|
||||
// labTarget's output instead of building `http://${env.internal_ip}:${env.internal_port}`
|
||||
// itself. See lab-endpoint.js's header comment and docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1.
|
||||
const { labTarget, labRequest } = require('./lab-endpoint');
|
||||
|
||||
const PORT = Number(process.env.GATEWAY_PORT) || 4200;
|
||||
const PLATFORM_API_URL = process.env.PLATFORM_API_URL || 'http://localhost:4300';
|
||||
@@ -90,6 +95,9 @@ async function resolveByToken(token) {
|
||||
return body;
|
||||
}
|
||||
|
||||
// internal_ip is set by the Provisioner as soon as a container exists, regardless of routing
|
||||
// mode (it's a Docker-internal fact, independent of how the Gateway will reach it) — so this
|
||||
// check stays valid unchanged for both 'direct' and 'router' lab_hosts.
|
||||
function isReachable(env) {
|
||||
return env && env.status === 'running' && env.internal_ip;
|
||||
}
|
||||
@@ -116,9 +124,9 @@ async function checkOwnership(req, env) {
|
||||
|
||||
// ── Progress relay (unchanged mechanism from Phase 2A, now passes environment_id +
|
||||
// validation_count through so Platform API can log a real scenario_attempts row) ──────────
|
||||
async function relayScenarioOutcome({ env, req, res, scenarioPath, extractOutcome }) {
|
||||
async function relayScenarioOutcome({ env, token, req, res, scenarioPath, extractOutcome }) {
|
||||
if (!isReachable(env)) return sendLabError(res, 503, 'lab_unavailable', 'Your lab is not currently running.');
|
||||
const labRes = await fetch(`http://${env.internal_ip}:${env.internal_port}${scenarioPath}`, {
|
||||
const labRes = await labRequest(env, token, scenarioPath, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(req.body || {}),
|
||||
@@ -157,7 +165,7 @@ app.post('/lab/:token/api/scenarios/:scenarioId/validate', express.json(), async
|
||||
const own = await checkOwnership(req, env);
|
||||
if (!own.ok) return sendLabError(res, own.reason === 'invalid_token' ? 404 : 403, own.reason, 'This lab is not available to you.');
|
||||
await relayScenarioOutcome({
|
||||
env, req, res, scenarioPath: `/api/scenarios/${req.params.scenarioId}/validate`,
|
||||
env, token: req.params.token, req, res, scenarioPath: `/api/scenarios/${req.params.scenarioId}/validate`,
|
||||
extractOutcome: (body) => (typeof body.passed === 'boolean' ? body.passed : null),
|
||||
});
|
||||
});
|
||||
@@ -167,7 +175,7 @@ app.post('/lab/:token/api/scenarios/:scenarioId/answer', express.json(), async (
|
||||
const own = await checkOwnership(req, env);
|
||||
if (!own.ok) return sendLabError(res, own.reason === 'invalid_token' ? 404 : 403, own.reason, 'This lab is not available to you.');
|
||||
await relayScenarioOutcome({
|
||||
env, req, res, scenarioPath: `/api/scenarios/${req.params.scenarioId}/answer`,
|
||||
env, token: req.params.token, req, res, scenarioPath: `/api/scenarios/${req.params.scenarioId}/answer`,
|
||||
extractOutcome: (body) => (typeof body.correct === 'boolean' ? body.correct : null),
|
||||
});
|
||||
});
|
||||
@@ -223,7 +231,7 @@ app.get('/lab/:token/api/scenarios/:scenarioId', async (req, res) => {
|
||||
if (!own.ok) return sendLabError(res, own.reason === 'invalid_token' ? 404 : 403, own.reason, 'This lab is not available to you.');
|
||||
if (!isReachable(env)) return sendLabError(res, 503, 'lab_unavailable', 'Your lab is not currently running.');
|
||||
try {
|
||||
const labRes = await fetch(`http://${env.internal_ip}:${env.internal_port}/api/scenarios/${req.params.scenarioId}`);
|
||||
const labRes = await labRequest(env, req.params.token, `/api/scenarios/${req.params.scenarioId}`);
|
||||
const body = await labRes.json().catch(() => ({}));
|
||||
const { hints, ...withoutHints } = body;
|
||||
res.status(labRes.status).json({ ...withoutHints, hint_count: Array.isArray(hints) ? hints.length : 0 });
|
||||
@@ -259,7 +267,8 @@ app.use('/lab/:token', async (req, res, next) => {
|
||||
// for this browser.
|
||||
res.append('Set-Cookie', `${LAB_ACTIVE_COOKIE}=1; Path=/; HttpOnly; SameSite=Lax; Max-Age=28800`);
|
||||
req.url = req.url.replace(/^\/lab\/[^/]+/, '') || '/';
|
||||
proxy.web(req, res, { target: `http://${env.internal_ip}:${env.internal_port}` });
|
||||
const target = labTarget(env, token);
|
||||
proxy.web(req, res, { target: target.url, agent: target.agent, headers: target.headers, changeOrigin: target.changeOrigin });
|
||||
});
|
||||
|
||||
// Resolves "the environment this browser's session currently owns and can reach," for the
|
||||
@@ -268,20 +277,23 @@ app.use('/lab/:token', async (req, res, next) => {
|
||||
// the WS upgrade handler so there's exactly one place this logic lives. Gated on
|
||||
// LAB_ACTIVE_COOKIE first — see its definition above for why "has a running env" alone isn't
|
||||
// a safe signal (it's true on the dashboard too).
|
||||
// Returns { env, token } (not just env) — Phase 4.1's labTarget needs the routing token
|
||||
// alongside the environment row for router-mode lab_hosts, and this is the only place that
|
||||
// derives the token from launch_path rather than the URL, so it has to hand both back.
|
||||
async function resolveOwnedRunningEnvironment(req) {
|
||||
const cookies = parseCookies(req);
|
||||
if (!cookies[LAB_ACTIVE_COOKIE]) return null;
|
||||
const token = cookies[STUDENT_COOKIE];
|
||||
if (!token) return null;
|
||||
const sessionToken = cookies[STUDENT_COOKIE];
|
||||
if (!sessionToken) return null;
|
||||
const studentId = await resolveSessionStudentId(req);
|
||||
if (!studentId) return null;
|
||||
try {
|
||||
const envRes = await fetch(`${PLATFORM_API_URL}/api/environment/status`, { headers: { Cookie: `${STUDENT_COOKIE}=${token}` } });
|
||||
const envRes = await fetch(`${PLATFORM_API_URL}/api/environment/status`, { headers: { Cookie: `${STUDENT_COOKIE}=${sessionToken}` } });
|
||||
const envInfo = await envRes.json().catch(() => null);
|
||||
if (envInfo?.status !== 'running' || !envInfo.launch_path) return null;
|
||||
const lookupToken = envInfo.launch_path.split('/')[2];
|
||||
const env = await resolveByToken(lookupToken);
|
||||
return isReachable(env) && env.student_id === studentId ? env : null;
|
||||
return isReachable(env) && env.student_id === studentId ? { env, token: lookupToken } : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
@@ -303,8 +315,11 @@ function looksLikeLabPath(pathname) {
|
||||
// or the Platform API (student dashboard, admin portal, roster/curriculum/progress) ────────
|
||||
app.use(async (req, res) => {
|
||||
if (looksLikeLabPath(req.path)) {
|
||||
const env = await resolveOwnedRunningEnvironment(req);
|
||||
if (env) return proxy.web(req, res, { target: `http://${env.internal_ip}:${env.internal_port}` });
|
||||
const owned = await resolveOwnedRunningEnvironment(req);
|
||||
if (owned) {
|
||||
const target = labTarget(owned.env, owned.token);
|
||||
return proxy.web(req, res, { target: target.url, agent: target.agent, headers: target.headers, changeOrigin: target.changeOrigin });
|
||||
}
|
||||
}
|
||||
proxy.web(req, res, { target: PLATFORM_API_URL });
|
||||
});
|
||||
@@ -319,12 +334,16 @@ server.on('upgrade', async (req, socket, head) => {
|
||||
const own = await checkOwnership(req, env);
|
||||
if (!own.ok || !isReachable(env)) return socket.destroy();
|
||||
req.url = rest || '/';
|
||||
return proxy.ws(req, socket, head, { target: `http://${env.internal_ip}:${env.internal_port}` });
|
||||
const target = labTarget(env, token);
|
||||
return proxy.ws(req, socket, head, { target: target.url, agent: target.agent, headers: target.headers, changeOrigin: target.changeOrigin });
|
||||
}
|
||||
|
||||
// Unprefixed WS (the existing Lab UI's /shell-ws, hit while "inside" a lab session).
|
||||
const env = await resolveOwnedRunningEnvironment(req);
|
||||
if (env) return proxy.ws(req, socket, head, { target: `http://${env.internal_ip}:${env.internal_port}` });
|
||||
const owned = await resolveOwnedRunningEnvironment(req);
|
||||
if (owned) {
|
||||
const target = labTarget(owned.env, owned.token);
|
||||
return proxy.ws(req, socket, head, { target: target.url, agent: target.agent, headers: target.headers, changeOrigin: target.changeOrigin });
|
||||
}
|
||||
socket.destroy();
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
FROM node:20-slim
|
||||
WORKDIR /app
|
||||
COPY package.json ./
|
||||
RUN npm install --production
|
||||
COPY src/ ./src/
|
||||
EXPOSE 8443
|
||||
CMD ["node", "src/server.js"]
|
||||
Generated
+706
@@ -0,0 +1,706 @@
|
||||
{
|
||||
"name": "rootnode-lab-router",
|
||||
"version": "0.1.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "rootnode-lab-router",
|
||||
"version": "0.1.0",
|
||||
"dependencies": {
|
||||
"dockerode": "^4.0.2",
|
||||
"http-proxy": "^1.18.1"
|
||||
}
|
||||
},
|
||||
"node_modules/@balena/dockerignore": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/@balena/dockerignore/-/dockerignore-1.0.2.tgz",
|
||||
"integrity": "sha512-wMue2Sy4GAVTk6Ic4tJVcnfdau+gx2EnG7S+uAEe+TWJFqE4YoWN4/H8MSLj4eYJKxGg26lZwboEniNiNwZQ6Q=="
|
||||
},
|
||||
"node_modules/@grpc/grpc-js": {
|
||||
"version": "1.14.4",
|
||||
"resolved": "https://registry.npmjs.org/@grpc/grpc-js/-/grpc-js-1.14.4.tgz",
|
||||
"integrity": "sha512-k9Dj3DV/itK9D06Y8f190Qgop7/Ui+D0njFV3LHMPwPT75DpXLQohE9Wmz0QElrJnzsjB7KPWiKJbOl7IPDArQ==",
|
||||
"dependencies": {
|
||||
"@grpc/proto-loader": "^0.8.0",
|
||||
"@js-sdsl/ordered-map": "^4.4.2"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@grpc/grpc-js/node_modules/@grpc/proto-loader": {
|
||||
"version": "0.8.1",
|
||||
"resolved": "https://registry.npmjs.org/@grpc/proto-loader/-/proto-loader-0.8.1.tgz",
|
||||
"integrity": "sha512-wtF6h+DY6M3YaDBPAmvuuA6jV8Sif9MjtOI5euKFWRgCDl5PeDpPsHR9u2l6St5ceY8AZgoNDww5+HvEsXFsGg==",
|
||||
"dependencies": {
|
||||
"lodash.camelcase": "^4.3.0",
|
||||
"long": "^5.0.0",
|
||||
"protobufjs": "^7.5.5",
|
||||
"yargs": "^17.7.2"
|
||||
},
|
||||
"bin": {
|
||||
"proto-loader-gen-types": "build/bin/proto-loader-gen-types.js"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=6"
|
||||
}
|
||||
},
|
||||
"node_modules/@grpc/proto-loader": {
|
||||
"version": "0.7.15",
|
||||
"resolved": "https://registry.npmjs.org/@grpc/proto-loader/-/proto-loader-0.7.15.tgz",
|
||||
"integrity": "sha512-tMXdRCfYVixjuFK+Hk0Q1s38gV9zDiDJfWL3h1rv4Qc39oILCu1TRTDt7+fGUI8K4G1Fj125Hx/ru3azECWTyQ==",
|
||||
"dependencies": {
|
||||
"lodash.camelcase": "^4.3.0",
|
||||
"long": "^5.0.0",
|
||||
"protobufjs": "^7.2.5",
|
||||
"yargs": "^17.7.2"
|
||||
},
|
||||
"bin": {
|
||||
"proto-loader-gen-types": "build/bin/proto-loader-gen-types.js"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=6"
|
||||
}
|
||||
},
|
||||
"node_modules/@js-sdsl/ordered-map": {
|
||||
"version": "4.4.2",
|
||||
"resolved": "https://registry.npmjs.org/@js-sdsl/ordered-map/-/ordered-map-4.4.2.tgz",
|
||||
"integrity": "sha512-iUKgm52T8HOE/makSxjqoWhe95ZJA1/G1sYsGev2JDKUSS14KAgg1LHb+Ba+IPow0xflbnSkOsZcO08C7w1gYw==",
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/js-sdsl"
|
||||
}
|
||||
},
|
||||
"node_modules/@protobufjs/aspromise": {
|
||||
"version": "1.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz",
|
||||
"integrity": "sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ=="
|
||||
},
|
||||
"node_modules/@protobufjs/base64": {
|
||||
"version": "1.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@protobufjs/base64/-/base64-1.1.2.tgz",
|
||||
"integrity": "sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg=="
|
||||
},
|
||||
"node_modules/@protobufjs/codegen": {
|
||||
"version": "2.0.5",
|
||||
"resolved": "https://registry.npmjs.org/@protobufjs/codegen/-/codegen-2.0.5.tgz",
|
||||
"integrity": "sha512-zgXFLzW3Ap33e6d0Wlj4MGIm6Ce8O89n/apUaGNB/jx+hw+ruWEp7EwGUshdLKVRCxZW12fp9r40E1mQrf/34g=="
|
||||
},
|
||||
"node_modules/@protobufjs/eventemitter": {
|
||||
"version": "1.1.1",
|
||||
"resolved": "https://registry.npmjs.org/@protobufjs/eventemitter/-/eventemitter-1.1.1.tgz",
|
||||
"integrity": "sha512-vW1GmwMZNnL+gMRaovlh9yZX74kc+TTU3FObkkurpMaRtBfLP3ldjS9KQWlwZgraRE0+dheEEoAxdzcJQ8eXZg=="
|
||||
},
|
||||
"node_modules/@protobufjs/fetch": {
|
||||
"version": "1.1.1",
|
||||
"resolved": "https://registry.npmjs.org/@protobufjs/fetch/-/fetch-1.1.1.tgz",
|
||||
"integrity": "sha512-GpptLrs57adMSuHi3VNj0mAF8dwh36LMaYF6XyJ6JMWlVsc+t42tm1HSEDmOs3A8fC9yyeisgLhsTVQokOZ0zw==",
|
||||
"dependencies": {
|
||||
"@protobufjs/aspromise": "^1.1.1"
|
||||
}
|
||||
},
|
||||
"node_modules/@protobufjs/float": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/@protobufjs/float/-/float-1.0.2.tgz",
|
||||
"integrity": "sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ=="
|
||||
},
|
||||
"node_modules/@protobufjs/path": {
|
||||
"version": "1.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@protobufjs/path/-/path-1.1.2.tgz",
|
||||
"integrity": "sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA=="
|
||||
},
|
||||
"node_modules/@protobufjs/pool": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/@protobufjs/pool/-/pool-1.1.0.tgz",
|
||||
"integrity": "sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw=="
|
||||
},
|
||||
"node_modules/@protobufjs/utf8": {
|
||||
"version": "1.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@protobufjs/utf8/-/utf8-1.1.2.tgz",
|
||||
"integrity": "sha512-b1UQwcEZ4yCnMCD8DAL1VlbvBJE9/IX4FTIp7BG1xYpf29SLazLSrqUkj4w7Y5y7cCVP6E5tcqqcI0xemPkHug=="
|
||||
},
|
||||
"node_modules/@types/node": {
|
||||
"version": "26.2.0",
|
||||
"resolved": "https://registry.npmjs.org/@types/node/-/node-26.2.0.tgz",
|
||||
"integrity": "sha512-5IviulTZeRNp2vAJ514cc/HUlY5nZ9fCbq9DMyC52BrhFZACo3nI0R7qBxhQmo/d27NFe96ur/b7Wwxklda+kg==",
|
||||
"dependencies": {
|
||||
"undici-types": "~8.3.0"
|
||||
}
|
||||
},
|
||||
"node_modules/ansi-regex": {
|
||||
"version": "5.0.1",
|
||||
"resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz",
|
||||
"integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==",
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/ansi-styles": {
|
||||
"version": "4.3.0",
|
||||
"resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-4.3.0.tgz",
|
||||
"integrity": "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg==",
|
||||
"dependencies": {
|
||||
"color-convert": "^2.0.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/chalk/ansi-styles?sponsor=1"
|
||||
}
|
||||
},
|
||||
"node_modules/asn1": {
|
||||
"version": "0.2.6",
|
||||
"resolved": "https://registry.npmjs.org/asn1/-/asn1-0.2.6.tgz",
|
||||
"integrity": "sha512-ix/FxPn0MDjeyJ7i/yoHGFt/EX6LyNbxSEhPPXODPL+KB0VPk86UYfL0lMdy+KCnv+fmvIzySwaK5COwqVbWTQ==",
|
||||
"dependencies": {
|
||||
"safer-buffer": "~2.1.0"
|
||||
}
|
||||
},
|
||||
"node_modules/base64-js": {
|
||||
"version": "1.5.1",
|
||||
"resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz",
|
||||
"integrity": "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==",
|
||||
"funding": [
|
||||
{
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/feross"
|
||||
},
|
||||
{
|
||||
"type": "patreon",
|
||||
"url": "https://www.patreon.com/feross"
|
||||
},
|
||||
{
|
||||
"type": "consulting",
|
||||
"url": "https://feross.org/support"
|
||||
}
|
||||
]
|
||||
},
|
||||
"node_modules/bcrypt-pbkdf": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/bcrypt-pbkdf/-/bcrypt-pbkdf-1.0.2.tgz",
|
||||
"integrity": "sha512-qeFIXtP4MSoi6NLqO12WfqARWWuCKi2Rn/9hJLEmtB5yTNr9DqFWkJRCf2qShWzPeAMRnOgCrq0sg/KLv5ES9w==",
|
||||
"dependencies": {
|
||||
"tweetnacl": "^0.14.3"
|
||||
}
|
||||
},
|
||||
"node_modules/bl": {
|
||||
"version": "4.1.0",
|
||||
"resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz",
|
||||
"integrity": "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==",
|
||||
"dependencies": {
|
||||
"buffer": "^5.5.0",
|
||||
"inherits": "^2.0.4",
|
||||
"readable-stream": "^3.4.0"
|
||||
}
|
||||
},
|
||||
"node_modules/buffer": {
|
||||
"version": "5.7.1",
|
||||
"resolved": "https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz",
|
||||
"integrity": "sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ==",
|
||||
"funding": [
|
||||
{
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/feross"
|
||||
},
|
||||
{
|
||||
"type": "patreon",
|
||||
"url": "https://www.patreon.com/feross"
|
||||
},
|
||||
{
|
||||
"type": "consulting",
|
||||
"url": "https://feross.org/support"
|
||||
}
|
||||
],
|
||||
"dependencies": {
|
||||
"base64-js": "^1.3.1",
|
||||
"ieee754": "^1.1.13"
|
||||
}
|
||||
},
|
||||
"node_modules/buildcheck": {
|
||||
"version": "0.0.7",
|
||||
"resolved": "https://registry.npmjs.org/buildcheck/-/buildcheck-0.0.7.tgz",
|
||||
"integrity": "sha512-lHblz4ahamxpTmnsk+MNTRWsjYKv965MwOrSJyeD588rR3Jcu7swE+0wN5F+PbL5cjgu/9ObkhfzEPuofEMwLA==",
|
||||
"optional": true,
|
||||
"engines": {
|
||||
"node": ">=10.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/chownr": {
|
||||
"version": "1.1.4",
|
||||
"resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz",
|
||||
"integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg=="
|
||||
},
|
||||
"node_modules/cliui": {
|
||||
"version": "8.0.1",
|
||||
"resolved": "https://registry.npmjs.org/cliui/-/cliui-8.0.1.tgz",
|
||||
"integrity": "sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ==",
|
||||
"dependencies": {
|
||||
"string-width": "^4.2.0",
|
||||
"strip-ansi": "^6.0.1",
|
||||
"wrap-ansi": "^7.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/color-convert": {
|
||||
"version": "2.0.1",
|
||||
"resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz",
|
||||
"integrity": "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==",
|
||||
"dependencies": {
|
||||
"color-name": "~1.1.4"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=7.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/color-name": {
|
||||
"version": "1.1.4",
|
||||
"resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz",
|
||||
"integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA=="
|
||||
},
|
||||
"node_modules/cpu-features": {
|
||||
"version": "0.0.10",
|
||||
"resolved": "https://registry.npmjs.org/cpu-features/-/cpu-features-0.0.10.tgz",
|
||||
"integrity": "sha512-9IkYqtX3YHPCzoVg1Py+o9057a3i0fp7S530UWokCSaFVTc7CwXPRiOjRjBQQ18ZCNafx78YfnG+HALxtVmOGA==",
|
||||
"hasInstallScript": true,
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"buildcheck": "~0.0.6",
|
||||
"nan": "^2.19.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=10.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/debug": {
|
||||
"version": "4.4.3",
|
||||
"resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",
|
||||
"integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==",
|
||||
"dependencies": {
|
||||
"ms": "^2.1.3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=6.0"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"supports-color": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/docker-modem": {
|
||||
"version": "5.0.7",
|
||||
"resolved": "https://registry.npmjs.org/docker-modem/-/docker-modem-5.0.7.tgz",
|
||||
"integrity": "sha512-XJgGhoR/CLpqshm4d3L7rzH6t8NgDFUIIpztYlLHIApeJjMZKYJMz2zxPsYxnejq5h3ELYSw/RBsi3t5h7gNTA==",
|
||||
"dependencies": {
|
||||
"debug": "^4.1.1",
|
||||
"readable-stream": "^3.5.0",
|
||||
"split-ca": "^1.0.1",
|
||||
"ssh2": "^1.15.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 8.0"
|
||||
}
|
||||
},
|
||||
"node_modules/dockerode": {
|
||||
"version": "4.0.12",
|
||||
"resolved": "https://registry.npmjs.org/dockerode/-/dockerode-4.0.12.tgz",
|
||||
"integrity": "sha512-/bCZd6KlGcjZO8Buqmi/vXuqEGVEZ0PNjx/biBNqJD3MhK9DmdiAuKxqfNhflgDESDIiBz3qF+0e55+CpnrUcw==",
|
||||
"dependencies": {
|
||||
"@balena/dockerignore": "^1.0.2",
|
||||
"@grpc/grpc-js": "^1.11.1",
|
||||
"@grpc/proto-loader": "^0.7.13",
|
||||
"docker-modem": "^5.0.7",
|
||||
"protobufjs": "^7.3.2",
|
||||
"tar-fs": "^2.1.4",
|
||||
"uuid": "^10.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 8.0"
|
||||
}
|
||||
},
|
||||
"node_modules/emoji-regex": {
|
||||
"version": "8.0.0",
|
||||
"resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz",
|
||||
"integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A=="
|
||||
},
|
||||
"node_modules/end-of-stream": {
|
||||
"version": "1.4.5",
|
||||
"resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz",
|
||||
"integrity": "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==",
|
||||
"dependencies": {
|
||||
"once": "^1.4.0"
|
||||
}
|
||||
},
|
||||
"node_modules/escalade": {
|
||||
"version": "3.2.0",
|
||||
"resolved": "https://registry.npmjs.org/escalade/-/escalade-3.2.0.tgz",
|
||||
"integrity": "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA==",
|
||||
"engines": {
|
||||
"node": ">=6"
|
||||
}
|
||||
},
|
||||
"node_modules/eventemitter3": {
|
||||
"version": "4.0.7",
|
||||
"resolved": "https://registry.npmjs.org/eventemitter3/-/eventemitter3-4.0.7.tgz",
|
||||
"integrity": "sha512-8guHBZCwKnFhYdHr2ysuRWErTwhoN2X8XELRlrRwpmfeY2jjuUN4taQMsULKUVo1K4DvZl+0pgfyoysHxvmvEw=="
|
||||
},
|
||||
"node_modules/follow-redirects": {
|
||||
"version": "1.16.0",
|
||||
"resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.16.0.tgz",
|
||||
"integrity": "sha512-y5rN/uOsadFT/JfYwhxRS5R7Qce+g3zG97+JrtFZlC9klX/W5hD7iiLzScI4nZqUS7DNUdhPgw4xI8W2LuXlUw==",
|
||||
"funding": [
|
||||
{
|
||||
"type": "individual",
|
||||
"url": "https://github.com/sponsors/RubenVerborgh"
|
||||
}
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=4.0"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"debug": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/fs-constants": {
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz",
|
||||
"integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow=="
|
||||
},
|
||||
"node_modules/get-caller-file": {
|
||||
"version": "2.0.5",
|
||||
"resolved": "https://registry.npmjs.org/get-caller-file/-/get-caller-file-2.0.5.tgz",
|
||||
"integrity": "sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg==",
|
||||
"engines": {
|
||||
"node": "6.* || 8.* || >= 10.*"
|
||||
}
|
||||
},
|
||||
"node_modules/http-proxy": {
|
||||
"version": "1.18.1",
|
||||
"resolved": "https://registry.npmjs.org/http-proxy/-/http-proxy-1.18.1.tgz",
|
||||
"integrity": "sha512-7mz/721AbnJwIVbnaSv1Cz3Am0ZLT/UBwkC92VlxhXv/k/BBQfM2fXElQNC27BVGr0uwUpplYPQM9LnaBMR5NQ==",
|
||||
"dependencies": {
|
||||
"eventemitter3": "^4.0.0",
|
||||
"follow-redirects": "^1.0.0",
|
||||
"requires-port": "^1.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=8.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/ieee754": {
|
||||
"version": "1.2.1",
|
||||
"resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz",
|
||||
"integrity": "sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==",
|
||||
"funding": [
|
||||
{
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/feross"
|
||||
},
|
||||
{
|
||||
"type": "patreon",
|
||||
"url": "https://www.patreon.com/feross"
|
||||
},
|
||||
{
|
||||
"type": "consulting",
|
||||
"url": "https://feross.org/support"
|
||||
}
|
||||
]
|
||||
},
|
||||
"node_modules/inherits": {
|
||||
"version": "2.0.4",
|
||||
"resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
|
||||
"integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="
|
||||
},
|
||||
"node_modules/is-fullwidth-code-point": {
|
||||
"version": "3.0.0",
|
||||
"resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz",
|
||||
"integrity": "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==",
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/lodash.camelcase": {
|
||||
"version": "4.3.0",
|
||||
"resolved": "https://registry.npmjs.org/lodash.camelcase/-/lodash.camelcase-4.3.0.tgz",
|
||||
"integrity": "sha512-TwuEnCnxbc3rAvhf/LbG7tJUDzhqXyFnv3dtzLOPgCG/hODL7WFnsbwktkD7yUV0RrreP/l1PALq/YSg6VvjlA=="
|
||||
},
|
||||
"node_modules/long": {
|
||||
"version": "5.3.2",
|
||||
"resolved": "https://registry.npmjs.org/long/-/long-5.3.2.tgz",
|
||||
"integrity": "sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA=="
|
||||
},
|
||||
"node_modules/mkdirp-classic": {
|
||||
"version": "0.5.3",
|
||||
"resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz",
|
||||
"integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A=="
|
||||
},
|
||||
"node_modules/ms": {
|
||||
"version": "2.1.3",
|
||||
"resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
|
||||
"integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="
|
||||
},
|
||||
"node_modules/nan": {
|
||||
"version": "2.28.0",
|
||||
"resolved": "https://registry.npmjs.org/nan/-/nan-2.28.0.tgz",
|
||||
"integrity": "sha512-fTsDz99OTq2sVePhGdp4qQhggZFtKr64ZNVyVajRKtMOkJxYekplBh577PiJB12v/D3s2E5cGtOI45LWp6rnLQ==",
|
||||
"optional": true
|
||||
},
|
||||
"node_modules/once": {
|
||||
"version": "1.4.0",
|
||||
"resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
|
||||
"integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==",
|
||||
"dependencies": {
|
||||
"wrappy": "1"
|
||||
}
|
||||
},
|
||||
"node_modules/protobufjs": {
|
||||
"version": "7.6.5",
|
||||
"resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-7.6.5.tgz",
|
||||
"integrity": "sha512-/FPD0nUc9jH6rfFjji9IBqOz4pcSE3CsT1m7Ep6Mdb0LxSUMj8hgl6GomOvZzpNpAqqGaXA0P3VSrZLFzIhQrw==",
|
||||
"hasInstallScript": true,
|
||||
"dependencies": {
|
||||
"@protobufjs/aspromise": "^1.1.2",
|
||||
"@protobufjs/base64": "^1.1.2",
|
||||
"@protobufjs/codegen": "^2.0.5",
|
||||
"@protobufjs/eventemitter": "^1.1.1",
|
||||
"@protobufjs/fetch": "^1.1.1",
|
||||
"@protobufjs/float": "^1.0.2",
|
||||
"@protobufjs/path": "^1.1.2",
|
||||
"@protobufjs/pool": "^1.1.0",
|
||||
"@protobufjs/utf8": "^1.1.1",
|
||||
"@types/node": ">=13.7.0",
|
||||
"long": "^5.3.2"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/pump": {
|
||||
"version": "3.0.4",
|
||||
"resolved": "https://registry.npmjs.org/pump/-/pump-3.0.4.tgz",
|
||||
"integrity": "sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA==",
|
||||
"dependencies": {
|
||||
"end-of-stream": "^1.1.0",
|
||||
"once": "^1.3.1"
|
||||
}
|
||||
},
|
||||
"node_modules/readable-stream": {
|
||||
"version": "3.6.2",
|
||||
"resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz",
|
||||
"integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==",
|
||||
"dependencies": {
|
||||
"inherits": "^2.0.3",
|
||||
"string_decoder": "^1.1.1",
|
||||
"util-deprecate": "^1.0.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 6"
|
||||
}
|
||||
},
|
||||
"node_modules/require-directory": {
|
||||
"version": "2.1.1",
|
||||
"resolved": "https://registry.npmjs.org/require-directory/-/require-directory-2.1.1.tgz",
|
||||
"integrity": "sha512-fGxEI7+wsG9xrvdjsrlmL22OMTTiHRwAMroiEeMgq8gzoLC/PQr7RsRDSTLUg/bZAZtF+TVIkHc6/4RIKrui+Q==",
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/requires-port": {
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/requires-port/-/requires-port-1.0.0.tgz",
|
||||
"integrity": "sha512-KigOCHcocU3XODJxsu8i/j8T9tzT4adHiecwORRQ0ZZFcp7ahwXuRU1m+yuO90C5ZUyGeGfocHDI14M3L3yDAQ=="
|
||||
},
|
||||
"node_modules/safe-buffer": {
|
||||
"version": "5.2.1",
|
||||
"resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz",
|
||||
"integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==",
|
||||
"funding": [
|
||||
{
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/feross"
|
||||
},
|
||||
{
|
||||
"type": "patreon",
|
||||
"url": "https://www.patreon.com/feross"
|
||||
},
|
||||
{
|
||||
"type": "consulting",
|
||||
"url": "https://feross.org/support"
|
||||
}
|
||||
]
|
||||
},
|
||||
"node_modules/safer-buffer": {
|
||||
"version": "2.1.2",
|
||||
"resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz",
|
||||
"integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="
|
||||
},
|
||||
"node_modules/split-ca": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/split-ca/-/split-ca-1.0.1.tgz",
|
||||
"integrity": "sha512-Q5thBSxp5t8WPTTJQS59LrGqOZqOsrhDGDVm8azCqIBjSBd7nd9o2PM+mDulQQkh8h//4U6hFZnc/mul8t5pWQ=="
|
||||
},
|
||||
"node_modules/ssh2": {
|
||||
"version": "1.17.0",
|
||||
"resolved": "https://registry.npmjs.org/ssh2/-/ssh2-1.17.0.tgz",
|
||||
"integrity": "sha512-wPldCk3asibAjQ/kziWQQt1Wh3PgDFpC0XpwclzKcdT1vql6KeYxf5LIt4nlFkUeR8WuphYMKqUA56X4rjbfgQ==",
|
||||
"hasInstallScript": true,
|
||||
"dependencies": {
|
||||
"asn1": "^0.2.6",
|
||||
"bcrypt-pbkdf": "^1.0.2"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=10.16.0"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"cpu-features": "~0.0.10",
|
||||
"nan": "^2.23.0"
|
||||
}
|
||||
},
|
||||
"node_modules/string_decoder": {
|
||||
"version": "1.3.0",
|
||||
"resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz",
|
||||
"integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==",
|
||||
"dependencies": {
|
||||
"safe-buffer": "~5.2.0"
|
||||
}
|
||||
},
|
||||
"node_modules/string-width": {
|
||||
"version": "4.2.3",
|
||||
"resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz",
|
||||
"integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==",
|
||||
"dependencies": {
|
||||
"emoji-regex": "^8.0.0",
|
||||
"is-fullwidth-code-point": "^3.0.0",
|
||||
"strip-ansi": "^6.0.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/strip-ansi": {
|
||||
"version": "6.0.1",
|
||||
"resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz",
|
||||
"integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==",
|
||||
"dependencies": {
|
||||
"ansi-regex": "^5.0.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/tar-fs": {
|
||||
"version": "2.1.5",
|
||||
"resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.5.tgz",
|
||||
"integrity": "sha512-OboTd8mmMhZDNPV+UjQcK9yKAatXu2aJ+r1w4im1Otd4M4fl2hwvdoXUxIYHFTHWK/3y3FarBP70v3vwmGlOxw==",
|
||||
"dependencies": {
|
||||
"chownr": "^1.1.1",
|
||||
"mkdirp-classic": "^0.5.2",
|
||||
"pump": "^3.0.0",
|
||||
"tar-stream": "^2.1.4"
|
||||
}
|
||||
},
|
||||
"node_modules/tar-stream": {
|
||||
"version": "2.2.0",
|
||||
"resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz",
|
||||
"integrity": "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==",
|
||||
"dependencies": {
|
||||
"bl": "^4.0.3",
|
||||
"end-of-stream": "^1.4.1",
|
||||
"fs-constants": "^1.0.0",
|
||||
"inherits": "^2.0.3",
|
||||
"readable-stream": "^3.1.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=6"
|
||||
}
|
||||
},
|
||||
"node_modules/tweetnacl": {
|
||||
"version": "0.14.5",
|
||||
"resolved": "https://registry.npmjs.org/tweetnacl/-/tweetnacl-0.14.5.tgz",
|
||||
"integrity": "sha512-KXXFFdAbFXY4geFIwoyNK+f5Z1b7swfXABfL7HXCmoIWMKU3dmS26672A4EeQtDzLKy7SXmfBu51JolvEKwtGA=="
|
||||
},
|
||||
"node_modules/undici-types": {
|
||||
"version": "8.3.0",
|
||||
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-8.3.0.tgz",
|
||||
"integrity": "sha512-j375ScV60dom+YkPFIfTLcOiPxkN/buHz5GobjLhixFuANaNs3C9l4GmrWqejgXWJ7BbJcFYpTEUkS1Ge8bpZQ=="
|
||||
},
|
||||
"node_modules/util-deprecate": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",
|
||||
"integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw=="
|
||||
},
|
||||
"node_modules/uuid": {
|
||||
"version": "10.0.0",
|
||||
"resolved": "https://registry.npmjs.org/uuid/-/uuid-10.0.0.tgz",
|
||||
"integrity": "sha512-8XkAphELsDnEGrDxUOHB3RGvXz6TeuYSGEZBOjtTtPm2lwhGBjLgOzLHB63IUWfBpNucQjND6d3AOudO+H3RWQ==",
|
||||
"deprecated": "uuid@10 and below is no longer supported. For ESM codebases, update to uuid@latest. For CommonJS codebases, use uuid@11 (but be aware this version will likely be deprecated in 2028).",
|
||||
"funding": [
|
||||
"https://github.com/sponsors/broofa",
|
||||
"https://github.com/sponsors/ctavan"
|
||||
],
|
||||
"bin": {
|
||||
"uuid": "dist/bin/uuid"
|
||||
}
|
||||
},
|
||||
"node_modules/wrap-ansi": {
|
||||
"version": "7.0.0",
|
||||
"resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-7.0.0.tgz",
|
||||
"integrity": "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==",
|
||||
"dependencies": {
|
||||
"ansi-styles": "^4.0.0",
|
||||
"string-width": "^4.1.0",
|
||||
"strip-ansi": "^6.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=10"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/chalk/wrap-ansi?sponsor=1"
|
||||
}
|
||||
},
|
||||
"node_modules/wrappy": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
|
||||
"integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ=="
|
||||
},
|
||||
"node_modules/y18n": {
|
||||
"version": "5.0.8",
|
||||
"resolved": "https://registry.npmjs.org/y18n/-/y18n-5.0.8.tgz",
|
||||
"integrity": "sha512-0pfFzegeDWJHJIAmTLRP2DwHjdF5s7jo9tuztdQxAhINCdvS+3nGINqPd00AphqJR/0LhANUS6/+7SCb98YOfA==",
|
||||
"engines": {
|
||||
"node": ">=10"
|
||||
}
|
||||
},
|
||||
"node_modules/yargs": {
|
||||
"version": "17.7.3",
|
||||
"resolved": "https://registry.npmjs.org/yargs/-/yargs-17.7.3.tgz",
|
||||
"integrity": "sha512-GZtjxm/J/4TSxuL3FNYjCmLktBTnIw/rVmKSIyKeYAZpmJB2ig9VauCC5xsa82GNKVKDAqpOn3KVzNt0zmrU0g==",
|
||||
"dependencies": {
|
||||
"cliui": "^8.0.1",
|
||||
"escalade": "^3.1.1",
|
||||
"get-caller-file": "^2.0.5",
|
||||
"require-directory": "^2.1.1",
|
||||
"string-width": "^4.2.3",
|
||||
"y18n": "^5.0.5",
|
||||
"yargs-parser": "^21.1.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/yargs-parser": {
|
||||
"version": "21.1.1",
|
||||
"resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-21.1.1.tgz",
|
||||
"integrity": "sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw==",
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"name": "rootnode-lab-router",
|
||||
"version": "0.1.0",
|
||||
"description": "RootNode Academy Phase 4.1 — Lab Host Router. Runs on the same Docker host as student Labs; resolves an opaque routing token to a local container and reverse-proxies the Platform Gateway's data-plane traffic to it. See docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3.",
|
||||
"private": true,
|
||||
"main": "src/server.js",
|
||||
"scripts": {
|
||||
"start": "node src/server.js",
|
||||
"test": "node test/run.js"
|
||||
},
|
||||
"dependencies": {
|
||||
"dockerode": "^4.0.2",
|
||||
"http-proxy": "^1.18.1"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
// Token -> local container resolution, split out from server.js so it can be unit-tested
|
||||
// against a real (or fake) Docker client without needing TLS material or a bound port.
|
||||
// See server.js's header comment for the full design rationale.
|
||||
|
||||
const ROUTING_TOKEN_LABEL = 'rootnode.routing-token';
|
||||
|
||||
function tokenFingerprint(token) {
|
||||
return token ? `${token.slice(0, 6)}…(${token.length})` : '(none)';
|
||||
}
|
||||
|
||||
// Resolves a routing token to a live local container's reachable base URL, purely from local
|
||||
// Docker state — no Postgres, no Provisioner call. Returns null for "no such token" AND "token
|
||||
// belonged to a destroyed/replaced container" — both cases the caller treats identically (404).
|
||||
async function resolveTarget(docker, token) {
|
||||
if (!token || typeof token !== 'string' || token.length < 16) return null; // reject garbage before even hitting Docker
|
||||
const containers = await docker.listContainers({
|
||||
filters: JSON.stringify({ label: [`${ROUTING_TOKEN_LABEL}=${token}`], status: ['running'] }),
|
||||
});
|
||||
if (containers.length !== 1) return null; // 0 = unknown/destroyed; >1 should never happen (label is per-env-unique) but fail closed either way
|
||||
const info = containers[0];
|
||||
const networks = info.NetworkSettings && info.NetworkSettings.Networks;
|
||||
if (!networks) return null;
|
||||
const netName = Object.keys(networks)[0];
|
||||
const ip = netName && networks[netName].IPAddress;
|
||||
if (!ip) return null;
|
||||
return `http://${ip}:80`; // every Lab container listens on 80 internally — see docs/LAB_ARCHITECTURE.md
|
||||
}
|
||||
|
||||
module.exports = { resolveTarget, tokenFingerprint, ROUTING_TOKEN_LABEL };
|
||||
@@ -0,0 +1,103 @@
|
||||
// RootNode Academy — Lab Host Router (Phase 4.1).
|
||||
//
|
||||
// Runs on the SAME Docker host as student Labs (rn-lab-host-01 in production). Its only job:
|
||||
// take a request from the Platform Gateway carrying an opaque routing token, find the ONE
|
||||
// local container labeled with that token, and reverse-proxy the request to it. See
|
||||
// docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1-3 and docs/PRODUCTION_NETWORK_SECURITY.md.
|
||||
//
|
||||
// Deliberately narrow:
|
||||
// - Never talks to Postgres, the Provisioner, or anything off this host — it resolves purely
|
||||
// from local `docker ps` state (containers are labeled by the Provisioner at creation time,
|
||||
// see provisioner lifecycle.js), so a destroyed/reset environment's old token simply has no
|
||||
// matching container anymore. No separate "is this token still valid" check is needed.
|
||||
// - Only one route exists: proxy-by-token. No container listing, no Docker-API passthrough,
|
||||
// no endpoint that echoes back a container id, IP, or network name to the caller.
|
||||
// - Requires mTLS — the Node https server itself rejects any connection that doesn't present
|
||||
// a certificate signed by the configured CA (requestCert + rejectUnauthorized), so an
|
||||
// unauthenticated caller never reaches route-resolution logic at all. This is the ONLY
|
||||
// authentication; there is no bearer-token/API-key fallback (mTLS or nothing).
|
||||
// - Binds to an internal address only — this process has no idea what "public" means, that's
|
||||
// enforced by the firewall (infra/firewall/rn-lab-host-01.rules) and by never being given a
|
||||
// public-facing port mapping in the production Compose file. Defense in depth: even if the
|
||||
// firewall were misconfigured, mTLS still blocks anyone without the Gateway's client cert.
|
||||
|
||||
const fs = require('fs');
|
||||
const https = require('https');
|
||||
const httpProxy = require('http-proxy');
|
||||
const Docker = require('dockerode');
|
||||
const { resolveTarget, tokenFingerprint } = require('./resolve');
|
||||
|
||||
const PORT = Number(process.env.LAB_ROUTER_PORT) || 8443;
|
||||
const TLS_CA = process.env.LAB_ROUTER_TLS_CA;
|
||||
const TLS_CERT = process.env.LAB_ROUTER_TLS_CERT;
|
||||
const TLS_KEY = process.env.LAB_ROUTER_TLS_KEY;
|
||||
const ROUTE_TOKEN_HEADER = 'x-rootnode-route-token';
|
||||
|
||||
if (!TLS_CA || !TLS_CERT || !TLS_KEY) {
|
||||
console.error('FATAL: LAB_ROUTER_TLS_CA/CERT/KEY are not set. Refusing to start an unauthenticated Lab Host Router.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Always the local socket — the Router only ever inspects containers on ITS OWN host, never a
|
||||
// remote Docker daemon (that's the Provisioner's job, over a completely separate mTLS
|
||||
// connection — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md §10 "control plane vs data plane").
|
||||
const docker = new Docker({ socketPath: process.env.DOCKER_SOCKET || '/var/run/docker.sock' });
|
||||
|
||||
const proxy = httpProxy.createProxyServer({ ws: true });
|
||||
proxy.on('error', (err, req, res) => {
|
||||
console.error('[lab-router] proxy error:', err.message);
|
||||
if (res && res.writeHead && !res.headersSent) {
|
||||
res.writeHead(502, { 'Content-Type': 'application/json' });
|
||||
res.end(JSON.stringify({ error: 'lab_unreachable' }));
|
||||
} else if (res && res.destroy) {
|
||||
res.destroy();
|
||||
}
|
||||
});
|
||||
|
||||
async function handleRequest(req, res) {
|
||||
const token = req.headers[ROUTE_TOKEN_HEADER];
|
||||
let target;
|
||||
try {
|
||||
target = await resolveTarget(docker, token);
|
||||
} catch (e) {
|
||||
console.error('[lab-router] docker lookup failed:', e.message);
|
||||
res.writeHead(502, { 'Content-Type': 'application/json' });
|
||||
return res.end(JSON.stringify({ error: 'router_internal_error' }));
|
||||
}
|
||||
if (!target) {
|
||||
console.log(`[lab-router] unknown/stale token ${tokenFingerprint(token)} -> 404`);
|
||||
res.writeHead(404, { 'Content-Type': 'application/json' });
|
||||
return res.end(JSON.stringify({ error: 'unknown_environment' }));
|
||||
}
|
||||
proxy.web(req, res, { target });
|
||||
}
|
||||
|
||||
async function handleUpgrade(req, socket, head) {
|
||||
const token = req.headers[ROUTE_TOKEN_HEADER];
|
||||
let target;
|
||||
try {
|
||||
target = await resolveTarget(docker, token);
|
||||
} catch (e) {
|
||||
console.error('[lab-router] docker lookup failed (ws):', e.message);
|
||||
return socket.destroy();
|
||||
}
|
||||
if (!target) {
|
||||
console.log(`[lab-router] unknown/stale token ${tokenFingerprint(token)} -> ws rejected`);
|
||||
return socket.destroy();
|
||||
}
|
||||
proxy.ws(req, socket, head, { target });
|
||||
}
|
||||
|
||||
const tlsOptions = {
|
||||
ca: fs.readFileSync(TLS_CA),
|
||||
cert: fs.readFileSync(TLS_CERT),
|
||||
key: fs.readFileSync(TLS_KEY),
|
||||
requestCert: true,
|
||||
rejectUnauthorized: true, // the entire authentication mechanism — see file header
|
||||
};
|
||||
|
||||
const server = https.createServer(tlsOptions, handleRequest);
|
||||
server.on('upgrade', handleUpgrade);
|
||||
server.on('tlsClientError', (err) => console.warn('[lab-router] rejected unauthenticated/invalid TLS client:', err.message));
|
||||
|
||||
server.listen(PORT, () => console.log(`Lab Host Router listening on :${PORT} (mTLS required)`));
|
||||
@@ -0,0 +1,73 @@
|
||||
#!/usr/bin/env node
|
||||
// Fast, isolated unit test for resolve.js's token->container logic — no live Docker daemon, no
|
||||
// TLS material, no network. Uses a fake dockerode-shaped client so this runs in milliseconds
|
||||
// as part of every regression pass. The REAL cross-host proxying behavior (mTLS handshake,
|
||||
// actual container resolution, HTTP+WS traffic) is covered by the two-host simulation harness
|
||||
// instead — see infra/scripts/simulate-two-host/README.md and
|
||||
// docs/PHASE4_1_PREPRODUCTION_HARDENING.md §6, not duplicated here.
|
||||
|
||||
const assert = require('assert');
|
||||
const { resolveTarget, tokenFingerprint } = require('../src/resolve');
|
||||
|
||||
function fakeDocker(containers) {
|
||||
return {
|
||||
listContainers: async ({ filters }) => {
|
||||
const parsed = JSON.parse(filters);
|
||||
const wantLabel = parsed.label[0];
|
||||
const wantStatus = parsed.status[0];
|
||||
return containers.filter((c) => c.Labels && `${Object.keys(c.Labels)[0]}=${Object.values(c.Labels)[0]}` === wantLabel && c.State === wantStatus);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const runningContainer = {
|
||||
Labels: { 'rootnode.routing-token': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' },
|
||||
State: 'running',
|
||||
NetworkSettings: { Networks: { 'rootnode-lab-net-abc': { IPAddress: '172.20.0.5' } } },
|
||||
};
|
||||
|
||||
{
|
||||
const docker = fakeDocker([runningContainer]);
|
||||
const target = await resolveTarget(docker, 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa');
|
||||
assert.strictEqual(target, 'http://172.20.0.5:80', 'known running token resolves to its container');
|
||||
console.log('ok — known token resolves to the correct container address');
|
||||
}
|
||||
|
||||
{
|
||||
const docker = fakeDocker([runningContainer]);
|
||||
const target = await resolveTarget(docker, 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb');
|
||||
assert.strictEqual(target, null, 'unknown token resolves to nothing');
|
||||
console.log('ok — unknown token resolves to null (404 at the HTTP layer)');
|
||||
}
|
||||
|
||||
{
|
||||
// Simulates a destroyed/reset environment: the container that used to match this label is
|
||||
// gone (not present in the list at all — a real reset/destroy removes the old container).
|
||||
const docker = fakeDocker([]);
|
||||
const target = await resolveTarget(docker, 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa');
|
||||
assert.strictEqual(target, null, 'stale token (container destroyed) resolves to nothing');
|
||||
console.log('ok — stale/destroyed-environment token resolves to null, same as unknown');
|
||||
}
|
||||
|
||||
{
|
||||
const docker = fakeDocker([runningContainer]);
|
||||
for (const garbage of [undefined, null, '', 'short', 123, {}]) {
|
||||
const target = await resolveTarget(docker, garbage);
|
||||
assert.strictEqual(target, null, `garbage input ${JSON.stringify(garbage)} must not resolve`);
|
||||
}
|
||||
console.log('ok — garbage/malformed tokens are rejected before ever touching Docker');
|
||||
}
|
||||
|
||||
{
|
||||
assert.strictEqual(tokenFingerprint('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'), 'aaaaaa…(32)');
|
||||
assert.strictEqual(tokenFingerprint(undefined), '(none)');
|
||||
const fp = tokenFingerprint('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa');
|
||||
assert.ok(!fp.includes('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'), 'fingerprint never contains the full token — never logged in plaintext');
|
||||
console.log('ok — token fingerprint for logs never reveals the full token');
|
||||
}
|
||||
|
||||
console.log('\nAll lab-router unit tests passed.');
|
||||
}
|
||||
|
||||
main().catch((e) => { console.error(e); process.exit(1); });
|
||||
@@ -1,19 +1,31 @@
|
||||
# Phase 1 lab_hosts seed — applied to the DB at Provisioner startup (config.js
|
||||
# applyLabHostsSeed), not managed via a runtime API (no host-management endpoints in Phase 1,
|
||||
# matching "don't overbuild scheduling yet" — see docs/PHASE1_PLATFORM_FOUNDATION.md §4).
|
||||
# lab_hosts seed — applied to the DB at Provisioner startup (config.js applyLabHostsSeed), not
|
||||
# managed via a runtime API (no host-management endpoints yet — see
|
||||
# docs/PHASE1_PLATFORM_FOUNDATION.md §4, "don't overbuild scheduling yet").
|
||||
#
|
||||
# Exactly one enabled row for Phase 1. Swapping this for a remote tcp://<lab-host-vm>:2376
|
||||
# entry with TLS material is the entire code change needed to point at a real, separate Lab
|
||||
# Host VM in production — see docs/LAB_ARCHITECTURE.md §1 and correction D.
|
||||
# `endpoint`/`tls_*_path` — the Provisioner's CONTROL-plane connection to this host's Docker
|
||||
# daemon (create/start/stop/remove containers). Unchanged since Phase 1.
|
||||
#
|
||||
# `routing_mode`/`router_endpoint` — Phase 4.1 addition, the Gateway's DATA-plane path to a
|
||||
# running Lab on this host. See docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1 and
|
||||
# docs/PRODUCTION_NETWORK_SECURITY.md.
|
||||
# direct — Gateway reaches env.internal_ip:internal_port itself. Only correct when Gateway
|
||||
# and this lab_host's Docker daemon are on the same network namespace (dev,
|
||||
# single-host). This is the default and requires no router_endpoint.
|
||||
# router — Gateway reaches this lab_host's Lab Host Router (platform/lab-router) instead,
|
||||
# over mTLS, and never touches a Docker-internal address. Required once this
|
||||
# lab_host is a genuinely separate VM from the Gateway.
|
||||
|
||||
lab_hosts:
|
||||
- lab_host_id: dev-local
|
||||
display_name: "Local development Docker socket"
|
||||
endpoint: "unix:///var/run/docker.sock"
|
||||
enabled: true
|
||||
routing_mode: direct
|
||||
|
||||
# Production example (not active — shown for reference; uncomment and fill in when a real
|
||||
# Lab Host VM exists):
|
||||
# Production example (not active — shown for reference; uncomment and fill in once a real
|
||||
# Lab Host VM exists). CONTROL plane: Provisioner -> dockerd, mTLS, port 2376. DATA plane:
|
||||
# Gateway -> Lab Host Router, mTLS, router's own port (infra default 8443) — see
|
||||
# infra/docker/lab-host/docker-compose.production.yml and infra/ansible/roles/lab-router.
|
||||
# - lab_host_id: labhost-01
|
||||
# display_name: "RootNode Lab Host VM 01"
|
||||
# endpoint: "tcp://10.20.30.5:2376"
|
||||
@@ -21,3 +33,5 @@ lab_hosts:
|
||||
# tls_cert_path: /etc/rootnode/tls/labhost-01/cert.pem
|
||||
# tls_key_path: /etc/rootnode/tls/labhost-01/key.pem
|
||||
# enabled: true
|
||||
# routing_mode: router
|
||||
# router_endpoint: "https://10.20.30.5:8443"
|
||||
|
||||
@@ -20,3 +20,12 @@ health_check:
|
||||
path: /api/health
|
||||
timeout_ms: 240000 # k3s cold start in a nested container can be slow — matches the
|
||||
# ~60-attempt wait already in scripts/entrypoint.sh
|
||||
|
||||
# Phase 4.1 §26 — admission control. Disabled (no cap) by default, on purpose: don't fabricate
|
||||
# a concurrency number before it's measured against the real host (docs/CAPACITY_PLANNING.md).
|
||||
# Set max_concurrent_labs from infra/scripts/capacity-model.js's real hard-admission-control
|
||||
# output once the production host exists, so a full host rejects new Labs cleanly (503, "Lab
|
||||
# capacity is temporarily full") instead of failing mid-provision once Docker itself runs out
|
||||
# of room.
|
||||
# admission_control:
|
||||
# max_concurrent_labs: null
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
-- lab_hosts routing mode — Phase 4.1, docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1.
|
||||
--
|
||||
-- Removes the Gateway's dependency on env.internal_ip being reachable directly, which only
|
||||
-- holds when Gateway and the Lab container share a Docker daemon (true in dev, false once
|
||||
-- Labs run on a genuinely separate rn-lab-host-01 VM). A lab_host is either:
|
||||
-- 'direct' — Gateway reaches env.internal_ip:internal_port itself. Correct when Gateway and
|
||||
-- this lab_host's Docker daemon share a network namespace (dev, single-host).
|
||||
-- 'router' — Gateway reaches this lab_host's Lab Host Router (router_endpoint) over mTLS
|
||||
-- instead, carrying the environment's routing_token in a header; the Router
|
||||
-- resolves that token to a local container on ITS OWN host and reverse-proxies.
|
||||
-- Correct once this lab_host is a genuinely separate VM.
|
||||
--
|
||||
-- Existing 'dev-local' row keeps routing_mode='direct' (the default) — no behavior change for
|
||||
-- current dev/single-host setups. All existing lab_hosts rows and every environments row are
|
||||
-- untouched; this only adds columns.
|
||||
|
||||
ALTER TABLE lab_hosts ADD COLUMN IF NOT EXISTS routing_mode TEXT NOT NULL DEFAULT 'direct';
|
||||
ALTER TABLE lab_hosts ADD COLUMN IF NOT EXISTS router_endpoint TEXT;
|
||||
|
||||
DO $$ BEGIN
|
||||
ALTER TABLE lab_hosts ADD CONSTRAINT lab_hosts_routing_mode_check
|
||||
CHECK (routing_mode IN ('direct', 'router'));
|
||||
EXCEPTION
|
||||
WHEN duplicate_object THEN NULL;
|
||||
END $$;
|
||||
|
||||
DO $$ BEGIN
|
||||
ALTER TABLE lab_hosts ADD CONSTRAINT lab_hosts_router_endpoint_required_check
|
||||
CHECK (routing_mode = 'direct' OR router_endpoint IS NOT NULL);
|
||||
EXCEPTION
|
||||
WHEN duplicate_object THEN NULL;
|
||||
END $$;
|
||||
@@ -24,14 +24,19 @@ function loadLabHostsSeed() {
|
||||
async function applyLabHostsSeed(pool) {
|
||||
const hosts = loadLabHostsSeed();
|
||||
for (const h of hosts) {
|
||||
const routingMode = h.routing_mode || 'direct'; // Phase 4.1 — see migration 004's comment
|
||||
if (routingMode === 'router' && !h.router_endpoint) {
|
||||
throw new Error(`lab_host '${h.lab_host_id}' has routing_mode: router but no router_endpoint configured`);
|
||||
}
|
||||
await pool.query(
|
||||
`INSERT INTO lab_hosts (lab_host_id, display_name, endpoint, tls_ca_path, tls_cert_path, tls_key_path, enabled)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7)
|
||||
`INSERT INTO lab_hosts (lab_host_id, display_name, endpoint, tls_ca_path, tls_cert_path, tls_key_path, enabled, routing_mode, router_endpoint)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
|
||||
ON CONFLICT (lab_host_id) DO UPDATE SET
|
||||
display_name = excluded.display_name, endpoint = excluded.endpoint,
|
||||
tls_ca_path = excluded.tls_ca_path, tls_cert_path = excluded.tls_cert_path,
|
||||
tls_key_path = excluded.tls_key_path, enabled = excluded.enabled`,
|
||||
[h.lab_host_id, h.display_name, h.endpoint, h.tls_ca_path || null, h.tls_cert_path || null, h.tls_key_path || null, h.enabled !== false]
|
||||
tls_key_path = excluded.tls_key_path, enabled = excluded.enabled,
|
||||
routing_mode = excluded.routing_mode, router_endpoint = excluded.router_endpoint`,
|
||||
[h.lab_host_id, h.display_name, h.endpoint, h.tls_ca_path || null, h.tls_cert_path || null, h.tls_key_path || null, h.enabled !== false, routingMode, h.router_endpoint || null]
|
||||
);
|
||||
}
|
||||
return hosts.length;
|
||||
|
||||
@@ -40,6 +40,21 @@ async function getEnvironment(pool, environmentId) {
|
||||
return rows[0] || null;
|
||||
}
|
||||
|
||||
// Phase 4.1 — the Gateway's read path (GET /environments/by-token/:token) needs to know HOW
|
||||
// to reach the Lab, not just its Docker-internal address: is this lab_host 'direct' (Gateway
|
||||
// reaches env.internal_ip itself) or 'router' (Gateway must go through that lab_host's Lab
|
||||
// Host Router instead)? Joining lab_hosts here keeps that decision entirely data-driven —
|
||||
// the Gateway never hardcodes or guesses a host's routing mode.
|
||||
async function getEnvironmentByTokenWithRouting(pool, token) {
|
||||
const { rows } = await pool.query(
|
||||
`SELECT e.*, h.routing_mode AS lab_host_routing_mode, h.router_endpoint AS lab_host_router_endpoint
|
||||
FROM environments e JOIN lab_hosts h ON h.lab_host_id = e.lab_host_id
|
||||
WHERE e.routing_token = $1`,
|
||||
[token]
|
||||
);
|
||||
return rows[0] || null;
|
||||
}
|
||||
|
||||
async function listEnvironments(pool, studentId) {
|
||||
const { rows } = await pool.query(
|
||||
'SELECT * FROM environments WHERE student_id = $1 ORDER BY created_at DESC',
|
||||
@@ -90,6 +105,23 @@ async function provisionEnvironment({ pool, studentId, resourcePolicy, imageRef,
|
||||
throw Object.assign(new Error(`student '${studentId}' already has a non-terminal environment (${existing.environment_id})`), { code: 'ALREADY_EXISTS', existing });
|
||||
}
|
||||
|
||||
// Admission control (Phase 4.1 §26) — checked BEFORE touching Docker at all, so a full host
|
||||
// fails cleanly and fast (503, no partial network/container left behind to clean up) instead
|
||||
// of failing mid-provision with a generic 500 once Docker itself runs out of room. No-op
|
||||
// when maxConcurrentLabs isn't configured (see resourcePolicyFor's comment on why that's the
|
||||
// default).
|
||||
if (resourcePolicy.maxConcurrentLabs) {
|
||||
const { rows } = await pool.query(
|
||||
`SELECT count(*)::int AS n FROM environments WHERE status NOT IN ('destroyed', 'error')`
|
||||
);
|
||||
if (rows[0].n >= resourcePolicy.maxConcurrentLabs) {
|
||||
throw Object.assign(
|
||||
new Error('Lab capacity is temporarily full. Try again shortly.'),
|
||||
{ code: 'CAPACITY_EXCEEDED', current: rows[0].n, max: resourcePolicy.maxConcurrentLabs }
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const labHost = await selectLabHost(pool);
|
||||
const docker = clientFor(labHost);
|
||||
const routingToken = generateRoutingToken();
|
||||
@@ -145,6 +177,15 @@ async function provisionEnvironment({ pool, studentId, resourcePolicy, imageRef,
|
||||
const container = await docker.createContainer({
|
||||
name: containerName,
|
||||
Image: imageRef,
|
||||
// Phase 4.1: labeled by routing token (not environment_id alone) so the Lab Host Router
|
||||
// — which runs on this same Docker host and has no Postgres access — can resolve
|
||||
// "which local container does this opaque token address" purely from local `docker ps`,
|
||||
// with no cross-VM call back to the Provisioner needed at request time. See
|
||||
// docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1, platform/lab-router/src/router.js.
|
||||
Labels: {
|
||||
'rootnode.routing-token': routingToken,
|
||||
'rootnode.environment-id': env.environment_id,
|
||||
},
|
||||
HostConfig: {
|
||||
Privileged: true, // required for k3s — see docs/LAB_ARCHITECTURE.md §7, not avoidable
|
||||
NetworkMode: networkName,
|
||||
@@ -321,6 +362,13 @@ async function resetEnvironment({ pool, environmentId, resourcePolicy, egressCon
|
||||
const container = await docker.createContainer({
|
||||
name: containerName,
|
||||
Image: env.image_ref,
|
||||
// Phase 4.1 — same routing-token label as provisionEnvironment; a reset keeps the same
|
||||
// environment_id/routing_token (same DB row) so this must relabel the RECREATED container
|
||||
// the same way, or the Lab Host Router would resolve stale/no container for this token.
|
||||
Labels: {
|
||||
'rootnode.routing-token': env.routing_token,
|
||||
'rootnode.environment-id': environmentId,
|
||||
},
|
||||
HostConfig: {
|
||||
Privileged: true,
|
||||
NetworkMode: networkName,
|
||||
@@ -453,7 +501,7 @@ async function reconcileOnStartup(pool) {
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
generateRoutingToken, emitEvent, getEnvironment, listEnvironments, getActiveEnvironmentForStudent,
|
||||
generateRoutingToken, emitEvent, getEnvironment, getEnvironmentByTokenWithRouting, listEnvironments, getActiveEnvironmentForStudent,
|
||||
provisionEnvironment, destroyEnvironment, resetEnvironment,
|
||||
suspendEnvironment, resumeEnvironment, pollHealth, shortId, reconcileOnStartup,
|
||||
};
|
||||
|
||||
@@ -40,11 +40,18 @@ function egressConfig() {
|
||||
|
||||
function resourcePolicyFor() {
|
||||
const policy = getResourcePolicy().default;
|
||||
// Phase 4.1 §26 — optional; null/unset (the default) means no cap is enforced, matching the
|
||||
// Phase 4 rule against fabricating a concurrency number before it's measured. Set this from
|
||||
// infra/scripts/capacity-model.js's real output once the host exists (see
|
||||
// infra/environments/production/capacity.yml.example's hard-admission-control field) — see
|
||||
// docs/PHASE4_1_PREPRODUCTION_HARDENING.md §26.
|
||||
const admission = getResourcePolicy().admission_control || {};
|
||||
return {
|
||||
cpuLimit: policy.cpu_limit,
|
||||
memLimitMb: policy.mem_limit_mb,
|
||||
pidsLimit: policy.pids_limit,
|
||||
storageLimitMb: policy.storage_limit_mb,
|
||||
maxConcurrentLabs: admission.max_concurrent_labs || null,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -81,7 +88,10 @@ function isValidUUID(v) {
|
||||
function asyncRoute(fn) {
|
||||
return (req, res) => fn(req, res).catch((e) => {
|
||||
console.error(`[${req.requestId}]`, e);
|
||||
const status = e.code === 'NOT_FOUND' ? 404 : e.code === 'ALREADY_EXISTS' ? 409 : 500;
|
||||
// CAPACITY_EXCEEDED (Phase 4.1 §26) is 503, not 500 — the host being full is an expected,
|
||||
// recoverable operational state, not an application error. See lifecycle.js's admission
|
||||
// check, which runs BEFORE any Docker/network resources are touched.
|
||||
const status = e.code === 'NOT_FOUND' ? 404 : e.code === 'ALREADY_EXISTS' ? 409 : e.code === 'CAPACITY_EXCEEDED' ? 503 : 500;
|
||||
res.status(status).json({ error: e.message, code: e.code || 'INTERNAL', request_id: req.requestId });
|
||||
});
|
||||
}
|
||||
@@ -98,6 +108,14 @@ app.get('/health', asyncRoute(async (req, res) => {
|
||||
res.status(ok ? 200 : 503).json({ ok, db: dbOk, lab_hosts: hostChecks });
|
||||
}));
|
||||
|
||||
// GET /capacity — Phase 4.1 §26. Internal-only, same shared-secret auth as everything else
|
||||
// here — lets the Platform API's instructor dashboard show real capacity state instead of
|
||||
// instructors only discovering a full host when a student's launch fails.
|
||||
app.get('/capacity', asyncRoute(async (req, res) => {
|
||||
const { rows } = await getPool().query(`SELECT count(*)::int AS n FROM environments WHERE status NOT IN ('destroyed', 'error')`);
|
||||
res.json({ current: rows[0].n, max: resourcePolicyFor().maxConcurrentLabs });
|
||||
}));
|
||||
|
||||
// POST /environments — { student_id }
|
||||
app.post('/environments', asyncRoute(async (req, res) => {
|
||||
const { student_id } = req.body || {};
|
||||
@@ -130,9 +148,12 @@ app.get('/environments', asyncRoute(async (req, res) => {
|
||||
// owning student_id) before proxying. Internal-only, same shared-secret auth as every other
|
||||
// route here — the Gateway is the only caller, browsers never reach this directly.
|
||||
app.get('/environments/by-token/:token', asyncRoute(async (req, res) => {
|
||||
const { rows } = await getPool().query('SELECT * FROM environments WHERE routing_token = $1', [req.params.token]);
|
||||
if (!rows.length) return res.status(404).json({ error: 'not found', request_id: req.requestId });
|
||||
res.json(redactRow(rows[0]));
|
||||
// Phase 4.1: also joins lab_host_routing_mode/lab_host_router_endpoint so the Gateway can
|
||||
// decide direct-vs-router without knowing anything about Docker topology itself — see
|
||||
// lifecycle.getEnvironmentByTokenWithRouting and docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1.
|
||||
const env = await lifecycle.getEnvironmentByTokenWithRouting(getPool(), req.params.token);
|
||||
if (!env) return res.status(404).json({ error: 'not found', request_id: req.requestId });
|
||||
res.json(redactRow(env));
|
||||
}));
|
||||
|
||||
// GET /environments/:id/token — Phase 2A addition. Every other read path deliberately redacts
|
||||
|
||||
@@ -160,12 +160,16 @@ async function main() {
|
||||
});
|
||||
|
||||
await test('audit log: security events were actually recorded', async () => {
|
||||
// Phase 4.1 §16 — /api/admin/audit-events is now paginated (see platform/api/src/pagination.js);
|
||||
// the events land on the most recent page, but this test has produced fewer than one
|
||||
// page's worth by this point, so page 1's items are still every event so far.
|
||||
const { body } = await owner.request('GET', '/api/admin/audit-events');
|
||||
const actions = body.map((e) => e.action);
|
||||
const items = body.items;
|
||||
const actions = items.map((e) => e.action);
|
||||
assert.ok(actions.includes('login'));
|
||||
assert.ok(actions.includes('authz_denied'));
|
||||
assert.ok(actions.includes('instructor_update'));
|
||||
assert.ok(body.every((e) => !JSON.stringify(e.metadata || {}).match(/password/i)), 'audit metadata must never contain password material');
|
||||
assert.ok(items.every((e) => !JSON.stringify(e.metadata || {}).match(/password/i)), 'audit metadata must never contain password material');
|
||||
});
|
||||
|
||||
// ── Student session hardening ──────────────────────────────────────────────────────────
|
||||
|
||||
Reference in New Issue
Block a user