rootnode-academy/docs/ENGINEERING_SOP.md

# RootNode Academy — Engineering SOP

Living doc. Covers the "how" of building this platform — the "why" and business/curriculum
process live in the Business Pack (`~/Documents/CDAA Technologies/07 RootNode academy/RootNode_Academy_Business_Pack/`),
this doc doesn't duplicate that. Update this file in the same commit as the change it describes.

## 1. Where this actually lives

- **Repo:** `~/Dev/Gitea/ced4568/rootnode-academy` — remote is **Gitea**
  (`git.synthossystems.com/ced4568/rootnode-academy`), not GitHub.
- **Gotcha:** `~/Dev/GitHub/ced4568/kubekosh` is a *different* clone of the same original
  upstream fork, still pointed at GitHub, and it's behind (missing the RootNode logo commit
  and the Real Ops enforce-limits scenario as of this writing). It's not deployed anywhere.
  Don't edit it by mistake — always confirm `git remote -v` shows `gitea:` before making changes.
- Known-good baseline as of this doc: commit `6ddfb1c` ("add probe-port-mismatch and
  enforce-limits to rootnode-real-ops bundle"). Everything below builds on top of that,
  clean working tree.

## 2. Architecture at a glance

```
backend/server.js      Express API + SQLite (better-sqlite3) + WebSocket PTY terminal — the
                        "Lab image," single-tenant, unmodified by the Platform work below
frontend/               Vite + React SPA, built to frontend/dist and served by nginx
scenarios/data/         One JSON file per scenario (task or MCQ)
scenarios/bundles/      Flat scenario_id lists grouped by cert track (basics/appdev/admin/security/real-ops)
scenarios/rootnode-path.json   The RootNode guided learning path (staged, see §4)
scripts/entrypoint.sh   Container boot: starts k3s, nginx, node backend, sets the shell banner/PS1
Dockerfile              ubuntu:22.04 + k3s + node20 + nginx, single image, one port (80)

platform/provisioner/   Phase 1 Platform Provisioner — disposable per-student Lab lifecycle,
                        PostgreSQL registry, egress firewall. See §8 and
                        platform/provisioner/README.md.
platform/egress-proxy/  Shared allowlisting egress proxy every Lab environment's traffic must
                        pass through.
```

Single Express process, single SQLite file at `/data/progress.db` (or `$PROGRESS_DB`),
single embedded k3s cluster **per Lab environment**. The Lab image itself is still
single-tenant by design — see §5 for identity, and §8 for how Phase 1 multiplies it into a
real multi-student platform without modifying it.

## 3. Local dev / safe smoke-testing

Never test schema changes against the real `/data/progress.db` first. Point at a scratch file:

```bash
cd backend && npm install   # first time only
node --check server.js      # syntax check, catches typos fast
PROGRESS_DB=/tmp/scratch/progress.db node server.js &
curl -s http://localhost:4000/api/health
# ...exercise the endpoints you changed...
fuser -k 4000/tcp            # stop it
```

To test a migration against **real prior data** without touching prod: copy the actual
`/data/progress.db` out of the running container/volume, point `PROGRESS_DB` at the copy,
boot, and verify the data you expect is still there before ever pointing at the real file.

kubectl/k3s-backed endpoints (`/setup`, `/validate`, `/context`, the WS terminal) won't work
outside the container — that's expected. API-and-DB-only endpoints (roster, login, progress,
curriculum, bundles, scenarios list) all smoke-test fine on a bare host.

## 4. Curriculum content

Two separate structures, don't confuse them:

- **Bundles** (`scenarios/bundles/*.json`) — flat scenario_id lists, one per cert track.
  Loaded by `loadBundles()`/`GET /api/bundles`. Adding a bundle file here = adding a new
  cert-track library. Every scenario referenced must have a matching file in `scenarios/data/`.
- **RootNode path** (`scenarios/rootnode-path.json`) — the guided onboarding sequence,
  separate from bundles on purpose (it's *staged*, bundles are *flat*, and the bundle loader
  would choke on a staged file if it were dropped in `scenarios/bundles/`). Loaded by
  `loadCurriculumPath()`/`GET /api/curriculum/path` (raw content) and, as of Phase 3B,
  `computeCurriculumState()`/`GET /api/curriculum/state` (rule-driven lock/complete/next-scenario
  state — see `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` §4). Stages are Foundation → Workloads →
  Networking → Configuration → Troubleshooting → Real Ops → Capstone, built by reusing existing
  `1-k8s-basics` and `rootnode-real-ops` scenarios grouped by their existing `category` field —
  no new scenario content was written for this. Each stage carries a `completion_rule`
  (`all_required`/`threshold`/`prerequisites`). **Capstone is no longer a hardcoded placeholder**
  — it's `"type": "capstone"` with an `"unlock_rule"` and a `"capstone_assessment_id"` pointing
  at the `rootnode-capstone` assessment definition (Postgres, `assessment_definitions` table),
  which is where its actual scenario draw lives — see the Phase 3B doc §7. Don't put real
  scenarios directly into the JSON stage's `scenario_ids` for capstone; that field stays empty
  by design, the draw is defined in the assessment definition instead.

To add/reorder a stage: edit `rootnode-path.json` directly, referencing existing
`scenarios/data/*.json` ids. No backend code change needed — it's read fresh via
`POST /api/cache/reload` or on server restart.

## 5. Student identity — what's built, what isn't

**Current state (Phase 5A — see `docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md` for the
full design):**
- `students.student_id` (`student-NNN`, sequential) is still the FK every other table hangs off
  of and never changes across an account's lifecycle, including conversion.
- Real credentials: `email` and `login_identifier` (independently optional/unique — email for
  older/self-paced students, an instructor-assigned login ID for younger ones), `password_hash`
  reusing `instructors`' exact `scrypt` machinery (`src/auth/password.js`). A student with
  neither field set is a plain **pilot** account, unchanged since Phase 2A: `POST /api/login
  { student_id }`, no password, instructor-roster-only.
- First login/reset is one shared, single-use, hashed, time-limited token mechanism
  (`students-auth.js`) — instructors never see or set a student's actual password.
- Sessions are the same Postgres-backed `sessions` table instructors use (opaque, HttpOnly,
  hashed, revocable, CSRF-protected); login is rate-limited the same Postgres-backed way as
  instructor login (Phase 4.1), now also applied to students.
- Per-student `progress`/`scenario_attempts`/exam `sessions` isolation (below) and real
  concurrent multi-student Lab isolation (Phase 1's Provisioner/k3s-per-student model) were both
  already in place well before this — Phase 5A only changed how a student proves who they are,
  not how their data or Labs are isolated.

**Not built — deliberate non-goals, not gaps (Phase 5A §37):**
- No public self-registration — accounts are still instructor-created/converted/imported only.
- No email delivery infrastructure — password reset works via an instructor-issued token today;
  self-service reset returns a real token only in non-production, by design.
- No Parent/Guardian portal — the schema doesn't block adding one later, but none exists.

**Historical note, kept for context:** earlier drafts of this section described a pre-Phase-2A
state (`X-Student-Id` header, unauthenticated roster POST, one shared cluster) that no longer
reflects any deployed version of this codebase; that content has been superseded by the above.

## 6. Schema migration pattern

SQLite can't `ALTER` a primary key. The codebase's existing convention (see `progress` table
history) is:

1. `PRAGMA table_info(<table>)` → check if the new column/shape is already there (idempotent,
   safe to run on every boot).
2. Simple new nullable column → `ALTER TABLE ... ADD COLUMN`.
3. Primary-key or structural change → build `<table>_new` with the new shape, `INSERT ...
   SELECT` the old data across with sane defaults for new required columns, `DROP TABLE`,
   `RENAME TO`. Never drop data — give it a legacy/default value (`'unassigned'`, etc.)
   instead of discarding rows.
4. Test the migration against a **copy of real data** (§3) before it ever runs against
   `/data/progress.db` in the container.

## 7. Roadmap status

Order agreed 2026-08-17/18. Status as of this doc:

1. RootNode identity/branding pass — **partial.** Shell banner fixed
   (`⎈ RootNode Academy Lab — node: k8s-lab`). Upstream KubeKosh attribution left untouched in
   footer/README/LICENSE. Broader "make Real Ops first-class" UI work not started.
2. RootNode curriculum mode — **done**. Rule-driven stage completion/unlock as of Phase 3B
   (§12), student-facing tabs (My Learning/Practice/Assessments/Capstone/Achievements) at
   `frontend/src/platform/`.
3. Student accounts / persistent progress — **done**, Postgres-authoritative as of Phase 2B
   (§9), sessions persistent as of Phase 3A (§10). Real student credentials (email or
   login-identifier + password, first-login/reset token flow) closed the remaining "pick your
   name" gap as of Phase 5A — see §5 above and
   docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md. Pilot (no-login) accounts still work
   unchanged for anyone not yet converted.
4. Instructor dashboard — **done** (§9/§10). Full admin portal at `/admin`: dashboard, cohort
   and student management, classroom Prepare/Destroy Labs, per-instructor accounts with
   owner/admin/instructor roles, audit log.
5. Per-student isolation — **Phase 1 + Phase 2A + Phase 2B + Phase 3A done**, see §8/§9/§10.
   Ephemeral per-student Lab environments, Gateway-mediated routing with persistent
   server-controlled sessions, a full instructor/admin control plane with real per-person
   accounts and RBAC, and a working entry→dashboard→launch→lab→progress flow are implemented
   and end-to-end tested on a single dev machine, including a 3-student cohort classroom
   workflow and a real Platform API restart mid-session. **Still not cleared for minors** — the
   production two-VM Proxmox split this depends on doesn't exist yet, and OIDC/SSO is a
   documented adapter, not a working integration (docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md
   §16).
6. Scenarios → teaching modules (incident context/investigation/prevention) — not started.
7. Capstone/exam engine — **done** as of Phase 3B (§12). RootNode Capstone (Real Ops-weighted,
   unlockable by curriculum-completion rule, not a permanent placeholder) plus a general exam
   engine (definitions, cohort assignment, random scenario draw, server-authoritative timer).
8. Progressive hint levels — **done** as of Phase 3B (§12). Server-enforced sequential reveal,
   tracked per attempt, policy-aware under assessment mode (free/disabled/flagged).
9. Production observability — not started.
10. CI scenario testing — not started.
11. Container/build optimization — explicitly deprioritized by design (platform works, not urgent).

Next slice: production OIDC/SSO, the production two-VM Proxmox deployment, and teaching-module
scenario content are the biggest remaining items. See
`docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` §21 for the full Phase 4 boundary.

## 8. Platform Provisioner (Phase 1)

Full design: `docs/LAB_ARCHITECTURE.md` + `docs/PHASE1_PLATFORM_FOUNDATION.md`. Operational
reference (config, local dev, known limitations): `platform/provisioner/README.md`. This
section is just the map.

- **What it is:** an internal-only service that provisions a disposable, fully isolated Lab
  environment per student — its own Docker network, its own container running the existing,
  unmodified Lab image, its own firewall-enforced egress. Postgres-backed registry
  (`environments`, `environment_events`, `lab_hosts`), idle reaper, startup reconciliation.
- **What it is not:** directly reachable by a student or the Gateway anymore — as of Phase 2B,
  the Platform API (§9) is the *only* caller (student launch/reset AND admin prepare/destroy/
  reset both go through it). Still exercised directly via its internal API (shared-secret
  auth) or `platform/provisioner/test/run.js` for isolated Provisioner testing.
- **Where identity lives:** as of Phase 2B, Postgres is authoritative —
  `students`/`cohorts`/`progress`/`scenario_attempts` live in the same `rootnode_platform`
  database as `environments`, owned by the Platform API's own migrations. See §9. As of Phase
  3A, `instructors`/`sessions`/`cohort_instructors`/`audit_events` live there too. See §10.
  `environments.student_id` is still a plain opaque reference, not a real FK — that's a
  deliberate, documented choice (docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md §2), not an oversight.
- **Local dev:** `platform/provisioner/docker-compose.dev.yml` runs Postgres + the egress
  proxy + the Provisioner on one machine — the documented Phase 1 dev exception. Production
  is a two-VM Proxmox split (Platform VM / Lab Host VM); the dev compose file is explicitly
  not that.
- **Test procedure:** `platform/provisioner/test/run.js` — 12 automated integration tests
  against a live dev stack (isolation, firewall ordering, reset/destroy cleanup, audit
  events). Some properties (idle reaper timing, remote TLS lab-host, reconciliation-on-crash,
  real-image k3s behavior) were validated manually during implementation rather than
  automated — see the Phase 1 final report for what and why.

## 9. Platform API (Phase 2B)

Full design: `docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md`. Operational reference:
`platform/api/README.md`.

- **What it is:** `platform/api/` — a lightweight Node/Express service (no k3s, no
  `--privileged`, no Lab terminal runtime) owning students, cohorts, enrollments, curriculum
  assignments, progress, scenario attempts, and — as of Phase 3A — instructor accounts,
  persistent sessions, and the audit log (§10). The sole caller of the Provisioner for both the
  student launch/reset path and the instructor/admin control plane.
- **What it is not:** reachable directly by a browser — always through the Gateway
  (`platform/gateway`), same trust boundary the Provisioner has relative to this service.
- **Local dev:** part of the same `platform/docker-compose.dev.yml` stack as the Provisioner —
  no separate compose file.
- **Test procedure:** `platform/test/e2e-phase2a.js` (student flow) and
  `platform/test/e2e-phase2b.js` (classroom acceptance workflow) — both need a bootstrapped
  instructor account as of Phase 3A, see §10.

## 10. Identity and instructor access (Phase 3A)

Full design: `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md`. Operational reference:
`platform/api/README.md` (bootstrap, config).

- **What it is:** persistent Postgres-backed sessions (student + instructor, survive a Platform
  API restart), real per-instructor accounts (`owner`/`admin`/`instructor` roles, scrypt
  password hashing, per-instructor salt), server-enforced RBAC on every `/api/admin/*` route,
  cohort-scoped access for the plain `instructor` role, CSRF double-submit protection, login
  rate limiting, and a platform-level `audit_events` log.
- **What it replaced:** Phase 2B's in-memory session `Map`s (gone on every restart) and the
  single shared `ADMIN_PASSWORD` env var (no accountability, no roles, no revocation).
- **Bootstrap:** `node platform/api/bootstrap-owner.js <email> <name> <password>` — the only
  supported way to create the first instructor account; no manual database editing.
- **Test procedure:** `platform/test/e2e-phase3a.js` — 21 automated tests covering session
  persistence (including a real `docker compose restart api`), instructor auth, RBAC, cohort
  scoping, CSRF, rate limiting, password change, and audit logging.
- **Known limitations, future OIDC path, production HTTPS expectations:** see
  `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md` §15–17.

## 11. Dependency lock convention

Every first-party Node service commits its `package-lock.json`: `backend/`, `frontend/`,
`platform/api/`, `platform/gateway/`, `platform/provisioner/`. This wasn't consistently true
before Phase 3B — `platform/api/`'s lockfile existed locally but was never committed, and
`platform/gateway/`/`platform/provisioner/` never had one generated at all. Neither was a
deliberate policy; both were oversights, fixed in Phase 3B (`chore(deps)` commits).

Going forward: after any `npm install` that changes a service's dependencies, the updated
`package-lock.json` is part of the same commit as the `package.json` change — never left
untracked, never regenerated-and-ignored. Validate a lockfile actually matches its
`package.json` with `npm ci` (fails loudly on drift) before committing either.

## 12. Learning & Assessment Engine (Phase 3B)

Full design: `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`.

- **What it is:** the teaching/assessment layer on top of everything Phases 1–3A built —
  detailed per-attempt telemetry (`scenario_attempts`, evolved from Phase 2B's one-row-per-click
  shape into a real attempt lifecycle), server-enforced progressive hints, rule-driven
  curriculum stage/capstone unlock logic, a full exam + capstone assessment engine
  (`assessment_definitions`/`assessment_sessions`/`assessment_session_scenarios`), and
  persistent `achievements` records.
- **Where the boundary moved:** the Gateway now rewrites one Lab response
  (`GET /lab/:token/api/scenarios/:scenarioId`, stripping `hints`) so progressive hints can be
  genuinely server-gated rather than just UI-collapsed — the Lab image's own code/bytes are
  still untouched; this is a response rewrite at the Gateway boundary, same pattern as every
  other Lab-mediation decision made in earlier phases. The Lab's own frontend bundle *did*
  change this phase (`ScenarioPanel.jsx`, new `ProgressiveHints`/`AssessmentBanner` components)
  — Phase 3B explicitly authorized enhancing Lab UX, unlike prior phases.
- **Test procedure:** `platform/test/e2e-phase3b.js` — covers attempt telemetry, hint
  sequencing/persistence-through-Lab-reset, curriculum/capstone lock-unlock, the exam engine
  (random draw, timer, expiry, restart persistence), two-student isolation, and instructor
  analytics/RBAC.
- **Known limitations:** see the Phase 3B doc §20 — no automatic abandoned-attempt detection, no
  per-cohort needs-attention threshold configuration, raw-JSON pool_spec authoring in the admin
  UI (correct, not yet a guided form).

## 13. Production Infrastructure (Phase 4)

Full design: `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`, plus detail docs
`PRODUCTION_DEPLOYMENT.md`, `PRODUCTION_NETWORK_SECURITY.md`, `BACKUP_DR.md`,
`OBSERVABILITY.md`, `CAPACITY_PLANNING.md`, `PRODUCTION_VALIDATION_CHECKLIST.md`, and 12
`RUNBOOK_*.md` files, all under `docs/`. Everything lives in `infra/` (Ansible, Terraform,
Compose, firewall, monitoring, backup, CI, load-test tooling).

- **What it is:** the infrastructure-as-code needed to run rn-platform-01/rn-lab-host-01 on a
  real Hetzner/Proxmox host — buildable and locally testable now, even though the dedicated
  server hasn't been purchased. Status is explicitly **PHASE 4 BUILD COMPLETE / PRODUCTION
  DEPLOYMENT PENDING HARDWARE**, not "deployed."
- **Where the boundary held:** the Platform VM / Lab Host VM security split from
  `docs/LAB_ARCHITECTURE.md` is codified in firewall rulesets and mutual-TLS Docker config, not
  weakened anywhere.
- **Known gap surfaced, then resolved in Phase 4.1:** the Gateway's Lab-reachability mechanism
  didn't survive the move to two real separate VMs — flagged explicitly rather than silently
  fixed, then actually built and simulation-tested (a Lab Host Router,
  `platform/lab-router/`) in Phase 4.1. See §13 below.
- **Validated without the hardware:** real `tofu validate`, `ansible-playbook --syntax-check`,
  `docker compose config`, `caddy validate`, `promtool`/`amtool`, and a genuine end-to-end
  backup→encrypt→restore→verify cycle against disposable local Postgres with the real
  production migrations applied. None of this proves the real Hetzner network boundary — that
  needs `docs/PRODUCTION_VALIDATION_CHECKLIST.md` walked against the real server.

## 13. Pre-Production Hardening (Phase 4.1)

Full design: `docs/PHASE4_1_PREPRODUCTION_HARDENING.md`, plus
`docs/PRODUCTION_NETWORK_SECURITY.md` (routing resolution), `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`,
`docs/SECURITY_SCANNING.md`.

- **What it is:** closes the Gateway-to-Lab cross-VM gap Phase 4 flagged (a Lab Host Router,
  `platform/lab-router/`, plus a routing abstraction in the Gateway,
  `platform/gateway/src/lab-endpoint.js`), then hardens the application layer found along the
  way: abandoned-attempt handling (`scenario_attempts.status = 'abandoned'`, never a punitive
  failure), a Postgres-backed login rate limiter (survives a restart, ready to be
  shared/multi-instance), real pagination on audit/attempt/assessment-session history, a
  structured assessment authoring UI (owner-only raw JSON remains as a fallback), a minimal
  `role_in_cohort` enforcement (`teacher`/`viewer`), a `/api/version` endpoint, and admission
  control (a clean 503 "capacity full" instead of a 500 once a configured cap is reached).
- **Tested, not just designed:** a real two-Docker-network routing simulation
  (`infra/scripts/simulate-two-host/`, 13/13), a destructive Postgres restore drill
  (`infra/backup/test-destructive-restore-drill.sh`), a Lab Host loss/rebuild drill
  (`infra/scripts/drills/test-lab-host-loss.js`), a deployment rollback drill, a CI-fails-closed
  drill (which found and fixed a real pre-existing `set -e` bug in both smoke-test scripts —
  see that drill's own output), and a full fake-cohort dress rehearsal
  (`infra/scripts/drills/test-fake-cohort-rehearsal.js`) — all passing, all in
  `infra/scripts/drills/` or `infra/backup/`.
- **Known limitation carried forward:** real remote/self-paced delivery needs actual student
  authentication (today's `student_id`-only login relies on instructors controlling who gets a
  valid id) — see `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md` for the full accounting of what
  already works vs. what's genuinely Phase 5 scope. **Closed in Phase 5A, below.**

## 14. Student Identity & Remote Classroom (Phase 5A)

Full design: `docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md`, plus
`docs/RUNBOOK_STUDENT_ACCOUNT_LIFECYCLE.md` and `docs/RUNBOOK_REMOTE_CLASSROOM.md`.

- **What it is:** two deliverables built on the Phase 4.1 milestone. (A) Real student accounts —
  email-or-login-identifier + password, reusing `instructors`' exact credential machinery, a
  single first-login/reset token mechanism, deactivation enforced per-request — replacing pilot
  "pick your name" identification without touching a single existing pilot student's progress.
  (B) A remote classroom control center — scheduled/live/ended sessions, derived (not stored)
  presence via heartbeat, a live instructor dashboard combining real Provisioner Lab state and
  real academic records, a help-request queue, guidance-only announcements/current-activity, and
  a generic external meeting-URL field. Explicitly **not** video/voice/chat/shell-takeover —
  RootNode owns identity and coordination, an external service owns the call.
- **Tested, not just designed:** `platform/test/e2e-phase5a.js` (21 checks: the two-student RBAC
  matrix and full password-security checklist, including a real Platform API restart proving
  rate-limit state survives it) and `infra/scripts/drills/test-phase5a-rehearsal.js` (15 checks:
  5 pilot students with seeded real history, a live classroom, help requests, **a real
  mid-rehearsal API restart** proving classroom/session/presence recovery, then converting two
  students and confirming zero orphaned data) — both against a live `docker-compose.dev.yml`
  stack, not mocks. The full pre-existing regression suite was re-run given how much shared code
  this phase touched (`server.js`, `admin-routes.js`, `sessions.js`, `rbac.js`).

## 15. Self-Paced Learning & Enrollment Foundation (Phase 5B)

Full design: `docs/PHASE5B_SELF_PACED_ENROLLMENT.md`, plus
`docs/RUNBOOK_SELF_PACED_ENROLLMENT.md` and `docs/WEBSITE_ENROLLMENT_INTEGRATION.md`.

- **What it is:** `programs` (a first-class learning product, distinct from curriculum JSON —
  `curriculum_path` is just what it points at) and `enrollments` (a student's academic access to
  a program, independent of cohort membership — a self-paced enrollment has no cohort at all).
  Server-enforced entitlement (`enrollment.deriveEnrollmentAccess`, mirroring `classroom.js`'s
  derived-not-stored `derivePresence` pattern from Phase 5A) gates a new
  `POST /api/student/programs/:id/launch` in front of the exact same disposable-Lab sequence
  `/api/environment/launch` already used (extracted into `environment-view.js` so both share one
  implementation) — pending/future/expired/cancelled enrollments cannot launch, server-side, not
  just hidden in the UI. Completion is entirely server-derived from the existing capstone
  achievement mechanism (Phase 3B, unmodified) and idempotent. A new, separate
  `POST /api/integrations/enrollments` boundary (its own pre-shared-secret auth, no session
  cookie involved — same trust-boundary pattern as Gateway↔Provisioner's
  `PROVISIONER_SHARED_SECRET`) is what the *separate* public website will eventually call;
  idempotent via a DB unique constraint on the caller's own event id, proven safe under real
  concurrent duplicate requests, not just sequential retries.
- **A real bug found and fixed:** `POST /api/progress/record` predated the `wrap()`
  async-error-handling convention — an unexpected internal error there (a malformed
  `environment_id`) crashed the entire Platform API process, not just that one request. Fixed
  (wrapped + input-validated) and regression-tested.
- **Tested, not just designed:** `platform/test/e2e-phase5b.js` (32 checks) and
  `infra/scripts/drills/test-phase5b-selfpaced-rehearsal.js` (15 checks) — both against a live
  stack, including real Lab destroy/reaper-reconciliation/recreate persistence proofs, a full
  real capstone completion, integration-endpoint concurrency, and a real Platform API restart.
  Migration safety proven explicitly against a fresh Postgres seeded with realistic pre-5B data
  (see that doc §14).

## 16. Enrollment Operations & Communications (Phase 5C)

Full design: `docs/PHASE5C_ENROLLMENT_COMMUNICATIONS.md`, plus
`docs/RUNBOOK_GOOGLE_WORKSPACE_EMAIL.md` and `docs/RUNBOOK_STUDENT_ACTIVATION.md`.

- **What it is:** a durable, provider-neutral notification pipeline (`notification_events`, a
  `FOR UPDATE SKIP LOCKED` worker polling on a plain `setInterval` — same pattern as the
  existing session/login-attempt sweepers, no new infrastructure) that finally sends real email
  for activation, enrollment confirmation, password reset, and password-changed — closing the
  gap Phase 5A explicitly left open. Google Workspace adapter: Gmail API via a service account
  with domain-wide delegation (no human password, no SMTP basic auth — see that doc §2 for why),
  behind the same provider interface a deterministic dev provider implements, so no test or dev
  run ever reaches a real inbox. Reuses Phase 5A's activation/reset token mechanism entirely — no
  competing invitation-token system.
- **A real bug found and fixed:** `request-password-reset` would mint a reset token for a
  never-activated student, which doesn't make sense (there's no password yet to reset) and would
  have sent a confusing email. Fixed to fall through to the same anti-enumeration response as a
  nonexistent account.
- **Tested, not just designed:** `platform/test/e2e-phase5c.js` (28 checks: activation vs.
  confirmation routing, duplicate/concurrent webhook idempotency, temporary/permanent/exhausted
  retry classification, resend/regenerate token invalidation, production token-secrecy gating
  via a real container recreate with `COOKIE_SECURE=1`, minor/contact-email resolution, RBAC
  scoping, metrics/audit hygiene, a real restart) and
  `infra/scripts/drills/test-phase5c-enrollment-activation.js` (10 checks: one continuous story
  from website enrollment through a real rendered/extracted activation email, a real Lab launch
  and scenario completion, a second enrollment's confirmation email, and a full password-reset
  round trip) — both against a live stack, using only the dev provider (no real network calls).
  Live Google Workspace delivery itself is implemented and unit-level ready
  (`infra/scripts/gws-email-test.js`, opt-in) but not yet validated against a real Workspace
  domain — see that doc's final section.