e2e-phase6a.js (21 checks): hierarchy/ordering, honest READY/PARTIAL/
PLANNED states, all three completion behaviors (manual marker/
scenario-derived/assessment-derived), prerequisite locking (including
the zero-content-is-vacuously-satisfied rule), resume, persistence
across logout/login, cross-student isolation, RBAC for self-paced
no-cohort students, draft-content hiding, and content-version
stability.
test-phase6a-course-rehearsal.js + run-phase6a-course-rehearsal.sh: a
real-k3s dress rehearsal of the gold-standard module end to end — a
student reads the real lesson content, launches a real disposable Lab,
the Lab's own POST /api/scenarios/:id/setup (called through the
Gateway's existing /lab/:token/* proxy exactly as the browser UI does)
stands up the broken api-gateway deployment, a real kubectl patch fixes
the readiness probe, and the module completes and unlocks the next one.
No mocked KubeKosh anywhere in this suite.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
- platform/test/e2e-phase5c.js (29 checks): activation-vs-confirmation routing,
duplicate/concurrent webhook idempotency, temporary/permanent/exhausted retry
classification, resend/regenerate token invalidation, activation/reset single-use and
expiration, HTML-injection regression, production token-secrecy gating via a real container
recreate with COOKIE_SECURE=1, minor/contact-email resolution, RBAC scoping, metrics/audit
hygiene, and a real Platform API restart.
- infra/scripts/drills/test-phase5c-enrollment-activation.js (10 checks): one continuous
story — website enrollment through a real rendered/extracted activation email, a real Lab
launch and scenario completion, a second enrollment's confirmation email, a full
password-reset round trip, and the minor/no-email contact-email path.
- infra/scripts/gws-email-test.js: the opt-in (RUN_GWS_EMAIL_TEST=true) live Google Workspace
delivery test — does nothing without the explicit flag, sends exactly one message to one
operator-configured recipient when enabled.
- platform/test/e2e-phase5b.js (32 checks): program seed, entitlement gating
(pending/future/expired/cancelled), cross-student isolation, real Lab launch/destroy/
reaper-reconciliation/recreate persistence, server-derived idempotent completion via a full
real capstone completion, the integration boundary (auth, idempotency, real concurrency,
unknown/inactive program rejection, minor/no-email path, no card data), RBAC scoping, a
regression test for the process-crash bug this suite found, Phase 5A smoke, and a real
Platform API restart.
- infra/scripts/drills/test-phase5b-selfpaced-rehearsal.js (15 checks): one continuous story —
activation, a simulated website enrollment event, real Lab work, Lab destroy, a new session,
Continue Learning, a fresh Lab, full real capstone completion, Completed transition, and a
second uninvolved student proven unable to reach any of it.
- platform/test/e2e-phase5a.js (21 checks): the two-student RBAC matrix (student can't reach
any /api/admin/* route, assigned/unassigned/viewer instructor access) and the full password
security checklist (correct/wrong/unknown-identifier/deactivated/immediate-deactivation/
rate-limit/rate-limit-survives-a-real-restart/change-password/session-revocation/
forgot-password anti-enumeration/admin-issued reset/CSRF), plus a classroom lifecycle smoke
pass. Replaces the throwaway smoke5a/smoke5b scripts used during development.
- infra/scripts/drills/test-phase5a-rehearsal.js (15 checks): 5 pilot students with real
seeded pre-existing progress/attempt/achievement history, a live classroom, help requests,
announcements, a REAL Platform API restart mid-rehearsal proving classroom/session/presence
recovery, then converting two students mid-rehearsal and confirming zero data was orphaned.
- Both run against a live docker-compose.dev.yml stack over real HTTP, not mocks.
- admin-routes.js: create-with-credentials, /convert, /reset-password, /revoke-sessions,
/sessions, /import (transactional CSV import — whole file validated, one Postgres
transaction, never accepts a password/hash column from input). Fixed GET
/api/admin/students to select an explicit safe column list instead of SELECT * — the new
password_hash/credential_token_hash columns must never be exposed through this route.
- server.js: /api/login now also blocks on credential_token_hash (not just password_hash) —
closes a real window where a student mid-conversion stayed reachable via the old
bare-student_id path. studentSession middleware now re-verifies status='active' on every
request, not just at session creation, matching instructorSession's existing behavior.
- sessions.js: revokeAllStudentSessions, listStudentSessions.
- rbac.js: extracted checkCohortAccess/checkCohortWriteAccess as shared core functions so the
new classroom/help-request RBAC (next commit) reuses the exact same authorization rule
instead of a parallel copy.
- student-csv.js + test: hand-rolled parser (no new dependency), unit tested.
- prometheus.yml.j2: activated the platform-api scrape job.
Master overview (PHASE4_1_PREPRODUCTION_HARDENING.md) covering all 27
substantive items, plus REMOTE_AND_SELF_PACED_ARCHITECTURE.md (reviews
what already works for remote/self-paced delivery - student-driven
provisioning, cohort-less students, idle-reaper-destroy-then-continue -
and the one real gap: student authentication, explicitly Phase 5 scope,
not built here).
Updated PRODUCTION_NETWORK_SECURITY.md (the Gateway-to-Lab gap is now
RESOLVED, with the full design writeup and a new control-plane-vs-data-plane
section), PRODUCTION_DEPLOYMENT.md (four-cert TLS bundle),
PRODUCTION_VALIDATION_CHECKLIST.md (Router mTLS check, admission control
checks), PHASE4_PRODUCTION_INFRASTRUCTURE.md and ENGINEERING_SOP.md and
LAB_ARCHITECTURE.md (stale 'known gap' pointers updated to point at the
resolution), BACKUP_DR.md (destructive drill), README.md and
platform/README.md (Phase 4.1 pointers, lab-router component, updated
known-limitations).
Also fixes a real pre-existing bug found while auditing what gets
committed: infra/.gitignore's patterns were written repo-root-relative
(e.g. "infra/terraform/...") but the file lives in infra/, so git resolved
them relative to THAT directory and none of them ever matched anything -
a real .terraform/ provider cache showed up as untracked when it should
have been ignored. Fixed by dropping the redundant leading "infra/" from
every pattern.
Status: PHASE 4.1 COMPLETE / PRE-PRODUCTION HARDENING COMPLETE / READY FOR
HARDWARE DEPLOYMENT.
- infra/scripts/simulate-two-host/: real two-Docker-network simulation
(Gateway genuinely isolated from any Lab bridge network) proving the
Lab Host Router fix - 13/13 real assertions (HTTP+WebSocket through the
Router, Alice/Bob isolation, garbage/destroyed tokens, mTLS-only Router
access).
- infra/scripts/drills/: Lab Host loss/rebuild (9/9), deployment rollback,
CI-fails-closed (6 required failure modes), fake cohort dress rehearsal
(14/14, one continuous narrative through the real app).
- infra/ci/gitea-workflows/build-and-test.yml: new security-scan job
(npm audit, gitleaks, hadolint, shellcheck, trivy - all free/open-source),
severity policy in docs/SECURITY_SCANNING.md (CRITICAL/HIGH blocks,
MEDIUM/LOW reported only). Gates build-images alongside regression.
Found and fixed two real pre-existing bugs via the CI-fails-closed drill:
staging-smoke-test.sh and production-smoke-test.sh both had a classic
set -e gotcha (curl ... && echo "OK" as one statement - a failing command
that isn't last in an && chain doesn't trigger -e) that let an
unreachable/broken endpoint fall through silently instead of failing the
smoke test.
platform/test/e2e-phase3a.js updated for the new paginated
/api/admin/audit-events response shape (Phase 4.1 pagination work).
infra/backup/test-destructive-restore-drill.sh - realistic academy data
(owner, admin instructor, cohort, 3 students, progress, attempts, an
assessment definition, achievements) seeded through the REAL Platform API,
encrypted-backed-up, the source Postgres container genuinely destroyed
(docker rm, not just left running alongside a second one), a fresh
Postgres created from nothing, restored, and the real Platform API booted
again against the restored database and used to log in as the restored
admin instructor over HTTP. Never touches the developer's real database.
Confirmed passing.
- Real server-side pagination (platform/api/src/pagination.js) on the audit
log (was a flat LIMIT 200), student attempt history (was LIMIT 100), and
assessment session history (was fully unbounded - the one genuine growth
risk found). Frontend updated (AuditView, AssessmentsView, StudentsView).
- Admission control: resource-policy.yaml's new (disabled-by-default)
admission_control.max_concurrent_labs, checked before touching Docker at
all, returns a clean 503 CAPACITY_EXCEEDED instead of failing mid-provision.
Relayed as a clean student-facing message, never a raw 500. New
GET /capacity endpoint feeds the instructor dashboard.
- cohort_instructors.role_in_cohort (existed since Phase 2B, never read) is
now enforced: teacher vs viewer, rbac.js's
requireCohortWriteAccess/requireStudentCohortWriteAccess.
- GET /api/version - app version, deployed Git SHA, latest migration as a
schema-version stand-in.
- Structured assessment authoring UI (AssessmentsView.jsx's PoolSpecEditor)
replaces the raw-JSON pool_spec textarea, backed by a new
GET /api/admin/scenario-catalog endpoint. Owner-only raw-JSON fallback
remains for anything the structured editor doesn't cover.
- NEEDS_ATTENTION_FAIL_THRESHOLD/STALE_DAYS moved to env-var configuration.
Full regression suite (e2e-phase2a/2b/3a/3b, 67/67) re-run clean after
these changes; frontend production build confirmed clean.
Adds platform/lab-router/ — a narrow service that runs on the Lab Host,
authenticates the Gateway via mTLS, and resolves an opaque routing token to
a local container purely from Docker label state (no Postgres access, no
callback to the Provisioner). Replaces the Gateway's env.internal_ip
assumption (only valid when Gateway and Lab share a Docker daemon) with a
data-driven direct/router decision per lab_host
(lab_hosts.routing_mode/router_endpoint, migration 004), resolved in one
place (platform/gateway/src/lab-endpoint.js's labTarget()) that every
proxy/fetch call site in server.js now goes through.
Extends the existing Docker TLS CA (infra/scripts/docker-tls/generate-docker-tls.sh)
with a second mTLS leaf pair (router-server/gateway-client) for this new
data-plane connection, deployed by the docker-tls Ansible role into a
separate directory from the control-plane pair. Firewall
(rn-lab-host-01.rules) replaces the old speculative wide port-range
allowance with one narrow rule: platform_ip -> the Router's port only.
dev-local stays routing_mode: direct — zero behavior change for local dev,
confirmed by the full existing regression suite passing unmodified.
boot-labs.js drives the real admin Prepare-Labs endpoint (reusing the
existing session-helper.js from Phase 3A) and records boot time/failure
rate; sample-resources.sh samples real docker stats for running Labs.
capacity-model.js turns measured per-Lab CPU/memory into a recommended/
warning/hard-admission-control concurrency threshold - no concurrency
number is hardcoded as a marketing figure, every input is explicitly
placeholder data until measured against the real host.
Plus deploy/rollback/smoke-test scripts used by the CI/CD and deployment
runbooks. All syntax-checked; validate-compose.sh and capacity-model.js
actually run against real inputs.
Automatic test/build/tag/staging-deploy on push to main; production
promotion stays a manual, human-run step (deploy-production.sh) - never
automated. SHA-tagged images always, semantic tags only on deliberate
release, never :latest alone in staging/production. No paid CI provider or
registry required (uses Gitea's own). Real regression job installs all
services, applies real migrations against a service Postgres, runs the
infra validation scripts.
Validated as real YAML (python3 yaml parse).
pg_dump -> gzip -> gpg AES-256 symmetric encryption, plaintext deleted
immediately, optional off-host rsync-over-SSH copy (Hetzner Storage Box or
any SSH target), retention pruning, Prometheus textfile-collector metrics
for backup success/age/size.
Genuinely tested end-to-end, not just written: real production migrations
(17 total, correct cross-service order) applied to a disposable Postgres,
sample data inserted, backed up, restored into a second disposable
Postgres, verified table-for-table and byte-for-byte. Exit 0.
Named-bridge-network production Compose (not host networking, unlike dev)
for rn-platform-01; egress-proxy + agents for rn-lab-host-01. Caddy reverse
proxy, single public origin, Grafana restricted to management_cidr.
iptables-restore firewall rulesets for all three hosts (Proxmox host,
rn-platform-01, rn-lab-host-01) codifying the Platform/Lab-Host boundary:
no student-to-student traffic, no direct Proxmox exposure, no direct
Postgres/Provisioner exposure from the Lab Host.
Mutual TLS for the Provisioner<->Lab Host Docker connection: certificate
generation/verification scripts, dockerd TLS config, firewall-restricted
port 2376. Application support already existed since Phase 1; this is
purely the operational half.
Validated: docker compose config, caddy validate, firewall structural lint,
and a real openssl s_server/s_client mTLS handshake proving the cert chain
and mutual-auth rejection (legitimate client verified, rogue client
correctly rejected with verify code 19).
common/firewall/docker-tls/platform/lab-host/monitoring-agent/backup roles,
orchestrated by site.yml with the ordering that matters (firewall/TLS land
before the services that depend on them). Ansible Vault for secrets, no new
tool introduced for this project's modest secret surface. Explicit
dev/staging/production environment separation, no secrets committed
(.gitignore covers vault plaintext, TLS keys, Terraform state, *.env).
Validated: ansible-playbook --syntax-check clean.
ced4568andClaude Sonnet 5