Adds platform/test/e2e-phase6d.js (20 checks): ROOT 01 hierarchy, ROOT 02-04 honestly empty
(not padded), activity ordering, lab-cannot-be-manually-completed, full course completion via
lesson/lab/assessment, the prerequisite chain proven on two students (one who completed ROOT 01,
one fresh), existing ROOT 05+/Capstone state unaffected, is_required unaffected, self-paced
no-instructor-deferral check, instructor-notes exclusion, RBAC, logout/login persistence,
cross-student isolation, version-bump stability, and content validation. 20/20 pass against both
a warm and a freshly-reset dev stack.
Adds infra/scripts/drills/{run,test}-phase6d-foundations-rehearsal — the real disposable-Lab
dress rehearsal: login -> enroll -> lesson/guided-exercise -> real Lab launch -> real bash
hardware inspection (nproc/free/df, written to a real file) -> real validation -> a genuine
permission denial (a second Linux user actually denied via `su -`) then a real chmod fix ->
remaining modules -> assessment -> course completion -> ROOT 02 unlock -> logout/login
persistence -> instructor visibility of the resulting learning state. 11/11 pass against a real
Lab container, no mocking. (First run caught a real triple-escaping bug in the hardware-report
step's inline heredoc — fixed by writing the script to a file and docker-cp-ing it in instead.)
Fixes two stale hardcoded assumptions in e2e-phase6a.js and e2e-phase6b.js that this phase's own
ROOT 01 addition broke: both suites asserted `resume` landed on ROOT 05's gold module as "the
first real content," which was true before this phase and is now ROOT 01. Both suites' original
intent (resume must never land on a bare-lab bank) is preserved, just against the current front
door. Both suites re-verified green (21/21 and 24/24) against a fresh stack.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Adds platform/test/e2e-phase6c.js (17 checks): upgraded-module activity
chains, documented-exception promotions, optional-lab non-blocking behavior,
no-fake-completion-from-viewing, idempotent completion, instructor-notes
exclusion and RBAC on new content, security-assessment pool/draw/cross-student
isolation/completion, the capstone gate's documented indifference to ROOT 09,
ROOT 11's honest partial/lesson-only rendering, and version-bump content
stability — plus running validate-course-content.js as part of the suite.
All 17 pass against both a warm and a freshly-reset dev stack.
Adds infra/scripts/drills/{run,test}-phase6c-security-config-rehearsal —
Section 22's real-k3s dress rehearsal: login -> program -> course -> lesson ->
demonstration -> knowledge-check -> guided-exercise -> real Lab -> real
kubectl (create Secret, apply Pod, wait for Ready) -> validation -> progress
persistence across logout/login -> instructor visibility of the resulting
learning state, walking root-07-m02-configuration-and-secrets (upgraded from
PARTIAL this phase). 10/10 pass against a real k3s Lab, no mocking.
Fixes two stale hardcoded assumptions in e2e-phase6a.js and e2e-phase6b.js
that Phase 6C's own legitimate content changes broke: the gold module
(root-10-m01) now has 7 activities (6 required + 1 new optional lab, not 6),
and root-05-m01 now has 2 required manual-marker activities the course-
completion path must also complete (its new bookending lesson/reflection).
Both suites are back to green (21/21 and 24/24) against a fresh stack.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Upgrades six PARTIAL modules to the gold-module standard (lesson/demonstration/
knowledge-check/guided-exercise/lab(s)/reflection), preserving every existing lab
activity_id and scenario_id: root-06-m03 (jobs/cronjobs/daemonsets), root-07-m04
(ingress/networkpolicy), root-07-m02 (configuration/secrets, converted in place
from a bare 5-lab bank), root-09-m02 (serviceaccounts/rbac), root-09-m03
(network/supply-chain security), root-10-m03 (troubleshooting framework).
root-05-m01 (20-task kubectl-fluency bank) gets a minimal bookending lesson +
reflection rather than a full redesign, since each task already carries its own
explanation. root-12-m01, root-capstone-m01, and root-09-m04 are promoted to
READY via documented exceptions (no new activities) where adding one would
misrepresent their intentional design.
Wires a new rootnode-security-assessment (exam, 60min, hint_policy disabled)
using the existing Phase 3B assessment engine: 3 required hands-on CORE
scenarios + 4 random-draw pools covering SecurityContext, RBAC, network
policy, and the 7 previously-unwired CKS MCQs. Wires the previously-unwired
readiness-probe-task scenario as an OPTIONAL practice lab in root-10-m01,
using the existing is_required model.
Adds root-11-m01-devops-platform-thinking — an honest, lesson-only ROOT 11
orientation module naming DevOps/platform engineering practices already
exercised in ROOT 05-10/12, explicit that GitOps/CI-CD/IaC/observability need
lab infrastructure this platform doesn't yet provide. No fake labs.
Adds infra/scripts/validate-course-content.js: cross-checks the DB against
scenario/content files for duplicate ids, orphans, invalid references, cycles,
sequence collisions, instructor-notes leakage, and (new) self-paced content
deferring explanation to a live instructor. 0 errors against the resulting
22 courses/modules, 163 activities, 98 scenario files.
No application code changed — every gap was satisfiable with existing tables
(course_modules, module_activities, assessment_definitions) and the existing
content-delivery pipeline.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
e2e-phase6b.js (24 checks): hierarchy for the 13 new modules across
ROOT 05-10/12/Capstone, prerequisite chains through the newly-inserted
front-door modules, lesson/knowledge-check completion in brand-new
modules, a newly-wired scenario (job-task) completing correctly, the
optional-vs-required (is_required) completion rule, persistence/
isolation/RBAC/draft-hiding proven against the new content, and the
capstone server-side unlock fix in both directions (a direct bypass
attempt rejected with blocking_stage_ids, an ordinary exam unaffected,
a legitimately unlocked attempt still accepted and its completion
reflected in the course content model).
test-phase6b-root-course-rehearsal.js + wrapper (11 checks, real k3s):
one continuous student narrative through a brand-new module (ROOT 07
Services & Service Discovery, not the Phase 6A gold module) — real Lab
launch, the Lab's own setup_commands, a real 'kubectl expose' against
the live cluster, real validate, logout/login/Resume Learning, Lab
destroy/recreate with academic progress intact — plus an instructor
narrative proving Curriculum view, Prepare Labs, and classroom/learning
tooling all still work against the expanded curriculum.
e2e-phase6a.js updated for two of this phase's own intentional,
documented behavior changes (not defects): resume now points at
root-05's new mental-model module instead of the old bare-lab bank, and
root-10's gold module now unlocks the new troubleshooting-framework
module rather than the bare-lab bank directly.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
e2e-phase6a.js (21 checks): hierarchy/ordering, honest READY/PARTIAL/
PLANNED states, all three completion behaviors (manual marker/
scenario-derived/assessment-derived), prerequisite locking (including
the zero-content-is-vacuously-satisfied rule), resume, persistence
across logout/login, cross-student isolation, RBAC for self-paced
no-cohort students, draft-content hiding, and content-version
stability.
test-phase6a-course-rehearsal.js + run-phase6a-course-rehearsal.sh: a
real-k3s dress rehearsal of the gold-standard module end to end — a
student reads the real lesson content, launches a real disposable Lab,
the Lab's own POST /api/scenarios/:id/setup (called through the
Gateway's existing /lab/:token/* proxy exactly as the browser UI does)
stands up the broken api-gateway deployment, a real kubectl patch fixes
the readiness probe, and the module completes and unlocks the next one.
No mocked KubeKosh anywhere in this suite.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
- platform/test/e2e-phase5c.js (29 checks): activation-vs-confirmation routing,
duplicate/concurrent webhook idempotency, temporary/permanent/exhausted retry
classification, resend/regenerate token invalidation, activation/reset single-use and
expiration, HTML-injection regression, production token-secrecy gating via a real container
recreate with COOKIE_SECURE=1, minor/contact-email resolution, RBAC scoping, metrics/audit
hygiene, and a real Platform API restart.
- infra/scripts/drills/test-phase5c-enrollment-activation.js (10 checks): one continuous
story — website enrollment through a real rendered/extracted activation email, a real Lab
launch and scenario completion, a second enrollment's confirmation email, a full
password-reset round trip, and the minor/no-email contact-email path.
- infra/scripts/gws-email-test.js: the opt-in (RUN_GWS_EMAIL_TEST=true) live Google Workspace
delivery test — does nothing without the explicit flag, sends exactly one message to one
operator-configured recipient when enabled.
- platform/test/e2e-phase5b.js (32 checks): program seed, entitlement gating
(pending/future/expired/cancelled), cross-student isolation, real Lab launch/destroy/
reaper-reconciliation/recreate persistence, server-derived idempotent completion via a full
real capstone completion, the integration boundary (auth, idempotency, real concurrency,
unknown/inactive program rejection, minor/no-email path, no card data), RBAC scoping, a
regression test for the process-crash bug this suite found, Phase 5A smoke, and a real
Platform API restart.
- infra/scripts/drills/test-phase5b-selfpaced-rehearsal.js (15 checks): one continuous story —
activation, a simulated website enrollment event, real Lab work, Lab destroy, a new session,
Continue Learning, a fresh Lab, full real capstone completion, Completed transition, and a
second uninvolved student proven unable to reach any of it.
- platform/test/e2e-phase5a.js (21 checks): the two-student RBAC matrix (student can't reach
any /api/admin/* route, assigned/unassigned/viewer instructor access) and the full password
security checklist (correct/wrong/unknown-identifier/deactivated/immediate-deactivation/
rate-limit/rate-limit-survives-a-real-restart/change-password/session-revocation/
forgot-password anti-enumeration/admin-issued reset/CSRF), plus a classroom lifecycle smoke
pass. Replaces the throwaway smoke5a/smoke5b scripts used during development.
- infra/scripts/drills/test-phase5a-rehearsal.js (15 checks): 5 pilot students with real
seeded pre-existing progress/attempt/achievement history, a live classroom, help requests,
announcements, a REAL Platform API restart mid-rehearsal proving classroom/session/presence
recovery, then converting two students mid-rehearsal and confirming zero data was orphaned.
- Both run against a live docker-compose.dev.yml stack over real HTTP, not mocks.
- infra/scripts/simulate-two-host/: real two-Docker-network simulation
(Gateway genuinely isolated from any Lab bridge network) proving the
Lab Host Router fix - 13/13 real assertions (HTTP+WebSocket through the
Router, Alice/Bob isolation, garbage/destroyed tokens, mTLS-only Router
access).
- infra/scripts/drills/: Lab Host loss/rebuild (9/9), deployment rollback,
CI-fails-closed (6 required failure modes), fake cohort dress rehearsal
(14/14, one continuous narrative through the real app).
- infra/ci/gitea-workflows/build-and-test.yml: new security-scan job
(npm audit, gitleaks, hadolint, shellcheck, trivy - all free/open-source),
severity policy in docs/SECURITY_SCANNING.md (CRITICAL/HIGH blocks,
MEDIUM/LOW reported only). Gates build-images alongside regression.
Found and fixed two real pre-existing bugs via the CI-fails-closed drill:
staging-smoke-test.sh and production-smoke-test.sh both had a classic
set -e gotcha (curl ... && echo "OK" as one statement - a failing command
that isn't last in an && chain doesn't trigger -e) that let an
unreachable/broken endpoint fall through silently instead of failing the
smoke test.
platform/test/e2e-phase3a.js updated for the new paginated
/api/admin/audit-events response shape (Phase 4.1 pagination work).
Adds platform/lab-router/ — a narrow service that runs on the Lab Host,
authenticates the Gateway via mTLS, and resolves an opaque routing token to
a local container purely from Docker label state (no Postgres access, no
callback to the Provisioner). Replaces the Gateway's env.internal_ip
assumption (only valid when Gateway and Lab share a Docker daemon) with a
data-driven direct/router decision per lab_host
(lab_hosts.routing_mode/router_endpoint, migration 004), resolved in one
place (platform/gateway/src/lab-endpoint.js's labTarget()) that every
proxy/fetch call site in server.js now goes through.
Extends the existing Docker TLS CA (infra/scripts/docker-tls/generate-docker-tls.sh)
with a second mTLS leaf pair (router-server/gateway-client) for this new
data-plane connection, deployed by the docker-tls Ansible role into a
separate directory from the control-plane pair. Firewall
(rn-lab-host-01.rules) replaces the old speculative wide port-range
allowance with one narrow rule: platform_ip -> the Router's port only.
dev-local stays routing_mode: direct — zero behavior change for local dev,
confirmed by the full existing regression suite passing unmodified.
boot-labs.js drives the real admin Prepare-Labs endpoint (reusing the
existing session-helper.js from Phase 3A) and records boot time/failure
rate; sample-resources.sh samples real docker stats for running Labs.
capacity-model.js turns measured per-Lab CPU/memory into a recommended/
warning/hard-admission-control concurrency threshold - no concurrency
number is hardcoded as a marketing figure, every input is explicitly
placeholder data until measured against the real host.
Plus deploy/rollback/smoke-test scripts used by the CI/CD and deployment
runbooks. All syntax-checked; validate-compose.sh and capacity-model.js
actually run against real inputs.
Automatic test/build/tag/staging-deploy on push to main; production
promotion stays a manual, human-run step (deploy-production.sh) - never
automated. SHA-tagged images always, semantic tags only on deliberate
release, never :latest alone in staging/production. No paid CI provider or
registry required (uses Gitea's own). Real regression job installs all
services, applies real migrations against a service Postgres, runs the
infra validation scripts.
Validated as real YAML (python3 yaml parse).
Named-bridge-network production Compose (not host networking, unlike dev)
for rn-platform-01; egress-proxy + agents for rn-lab-host-01. Caddy reverse
proxy, single public origin, Grafana restricted to management_cidr.
iptables-restore firewall rulesets for all three hosts (Proxmox host,
rn-platform-01, rn-lab-host-01) codifying the Platform/Lab-Host boundary:
no student-to-student traffic, no direct Proxmox exposure, no direct
Postgres/Provisioner exposure from the Lab Host.
Mutual TLS for the Provisioner<->Lab Host Docker connection: certificate
generation/verification scripts, dockerd TLS config, firewall-restricted
port 2376. Application support already existed since Phase 1; this is
purely the operational half.
Validated: docker compose config, caddy validate, firewall structural lint,
and a real openssl s_server/s_client mTLS handshake proving the cert chain
and mutual-auth rejection (legitimate client verified, rogue client
correctly rejected with verify code 19).
ced4568andClaude Sonnet 5