Adds platform/test/e2e-phase6c.js (17 checks): upgraded-module activity
chains, documented-exception promotions, optional-lab non-blocking behavior,
no-fake-completion-from-viewing, idempotent completion, instructor-notes
exclusion and RBAC on new content, security-assessment pool/draw/cross-student
isolation/completion, the capstone gate's documented indifference to ROOT 09,
ROOT 11's honest partial/lesson-only rendering, and version-bump content
stability — plus running validate-course-content.js as part of the suite.
All 17 pass against both a warm and a freshly-reset dev stack.
Adds infra/scripts/drills/{run,test}-phase6c-security-config-rehearsal —
Section 22's real-k3s dress rehearsal: login -> program -> course -> lesson ->
demonstration -> knowledge-check -> guided-exercise -> real Lab -> real
kubectl (create Secret, apply Pod, wait for Ready) -> validation -> progress
persistence across logout/login -> instructor visibility of the resulting
learning state, walking root-07-m02-configuration-and-secrets (upgraded from
PARTIAL this phase). 10/10 pass against a real k3s Lab, no mocking.
Fixes two stale hardcoded assumptions in e2e-phase6a.js and e2e-phase6b.js
that Phase 6C's own legitimate content changes broke: the gold module
(root-10-m01) now has 7 activities (6 required + 1 new optional lab, not 6),
and root-05-m01 now has 2 required manual-marker activities the course-
completion path must also complete (its new bookending lesson/reflection).
Both suites are back to green (21/21 and 24/24) against a fresh stack.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Upgrades six PARTIAL modules to the gold-module standard (lesson/demonstration/
knowledge-check/guided-exercise/lab(s)/reflection), preserving every existing lab
activity_id and scenario_id: root-06-m03 (jobs/cronjobs/daemonsets), root-07-m04
(ingress/networkpolicy), root-07-m02 (configuration/secrets, converted in place
from a bare 5-lab bank), root-09-m02 (serviceaccounts/rbac), root-09-m03
(network/supply-chain security), root-10-m03 (troubleshooting framework).
root-05-m01 (20-task kubectl-fluency bank) gets a minimal bookending lesson +
reflection rather than a full redesign, since each task already carries its own
explanation. root-12-m01, root-capstone-m01, and root-09-m04 are promoted to
READY via documented exceptions (no new activities) where adding one would
misrepresent their intentional design.
Wires a new rootnode-security-assessment (exam, 60min, hint_policy disabled)
using the existing Phase 3B assessment engine: 3 required hands-on CORE
scenarios + 4 random-draw pools covering SecurityContext, RBAC, network
policy, and the 7 previously-unwired CKS MCQs. Wires the previously-unwired
readiness-probe-task scenario as an OPTIONAL practice lab in root-10-m01,
using the existing is_required model.
Adds root-11-m01-devops-platform-thinking — an honest, lesson-only ROOT 11
orientation module naming DevOps/platform engineering practices already
exercised in ROOT 05-10/12, explicit that GitOps/CI-CD/IaC/observability need
lab infrastructure this platform doesn't yet provide. No fake labs.
Adds infra/scripts/validate-course-content.js: cross-checks the DB against
scenario/content files for duplicate ids, orphans, invalid references, cycles,
sequence collisions, instructor-notes leakage, and (new) self-paced content
deferring explanation to a live instructor. 0 errors against the resulting
22 courses/modules, 163 activities, 98 scenario files.
No application code changed — every gap was satisfiable with existing tables
(course_modules, module_activities, assessment_definitions) and the existing
content-delivery pipeline.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
e2e-phase6b.js (24 checks): hierarchy for the 13 new modules across
ROOT 05-10/12/Capstone, prerequisite chains through the newly-inserted
front-door modules, lesson/knowledge-check completion in brand-new
modules, a newly-wired scenario (job-task) completing correctly, the
optional-vs-required (is_required) completion rule, persistence/
isolation/RBAC/draft-hiding proven against the new content, and the
capstone server-side unlock fix in both directions (a direct bypass
attempt rejected with blocking_stage_ids, an ordinary exam unaffected,
a legitimately unlocked attempt still accepted and its completion
reflected in the course content model).
test-phase6b-root-course-rehearsal.js + wrapper (11 checks, real k3s):
one continuous student narrative through a brand-new module (ROOT 07
Services & Service Discovery, not the Phase 6A gold module) — real Lab
launch, the Lab's own setup_commands, a real 'kubectl expose' against
the live cluster, real validate, logout/login/Resume Learning, Lab
destroy/recreate with academic progress intact — plus an instructor
narrative proving Curriculum view, Prepare Labs, and classroom/learning
tooling all still work against the expanded curriculum.
e2e-phase6a.js updated for two of this phase's own intentional,
documented behavior changes (not defects): resume now points at
root-05's new mental-model module instead of the old bare-lab bank, and
root-10's gold module now unlocks the new troubleshooting-framework
module rather than the bare-lab bank directly.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
e2e-phase6a.js (21 checks): hierarchy/ordering, honest READY/PARTIAL/
PLANNED states, all three completion behaviors (manual marker/
scenario-derived/assessment-derived), prerequisite locking (including
the zero-content-is-vacuously-satisfied rule), resume, persistence
across logout/login, cross-student isolation, RBAC for self-paced
no-cohort students, draft-content hiding, and content-version
stability.
test-phase6a-course-rehearsal.js + run-phase6a-course-rehearsal.sh: a
real-k3s dress rehearsal of the gold-standard module end to end — a
student reads the real lesson content, launches a real disposable Lab,
the Lab's own POST /api/scenarios/:id/setup (called through the
Gateway's existing /lab/:token/* proxy exactly as the browser UI does)
stands up the broken api-gateway deployment, a real kubectl patch fixes
the readiness probe, and the module completes and unlocks the next one.
No mocked KubeKosh anywhere in this suite.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
- platform/test/e2e-phase5c.js (29 checks): activation-vs-confirmation routing,
duplicate/concurrent webhook idempotency, temporary/permanent/exhausted retry
classification, resend/regenerate token invalidation, activation/reset single-use and
expiration, HTML-injection regression, production token-secrecy gating via a real container
recreate with COOKIE_SECURE=1, minor/contact-email resolution, RBAC scoping, metrics/audit
hygiene, and a real Platform API restart.
- infra/scripts/drills/test-phase5c-enrollment-activation.js (10 checks): one continuous
story — website enrollment through a real rendered/extracted activation email, a real Lab
launch and scenario completion, a second enrollment's confirmation email, a full
password-reset round trip, and the minor/no-email contact-email path.
- infra/scripts/gws-email-test.js: the opt-in (RUN_GWS_EMAIL_TEST=true) live Google Workspace
delivery test — does nothing without the explicit flag, sends exactly one message to one
operator-configured recipient when enabled.
- platform/test/e2e-phase5b.js (32 checks): program seed, entitlement gating
(pending/future/expired/cancelled), cross-student isolation, real Lab launch/destroy/
reaper-reconciliation/recreate persistence, server-derived idempotent completion via a full
real capstone completion, the integration boundary (auth, idempotency, real concurrency,
unknown/inactive program rejection, minor/no-email path, no card data), RBAC scoping, a
regression test for the process-crash bug this suite found, Phase 5A smoke, and a real
Platform API restart.
- infra/scripts/drills/test-phase5b-selfpaced-rehearsal.js (15 checks): one continuous story —
activation, a simulated website enrollment event, real Lab work, Lab destroy, a new session,
Continue Learning, a fresh Lab, full real capstone completion, Completed transition, and a
second uninvolved student proven unable to reach any of it.
- platform/test/e2e-phase5a.js (21 checks): the two-student RBAC matrix (student can't reach
any /api/admin/* route, assigned/unassigned/viewer instructor access) and the full password
security checklist (correct/wrong/unknown-identifier/deactivated/immediate-deactivation/
rate-limit/rate-limit-survives-a-real-restart/change-password/session-revocation/
forgot-password anti-enumeration/admin-issued reset/CSRF), plus a classroom lifecycle smoke
pass. Replaces the throwaway smoke5a/smoke5b scripts used during development.
- infra/scripts/drills/test-phase5a-rehearsal.js (15 checks): 5 pilot students with real
seeded pre-existing progress/attempt/achievement history, a live classroom, help requests,
announcements, a REAL Platform API restart mid-rehearsal proving classroom/session/presence
recovery, then converting two students mid-rehearsal and confirming zero data was orphaned.
- Both run against a live docker-compose.dev.yml stack over real HTTP, not mocks.
- admin-routes.js: create-with-credentials, /convert, /reset-password, /revoke-sessions,
/sessions, /import (transactional CSV import — whole file validated, one Postgres
transaction, never accepts a password/hash column from input). Fixed GET
/api/admin/students to select an explicit safe column list instead of SELECT * — the new
password_hash/credential_token_hash columns must never be exposed through this route.
- server.js: /api/login now also blocks on credential_token_hash (not just password_hash) —
closes a real window where a student mid-conversion stayed reachable via the old
bare-student_id path. studentSession middleware now re-verifies status='active' on every
request, not just at session creation, matching instructorSession's existing behavior.
- sessions.js: revokeAllStudentSessions, listStudentSessions.
- rbac.js: extracted checkCohortAccess/checkCohortWriteAccess as shared core functions so the
new classroom/help-request RBAC (next commit) reuses the exact same authorization rule
instead of a parallel copy.
- student-csv.js + test: hand-rolled parser (no new dependency), unit tested.
- prometheus.yml.j2: activated the platform-api scrape job.
Master overview (PHASE4_1_PREPRODUCTION_HARDENING.md) covering all 27
substantive items, plus REMOTE_AND_SELF_PACED_ARCHITECTURE.md (reviews
what already works for remote/self-paced delivery - student-driven
provisioning, cohort-less students, idle-reaper-destroy-then-continue -
and the one real gap: student authentication, explicitly Phase 5 scope,
not built here).
Updated PRODUCTION_NETWORK_SECURITY.md (the Gateway-to-Lab gap is now
RESOLVED, with the full design writeup and a new control-plane-vs-data-plane
section), PRODUCTION_DEPLOYMENT.md (four-cert TLS bundle),
PRODUCTION_VALIDATION_CHECKLIST.md (Router mTLS check, admission control
checks), PHASE4_PRODUCTION_INFRASTRUCTURE.md and ENGINEERING_SOP.md and
LAB_ARCHITECTURE.md (stale 'known gap' pointers updated to point at the
resolution), BACKUP_DR.md (destructive drill), README.md and
platform/README.md (Phase 4.1 pointers, lab-router component, updated
known-limitations).
Also fixes a real pre-existing bug found while auditing what gets
committed: infra/.gitignore's patterns were written repo-root-relative
(e.g. "infra/terraform/...") but the file lives in infra/, so git resolved
them relative to THAT directory and none of them ever matched anything -
a real .terraform/ provider cache showed up as untracked when it should
have been ignored. Fixed by dropping the redundant leading "infra/" from
every pattern.
Status: PHASE 4.1 COMPLETE / PRE-PRODUCTION HARDENING COMPLETE / READY FOR
HARDWARE DEPLOYMENT.
- infra/scripts/simulate-two-host/: real two-Docker-network simulation
(Gateway genuinely isolated from any Lab bridge network) proving the
Lab Host Router fix - 13/13 real assertions (HTTP+WebSocket through the
Router, Alice/Bob isolation, garbage/destroyed tokens, mTLS-only Router
access).
- infra/scripts/drills/: Lab Host loss/rebuild (9/9), deployment rollback,
CI-fails-closed (6 required failure modes), fake cohort dress rehearsal
(14/14, one continuous narrative through the real app).
- infra/ci/gitea-workflows/build-and-test.yml: new security-scan job
(npm audit, gitleaks, hadolint, shellcheck, trivy - all free/open-source),
severity policy in docs/SECURITY_SCANNING.md (CRITICAL/HIGH blocks,
MEDIUM/LOW reported only). Gates build-images alongside regression.
Found and fixed two real pre-existing bugs via the CI-fails-closed drill:
staging-smoke-test.sh and production-smoke-test.sh both had a classic
set -e gotcha (curl ... && echo "OK" as one statement - a failing command
that isn't last in an && chain doesn't trigger -e) that let an
unreachable/broken endpoint fall through silently instead of failing the
smoke test.
platform/test/e2e-phase3a.js updated for the new paginated
/api/admin/audit-events response shape (Phase 4.1 pagination work).
infra/backup/test-destructive-restore-drill.sh - realistic academy data
(owner, admin instructor, cohort, 3 students, progress, attempts, an
assessment definition, achievements) seeded through the REAL Platform API,
encrypted-backed-up, the source Postgres container genuinely destroyed
(docker rm, not just left running alongside a second one), a fresh
Postgres created from nothing, restored, and the real Platform API booted
again against the restored database and used to log in as the restored
admin instructor over HTTP. Never touches the developer's real database.
Confirmed passing.
- Real server-side pagination (platform/api/src/pagination.js) on the audit
log (was a flat LIMIT 200), student attempt history (was LIMIT 100), and
assessment session history (was fully unbounded - the one genuine growth
risk found). Frontend updated (AuditView, AssessmentsView, StudentsView).
- Admission control: resource-policy.yaml's new (disabled-by-default)
admission_control.max_concurrent_labs, checked before touching Docker at
all, returns a clean 503 CAPACITY_EXCEEDED instead of failing mid-provision.
Relayed as a clean student-facing message, never a raw 500. New
GET /capacity endpoint feeds the instructor dashboard.
- cohort_instructors.role_in_cohort (existed since Phase 2B, never read) is
now enforced: teacher vs viewer, rbac.js's
requireCohortWriteAccess/requireStudentCohortWriteAccess.
- GET /api/version - app version, deployed Git SHA, latest migration as a
schema-version stand-in.
- Structured assessment authoring UI (AssessmentsView.jsx's PoolSpecEditor)
replaces the raw-JSON pool_spec textarea, backed by a new
GET /api/admin/scenario-catalog endpoint. Owner-only raw-JSON fallback
remains for anything the structured editor doesn't cover.
- NEEDS_ATTENTION_FAIL_THRESHOLD/STALE_DAYS moved to env-var configuration.
Full regression suite (e2e-phase2a/2b/3a/3b, 67/67) re-run clean after
these changes; frontend production build confirmed clean.
Adds platform/lab-router/ — a narrow service that runs on the Lab Host,
authenticates the Gateway via mTLS, and resolves an opaque routing token to
a local container purely from Docker label state (no Postgres access, no
callback to the Provisioner). Replaces the Gateway's env.internal_ip
assumption (only valid when Gateway and Lab share a Docker daemon) with a
data-driven direct/router decision per lab_host
(lab_hosts.routing_mode/router_endpoint, migration 004), resolved in one
place (platform/gateway/src/lab-endpoint.js's labTarget()) that every
proxy/fetch call site in server.js now goes through.
Extends the existing Docker TLS CA (infra/scripts/docker-tls/generate-docker-tls.sh)
with a second mTLS leaf pair (router-server/gateway-client) for this new
data-plane connection, deployed by the docker-tls Ansible role into a
separate directory from the control-plane pair. Firewall
(rn-lab-host-01.rules) replaces the old speculative wide port-range
allowance with one narrow rule: platform_ip -> the Router's port only.
dev-local stays routing_mode: direct — zero behavior change for local dev,
confirmed by the full existing regression suite passing unmodified.
boot-labs.js drives the real admin Prepare-Labs endpoint (reusing the
existing session-helper.js from Phase 3A) and records boot time/failure
rate; sample-resources.sh samples real docker stats for running Labs.
capacity-model.js turns measured per-Lab CPU/memory into a recommended/
warning/hard-admission-control concurrency threshold - no concurrency
number is hardcoded as a marketing figure, every input is explicitly
placeholder data until measured against the real host.
Plus deploy/rollback/smoke-test scripts used by the CI/CD and deployment
runbooks. All syntax-checked; validate-compose.sh and capacity-model.js
actually run against real inputs.
Automatic test/build/tag/staging-deploy on push to main; production
promotion stays a manual, human-run step (deploy-production.sh) - never
automated. SHA-tagged images always, semantic tags only on deliberate
release, never :latest alone in staging/production. No paid CI provider or
registry required (uses Gitea's own). Real regression job installs all
services, applies real migrations against a service Postgres, runs the
infra validation scripts.
Validated as real YAML (python3 yaml parse).
pg_dump -> gzip -> gpg AES-256 symmetric encryption, plaintext deleted
immediately, optional off-host rsync-over-SSH copy (Hetzner Storage Box or
any SSH target), retention pruning, Prometheus textfile-collector metrics
for backup success/age/size.
Genuinely tested end-to-end, not just written: real production migrations
(17 total, correct cross-service order) applied to a disposable Postgres,
sample data inserted, backed up, restored into a second disposable
Postgres, verified table-for-table and byte-for-byte. Exit 0.
Named-bridge-network production Compose (not host networking, unlike dev)
for rn-platform-01; egress-proxy + agents for rn-lab-host-01. Caddy reverse
proxy, single public origin, Grafana restricted to management_cidr.
iptables-restore firewall rulesets for all three hosts (Proxmox host,
rn-platform-01, rn-lab-host-01) codifying the Platform/Lab-Host boundary:
no student-to-student traffic, no direct Proxmox exposure, no direct
Postgres/Provisioner exposure from the Lab Host.
Mutual TLS for the Provisioner<->Lab Host Docker connection: certificate
generation/verification scripts, dockerd TLS config, firewall-restricted
port 2376. Application support already existed since Phase 1; this is
purely the operational half.
Validated: docker compose config, caddy validate, firewall structural lint,
and a real openssl s_server/s_client mTLS handshake proving the cert chain
and mutual-auth rejection (legitimate client verified, rogue client
correctly rejected with verify code 19).
common/firewall/docker-tls/platform/lab-host/monitoring-agent/backup roles,
orchestrated by site.yml with the ordering that matters (firewall/TLS land
before the services that depend on them). Ansible Vault for secrets, no new
tool introduced for this project's modest secret surface. Explicit
dev/staging/production environment separation, no secrets committed
(.gitignore covers vault plaintext, TLS keys, Terraform state, *.env).
Validated: ansible-playbook --syntax-check clean.
ced4568andClaude Sonnet 5