Master overview (PHASE4_1_PREPRODUCTION_HARDENING.md) covering all 27
substantive items, plus REMOTE_AND_SELF_PACED_ARCHITECTURE.md (reviews
what already works for remote/self-paced delivery - student-driven
provisioning, cohort-less students, idle-reaper-destroy-then-continue -
and the one real gap: student authentication, explicitly Phase 5 scope,
not built here).
Updated PRODUCTION_NETWORK_SECURITY.md (the Gateway-to-Lab gap is now
RESOLVED, with the full design writeup and a new control-plane-vs-data-plane
section), PRODUCTION_DEPLOYMENT.md (four-cert TLS bundle),
PRODUCTION_VALIDATION_CHECKLIST.md (Router mTLS check, admission control
checks), PHASE4_PRODUCTION_INFRASTRUCTURE.md and ENGINEERING_SOP.md and
LAB_ARCHITECTURE.md (stale 'known gap' pointers updated to point at the
resolution), BACKUP_DR.md (destructive drill), README.md and
platform/README.md (Phase 4.1 pointers, lab-router component, updated
known-limitations).
Also fixes a real pre-existing bug found while auditing what gets
committed: infra/.gitignore's patterns were written repo-root-relative
(e.g. "infra/terraform/...") but the file lives in infra/, so git resolved
them relative to THAT directory and none of them ever matched anything -
a real .terraform/ provider cache showed up as untracked when it should
have been ignored. Fixed by dropping the redundant leading "infra/" from
every pattern.
Status: PHASE 4.1 COMPLETE / PRE-PRODUCTION HARDENING COMPLETE / READY FOR
HARDWARE DEPLOYMENT.
- infra/scripts/simulate-two-host/: real two-Docker-network simulation
(Gateway genuinely isolated from any Lab bridge network) proving the
Lab Host Router fix - 13/13 real assertions (HTTP+WebSocket through the
Router, Alice/Bob isolation, garbage/destroyed tokens, mTLS-only Router
access).
- infra/scripts/drills/: Lab Host loss/rebuild (9/9), deployment rollback,
CI-fails-closed (6 required failure modes), fake cohort dress rehearsal
(14/14, one continuous narrative through the real app).
- infra/ci/gitea-workflows/build-and-test.yml: new security-scan job
(npm audit, gitleaks, hadolint, shellcheck, trivy - all free/open-source),
severity policy in docs/SECURITY_SCANNING.md (CRITICAL/HIGH blocks,
MEDIUM/LOW reported only). Gates build-images alongside regression.
Found and fixed two real pre-existing bugs via the CI-fails-closed drill:
staging-smoke-test.sh and production-smoke-test.sh both had a classic
set -e gotcha (curl ... && echo "OK" as one statement - a failing command
that isn't last in an && chain doesn't trigger -e) that let an
unreachable/broken endpoint fall through silently instead of failing the
smoke test.
platform/test/e2e-phase3a.js updated for the new paginated
/api/admin/audit-events response shape (Phase 4.1 pagination work).
infra/backup/test-destructive-restore-drill.sh - realistic academy data
(owner, admin instructor, cohort, 3 students, progress, attempts, an
assessment definition, achievements) seeded through the REAL Platform API,
encrypted-backed-up, the source Postgres container genuinely destroyed
(docker rm, not just left running alongside a second one), a fresh
Postgres created from nothing, restored, and the real Platform API booted
again against the restored database and used to log in as the restored
admin instructor over HTTP. Never touches the developer's real database.
Confirmed passing.
- Real server-side pagination (platform/api/src/pagination.js) on the audit
log (was a flat LIMIT 200), student attempt history (was LIMIT 100), and
assessment session history (was fully unbounded - the one genuine growth
risk found). Frontend updated (AuditView, AssessmentsView, StudentsView).
- Admission control: resource-policy.yaml's new (disabled-by-default)
admission_control.max_concurrent_labs, checked before touching Docker at
all, returns a clean 503 CAPACITY_EXCEEDED instead of failing mid-provision.
Relayed as a clean student-facing message, never a raw 500. New
GET /capacity endpoint feeds the instructor dashboard.
- cohort_instructors.role_in_cohort (existed since Phase 2B, never read) is
now enforced: teacher vs viewer, rbac.js's
requireCohortWriteAccess/requireStudentCohortWriteAccess.
- GET /api/version - app version, deployed Git SHA, latest migration as a
schema-version stand-in.
- Structured assessment authoring UI (AssessmentsView.jsx's PoolSpecEditor)
replaces the raw-JSON pool_spec textarea, backed by a new
GET /api/admin/scenario-catalog endpoint. Owner-only raw-JSON fallback
remains for anything the structured editor doesn't cover.
- NEEDS_ATTENTION_FAIL_THRESHOLD/STALE_DAYS moved to env-var configuration.
Full regression suite (e2e-phase2a/2b/3a/3b, 67/67) re-run clean after
these changes; frontend production build confirmed clean.
Adds platform/lab-router/ — a narrow service that runs on the Lab Host,
authenticates the Gateway via mTLS, and resolves an opaque routing token to
a local container purely from Docker label state (no Postgres access, no
callback to the Provisioner). Replaces the Gateway's env.internal_ip
assumption (only valid when Gateway and Lab share a Docker daemon) with a
data-driven direct/router decision per lab_host
(lab_hosts.routing_mode/router_endpoint, migration 004), resolved in one
place (platform/gateway/src/lab-endpoint.js's labTarget()) that every
proxy/fetch call site in server.js now goes through.
Extends the existing Docker TLS CA (infra/scripts/docker-tls/generate-docker-tls.sh)
with a second mTLS leaf pair (router-server/gateway-client) for this new
data-plane connection, deployed by the docker-tls Ansible role into a
separate directory from the control-plane pair. Firewall
(rn-lab-host-01.rules) replaces the old speculative wide port-range
allowance with one narrow rule: platform_ip -> the Router's port only.
dev-local stays routing_mode: direct — zero behavior change for local dev,
confirmed by the full existing regression suite passing unmodified.
boot-labs.js drives the real admin Prepare-Labs endpoint (reusing the
existing session-helper.js from Phase 3A) and records boot time/failure
rate; sample-resources.sh samples real docker stats for running Labs.
capacity-model.js turns measured per-Lab CPU/memory into a recommended/
warning/hard-admission-control concurrency threshold - no concurrency
number is hardcoded as a marketing figure, every input is explicitly
placeholder data until measured against the real host.
Plus deploy/rollback/smoke-test scripts used by the CI/CD and deployment
runbooks. All syntax-checked; validate-compose.sh and capacity-model.js
actually run against real inputs.
Automatic test/build/tag/staging-deploy on push to main; production
promotion stays a manual, human-run step (deploy-production.sh) - never
automated. SHA-tagged images always, semantic tags only on deliberate
release, never :latest alone in staging/production. No paid CI provider or
registry required (uses Gitea's own). Real regression job installs all
services, applies real migrations against a service Postgres, runs the
infra validation scripts.
Validated as real YAML (python3 yaml parse).
pg_dump -> gzip -> gpg AES-256 symmetric encryption, plaintext deleted
immediately, optional off-host rsync-over-SSH copy (Hetzner Storage Box or
any SSH target), retention pruning, Prometheus textfile-collector metrics
for backup success/age/size.
Genuinely tested end-to-end, not just written: real production migrations
(17 total, correct cross-service order) applied to a disposable Postgres,
sample data inserted, backed up, restored into a second disposable
Postgres, verified table-for-table and byte-for-byte. Exit 0.
Named-bridge-network production Compose (not host networking, unlike dev)
for rn-platform-01; egress-proxy + agents for rn-lab-host-01. Caddy reverse
proxy, single public origin, Grafana restricted to management_cidr.
iptables-restore firewall rulesets for all three hosts (Proxmox host,
rn-platform-01, rn-lab-host-01) codifying the Platform/Lab-Host boundary:
no student-to-student traffic, no direct Proxmox exposure, no direct
Postgres/Provisioner exposure from the Lab Host.
Mutual TLS for the Provisioner<->Lab Host Docker connection: certificate
generation/verification scripts, dockerd TLS config, firewall-restricted
port 2376. Application support already existed since Phase 1; this is
purely the operational half.
Validated: docker compose config, caddy validate, firewall structural lint,
and a real openssl s_server/s_client mTLS handshake proving the cert chain
and mutual-auth rejection (legitimate client verified, rogue client
correctly rejected with verify code 19).
common/firewall/docker-tls/platform/lab-host/monitoring-agent/backup roles,
orchestrated by site.yml with the ordering that matters (firewall/TLS land
before the services that depend on them). Ansible Vault for secrets, no new
tool introduced for this project's modest secret surface. Explicit
dev/staging/production environment separation, no secrets committed
(.gitignore covers vault plaintext, TLS keys, Terraform state, *.env).
Validated: ansible-playbook --syntax-check clean.
ced4568