docs(phase4.1): pre-production hardening documentation
Release to Docker Hub / Build & Push Docker Image (push) Canceled after 0s
Release to Docker Hub / Create GitHub Release (push) Canceled after 0s

Master overview (PHASE4_1_PREPRODUCTION_HARDENING.md) covering all 27
substantive items, plus REMOTE_AND_SELF_PACED_ARCHITECTURE.md (reviews
what already works for remote/self-paced delivery - student-driven
provisioning, cohort-less students, idle-reaper-destroy-then-continue -
and the one real gap: student authentication, explicitly Phase 5 scope,
not built here).

Updated PRODUCTION_NETWORK_SECURITY.md (the Gateway-to-Lab gap is now
RESOLVED, with the full design writeup and a new control-plane-vs-data-plane
section), PRODUCTION_DEPLOYMENT.md (four-cert TLS bundle),
PRODUCTION_VALIDATION_CHECKLIST.md (Router mTLS check, admission control
checks), PHASE4_PRODUCTION_INFRASTRUCTURE.md and ENGINEERING_SOP.md and
LAB_ARCHITECTURE.md (stale 'known gap' pointers updated to point at the
resolution), BACKUP_DR.md (destructive drill), README.md and
platform/README.md (Phase 4.1 pointers, lab-router component, updated
known-limitations).

Also fixes a real pre-existing bug found while auditing what gets
committed: infra/.gitignore's patterns were written repo-root-relative
(e.g. "infra/terraform/...") but the file lives in infra/, so git resolved
them relative to THAT directory and none of them ever matched anything -
a real .terraform/ provider cache showed up as untracked when it should
have been ignored. Fixed by dropping the redundant leading "infra/" from
every pattern.

Status: PHASE 4.1 COMPLETE / PRE-PRODUCTION HARDENING COMPLETE / READY FOR
HARDWARE DEPLOYMENT.
This commit is contained in:
2026-08-24 20:16:20 -05:00
parent f3c356951c
commit 5bc0743b08
12 changed files with 520 additions and 69 deletions
+32 -4
View File
@@ -296,12 +296,40 @@ Compose, firewall, monitoring, backup, CI, load-test tooling).
- **Where the boundary held:** the Platform VM / Lab Host VM security split from
`docs/LAB_ARCHITECTURE.md` is codified in firewall rulesets and mutual-TLS Docker config, not
weakened anywhere.
- **Known gap surfaced, not silently fixed:** the Gateway's Lab-reachability mechanism doesn't
survive the move to two real separate VMs — see `docs/PRODUCTION_NETWORK_SECURITY.md`'s
"Known gap" section for what's built (firewall policy) vs. what's an explicit, reviewed,
not-yet-built application follow-up (Provisioner/Gateway address handling).
- **Known gap surfaced, then resolved in Phase 4.1:** the Gateway's Lab-reachability mechanism
didn't survive the move to two real separate VMs — flagged explicitly rather than silently
fixed, then actually built and simulation-tested (a Lab Host Router,
`platform/lab-router/`) in Phase 4.1. See §13 below.
- **Validated without the hardware:** real `tofu validate`, `ansible-playbook --syntax-check`,
`docker compose config`, `caddy validate`, `promtool`/`amtool`, and a genuine end-to-end
backup→encrypt→restore→verify cycle against disposable local Postgres with the real
production migrations applied. None of this proves the real Hetzner network boundary — that
needs `docs/PRODUCTION_VALIDATION_CHECKLIST.md` walked against the real server.
## 13. Pre-Production Hardening (Phase 4.1)
Full design: `docs/PHASE4_1_PREPRODUCTION_HARDENING.md`, plus
`docs/PRODUCTION_NETWORK_SECURITY.md` (routing resolution), `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md`,
`docs/SECURITY_SCANNING.md`.
- **What it is:** closes the Gateway-to-Lab cross-VM gap Phase 4 flagged (a Lab Host Router,
`platform/lab-router/`, plus a routing abstraction in the Gateway,
`platform/gateway/src/lab-endpoint.js`), then hardens the application layer found along the
way: abandoned-attempt handling (`scenario_attempts.status = 'abandoned'`, never a punitive
failure), a Postgres-backed login rate limiter (survives a restart, ready to be
shared/multi-instance), real pagination on audit/attempt/assessment-session history, a
structured assessment authoring UI (owner-only raw JSON remains as a fallback), a minimal
`role_in_cohort` enforcement (`teacher`/`viewer`), a `/api/version` endpoint, and admission
control (a clean 503 "capacity full" instead of a 500 once a configured cap is reached).
- **Tested, not just designed:** a real two-Docker-network routing simulation
(`infra/scripts/simulate-two-host/`, 13/13), a destructive Postgres restore drill
(`infra/backup/test-destructive-restore-drill.sh`), a Lab Host loss/rebuild drill
(`infra/scripts/drills/test-lab-host-loss.js`), a deployment rollback drill, a CI-fails-closed
drill (which found and fixed a real pre-existing `set -e` bug in both smoke-test scripts —
see that drill's own output), and a full fake-cohort dress rehearsal
(`infra/scripts/drills/test-fake-cohort-rehearsal.js`) — all passing, all in
`infra/scripts/drills/` or `infra/backup/`.
- **Known limitation carried forward:** real remote/self-paced delivery needs actual student
authentication (today's `student_id`-only login relies on instructors controlling who gets a
valid id) — see `docs/REMOTE_AND_SELF_PACED_ARCHITECTURE.md` for the full accounting of what
already works vs. what's genuinely Phase 5 scope.