docs(phase4.1): pre-production hardening documentation
Release to Docker Hub / Build & Push Docker Image (push) Canceled after 0s
Release to Docker Hub / Create GitHub Release (push) Canceled after 0s

Master overview (PHASE4_1_PREPRODUCTION_HARDENING.md) covering all 27
substantive items, plus REMOTE_AND_SELF_PACED_ARCHITECTURE.md (reviews
what already works for remote/self-paced delivery - student-driven
provisioning, cohort-less students, idle-reaper-destroy-then-continue -
and the one real gap: student authentication, explicitly Phase 5 scope,
not built here).

Updated PRODUCTION_NETWORK_SECURITY.md (the Gateway-to-Lab gap is now
RESOLVED, with the full design writeup and a new control-plane-vs-data-plane
section), PRODUCTION_DEPLOYMENT.md (four-cert TLS bundle),
PRODUCTION_VALIDATION_CHECKLIST.md (Router mTLS check, admission control
checks), PHASE4_PRODUCTION_INFRASTRUCTURE.md and ENGINEERING_SOP.md and
LAB_ARCHITECTURE.md (stale 'known gap' pointers updated to point at the
resolution), BACKUP_DR.md (destructive drill), README.md and
platform/README.md (Phase 4.1 pointers, lab-router component, updated
known-limitations).

Also fixes a real pre-existing bug found while auditing what gets
committed: infra/.gitignore's patterns were written repo-root-relative
(e.g. "infra/terraform/...") but the file lives in infra/, so git resolved
them relative to THAT directory and none of them ever matched anything -
a real .terraform/ provider cache showed up as untracked when it should
have been ignored. Fixed by dropping the redundant leading "infra/" from
every pattern.

Status: PHASE 4.1 COMPLETE / PRE-PRODUCTION HARDENING COMPLETE / READY FOR
HARDWARE DEPLOYMENT.
This commit is contained in:
2026-08-24 20:16:20 -05:00
parent f3c356951c
commit 5bc0743b08
12 changed files with 520 additions and 69 deletions
+10 -2
View File
@@ -19,8 +19,9 @@ can prove in advance.
## Lab Host VM
- [ ] Docker daemon TLS verified: `docker --tlsverify --tlscacert=... -H tcp://<lab_host_ip>:2376 info` succeeds from the Platform VM, fails from anywhere else
- [ ] Lab Host Router mTLS verified: a request to `https://<lab_host_ip>:8443` with no client cert is rejected; with the Gateway's real client cert it isn't (mirrors `infra/scripts/simulate-two-host/test-remote-routing.js`'s "direct public access: NOT AVAILABLE" case, now against the real host)
- [ ] Provisioner successfully creates a real Lab against the real Lab Host
- [ ] Gateway can actually reach that Lab's terminal — **this depends on the not-yet-built application change documented in `docs/PRODUCTION_NETWORK_SECURITY.md`'s "Known gap" section; do not mark this item done until that change has landed and been verified for real**
- [ ] Gateway can actually reach that Lab's terminal (HTTP UI load + WebSocket `kubectl` command) — application mechanism built and simulation-tested in Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1/§6/§7); this line is the real-hardware confirmation that the same result holds across the actual two VMs, not two Docker networks on one daemon
- [ ] Egress proxy correctly restricts a Lab's outbound traffic (Provisioner's per-Lab `DOCKER-USER` chain)
## Security
@@ -38,7 +39,7 @@ can prove in advance.
## Backup
- [ ] Manual `backup-postgres.sh` run succeeds against real Postgres, real off-host target reachable
- [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness
- [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness — Phase 4.1's `infra/backup/test-destructive-restore-drill.sh` proved the full mechanism (realistic data, real storage destruction, restore, real Platform API boot) locally; this line is the real-hardware confirmation
- [ ] Cron job confirmed scheduled (`crontab -l`)
## Observability
@@ -46,5 +47,12 @@ can prove in advance.
- [ ] Trigger one alert deliberately (e.g. stop a service) and confirm it fires and reaches the webhook
- [ ] Loki has logs flowing from both VMs
## Pre-production hardening (Phase 4.1 — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md)
- [ ] `resource-policy.yaml`'s `admission_control.max_concurrent_labs` set from a real measured
`infra/scripts/capacity-model.js` run against the real Lab Host, not left disabled
- [ ] A deliberate over-capacity attempt returns the clean 503 "Lab capacity is temporarily
full" message, not a 500 — confirms the real host's admission control matches the local test
- [ ] `docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md`'s steps still make sense once capacity numbers are real
Only once every box above is checked against the real server does the deployment graduate from
"infrastructure package built" to "actually in production."