docs(phase4.1): pre-production hardening documentation
Master overview (PHASE4_1_PREPRODUCTION_HARDENING.md) covering all 27 substantive items, plus REMOTE_AND_SELF_PACED_ARCHITECTURE.md (reviews what already works for remote/self-paced delivery - student-driven provisioning, cohort-less students, idle-reaper-destroy-then-continue - and the one real gap: student authentication, explicitly Phase 5 scope, not built here). Updated PRODUCTION_NETWORK_SECURITY.md (the Gateway-to-Lab gap is now RESOLVED, with the full design writeup and a new control-plane-vs-data-plane section), PRODUCTION_DEPLOYMENT.md (four-cert TLS bundle), PRODUCTION_VALIDATION_CHECKLIST.md (Router mTLS check, admission control checks), PHASE4_PRODUCTION_INFRASTRUCTURE.md and ENGINEERING_SOP.md and LAB_ARCHITECTURE.md (stale 'known gap' pointers updated to point at the resolution), BACKUP_DR.md (destructive drill), README.md and platform/README.md (Phase 4.1 pointers, lab-router component, updated known-limitations). Also fixes a real pre-existing bug found while auditing what gets committed: infra/.gitignore's patterns were written repo-root-relative (e.g. "infra/terraform/...") but the file lives in infra/, so git resolved them relative to THAT directory and none of them ever matched anything - a real .terraform/ provider cache showed up as untracked when it should have been ignored. Fixed by dropping the redundant leading "infra/" from every pattern. Status: PHASE 4.1 COMPLETE / PRE-PRODUCTION HARDENING COMPLETE / READY FOR HARDWARE DEPLOYMENT.
This commit is contained in:
@@ -19,8 +19,9 @@ can prove in advance.
|
||||
|
||||
## Lab Host VM
|
||||
- [ ] Docker daemon TLS verified: `docker --tlsverify --tlscacert=... -H tcp://<lab_host_ip>:2376 info` succeeds from the Platform VM, fails from anywhere else
|
||||
- [ ] Lab Host Router mTLS verified: a request to `https://<lab_host_ip>:8443` with no client cert is rejected; with the Gateway's real client cert it isn't (mirrors `infra/scripts/simulate-two-host/test-remote-routing.js`'s "direct public access: NOT AVAILABLE" case, now against the real host)
|
||||
- [ ] Provisioner successfully creates a real Lab against the real Lab Host
|
||||
- [ ] Gateway can actually reach that Lab's terminal — **this depends on the not-yet-built application change documented in `docs/PRODUCTION_NETWORK_SECURITY.md`'s "Known gap" section; do not mark this item done until that change has landed and been verified for real**
|
||||
- [ ] Gateway can actually reach that Lab's terminal (HTTP UI load + WebSocket `kubectl` command) — application mechanism built and simulation-tested in Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1/§6/§7); this line is the real-hardware confirmation that the same result holds across the actual two VMs, not two Docker networks on one daemon
|
||||
- [ ] Egress proxy correctly restricts a Lab's outbound traffic (Provisioner's per-Lab `DOCKER-USER` chain)
|
||||
|
||||
## Security
|
||||
@@ -38,7 +39,7 @@ can prove in advance.
|
||||
|
||||
## Backup
|
||||
- [ ] Manual `backup-postgres.sh` run succeeds against real Postgres, real off-host target reachable
|
||||
- [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness
|
||||
- [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness — Phase 4.1's `infra/backup/test-destructive-restore-drill.sh` proved the full mechanism (realistic data, real storage destruction, restore, real Platform API boot) locally; this line is the real-hardware confirmation
|
||||
- [ ] Cron job confirmed scheduled (`crontab -l`)
|
||||
|
||||
## Observability
|
||||
@@ -46,5 +47,12 @@ can prove in advance.
|
||||
- [ ] Trigger one alert deliberately (e.g. stop a service) and confirm it fires and reaches the webhook
|
||||
- [ ] Loki has logs flowing from both VMs
|
||||
|
||||
## Pre-production hardening (Phase 4.1 — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md)
|
||||
- [ ] `resource-policy.yaml`'s `admission_control.max_concurrent_labs` set from a real measured
|
||||
`infra/scripts/capacity-model.js` run against the real Lab Host, not left disabled
|
||||
- [ ] A deliberate over-capacity attempt returns the clean 503 "Lab capacity is temporarily
|
||||
full" message, not a 500 — confirms the real host's admission control matches the local test
|
||||
- [ ] `docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md`'s steps still make sense once capacity numbers are real
|
||||
|
||||
Only once every box above is checked against the real server does the deployment graduate from
|
||||
"infrastructure package built" to "actually in production."
|
||||
|
||||
Reference in New Issue
Block a user