ced4568
5bc0743b08
Master overview (PHASE4_1_PREPRODUCTION_HARDENING.md) covering all 27 substantive items, plus REMOTE_AND_SELF_PACED_ARCHITECTURE.md (reviews what already works for remote/self-paced delivery - student-driven provisioning, cohort-less students, idle-reaper-destroy-then-continue - and the one real gap: student authentication, explicitly Phase 5 scope, not built here). Updated PRODUCTION_NETWORK_SECURITY.md (the Gateway-to-Lab gap is now RESOLVED, with the full design writeup and a new control-plane-vs-data-plane section), PRODUCTION_DEPLOYMENT.md (four-cert TLS bundle), PRODUCTION_VALIDATION_CHECKLIST.md (Router mTLS check, admission control checks), PHASE4_PRODUCTION_INFRASTRUCTURE.md and ENGINEERING_SOP.md and LAB_ARCHITECTURE.md (stale 'known gap' pointers updated to point at the resolution), BACKUP_DR.md (destructive drill), README.md and platform/README.md (Phase 4.1 pointers, lab-router component, updated known-limitations). Also fixes a real pre-existing bug found while auditing what gets committed: infra/.gitignore's patterns were written repo-root-relative (e.g. "infra/terraform/...") but the file lives in infra/, so git resolved them relative to THAT directory and none of them ever matched anything - a real .terraform/ provider cache showed up as untracked when it should have been ignored. Fixed by dropping the redundant leading "infra/" from every pattern. Status: PHASE 4.1 COMPLETE / PRE-PRODUCTION HARDENING COMPLETE / READY FOR HARDWARE DEPLOYMENT.
4.3 KiB
4.3 KiB
Production Validation Checklist
Walk this once, top to bottom, before calling a production deployment live. Every item here needs the real hardware — this is the "DEPLOY LATER" half of Phase 4, not something this repo can prove in advance.
Hardware / Proxmox
- Disk inspection done, layout chosen and matches what's actually in the server (
docs/RUNBOOK_DISK_FAILURE.mdinspection checklist,docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md§4) - Proxmox VE installed, reachable only from
management_cidr/proxmox_mgmt_cidr - VM template built from real cloud-init (
infra/proxmox/README.md) tofu applycreated both VMs; both boot and are SSH-reachable
Platform VM
ansible-playbook site.ymlcompleted clean, all rolesdocker compose psshows every service healthy (postgres, provisioner, api, gateway, reverse-proxy, prometheus, alertmanager, grafana, loki, promtail, exporters)curl https://<origin>/healthz(via Gateway) returns 200curl http://localhost:4300/healthz(Platform API, from the VM) returns{"ok":true,"db":"ok"}- TLS certificate issued and valid for the real domain
Lab Host VM
- Docker daemon TLS verified:
docker --tlsverify --tlscacert=... -H tcp://<lab_host_ip>:2376 infosucceeds from the Platform VM, fails from anywhere else - Lab Host Router mTLS verified: a request to
https://<lab_host_ip>:8443with no client cert is rejected; with the Gateway's real client cert it isn't (mirrorsinfra/scripts/simulate-two-host/test-remote-routing.js's "direct public access: NOT AVAILABLE" case, now against the real host) - Provisioner successfully creates a real Lab against the real Lab Host
- Gateway can actually reach that Lab's terminal (HTTP UI load + WebSocket
kubectlcommand) — application mechanism built and simulation-tested in Phase 4.1 (docs/PHASE4_1_PREPRODUCTION_HARDENING.md§1/§6/§7); this line is the real-hardware confirmation that the same result holds across the actual two VMs, not two Docker networks on one daemon - Egress proxy correctly restricts a Lab's outbound traffic (Provisioner's per-Lab
DOCKER-USERchain)
Security
infra/firewall/validate-firewall.shrun against the real hosts with realiptables-restore --test, not just the structural lint- Confirm from an unauthorized IP: Postgres, Provisioner, Docker TLS port, Proxmox UI are all unreachable
- Confirm Student Lab → another Lab is blocked; confirm Student Lab → Proxmox is blocked
COOKIE_SECURE/TRUST_PROXYactive (check a real Set-Cookie header over HTTPS)
Academy (application-level smoke test)
- Owner bootstrap login works
- Instructor can create a cohort, roster a student
- Student can log in, boot a Lab, complete a scenario, see progress
- An assessment can be assigned, taken, and graded end-to-end
- Instructor analytics dashboards populate with real data
Backup
- Manual
backup-postgres.shrun succeeds against real Postgres, real off-host target reachable - A real restore (
restore-postgres.sh+verify-restore.sh) has been rehearsed at least once against a throwaway target, not just the local test harness — Phase 4.1'sinfra/backup/test-destructive-restore-drill.shproved the full mechanism (realistic data, real storage destruction, restore, real Platform API boot) locally; this line is the real-hardware confirmation - Cron job confirmed scheduled (
crontab -l)
Observability
- Grafana dashboard shows real data for both VMs
- Trigger one alert deliberately (e.g. stop a service) and confirm it fires and reaches the webhook
- Loki has logs flowing from both VMs
Pre-production hardening (Phase 4.1 — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md)
resource-policy.yaml'sadmission_control.max_concurrent_labsset from a real measuredinfra/scripts/capacity-model.jsrun against the real Lab Host, not left disabled- A deliberate over-capacity attempt returns the clean 503 "Lab capacity is temporarily full" message, not a 500 — confirms the real host's admission control matches the local test
docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md's steps still make sense once capacity numbers are real
Only once every box above is checked against the real server does the deployment graduate from "infrastructure package built" to "actually in production."