# Production Validation Checklist Walk this once, top to bottom, before calling a production deployment live. Every item here needs the real hardware — this is the "DEPLOY LATER" half of Phase 4, not something this repo can prove in advance. ## Hardware / Proxmox - [ ] Disk inspection done, layout chosen and matches what's actually in the server (`docs/RUNBOOK_DISK_FAILURE.md` inspection checklist, `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4) - [ ] Proxmox VE installed, reachable only from `management_cidr`/`proxmox_mgmt_cidr` - [ ] VM template built from real cloud-init (`infra/proxmox/README.md`) - [ ] `tofu apply` created both VMs; both boot and are SSH-reachable ## Platform VM - [ ] `ansible-playbook site.yml` completed clean, all roles - [ ] `docker compose ps` shows every service healthy (postgres, provisioner, api, gateway, reverse-proxy, prometheus, alertmanager, grafana, loki, promtail, exporters) - [ ] `curl https:///healthz` (via Gateway) returns 200 - [ ] `curl http://localhost:4300/healthz` (Platform API, from the VM) returns `{"ok":true,"db":"ok"}` - [ ] TLS certificate issued and valid for the real domain ## Lab Host VM - [ ] Docker daemon TLS verified: `docker --tlsverify --tlscacert=... -H tcp://:2376 info` succeeds from the Platform VM, fails from anywhere else - [ ] Lab Host Router mTLS verified: a request to `https://:8443` with no client cert is rejected; with the Gateway's real client cert it isn't (mirrors `infra/scripts/simulate-two-host/test-remote-routing.js`'s "direct public access: NOT AVAILABLE" case, now against the real host) - [ ] Provisioner successfully creates a real Lab against the real Lab Host - [ ] Gateway can actually reach that Lab's terminal (HTTP UI load + WebSocket `kubectl` command) — application mechanism built and simulation-tested in Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1/§6/§7); this line is the real-hardware confirmation that the same result holds across the actual two VMs, not two Docker networks on one daemon - [ ] Egress proxy correctly restricts a Lab's outbound traffic (Provisioner's per-Lab `DOCKER-USER` chain) ## Security - [ ] `infra/firewall/validate-firewall.sh` run against the real hosts with real `iptables-restore --test`, not just the structural lint - [ ] Confirm from an unauthorized IP: Postgres, Provisioner, Docker TLS port, Proxmox UI are all unreachable - [ ] Confirm Student Lab → another Lab is blocked; confirm Student Lab → Proxmox is blocked - [ ] `COOKIE_SECURE`/`TRUST_PROXY` active (check a real Set-Cookie header over HTTPS) ## Academy (application-level smoke test) - [ ] Owner bootstrap login works - [ ] Instructor can create a cohort, roster a student - [ ] Student can log in, boot a Lab, complete a scenario, see progress - [ ] An assessment can be assigned, taken, and graded end-to-end - [ ] Instructor analytics dashboards populate with real data ## Backup - [ ] Manual `backup-postgres.sh` run succeeds against real Postgres, real off-host target reachable - [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness — Phase 4.1's `infra/backup/test-destructive-restore-drill.sh` proved the full mechanism (realistic data, real storage destruction, restore, real Platform API boot) locally; this line is the real-hardware confirmation - [ ] Cron job confirmed scheduled (`crontab -l`) ## Observability - [ ] Grafana dashboard shows real data for both VMs - [ ] Trigger one alert deliberately (e.g. stop a service) and confirm it fires and reaches the webhook - [ ] Loki has logs flowing from both VMs ## Pre-production hardening (Phase 4.1 — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md) - [ ] `resource-policy.yaml`'s `admission_control.max_concurrent_labs` set from a real measured `infra/scripts/capacity-model.js` run against the real Lab Host, not left disabled - [ ] A deliberate over-capacity attempt returns the clean 503 "Lab capacity is temporarily full" message, not a 500 — confirms the real host's admission control matches the local test - [ ] `docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md`'s steps still make sense once capacity numbers are real ## Student identity & remote classroom (Phase 5A — see docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md) - [ ] `COOKIE_SECURE=1` in the real production environment — confirms `/api/student/ request-password-reset` never returns `dev_reset_token` outside of dev/CI - [ ] A real activation/reset link (`?activate=`/`?reset=`) round-trips against the production domain, not just `localhost` — query-param parsing, no router library in play - [ ] Student login rate limiting still 429s after a restart against the real production Postgres, same as the local `e2e-phase5a.js` check - [ ] `GET /metrics` is reachable by the real Prometheus instance (`infra/monitoring/ prometheus/prometheus.yml.j2`'s `rootnode-platform-api` job) and no student/instructor ID appears in any label value - [ ] `docs/RUNBOOK_STUDENT_ACCOUNT_LIFECYCLE.md` and `docs/RUNBOOK_REMOTE_CLASSROOM.md`'s steps still make sense walked against the real admin portal ## Self-paced enrollment & website integration (Phase 5B — see docs/PHASE5B_SELF_PACED_ENROLLMENT.md) - [ ] `INTEGRATION_SHARED_SECRET` is set to a real, unique-to-production secret (not the dev value) and is stored only in the Platform API's own server-side environment — never committed, never in any browser-reachable config - [ ] The website team's production integration key was delivered out of band (not plaintext email/chat) and is confirmed working with one real, low-stakes test enrollment before go-live - [ ] A deliberate `POST /api/integrations/enrollments` call missing/with a wrong key returns a clean `401` against the real production endpoint, not a 500 or an information leak - [ ] `GET /metrics`'s `rootnode_enrollments_total`/`rootnode_program_completions_total` labels contain only status/delivery_mode/program_id values — confirm no student_id, email, or external_reference ever appears in a real production scrape - [ ] `docs/RUNBOOK_SELF_PACED_ENROLLMENT.md`'s steps still make sense walked against the real admin portal, and `docs/WEBSITE_ENROLLMENT_INTEGRATION.md` is what the website team actually integrated against (update it if reality drifted during that integration) Only once every box above is checked against the real server does the deployment graduate from "infrastructure package built" to "actually in production."