# Production Validation Checklist Walk this once, top to bottom, before calling a production deployment live. Every item here needs the real hardware — this is the "DEPLOY LATER" half of Phase 4, not something this repo can prove in advance. ## Hardware / Proxmox - [ ] Disk inspection done, layout chosen and matches what's actually in the server (`docs/RUNBOOK_DISK_FAILURE.md` inspection checklist, `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4) - [ ] Proxmox VE installed, reachable only from `management_cidr`/`proxmox_mgmt_cidr` - [ ] VM template built from real cloud-init (`infra/proxmox/README.md`) - [ ] `tofu apply` created both VMs; both boot and are SSH-reachable ## Platform VM - [ ] `ansible-playbook site.yml` completed clean, all roles - [ ] `docker compose ps` shows every service healthy (postgres, provisioner, api, gateway, reverse-proxy, prometheus, alertmanager, grafana, loki, promtail, exporters) - [ ] `curl https:///healthz` (via Gateway) returns 200 - [ ] `curl http://localhost:4300/healthz` (Platform API, from the VM) returns `{"ok":true,"db":"ok"}` - [ ] TLS certificate issued and valid for the real domain ## Lab Host VM - [ ] Docker daemon TLS verified: `docker --tlsverify --tlscacert=... -H tcp://:2376 info` succeeds from the Platform VM, fails from anywhere else - [ ] Lab Host Router mTLS verified: a request to `https://:8443` with no client cert is rejected; with the Gateway's real client cert it isn't (mirrors `infra/scripts/simulate-two-host/test-remote-routing.js`'s "direct public access: NOT AVAILABLE" case, now against the real host) - [ ] Provisioner successfully creates a real Lab against the real Lab Host - [ ] Gateway can actually reach that Lab's terminal (HTTP UI load + WebSocket `kubectl` command) — application mechanism built and simulation-tested in Phase 4.1 (`docs/PHASE4_1_PREPRODUCTION_HARDENING.md` §1/§6/§7); this line is the real-hardware confirmation that the same result holds across the actual two VMs, not two Docker networks on one daemon - [ ] Egress proxy correctly restricts a Lab's outbound traffic (Provisioner's per-Lab `DOCKER-USER` chain) ## Security - [ ] `infra/firewall/validate-firewall.sh` run against the real hosts with real `iptables-restore --test`, not just the structural lint - [ ] Confirm from an unauthorized IP: Postgres, Provisioner, Docker TLS port, Proxmox UI are all unreachable - [ ] Confirm Student Lab → another Lab is blocked; confirm Student Lab → Proxmox is blocked - [ ] `COOKIE_SECURE`/`TRUST_PROXY` active (check a real Set-Cookie header over HTTPS) ## Academy (application-level smoke test) - [ ] Owner bootstrap login works - [ ] Instructor can create a cohort, roster a student - [ ] Student can log in, boot a Lab, complete a scenario, see progress - [ ] An assessment can be assigned, taken, and graded end-to-end - [ ] Instructor analytics dashboards populate with real data ## Backup - [ ] Manual `backup-postgres.sh` run succeeds against real Postgres, real off-host target reachable - [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness — Phase 4.1's `infra/backup/test-destructive-restore-drill.sh` proved the full mechanism (realistic data, real storage destruction, restore, real Platform API boot) locally; this line is the real-hardware confirmation - [ ] Cron job confirmed scheduled (`crontab -l`) ## Observability - [ ] Grafana dashboard shows real data for both VMs - [ ] Trigger one alert deliberately (e.g. stop a service) and confirm it fires and reaches the webhook - [ ] Loki has logs flowing from both VMs ## Pre-production hardening (Phase 4.1 — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md) - [ ] `resource-policy.yaml`'s `admission_control.max_concurrent_labs` set from a real measured `infra/scripts/capacity-model.js` run against the real Lab Host, not left disabled - [ ] A deliberate over-capacity attempt returns the clean 503 "Lab capacity is temporarily full" message, not a 500 — confirms the real host's admission control matches the local test - [ ] `docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md`'s steps still make sense once capacity numbers are real ## Student identity & remote classroom (Phase 5A — see docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md) - [ ] `COOKIE_SECURE=1` in the real production environment — confirms `/api/student/ request-password-reset` never returns `dev_reset_token` outside of dev/CI - [ ] A real activation/reset link (`?activate=`/`?reset=`) round-trips against the production domain, not just `localhost` — query-param parsing, no router library in play - [ ] Student login rate limiting still 429s after a restart against the real production Postgres, same as the local `e2e-phase5a.js` check - [ ] `GET /metrics` is reachable by the real Prometheus instance (`infra/monitoring/ prometheus/prometheus.yml.j2`'s `rootnode-platform-api` job) and no student/instructor ID appears in any label value - [ ] `docs/RUNBOOK_STUDENT_ACCOUNT_LIFECYCLE.md` and `docs/RUNBOOK_REMOTE_CLASSROOM.md`'s steps still make sense walked against the real admin portal ## Self-paced enrollment & website integration (Phase 5B — see docs/PHASE5B_SELF_PACED_ENROLLMENT.md) - [ ] `INTEGRATION_SHARED_SECRET` is set to a real, unique-to-production secret (not the dev value) and is stored only in the Platform API's own server-side environment — never committed, never in any browser-reachable config - [ ] The website team's production integration key was delivered out of band (not plaintext email/chat) and is confirmed working with one real, low-stakes test enrollment before go-live - [ ] A deliberate `POST /api/integrations/enrollments` call missing/with a wrong key returns a clean `401` against the real production endpoint, not a 500 or an information leak - [ ] `GET /metrics`'s `rootnode_enrollments_total`/`rootnode_program_completions_total` labels contain only status/delivery_mode/program_id values — confirm no student_id, email, or external_reference ever appears in a real production scrape - [ ] `docs/RUNBOOK_SELF_PACED_ENROLLMENT.md`'s steps still make sense walked against the real admin portal, and `docs/WEBSITE_ENROLLMENT_INTEGRATION.md` is what the website team actually integrated against (update it if reality drifted during that integration) ## Enrollment communications / Google Workspace email (Phase 5C — see docs/PHASE5C_ENROLLMENT_COMMUNICATIONS.md) - [ ] Every step in `docs/RUNBOOK_GOOGLE_WORKSPACE_EMAIL.md` marked REQUIRES GOOGLE WORKSPACE ADMIN ACTION has actually been completed (sending mailbox, service account, domain-wide delegation scoped to `gmail.send` only, SPF/DKIM/DMARC) - [ ] `EMAIL_PROVIDER=gws` set in the real production environment — confirm it is NOT left at the dev-provider default, which would silently mean no real student ever receives email - [ ] `GOOGLE_SERVICE_ACCOUNT_JSON`, `EMAIL_FROM_ADDRESS`, `EMAIL_FROM_NAME`, `EMAIL_REPLY_TO`, and `PUBLIC_APP_ORIGIN` are all set from the real production secret store, not left at any dev/example value - [ ] `infra/scripts/gws-email-test.js` (opt-in, `RUN_GWS_EMAIL_TEST=true`) run once against a real `TEST_EMAIL_RECIPIENT` and confirmed delivered — the one and only "does this actually work" proof this checklist can't fake - [ ] `COOKIE_SECURE=1` in the real production environment — confirms the dev-preview notification route (`GET /api/admin/notifications/:id/dev-preview`) is unreachable and no activation/reset token is ever returned by any API response - [ ] `GET /metrics`'s notification gauges/counters contain no student_id/email/token/ notification_id/external_event_id in any label value on a real production scrape - [ ] `docs/RUNBOOK_STUDENT_ACTIVATION.md`'s steps still make sense walked against the real admin portal, including a real (not dev-provider) activation email actually arriving Only once every box above is checked against the real server does the deployment graduate from "infrastructure package built" to "actually in production."