rootnode-academy/docs/PRODUCTION_VALIDATION_CHECKLIST.md

Production Validation Checklist

Walk this once, top to bottom, before calling a production deployment live. Every item here needs the real hardware — this is the "DEPLOY LATER" half of Phase 4, not something this repo can prove in advance.

Hardware / Proxmox

• Disk inspection done, layout chosen and matches what's actually in the server (docs/RUNBOOK_DISK_FAILURE.md inspection checklist, docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md §4)
• Proxmox VE installed, reachable only from management_cidr/proxmox_mgmt_cidr
• VM template built from real cloud-init (infra/proxmox/README.md)
• tofu apply created both VMs; both boot and are SSH-reachable

Platform VM

• ansible-playbook site.yml completed clean, all roles
• docker compose ps shows every service healthy (postgres, provisioner, api, gateway, reverse-proxy, prometheus, alertmanager, grafana, loki, promtail, exporters)
• curl https://<origin>/healthz (via Gateway) returns 200
• curl http://localhost:4300/healthz (Platform API, from the VM) returns {"ok":true,"db":"ok"}
• TLS certificate issued and valid for the real domain

Lab Host VM

• Docker daemon TLS verified: docker --tlsverify --tlscacert=... -H tcp://<lab_host_ip>:2376 info succeeds from the Platform VM, fails from anywhere else
• Lab Host Router mTLS verified: a request to https://<lab_host_ip>:8443 with no client cert is rejected; with the Gateway's real client cert it isn't (mirrors infra/scripts/simulate-two-host/test-remote-routing.js's "direct public access: NOT AVAILABLE" case, now against the real host)
• Provisioner successfully creates a real Lab against the real Lab Host
• Gateway can actually reach that Lab's terminal (HTTP UI load + WebSocket kubectl command) — application mechanism built and simulation-tested in Phase 4.1 (docs/PHASE4_1_PREPRODUCTION_HARDENING.md §1/§6/§7); this line is the real-hardware confirmation that the same result holds across the actual two VMs, not two Docker networks on one daemon
• Egress proxy correctly restricts a Lab's outbound traffic (Provisioner's per-Lab DOCKER-USER chain)

Security

• infra/firewall/validate-firewall.sh run against the real hosts with real iptables-restore --test, not just the structural lint
• Confirm from an unauthorized IP: Postgres, Provisioner, Docker TLS port, Proxmox UI are all unreachable
• Confirm Student Lab → another Lab is blocked; confirm Student Lab → Proxmox is blocked
• COOKIE_SECURE/TRUST_PROXY active (check a real Set-Cookie header over HTTPS)

Academy (application-level smoke test)

• Owner bootstrap login works
• Instructor can create a cohort, roster a student
• Student can log in, boot a Lab, complete a scenario, see progress
• An assessment can be assigned, taken, and graded end-to-end
• Instructor analytics dashboards populate with real data

Backup

• Manual backup-postgres.sh run succeeds against real Postgres, real off-host target reachable
• A real restore (restore-postgres.sh + verify-restore.sh) has been rehearsed at least once against a throwaway target, not just the local test harness — Phase 4.1's infra/backup/test-destructive-restore-drill.sh proved the full mechanism (realistic data, real storage destruction, restore, real Platform API boot) locally; this line is the real-hardware confirmation
• Cron job confirmed scheduled (crontab -l)

Observability

• Grafana dashboard shows real data for both VMs
• Trigger one alert deliberately (e.g. stop a service) and confirm it fires and reaches the webhook
• Loki has logs flowing from both VMs

Pre-production hardening (Phase 4.1 — see docs/PHASE4_1_PREPRODUCTION_HARDENING.md)

• resource-policy.yaml's admission_control.max_concurrent_labs set from a real measured infra/scripts/capacity-model.js run against the real Lab Host, not left disabled
• A deliberate over-capacity attempt returns the clean 503 "Lab capacity is temporarily full" message, not a 500 — confirms the real host's admission control matches the local test
• docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md's steps still make sense once capacity numbers are real

Student identity & remote classroom (Phase 5A — see docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md)

• COOKIE_SECURE=1 in the real production environment — confirms /api/student/ request-password-reset never returns dev_reset_token outside of dev/CI
• A real activation/reset link (?activate=/?reset=) round-trips against the production domain, not just localhost — query-param parsing, no router library in play
• Student login rate limiting still 429s after a restart against the real production Postgres, same as the local e2e-phase5a.js check
• GET /metrics is reachable by the real Prometheus instance (infra/monitoring/ prometheus/prometheus.yml.j2's rootnode-platform-api job) and no student/instructor ID appears in any label value
• docs/RUNBOOK_STUDENT_ACCOUNT_LIFECYCLE.md and docs/RUNBOOK_REMOTE_CLASSROOM.md's steps still make sense walked against the real admin portal

Self-paced enrollment & website integration (Phase 5B — see docs/PHASE5B_SELF_PACED_ENROLLMENT.md)

• INTEGRATION_SHARED_SECRET is set to a real, unique-to-production secret (not the dev value) and is stored only in the Platform API's own server-side environment — never committed, never in any browser-reachable config
• The website team's production integration key was delivered out of band (not plaintext email/chat) and is confirmed working with one real, low-stakes test enrollment before go-live
• A deliberate POST /api/integrations/enrollments call missing/with a wrong key returns a clean 401 against the real production endpoint, not a 500 or an information leak
• GET /metrics's rootnode_enrollments_total/rootnode_program_completions_total labels contain only status/delivery_mode/program_id values — confirm no student_id, email, or external_reference ever appears in a real production scrape
• docs/RUNBOOK_SELF_PACED_ENROLLMENT.md's steps still make sense walked against the real admin portal, and docs/WEBSITE_ENROLLMENT_INTEGRATION.md is what the website team actually integrated against (update it if reality drifted during that integration)

Enrollment communications / Google Workspace email (Phase 5C — see docs/PHASE5C_ENROLLMENT_COMMUNICATIONS.md)

• Every step in docs/RUNBOOK_GOOGLE_WORKSPACE_EMAIL.md marked REQUIRES GOOGLE WORKSPACE ADMIN ACTION has actually been completed (sending mailbox, service account, domain-wide delegation scoped to gmail.send only, SPF/DKIM/DMARC)
• EMAIL_PROVIDER=gws set in the real production environment — confirm it is NOT left at the dev-provider default, which would silently mean no real student ever receives email
• GOOGLE_SERVICE_ACCOUNT_JSON, EMAIL_FROM_ADDRESS, EMAIL_FROM_NAME, EMAIL_REPLY_TO, and PUBLIC_APP_ORIGIN are all set from the real production secret store, not left at any dev/example value
• infra/scripts/gws-email-test.js (opt-in, RUN_GWS_EMAIL_TEST=true) run once against a real TEST_EMAIL_RECIPIENT and confirmed delivered — the one and only "does this actually work" proof this checklist can't fake
• COOKIE_SECURE=1 in the real production environment — confirms the dev-preview notification route (GET /api/admin/notifications/:id/dev-preview) is unreachable and no activation/reset token is ever returned by any API response
• GET /metrics's notification gauges/counters contain no student_id/email/token/ notification_id/external_event_id in any label value on a real production scrape
• docs/RUNBOOK_STUDENT_ACTIVATION.md's steps still make sense walked against the real admin portal, including a real (not dev-provider) activation email actually arriving

Only once every box above is checked against the real server does the deployment graduate from "infrastructure package built" to "actually in production."