# Production Validation Checklist

Walk this once, top to bottom, before calling a production deployment live. Every item here
needs the real hardware — this is the "DEPLOY LATER" half of Phase 4, not something this repo
can prove in advance.

## Hardware / Proxmox
- [ ] Disk inspection done, layout chosen and matches what's actually in the server (`docs/RUNBOOK_DISK_FAILURE.md` inspection checklist, `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4)
- [ ] Proxmox VE installed, reachable only from `management_cidr`/`proxmox_mgmt_cidr`
- [ ] VM template built from real cloud-init (`infra/proxmox/README.md`)
- [ ] `tofu apply` created both VMs; both boot and are SSH-reachable

## Platform VM
- [ ] `ansible-playbook site.yml` completed clean, all roles
- [ ] `docker compose ps` shows every service healthy (postgres, provisioner, api, gateway, reverse-proxy, prometheus, alertmanager, grafana, loki, promtail, exporters)
- [ ] `curl https://<origin>/healthz` (via Gateway) returns 200
- [ ] `curl http://localhost:4300/healthz` (Platform API, from the VM) returns `{"ok":true,"db":"ok"}`
- [ ] TLS certificate issued and valid for the real domain

## Lab Host VM
- [ ] Docker daemon TLS verified: `docker --tlsverify --tlscacert=... -H tcp://<lab_host_ip>:2376 info` succeeds from the Platform VM, fails from anywhere else
- [ ] Provisioner successfully creates a real Lab against the real Lab Host
- [ ] Gateway can actually reach that Lab's terminal — **this depends on the not-yet-built application change documented in `docs/PRODUCTION_NETWORK_SECURITY.md`'s "Known gap" section; do not mark this item done until that change has landed and been verified for real**
- [ ] Egress proxy correctly restricts a Lab's outbound traffic (Provisioner's per-Lab `DOCKER-USER` chain)

## Security
- [ ] `infra/firewall/validate-firewall.sh` run against the real hosts with real `iptables-restore --test`, not just the structural lint
- [ ] Confirm from an unauthorized IP: Postgres, Provisioner, Docker TLS port, Proxmox UI are all unreachable
- [ ] Confirm Student Lab → another Lab is blocked; confirm Student Lab → Proxmox is blocked
- [ ] `COOKIE_SECURE`/`TRUST_PROXY` active (check a real Set-Cookie header over HTTPS)

## Academy (application-level smoke test)
- [ ] Owner bootstrap login works
- [ ] Instructor can create a cohort, roster a student
- [ ] Student can log in, boot a Lab, complete a scenario, see progress
- [ ] An assessment can be assigned, taken, and graded end-to-end
- [ ] Instructor analytics dashboards populate with real data

## Backup
- [ ] Manual `backup-postgres.sh` run succeeds against real Postgres, real off-host target reachable
- [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness
- [ ] Cron job confirmed scheduled (`crontab -l`)

## Observability
- [ ] Grafana dashboard shows real data for both VMs
- [ ] Trigger one alert deliberately (e.g. stop a service) and confirm it fires and reaches the webhook
- [ ] Loki has logs flowing from both VMs

Only once every box above is checked against the real server does the deployment graduate from
"infrastructure package built" to "actually in production."