ced4568
bf1e1b5c76
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs (PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR, OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST, and 12 runbooks (first deployment, normal deploy, rollback, Postgres restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure). PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine architectural gap this phase surfaced: the Gateway's existing env.internal_ip mechanism for reaching a Lab container assumes Gateway and Lab share a Docker daemon, which breaks once Labs run on a genuinely separate rn-lab-host-01 VM. The network-policy half is built (a firewall allowance for Gateway->Lab-Host on a published port range); the application half (Provisioner/Gateway address handling) is an explicit, reviewed, NOT-YET-BUILT follow-up flagged there rather than silently decided. Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and platform/README.md with Phase 4 pointers, consistent with how every prior phase updated these. Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
3.2 KiB
3.2 KiB
Production Validation Checklist
Walk this once, top to bottom, before calling a production deployment live. Every item here needs the real hardware — this is the "DEPLOY LATER" half of Phase 4, not something this repo can prove in advance.
Hardware / Proxmox
- Disk inspection done, layout chosen and matches what's actually in the server (
docs/RUNBOOK_DISK_FAILURE.mdinspection checklist,docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md§4) - Proxmox VE installed, reachable only from
management_cidr/proxmox_mgmt_cidr - VM template built from real cloud-init (
infra/proxmox/README.md) tofu applycreated both VMs; both boot and are SSH-reachable
Platform VM
ansible-playbook site.ymlcompleted clean, all rolesdocker compose psshows every service healthy (postgres, provisioner, api, gateway, reverse-proxy, prometheus, alertmanager, grafana, loki, promtail, exporters)curl https://<origin>/healthz(via Gateway) returns 200curl http://localhost:4300/healthz(Platform API, from the VM) returns{"ok":true,"db":"ok"}- TLS certificate issued and valid for the real domain
Lab Host VM
- Docker daemon TLS verified:
docker --tlsverify --tlscacert=... -H tcp://<lab_host_ip>:2376 infosucceeds from the Platform VM, fails from anywhere else - Provisioner successfully creates a real Lab against the real Lab Host
- Gateway can actually reach that Lab's terminal — this depends on the not-yet-built application change documented in
docs/PRODUCTION_NETWORK_SECURITY.md's "Known gap" section; do not mark this item done until that change has landed and been verified for real - Egress proxy correctly restricts a Lab's outbound traffic (Provisioner's per-Lab
DOCKER-USERchain)
Security
infra/firewall/validate-firewall.shrun against the real hosts with realiptables-restore --test, not just the structural lint- Confirm from an unauthorized IP: Postgres, Provisioner, Docker TLS port, Proxmox UI are all unreachable
- Confirm Student Lab → another Lab is blocked; confirm Student Lab → Proxmox is blocked
COOKIE_SECURE/TRUST_PROXYactive (check a real Set-Cookie header over HTTPS)
Academy (application-level smoke test)
- Owner bootstrap login works
- Instructor can create a cohort, roster a student
- Student can log in, boot a Lab, complete a scenario, see progress
- An assessment can be assigned, taken, and graded end-to-end
- Instructor analytics dashboards populate with real data
Backup
- Manual
backup-postgres.shrun succeeds against real Postgres, real off-host target reachable - A real restore (
restore-postgres.sh+verify-restore.sh) has been rehearsed at least once against a throwaway target, not just the local test harness - Cron job confirmed scheduled (
crontab -l)
Observability
- Grafana dashboard shows real data for both VMs
- Trigger one alert deliberately (e.g. stop a service) and confirm it fires and reaches the webhook
- Loki has logs flowing from both VMs
Only once every box above is checked against the real server does the deployment graduate from "infrastructure package built" to "actually in production."