ced4568
5bc0743b08
Master overview (PHASE4_1_PREPRODUCTION_HARDENING.md) covering all 27 substantive items, plus REMOTE_AND_SELF_PACED_ARCHITECTURE.md (reviews what already works for remote/self-paced delivery - student-driven provisioning, cohort-less students, idle-reaper-destroy-then-continue - and the one real gap: student authentication, explicitly Phase 5 scope, not built here). Updated PRODUCTION_NETWORK_SECURITY.md (the Gateway-to-Lab gap is now RESOLVED, with the full design writeup and a new control-plane-vs-data-plane section), PRODUCTION_DEPLOYMENT.md (four-cert TLS bundle), PRODUCTION_VALIDATION_CHECKLIST.md (Router mTLS check, admission control checks), PHASE4_PRODUCTION_INFRASTRUCTURE.md and ENGINEERING_SOP.md and LAB_ARCHITECTURE.md (stale 'known gap' pointers updated to point at the resolution), BACKUP_DR.md (destructive drill), README.md and platform/README.md (Phase 4.1 pointers, lab-router component, updated known-limitations). Also fixes a real pre-existing bug found while auditing what gets committed: infra/.gitignore's patterns were written repo-root-relative (e.g. "infra/terraform/...") but the file lives in infra/, so git resolved them relative to THAT directory and none of them ever matched anything - a real .terraform/ provider cache showed up as untracked when it should have been ignored. Fixed by dropping the redundant leading "infra/" from every pattern. Status: PHASE 4.1 COMPLETE / PRE-PRODUCTION HARDENING COMPLETE / READY FOR HARDWARE DEPLOYMENT.
91 lines
3.7 KiB
Markdown
91 lines
3.7 KiB
Markdown
# Production Deployment
|
|
|
|
Step-by-step, for when the real Hetzner host exists. See
|
|
`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` for the architecture this executes.
|
|
|
|
## 0. One-time, manual, not automated (deliberately)
|
|
|
|
1. Install Proxmox VE on the bare-metal host — `infra/proxmox/README.md`.
|
|
2. Inspect the real disks, pick a ZFS layout — `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4,
|
|
`docs/RUNBOOK_DISK_FAILURE.md`'s inspection checklist.
|
|
3. Build the cloud-init VM template Terraform will clone —
|
|
`infra/proxmox/cloud-init/user-data.yaml` + `infra/proxmox/README.md`.
|
|
4. Fill in `infra/terraform/terraform.tfvars` (copy from `.tfvars.example`) with the real
|
|
Proxmox API endpoint, node name, storage/datastore names, and chosen VM sizes.
|
|
5. Fill in `infra/ansible/inventory/production.ini` with the real VM IPs, and
|
|
`infra/ansible/group_vars/production.yml` with `platform_ip`, `lab_host_ip`,
|
|
`management_cidr`, `proxmox_mgmt_cidr`.
|
|
6. Copy `infra/ansible/group_vars/production.vault.yml.template` →
|
|
`production.vault.yml`, fill in every `vault_*` secret with real generated values, then
|
|
`ansible-vault encrypt group_vars/production.vault.yml`. Never commit the plaintext.
|
|
|
|
## 1. Create the VMs
|
|
|
|
```bash
|
|
cd infra/terraform
|
|
tofu init
|
|
tofu plan # review before applying against real hardware
|
|
tofu apply
|
|
```
|
|
|
|
Creates `rn-platform-01` and `rn-lab-host-01` from the cloud-init template. Confirm both boot
|
|
and are SSH-reachable from the management network before continuing.
|
|
|
|
## 2. Generate TLS certificates (offline, once)
|
|
|
|
```bash
|
|
export ROOTNODE_TLS_BUNDLE_DIR=/secure/offline/path # never inside the repo
|
|
./infra/scripts/docker-tls/generate-docker-tls.sh
|
|
./infra/scripts/docker-tls/verify-docker-tls.sh
|
|
```
|
|
|
|
Generates FOUR leaf certs under one CA (Phase 4.1 extended this from two): the CONTROL-plane
|
|
pair (`server-*`/`provisioner-client-*`, for Provisioner↔dockerd) and the DATA-plane pair
|
|
(`router-server-*`/`gateway-client-*`, for Gateway↔Lab Host Router — see
|
|
`docs/PRODUCTION_NETWORK_SECURITY.md`'s "control plane vs data plane" section). The CA private
|
|
key stays in `ROOTNODE_TLS_BUNDLE_DIR` and never touches either VM — only the CA cert (public)
|
|
and each host's own leaf cert+key go to each host, both pairs deployed by the same `docker-tls`
|
|
Ansible role into two separate directories (`/etc/rootnode/docker-tls/` and
|
|
`/etc/rootnode/lab-router-tls/`).
|
|
|
|
## 3. Configure both hosts
|
|
|
|
```bash
|
|
cd infra/ansible
|
|
ansible-playbook -i inventory/production.ini site.yml --ask-vault-pass
|
|
```
|
|
|
|
Role order matters and is fixed in `site.yml`: `common` → `firewall` → `docker-tls` →
|
|
`platform`/`lab-host` → `monitoring-agent` → `backup`. Firewall and TLS land before anything
|
|
that depends on them.
|
|
|
|
## 4. First application deploy
|
|
|
|
```bash
|
|
./infra/scripts/deploy-production.sh <image-tag>
|
|
```
|
|
|
|
Pulls the tagged images, runs the compose stack, applies Postgres migrations (automatic, on
|
|
Platform API/Provisioner startup — see `docs/RUNBOOK_FAILED_MIGRATION.md` if one fails).
|
|
|
|
## 5. Verify
|
|
|
|
```bash
|
|
./infra/scripts/production-smoke-test.sh
|
|
```
|
|
|
|
Curls `/healthz` on the Gateway and Platform API, confirms Postgres is reachable, confirms
|
|
Prometheus/Grafana/Alertmanager are up. Then walk
|
|
`docs/PRODUCTION_VALIDATION_CHECKLIST.md` in full before calling the deployment live.
|
|
|
|
## Normal (non-first) deploys
|
|
|
|
`docs/RUNBOOK_NORMAL_DEPLOY.md`. Rollback: `docs/RUNBOOK_ROLLBACK.md`.
|
|
|
|
## What this document does NOT cover
|
|
|
|
DNS/Cloudflare setup for the public origin (a one-time manual step outside this repo's scope —
|
|
point the A/AAAA record at the real host and confirm Caddy issues its certificate). Which
|
|
storage layout to pick (decision framework only, §4 of the master doc — the real answer depends
|
|
on the real disks).
|