ced4568
bf1e1b5c76
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs (PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR, OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST, and 12 runbooks (first deployment, normal deploy, rollback, Postgres restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure). PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine architectural gap this phase surfaced: the Gateway's existing env.internal_ip mechanism for reaching a Lab container assumes Gateway and Lab share a Docker daemon, which breaks once Labs run on a genuinely separate rn-lab-host-01 VM. The network-policy half is built (a firewall allowance for Gateway->Lab-Host on a published port range); the application half (Provisioner/Gateway address handling) is an explicit, reviewed, NOT-YET-BUILT follow-up flagged there rather than silently decided. Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and platform/README.md with Phase 4 pointers, consistent with how every prior phase updated these. Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
85 lines
3.2 KiB
Markdown
85 lines
3.2 KiB
Markdown
# Production Deployment
|
|
|
|
Step-by-step, for when the real Hetzner host exists. See
|
|
`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` for the architecture this executes.
|
|
|
|
## 0. One-time, manual, not automated (deliberately)
|
|
|
|
1. Install Proxmox VE on the bare-metal host — `infra/proxmox/README.md`.
|
|
2. Inspect the real disks, pick a ZFS layout — `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4,
|
|
`docs/RUNBOOK_DISK_FAILURE.md`'s inspection checklist.
|
|
3. Build the cloud-init VM template Terraform will clone —
|
|
`infra/proxmox/cloud-init/user-data.yaml` + `infra/proxmox/README.md`.
|
|
4. Fill in `infra/terraform/terraform.tfvars` (copy from `.tfvars.example`) with the real
|
|
Proxmox API endpoint, node name, storage/datastore names, and chosen VM sizes.
|
|
5. Fill in `infra/ansible/inventory/production.ini` with the real VM IPs, and
|
|
`infra/ansible/group_vars/production.yml` with `platform_ip`, `lab_host_ip`,
|
|
`management_cidr`, `proxmox_mgmt_cidr`.
|
|
6. Copy `infra/ansible/group_vars/production.vault.yml.template` →
|
|
`production.vault.yml`, fill in every `vault_*` secret with real generated values, then
|
|
`ansible-vault encrypt group_vars/production.vault.yml`. Never commit the plaintext.
|
|
|
|
## 1. Create the VMs
|
|
|
|
```bash
|
|
cd infra/terraform
|
|
tofu init
|
|
tofu plan # review before applying against real hardware
|
|
tofu apply
|
|
```
|
|
|
|
Creates `rn-platform-01` and `rn-lab-host-01` from the cloud-init template. Confirm both boot
|
|
and are SSH-reachable from the management network before continuing.
|
|
|
|
## 2. Generate Docker TLS certificates (offline, once)
|
|
|
|
```bash
|
|
export ROOTNODE_TLS_BUNDLE_DIR=/secure/offline/path # never inside the repo
|
|
./infra/scripts/docker-tls/generate-docker-tls.sh
|
|
./infra/scripts/docker-tls/verify-docker-tls.sh
|
|
```
|
|
|
|
The CA private key stays in `ROOTNODE_TLS_BUNDLE_DIR` and never touches either VM — only the CA
|
|
cert (public) and the relevant leaf cert+key go to each host.
|
|
|
|
## 3. Configure both hosts
|
|
|
|
```bash
|
|
cd infra/ansible
|
|
ansible-playbook -i inventory/production.ini site.yml --ask-vault-pass
|
|
```
|
|
|
|
Role order matters and is fixed in `site.yml`: `common` → `firewall` → `docker-tls` →
|
|
`platform`/`lab-host` → `monitoring-agent` → `backup`. Firewall and TLS land before anything
|
|
that depends on them.
|
|
|
|
## 4. First application deploy
|
|
|
|
```bash
|
|
./infra/scripts/deploy-production.sh <image-tag>
|
|
```
|
|
|
|
Pulls the tagged images, runs the compose stack, applies Postgres migrations (automatic, on
|
|
Platform API/Provisioner startup — see `docs/RUNBOOK_FAILED_MIGRATION.md` if one fails).
|
|
|
|
## 5. Verify
|
|
|
|
```bash
|
|
./infra/scripts/production-smoke-test.sh
|
|
```
|
|
|
|
Curls `/healthz` on the Gateway and Platform API, confirms Postgres is reachable, confirms
|
|
Prometheus/Grafana/Alertmanager are up. Then walk
|
|
`docs/PRODUCTION_VALIDATION_CHECKLIST.md` in full before calling the deployment live.
|
|
|
|
## Normal (non-first) deploys
|
|
|
|
`docs/RUNBOOK_NORMAL_DEPLOY.md`. Rollback: `docs/RUNBOOK_ROLLBACK.md`.
|
|
|
|
## What this document does NOT cover
|
|
|
|
DNS/Cloudflare setup for the public origin (a one-time manual step outside this repo's scope —
|
|
point the A/AAAA record at the real host and confirm Caddy issues its certificate). Which
|
|
storage layout to pick (decision framework only, §4 of the master doc — the real answer depends
|
|
on the real disks).
|