ced4568
bf1e1b5c76
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs (PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR, OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST, and 12 runbooks (first deployment, normal deploy, rollback, Postgres restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure). PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine architectural gap this phase surfaced: the Gateway's existing env.internal_ip mechanism for reaching a Lab container assumes Gateway and Lab share a Docker daemon, which breaks once Labs run on a genuinely separate rn-lab-host-01 VM. The network-policy half is built (a firewall allowance for Gateway->Lab-Host on a published port range); the application half (Provisioner/Gateway address handling) is an explicit, reviewed, NOT-YET-BUILT follow-up flagged there rather than silently decided. Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and platform/README.md with Phase 4 pointers, consistent with how every prior phase updated these. Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
32 lines
1.7 KiB
Markdown
32 lines
1.7 KiB
Markdown
# Runbook: Docker TLS Rotation
|
|
|
|
Rotate on a schedule (recommend yearly, well before the ~2.25-year cert lifetime
|
|
`generate-docker-tls.sh` defaults to) or immediately if a client cert may have leaked.
|
|
|
|
```bash
|
|
# 1. Generate a fresh bundle (new CA too — simplest, avoids any question about whether the old
|
|
# CA's private key was ever exposed)
|
|
cd infra/scripts/docker-tls
|
|
./generate-docker-tls.sh --out-dir ./out-$(date +%Y%m%d) --lab-host-ip <real-ip> --lab-host-dns rn-lab-host-01.internal
|
|
./verify-docker-tls.sh ./out-$(date +%Y%m%d)
|
|
|
|
# 2. Redeploy — this restarts dockerd on rn-lab-host-01 (brief interruption to any in-flight
|
|
# Docker API call from the Provisioner, NOT to already-running student Labs — their
|
|
# containers keep running through a dockerd restart)
|
|
cd ../../ansible
|
|
ROOTNODE_TLS_BUNDLE_DIR=../scripts/docker-tls/out-$(date +%Y%m%d) \
|
|
ansible-playbook site.yml -i inventory/production.ini --tags docker-tls --ask-vault-pass
|
|
|
|
# 3. Verify the Provisioner can still reach rn-lab-host-01
|
|
curl -s http://localhost:4100/health # from rn-platform-01
|
|
```
|
|
|
|
**Troubleshooting a failed rotation:**
|
|
- `x509: certificate signed by unknown authority` — the new ca.pem wasn't deployed to both
|
|
sides, or an old one is cached; re-run the docker-tls role on BOTH hosts.
|
|
- Provisioner logs `ECONNREFUSED` on port 2376 — dockerd didn't actually restart with the new
|
|
daemon.json; check `systemctl status docker` on rn-lab-host-01 and the ExecStart override
|
|
(`infra/ansible/roles/docker-tls/templates/docker-override.conf.j2`) actually applied.
|
|
- Firewall — confirm `infra/firewall/rn-lab-host-01.rules`' 2376 rule still matches
|
|
rn-platform-01's actual IP (didn't change as part of this rotation, but worth a sanity check).
|