ced4568
bf1e1b5c76
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs (PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR, OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST, and 12 runbooks (first deployment, normal deploy, rollback, Postgres restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure). PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine architectural gap this phase surfaced: the Gateway's existing env.internal_ip mechanism for reaching a Lab container assumes Gateway and Lab share a Docker daemon, which breaks once Labs run on a genuinely separate rn-lab-host-01 VM. The network-policy half is built (a firewall allowance for Gateway->Lab-Host on a published port range); the application half (Provisioner/Gateway address handling) is an explicit, reviewed, NOT-YET-BUILT follow-up flagged there rather than silently decided. Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and platform/README.md with Phase 4 pointers, consistent with how every prior phase updated these. Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
46 lines
2.1 KiB
Markdown
46 lines
2.1 KiB
Markdown
# Runbook: Postgres Restore
|
|
|
|
**When to use:** Postgres corruption/data loss on rn-platform-01, or verifying a backup is
|
|
actually good (do this periodically even without an incident — an untested backup is a
|
|
hypothesis, not a backup).
|
|
|
|
```bash
|
|
# 1. Find the backup to restore
|
|
ls -la /srv/rootnode-data/backups/ # or wherever backup_offsite_target points, if local copies are gone
|
|
|
|
# 2. Create a FRESH target database — never restore over the live one directly
|
|
docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
|
|
"CREATE DATABASE rootnode_platform_restore_test;"
|
|
|
|
# 3. Restore into it
|
|
source /etc/rootnode/backup.env # for BACKUP_ENCRYPTION_PASSPHRASE
|
|
/opt/rootnode/backup/restore-postgres.sh \
|
|
/srv/rootnode-data/backups/rootnode-platform-<timestamp>.sql.gz.gpg \
|
|
"postgres://rootnode_platform:$PGPASS@localhost:5432/rootnode_platform_restore_test"
|
|
# type RESTORE when prompted
|
|
|
|
# 4. Verify
|
|
/opt/rootnode/backup/verify-restore.sh \
|
|
"postgres://rootnode_platform:$PGPASS@localhost:5432/rootnode_platform_restore_test"
|
|
|
|
# 5a. If this was a drill: drop the test database
|
|
docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
|
|
"DROP DATABASE rootnode_platform_restore_test;"
|
|
|
|
# 5b. If this was a REAL incident: stop the app stack, rename the broken DB, rename the
|
|
# restored one into its place, restart.
|
|
systemctl stop rootnode-platform
|
|
docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
|
|
"ALTER DATABASE rootnode_platform RENAME TO rootnode_platform_broken_$(date +%s);"
|
|
docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
|
|
"ALTER DATABASE rootnode_platform_restore_test RENAME TO rootnode_platform;"
|
|
systemctl start rootnode-platform
|
|
```
|
|
|
|
Local, non-production dry run of this exact flow: `infra/backup/test-backup-restore.sh` (uses
|
|
disposable containers, safe to run anytime, no real data involved).
|
|
|
|
**Recovery point objective:** up to 24 hours of academic data (daily backup cadence) — see
|
|
`docs/BACKUP_DR.md` for the full RPO/RTO discussion and why that's the honest number, not an
|
|
aspirational one.
|