ced4568
bf1e1b5c76
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs (PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR, OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST, and 12 runbooks (first deployment, normal deploy, rollback, Postgres restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure). PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine architectural gap this phase surfaced: the Gateway's existing env.internal_ip mechanism for reaching a Lab container assumes Gateway and Lab share a Docker daemon, which breaks once Labs run on a genuinely separate rn-lab-host-01 VM. The network-policy half is built (a firewall allowance for Gateway->Lab-Host on a published port range); the application half (Provisioner/Gateway address handling) is an explicit, reviewed, NOT-YET-BUILT follow-up flagged there rather than silently decided. Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and platform/README.md with Phase 4 pointers, consistent with how every prior phase updated these. Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
36 lines
2.0 KiB
Markdown
36 lines
2.0 KiB
Markdown
# Runbook: Failed Migration
|
|
|
|
Both Platform API and Provisioner apply their own migrations automatically on startup
|
|
(`platform/api/src/db.js`, `platform/provisioner/src/db.js`), file-tracked in
|
|
`schema_migrations`, wrapped in a transaction per file — a migration either fully applies or
|
|
fully rolls back, it can't leave a table half-altered.
|
|
|
|
**If a deploy fails because a migration errored:**
|
|
|
|
```bash
|
|
# 1. Check what actually happened
|
|
docker logs rootnode-platform-api-1 --tail 50 # the error is right there — migration file + message
|
|
docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
|
|
"SELECT * FROM schema_migrations ORDER BY applied_at DESC LIMIT 10;"
|
|
|
|
# 2. Since the failing migration's transaction rolled back, the schema is still in its
|
|
# PREVIOUS valid state — the app just won't start until the migration succeeds. Fix the
|
|
# root cause (usually a real data issue the migration didn't anticipate, or the
|
|
# cross-service ordering dependency — Provisioner's migrations must run before Platform
|
|
# API's, since scenario_attempts has a real FK to environments; see
|
|
# docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md).
|
|
|
|
# 3. Redeploy once fixed — the migration runner is idempotent and will just pick up where it
|
|
# left off; already-applied migrations are skipped.
|
|
./infra/scripts/deploy-production.sh <same-or-fixed-tag>
|
|
```
|
|
|
|
**If you need to roll back the CODE but the migration already succeeded** (schema changed,
|
|
code that depended on it is being reverted): rolling back application images
|
|
(`docs/RUNBOOK_ROLLBACK.md`) does NOT undo a Postgres migration — there is no automatic
|
|
down-migration mechanism (deliberate, matches the existing check-before-alter, additive-only
|
|
migration convention documented in `docs/ENGINEERING_SOP.md` §6). If the old code genuinely
|
|
can't run against the new schema, restore from the pre-migration backup instead
|
|
(`docs/RUNBOOK_POSTGRES_RESTORE.md`) rather than trying to hand-write a down-migration under
|
|
pressure.
|