ced4568
bf1e1b5c76
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs (PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR, OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST, and 12 runbooks (first deployment, normal deploy, rollback, Postgres restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure). PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine architectural gap this phase surfaced: the Gateway's existing env.internal_ip mechanism for reaching a Lab container assumes Gateway and Lab share a Docker daemon, which breaks once Labs run on a genuinely separate rn-lab-host-01 VM. The network-policy half is built (a firewall allowance for Gateway->Lab-Host on a published port range); the application half (Provisioner/Gateway address handling) is an explicit, reviewed, NOT-YET-BUILT follow-up flagged there rather than silently decided. Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and platform/README.md with Phase 4 pointers, consistent with how every prior phase updated these. Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
41 lines
2.0 KiB
Markdown
41 lines
2.0 KiB
Markdown
# Runbook: Stuck Lab Cleanup
|
|
|
|
A Lab that's stuck `provisioning`/`resetting` past its expected boot window, or a container
|
|
that outlived its Postgres `environments` row (crash during teardown, etc.).
|
|
|
|
**Via the admin UI (preferred, no host access needed):** `/admin` -> Students -> [student] ->
|
|
Environment -> Destroy, then Provision again if they need a fresh one immediately.
|
|
|
|
**Via the API directly** (same effect, useful for scripting a batch cleanup):
|
|
```bash
|
|
curl -s -b <owner-session-cookie> -X DELETE https://academy.example.com/api/admin/environments/<environment_id>
|
|
```
|
|
|
|
**A container the Provisioner itself has lost track of** (rare — e.g. a crash during
|
|
provisioning left an orphaned container/network/firewall chain with no matching `environments`
|
|
row, or vice versa):
|
|
|
|
```bash
|
|
# On rn-lab-host-01 — list what's actually running
|
|
docker ps -a --format '{{.Names}}\t{{.Status}}' | grep '^rootnode-lab-'
|
|
|
|
# Cross-reference against the Provisioner's own view
|
|
curl -s http://localhost:4100/environments | jq '.[] | {environment_id, status, container_name}'
|
|
|
|
# Provisioner already does this automatically on its own restart (reconcileOnStartup,
|
|
# platform/provisioner/src/lifecycle.js) — a simple `systemctl restart rootnode-platform`
|
|
# (which restarts the provisioner container as part of the stack) is often the fastest fix for
|
|
# a batch of orphans rather than hand-cleaning each one.
|
|
|
|
# Manual last resort — remove the container/network/firewall chain directly:
|
|
docker rm -f rootnode-lab-<id>
|
|
docker network rm rootnode-lab-net-<id>
|
|
iptables -D DOCKER-USER -s <subnet> -j ROOTNODE-ENV-<id> # remove the jump rule FIRST
|
|
iptables -F ROOTNODE-ENV-<id> && iptables -X ROOTNODE-ENV-<id>
|
|
```
|
|
|
|
Never manually delete an `environments` row from Postgres directly unless you've confirmed the
|
|
container/network/firewall chain are already gone — the Provisioner treats that table as
|
|
authoritative and will behave unpredictably against real Docker/iptables state it doesn't know
|
|
matches.
|