# Runbook: Stuck Lab Cleanup

A Lab that's stuck `provisioning`/`resetting` past its expected boot window, or a container
that outlived its Postgres `environments` row (crash during teardown, etc.).

**Via the admin UI (preferred, no host access needed):** `/admin` -> Students -> [student] ->
Environment -> Destroy, then Provision again if they need a fresh one immediately.

**Via the API directly** (same effect, useful for scripting a batch cleanup):
```bash
curl -s -b <owner-session-cookie> -X DELETE https://academy.example.com/api/admin/environments/<environment_id>
```

**A container the Provisioner itself has lost track of** (rare — e.g. a crash during
provisioning left an orphaned container/network/firewall chain with no matching `environments`
row, or vice versa):

```bash
# On rn-lab-host-01 — list what's actually running
docker ps -a --format '{{.Names}}\t{{.Status}}' | grep '^rootnode-lab-'

# Cross-reference against the Provisioner's own view
curl -s http://localhost:4100/environments | jq '.[] | {environment_id, status, container_name}'

# Provisioner already does this automatically on its own restart (reconcileOnStartup,
# platform/provisioner/src/lifecycle.js) — a simple `systemctl restart rootnode-platform`
# (which restarts the provisioner container as part of the stack) is often the fastest fix for
# a batch of orphans rather than hand-cleaning each one.

# Manual last resort — remove the container/network/firewall chain directly:
docker rm -f rootnode-lab-<id>
docker network rm rootnode-lab-net-<id>
iptables -D DOCKER-USER -s <subnet> -j ROOTNODE-ENV-<id>   # remove the jump rule FIRST
iptables -F ROOTNODE-ENV-<id> && iptables -X ROOTNODE-ENV-<id>
```

Never manually delete an `environments` row from Postgres directly unless you've confirmed the
container/network/firewall chain are already gone — the Provisioner treats that table as
authoritative and will behave unpredictably against real Docker/iptables state it doesn't know
matches.