rootnode-academy/docs/PHASE1_PLATFORM_FOUNDATION.md

# RootNode Academy — Phase 1: Platform Foundation

Status: **IMPLEMENTED, tested on a single dev machine, 2026-08-18.** Code lives in
`platform/provisioner/` and `platform/egress-proxy/`. This doc still reflects the corrected
design (four architecture corrections: VM boundary, egress-as-firewall-not-app-config, opaque
`student_id`, lab-host abstraction) — the implementation matches it, with a handful of
deviations found and fixed during testing, listed in "Implementation notes" at the end of each
section below where relevant, and summarized in `platform/provisioner/README.md`'s "Known
limitations." Builds on `docs/LAB_ARCHITECTURE.md` (read that first).

**Scope — this phase only:**
Postgres-backed environment registry · lifecycle state model · provisioner service skeleton ·
per-student private Docker network · disposable Lab container creation from the existing
image · configurable CPU/RAM/PID/storage limits · restricted egress · unique opaque routing
token · environment event/audit log · destroy/reset · idle-reaper design · current
single-container dev mode kept working, untouched.

**Explicitly not in this phase:** instructor dashboard, self-paced public enrollment, any
scenario content change, unrestricted egress, removing local single-user mode. Nothing below
proposes touching `backend/server.js`, `Dockerfile`, `scripts/entrypoint.sh`, or
`scenarios/**` — the Lab image is used exactly as it exists today.

---

## 1. Service boundaries

Two new components. Neither is the Lab image; neither is public-facing.

- **Provisioner** (`platform/provisioner/`) — new Node service. Owns: the Postgres
  `environments`/`environment_events`/`lab_hosts` tables, the Docker Engine API connection (via
  `dockerode`, not shelling out to the `docker` CLI), network/container lifecycle, the idle
  reaper, and egress-firewall management for each student network. Exposes an internal HTTP API
  (§4) gated by a shared-secret header — same pattern already used for the instructor-passcode
  gate on the roster endpoint, so this doesn't introduce a new auth idiom.
  **Docker connection is always resolved through `lab_hosts` (§2), never hard-coded** — Phase 1
  configures exactly one row, pointing at `unix:///var/run/docker.sock` for single-machine dev
  or a `tcp://<lab-host-vm>:2376` TLS endpoint for the two-VM production topology in
  `docs/LAB_ARCHITECTURE.md` §1. Same code path either way; only the config differs.
- **Egress proxy** (`platform/egress-proxy/`) — one shared instance (not per-student), a
  minimal HTTP(S) forward proxy (Squid) with a static host allowlist. Every Lab container's
  `HTTP_PROXY`/`HTTPS_PROXY` env vars point at it; the student network's firewall rule permits
  egress *only* to this proxy and to DNS, so there's no way for a Lab container to reach the
  internet except through something the Platform's allowlist controls.

Nothing else changes. No new Platform API / roster-progress migration in this phase — that's
explicitly out of scope (see §2 note on `student_id`).

---

## 2. Exact DB schema (PostgreSQL)

New database, e.g. `rootnode_platform`. Only the environment registry ships this phase —
`students`/`progress`/`sessions` stay on the existing SQLite inside the Lab image for now;
migrating those onto this Postgres instance is a later phase, not bundled in here.

```sql
-- ── lab_hosts — correction D: the abstraction that keeps Docker connectivity out of code ──
CREATE TABLE lab_hosts (
  lab_host_id    TEXT PRIMARY KEY,          -- e.g. 'dev-local', 'labhost-01'
  display_name   TEXT NOT NULL,
  endpoint       TEXT NOT NULL,             -- 'unix:///var/run/docker.sock' (dev) or
                                              -- 'tcp://10.x.x.x:2376' (prod, TLS-verified)
  tls_ca_path    TEXT,                       -- null for the local-socket dev case
  tls_cert_path  TEXT,                       -- Provisioner's client cert for this host
  tls_key_path   TEXT,
  enabled        BOOLEAN NOT NULL DEFAULT true,
  created_at     TIMESTAMPTZ NOT NULL DEFAULT now()
  -- Future (not this phase): capacity/status columns for real scheduling once there's
  -- more than one host. Phase 1 selects "the one enabled row" — no scheduler.
);

CREATE TYPE environment_status AS ENUM (
  'requested', 'provisioning', 'running', 'resetting',
  'suspended',            -- experimental path only, see decision #3
  'destroying', 'destroyed', 'error'
);

CREATE TABLE environments (
  environment_id    UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  student_id         TEXT NOT NULL,        -- CORRECTION C: opaque external identifier, not a
                                             -- foreign key. students/progress stay on the
                                             -- existing SQLite through Phase 1 (see
                                             -- docs/LAB_ARCHITECTURE.md §3/§9) — there is no
                                             -- students table in this database yet, so there
                                             -- is nothing to reference. The Provisioner trusts
                                             -- whatever value its caller passes; it does not
                                             -- validate against a roster. A real FK gets added
                                             -- only when roster/progress actually migrate onto
                                             -- this Postgres instance, in a later, separate
                                             -- migration — not assumed or faked here.
  lab_host_id        TEXT NOT NULL REFERENCES lab_hosts(lab_host_id),  -- real FK: both tables
                                             -- are Provisioner-owned Postgres, correction D.
  status             environment_status NOT NULL DEFAULT 'requested',
  routing_token      TEXT NOT NULL,         -- CSPRNG, opaque, unique
  container_id       TEXT,                  -- null until provisioned
  container_name     TEXT,
  network_id         TEXT,
  network_name       TEXT,
  network_subnet     CIDR,                  -- needed to install/remove the per-env firewall chain
  image_ref          TEXT NOT NULL,         -- pinned Lab image, e.g. rootnode-academy:1.4.0
  cpu_limit          NUMERIC NOT NULL,
  mem_limit_mb       INTEGER NOT NULL,
  pids_limit         INTEGER NOT NULL,
  storage_limit_mb   INTEGER NOT NULL,
  internal_ip        TEXT,                  -- set once running
  internal_port      INTEGER NOT NULL DEFAULT 80,
  created_at         TIMESTAMPTZ NOT NULL DEFAULT now(),
  started_at         TIMESTAMPTZ,
  last_active_at     TIMESTAMPTZ,
  destroy_at         TIMESTAMPTZ,           -- idle-reaper's scheduled destroy time
  error_reason       TEXT
);

CREATE UNIQUE INDEX idx_environments_routing_token ON environments(routing_token);
CREATE INDEX idx_environments_student ON environments(student_id);
CREATE INDEX idx_environments_status ON environments(status);
CREATE INDEX idx_environments_lab_host ON environments(lab_host_id);
-- Phase 1 invariant: at most one non-terminal environment per student.
-- Enforced in application logic (see §4 POST /environments), not a DB constraint yet —
-- a partial unique index is the natural upgrade once this needs to be bulletproof under
-- concurrent requests, not needed for a single-instance provisioner in this phase.

CREATE TABLE environment_events (
  id              BIGSERIAL PRIMARY KEY,
  environment_id  UUID NOT NULL REFERENCES environments(environment_id),
  event_type      TEXT NOT NULL,   -- provision_requested | provisioned | started | reset
                                     -- | destroyed | error | reaped_idle | suspend | resume
  actor           TEXT NOT NULL,   -- 'system' | 'instructor:<id>' | 'student:<id>' | 'reaper'
  detail          JSONB,
  created_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX idx_env_events_environment ON environment_events(environment_id);
```

**Resource policy and the egress allowlist are config files, not tables.** There's no admin UI
to manage them via a database yet (that's dashboard work, explicitly deferred), so a
DB-backed policy table would just be a write path nothing uses. `config/resource-policy.yaml`
(cpu/mem/pids/storage defaults, reloadable) and `platform/egress-proxy/allowlist.conf` (Squid
ACL) are the actual source of truth — plain files an operator edits, matching how
`entrypoint.sh` already does config via env vars rather than a database.

---

## 3. Lifecycle state machine

Exactly what `docs/LAB_ARCHITECTURE.md` §2 now specifies, decision #3 applied:

```
requested → provisioning → running ─┬─→ resetting → running   (same environment_id + routing_token)
                                     └─→ destroying → destroyed
provisioning → error
running → error
error → destroying → destroyed
error → resetting → running     (when recoverable)

[EXPERIMENTAL — not required to pass for Phase 1 sign-off]
running → suspended → running   (resume)
running → suspended → error     (resume fails)
```

`reset` is implemented as `destroy` + `provision` against the same DB row (new container,
network, and internal_ip; same `environment_id`/`routing_token` so a student's saved link
doesn't change).

---

## 4. API endpoints (Provisioner, internal-only)

All routes require `X-Provisioner-Key: <shared secret>`. Nothing here is reachable by a
student or the public internet in this phase — there's no frontend/gateway wiring yet, this
is tested directly (curl / a test script), which is also what makes it independently
verifiable before any later phase builds on top of it.

| Method & path | Purpose |
|---|---|
| `POST /environments` | Body `{ student_id }` — `student_id` is opaque (correction C, §2), not validated against a roster. `lab_host_id` is not a caller-supplied parameter in Phase 1: the Provisioner selects the single enabled `lab_hosts` row itself (`SELECT ... WHERE enabled LIMIT 1`) — no scheduling logic, no host-selection API, matching "don't overbuild scheduling yet." Rejects if the student already has a non-terminal environment (Phase 1 invariant, see §2). Synchronous: creates network → installs firewall chain (§5) → creates+starts container → polls Lab `/api/health` → marks `running`, or `error` on timeout. Returns `{ environment_id, status, routing_token }`. |
| `GET /environments/:id` | Full row: status, container_id, limits, timestamps. |
| `GET /environments?student_id=` | A student's environments (expected: zero or one active). |
| `POST /environments/:id/reset` | Destroy + recreate, same `environment_id`/`routing_token`. |
| `DELETE /environments/:id` | Destroy: stop, remove container, remove network + its firewall rules, mark `destroyed`. |
| `POST /environments/:id/suspend` *(experimental)* | `docker stop`, mark `suspended`. Documented as unreliable — see test plan §7. |
| `POST /environments/:id/resume` *(experimental)* | `docker start` + re-poll readiness. On failure, mark `error` rather than leaving a wrong `suspended` status. |
| `GET /environments/:id/events` | Audit trail for that environment. |
| `GET /health` | Provisioner's own health: Docker reachable, Postgres reachable. |

Synchronous provisioning is a deliberate simplification for this phase — no job queue, no
webhook callback. Given expected Phase 1 usage (developer testing, not a live cohort), a
blocking HTTP call that takes a few seconds while k3s comes up is fine; converting to
async-with-polling is the obvious upgrade if provisioning latency becomes a real UX problem
once this is behind an actual frontend.

---

## 5. Docker / network operations (via `dockerode`, shown as CLI-equivalent for readability)

**Provision — network:**
```bash
docker network create --driver bridge rootnode-lab-net-<short-env-id>
# inspect to get the allocated subnet, store as environments.network_subnet
```

**Provision — egress firewall, corrected (§4 of `docs/LAB_ARCHITECTURE.md`, correction B).**
The previous draft chained individual `iptables -I DOCKER-USER ...` inserts in
ACCEPT-then-DROP order, which is backwards: `-I` without a rule number pushes each new rule to
*position 1*, so the rules end up installed in the *reverse* of the order they were run —
the final default-DROP would land at the top of `DOCKER-USER` and shadow everything below it,
including the proxy/DNS accepts. Fixed by giving each environment its own dedicated chain,
built with `-A` (append — rules land in the order added, no head-insertion trap), and reaching
it from `DOCKER-USER` with exactly one jump rule per environment:

```bash
NET_SUBNET=$(docker network inspect rootnode-lab-net-<id> -f '{{(index .IPAM.Config 0).Subnet}}')
CHAIN="ROOTNODE-ENV-<short-env-id>"

iptables -N "$CHAIN"                      # fresh, empty chain for this environment

# Required infrastructure — explicitly permitted FIRST, before any blanket deny:
iptables -A "$CHAIN" -d <egress-proxy-ip> -p tcp --dport 3128 -j ACCEPT
iptables -A "$CHAIN" -d <dns-resolver-ip> -p udp --dport 53   -j ACCEPT
iptables -A "$CHAIN" -d <dns-resolver-ip> -p tcp --dport 53   -j ACCEPT
# (any explicitly-required Platform endpoint would be an additional ACCEPT line here —
#  none needed in Phase 1; the Lab image doesn't call back to the Platform yet)

# Everything that must never be reachable, regardless of the above — unconditional, no
# allowlist entry can ever bypass these, satisfying "no route to trusted networks" as a
# hard rule rather than a policy convention:
iptables -A "$CHAIN" -d 10.0.0.0/8     -j DROP
iptables -A "$CHAIN" -d 172.16.0.0/12  -j DROP
iptables -A "$CHAIN" -d 192.168.0.0/16 -j DROP
iptables -A "$CHAIN" -j DROP           # default deny, catches everything else (public internet)

# Hook the student's subnet into this chain. -I here is fine and intentional: we want each
# new environment's jump rule evaluated ahead of any broader, pre-existing DOCKER-USER rule,
# and since every environment matches a disjoint source subnet, the relative order of jump
# rules against each other never matters — only that each is ahead of a catch-all, if one
# ever exists at the DOCKER-USER level.
iptables -I DOCKER-USER -s "$NET_SUBNET" -j "$CHAIN"
```

This also directly satisfies the explicit "must never reach" list from
`docs/LAB_ARCHITECTURE.md` §4: RootNode management networks, the Platform database, Docker/
Proxmox management interfaces (including the Lab Host VM's own Docker API — notably, since
that API is what the Provisioner itself talks to from the *Platform* VM, a Lab container has
no way to reach it either way, but the RFC1918 drop covers it as a second, independent layer),
CDAA infrastructure, home/trusted networks, and other students' lab networks all fall inside
one of the three RFC1918 ranges above and are dropped unconditionally by this same ruleset —
one mechanism, not five separate ones to keep in sync.

**Provision — container** (limits from `resource-policy.yaml`, no image/entrypoint change):
```bash
docker create \
  --name rootnode-lab-<short-env-id> \
  --network rootnode-lab-net-<short-env-id> \
  --privileged \
  --cpus <cpu_limit> --memory <mem_limit_mb>m --memory-swap <mem_limit_mb>m \
  --pids-limit <pids_limit> \
  --storage-opt size=<storage_limit_mb>m \
  -e HTTP_PROXY=http://<egress-proxy-host>:3128 -e HTTPS_PROXY=http://<egress-proxy-host>:3128 \
  -e NO_PROXY=localhost,127.0.0.1,10.42.0.0/16,10.43.0.0/16 \
  <pinned-lab-image-ref>
docker start rootnode-lab-<short-env-id>
```
**Implementation note: `<egress-proxy-host>` is resolved per environment, not a fixed
hostname/IP.** The egress proxy runs with host networking (`docker-compose.dev.yml`), so it's
present on every per-student bridge network's *gateway* interface — that gateway IP (from
`docker network inspect`'s `IPAM.Config[0].Gateway`, right alongside the subnet already used
for the firewall chain) is what's actually reachable from that specific environment's private
network. A single static proxy hostname would only be reachable from whichever one network
happened to be able to resolve/route to it — this wasn't apparent until testing with two
simultaneous environments on two different subnets. Same resolution is used for both the
firewall's `ACCEPT` rule destination (§ above) and this container's proxy env vars, so they
always agree.

`NO_PROXY` carves out k3s's own internal pod/service CIDRs so in-cluster traffic never routes
through the proxy — only the container's *outbound-to-the-internet* path does. containerd
(started by the existing, unmodified `entrypoint.sh`) honors `HTTP_PROXY`/`HTTPS_PROXY` from
its process environment automatically, which is why no Lab image or entrypoint change is
needed for well-behaved image pulls to go through the proxy.

**Correction B, stated plainly: these env vars configure containerd for a clean pull path —
they are not what makes egress restricted.** A process inside the container that ignores them
and dials out directly is stopped by the `ROOTNODE-ENV-<id>` firewall chain above regardless,
because that chain evaluates every packet leaving the container's network, independent of
what any process inside chose to do. If the env vars were somehow wrong or absent, the only
consequence is broken image pulls (a visible, debuggable failure) — never a silent hole in the
egress boundary, because the boundary was never the env vars in the first place.

Readiness: poll the container's internal IP (`docker inspect` → `NetworkSettings`) for
`GET /api/health` — the exact endpoint `entrypoint.sh` already waits on internally, so the
Provisioner is just doing the same check from outside instead of inventing a new one.

**Destroy:**
```bash
docker stop rootnode-lab-<id> --time 10 && docker rm rootnode-lab-<id>

# Unhook and remove this environment's dedicated chain — three ops, no rule-scanning/matching
# needed since it's one self-contained chain per environment, not tagged rules mixed into a
# shared one:
iptables -D DOCKER-USER -s "$NET_SUBNET" -j "$CHAIN"
iptables -F "$CHAIN"
iptables -X "$CHAIN"

docker network rm rootnode-lab-net-<id>
```

**Reset** = destroy, then provision again against the same `environments` row (new
`container_id`/`network_id`/`internal_ip`, unchanged `environment_id`/`routing_token`).

**Idle reaper** (background loop in the Provisioner, e.g. every 60s):
- Query `environments WHERE status='running' AND COALESCE(last_active_at, created_at) < now() - idle_threshold`.
- Past a shorter "idle" threshold: log a warning event (future: notify student/instructor —
  not built yet).
- Past a longer "abandoned" threshold: call the same destroy path as `DELETE /environments/:id`,
  log `event_type='reaped_idle'`. Config-driven thresholds, not hard-coded (same policy file
  as resource limits).

**Implementation notes (found during testing, not in the original design):**
- **`COALESCE(last_active_at, created_at)`, not bare `last_active_at`.** An environment that
  fails during provisioning (before ever reaching `running`) has a NULL `last_active_at` —
  `NULL < anything` is never true in SQL, so without the fallback such an environment would
  never age out via the reaper at all, regardless of how long ago it failed.
- **Provisioning failure now does best-effort cleanup of whatever was already created**
  (container/network/firewall chain) before marking the row `error`, rather than leaving that
  infrastructure orphaned. Found by testing: a deliberate readiness-timeout failure left a real
  running container and firewall chain in place with only the DB row marked `error` — exactly
  the kind of leak the reaper exists to prevent, except this case predates the reaper ever
  having something to catch. Cleanup on the failure path directly is more honest than relying
  on the reaper for a gap it wasn't actually catching. See `src/lifecycle.js`'s
  `provisionEnvironment` catch block and `src/reaper.js`.
- **Startup reconciliation (docs/LAB_ARCHITECTURE.md §8) is implemented** as
  `reconcileOnStartup()` in `src/lifecycle.js`, called once at Provisioner boot before the
  reaper starts. Tested by killing a container out-of-band and restarting the Provisioner —
  confirmed it correctly flags the mismatch as `error` rather than trusting the stale
  `running` status. Any environment caught mid-transition (`provisioning`/`resetting`/
  `destroying`) by a crash is also always marked `error` on restart, never assumed to have
  finished.

---

## 6. Security assumptions (stated explicitly, not implied)

- **Correction A — the Lab Host is not the Platform's trust boundary, the VM is.** In
  production, the host running privileged Lab containers is a separate VM from the one running
  Postgres/Provisioner/Platform API, with no route back from the Lab Host VM to the Platform VM
  at all — enforced at the hypervisor/network level, not by this phase's code. Phase 1's
  single-machine dev mode is a deliberate, documented exception (`docs/LAB_ARCHITECTURE.md` §1)
  — it does not redefine what "isolated" means for the real deployment.
- The Docker host(s) sit on a segment with no route to trusted RootNode/CDAA/home/business
  networks — a prerequisite this phase depends on but does not create (matches the existing
  Business Pack Technical Platform Plan). Everything below assumes that's already true.
- No other host firewall tooling (ufw, firewalld) is silently rewriting `DOCKER-USER` or
  removing the per-environment `ROOTNODE-ENV-<id>` chains — needs a one-time check on the real
  host before trusting the egress rules; not something this design can verify in advance.
- `--privileged` is required for k3s and is not being avoided (confirmed in
  `docs/LAB_ARCHITECTURE.md` §7) — the security boundary is the VM split plus network
  containment, not reduced container privilege.
- **Correction B — `HTTP_PROXY`/`HTTPS_PROXY` are not a security control**, stated here again
  because it's the assumption most likely to get silently reintroduced during implementation:
  the firewall chain enforces the boundary; the env vars only make well-behaved pulls work
  cleanly through it.
- The egress proxy's allowlist is operator-controlled (a file the Provisioner's operator
  edits), never student- or Lab-container-writable.
- The Provisioner API's shared secret is the only thing standing between "internal service"
  and "anyone who can create/destroy arbitrary privileged containers on the host" — treat that
  secret with the same care as the instructor passcode, not as a minor implementation detail.
  This matters more, not less, once the Provisioner is issuing commands to a remote Docker
  endpoint (correction D) — the TLS client cert for that connection carries the same weight.
- `storage_limit_mb` enforcement via `--storage-opt size=` depends on the host's Docker storage
  driver supporting it (overlay2 with specific backing filesystems does; not all do) — unverified
  until tested against the real host (§8).
- Nothing in this phase is internet-facing. The Provisioner API assumes internal-network-only
  reachability; it does not implement its own TLS. The Provisioner→Lab-Host Docker connection
  (correction D), by contrast, is TLS by design once it's not a local socket — that's a
  different boundary with different stakes and isn't optional the way the Provisioner API's
  own TLS currently is.

---

## 7. Files to be created

All new. Nothing existing is modified beyond the two docs already updated.

```
platform/provisioner/
├── package.json
├── src/
│   ├── server.js        # Express app, routes from §4
│   ├── db.js             # Postgres connection (pg), migration runner
│   ├── docker.js         # dockerode client factory — resolves a lab_hosts row to a connected
│   │                        client (local socket or remote TLS), never a hard-coded default
│   ├── lifecycle.js      # state machine + orchestration (calls docker.js + db.js)
│   ├── egress.js          # per-environment ROOTNODE-ENV-<id> chain create/hook/unhook/remove
│   └── reaper.js          # idle-reaper background loop
├── migrations/
│   ├── 001_lab_hosts.sql
│   ├── 002_environments.sql
│   └── 003_environment_events.sql
├── config/
│   ├── resource-policy.yaml     # cpu/mem/pids/storage defaults + idle/abandoned thresholds
│   └── lab-hosts.seed.yaml       # the one Phase 1 lab_hosts row (endpoint + TLS paths),
│                                   # applied by the migration runner — not a runtime API,
│                                   # per "don't overbuild scheduling yet"
├── Dockerfile
└── docker-compose.dev.yml       # Postgres + egress-proxy + provisioner, for local Phase 1 testing

platform/egress-proxy/
├── Dockerfile            # minimal Squid image
└── allowlist.conf        # Squid ACL: registry-1.docker.io, ghcr.io, etc. — starting set
                            # to be confirmed against what scenarios actually pull
```

No changes to `backend/server.js`, `Dockerfile`, `scripts/entrypoint.sh`,
`scripts/nginx.conf`, or anything under `scenarios/`.

---

## 8. Test plan

1. **Schema migration idempotency** — run `001_environments.sql` against a scratch Postgres
   twice, confirm the second run is a no-op (same pattern already used for the SQLite
   migrations: check-before-alter).
2. **Provision → destroy round trip** — create an environment, confirm the container is
   `running` and reachable on its internal IP, confirm `environment_events` has one row per
   transition, destroy, confirm container/network are gone and the environment's
   `ROOTNODE-ENV-<id>` chain no longer exists (`iptables -L ROOTNODE-ENV-<id>` errors "chain
   does not exist") and its `DOCKER-USER` jump rule is gone (`iptables -S DOCKER-USER | grep
   <id>` returns nothing).
3. **Reset** — provision, note `environment_id`/`routing_token`, reset, confirm both are
   unchanged, confirm `container_id` changed (genuinely new container), confirm no state
   carried over (fresh k3s).
4. **Isolation** — provision two environments, confirm A cannot reach B at all
   (`docker exec <A> ping <B's internal_ip>` times out) — this is the core security property
   and must be an explicit test, not an assumption.
5. **Egress allowlist, including the ordering fix specifically** — from inside a provisioned
   environment: an allowlisted registry pull succeeds via the proxy; a non-allowlisted host
   fails; a direct request to an RFC1918 address (simulating a trusted-network address) is
   dropped regardless of allowlist state. Critically: **the egress proxy's own IP is itself an
   RFC1918 address** — this test only actually exercises the fix from correction B if the proxy
   is reachable *and* the broader RFC1918 range is still blocked. If the proxy were unreachable
   here, that's the exact symptom the original ordering bug would have caused (the DROP rules
   shadowing the ACCEPT rules) — this test is what would have caught it before it shipped.
6. **Idle reaper** — provision, force `last_active_at` backward past the abandoned threshold,
   run one reaper tick, confirm the environment is destroyed and logged as `reaped_idle`.
7. **Suspend/resume (experimental, decision #3 — informational, not a pass/fail gate)** —
   `docker stop`/`docker start` a running Lab container, time how long it takes (if ever) to
   reach `/api/health` ready again versus a fresh `provision`. This is the actual answer to
   the question `docs/LAB_ARCHITECTURE.md` flagged as untested — record the result either way,
   but Phase 1 sign-off does not depend on this succeeding.
8. **Reconciliation on Provisioner restart** — manually create a container outside the
   Provisioner's knowledge, restart the Provisioner, confirm it doesn't silently trust stale
   `environments` rows — either adopts or flags the mismatch (per
   `docs/LAB_ARCHITECTURE.md` §8).
9. **Resource limits hold** — inside a provisioned environment, confirm `--cpus`/`--memory`/
   `--pids-limit` are actually enforced (e.g. a deliberate memory-hog process gets OOM-killed
   at the configured limit, not the host's full RAM); confirm whether `--storage-opt size=`
   is honored on the real host's storage driver — this is the resource-model unknown from
   `docs/LAB_ARCHITECTURE.md` §5, and this test is the first real benchmarking data point
   toward the 16-core/128GB target sizing.
10. **Local dev-mode regression check** — confirm `docker run --privileged -p 7554:80 ...
    rootnode-academy:latest` (today's standalone command) is completely unaffected —
    `git diff` scoped to `backend/`, `Dockerfile`, `scripts/` should show zero changes from
    this phase.
11. **Lab host abstraction actually abstracts (correction D)** — point the Provisioner's
    `lab_hosts` config at a Docker daemon exposed over TCP+TLS on the *same* dev machine
    (`dockerd -H tcp://127.0.0.1:2376 --tlsverify ...`) instead of the local Unix socket, with
    no code change — only the `lab_hosts` row differs. Provision/destroy round trip (test 2)
    must pass identically against that endpoint. This is the practical proof that "Docker is
    local" was actually removed as an assumption, not just renamed — it doesn't require a
    second real VM to verify, only a second way of reaching the same daemon.
12. **VM-boundary reasoning check (correction A, documentation-level, not a running test)** —
    walk the failure scenario explicitly: if a student escaped their container onto the Lab
    Host VM's OS, what could they reach? Confirm the answer stays "nothing on the Platform VM"
    given the topology in `docs/LAB_ARCHITECTURE.md` §1 — i.e. confirm the design doesn't rely
    on anything *inside* the Lab Host VM (a firewall rule, a credential, an agent) as the only
    thing standing between a compromised container and Postgres. This is a design-review check
    now; it becomes a real network-level test once a second VM exists to test against.

Tests 1–6, 8, 10, 11 are runnable in a plain Docker environment with no special hardware.
Tests 5 and 9 need the actual target host (or something close to it) to produce meaningful
numbers, test 7 needs the real Lab image, and test 12 needs a second VM to be more than a
design-review exercise — flagging that distinction so "passed the test plan" doesn't get read
as "validated on production-class, two-VM hardware" when parts of it were necessarily
validated on a single dev machine.