diff --git a/README.md b/README.md index 2bd1ab3..616d166 100644 --- a/README.md +++ b/README.md @@ -194,6 +194,11 @@ Production topology and what's built vs. not yet: `docs/LAB_ARCHITECTURE.md`, `docs/PHASE1_PLATFORM_FOUNDATION.md`, `docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md`, `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md`, `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`. +The production infrastructure package (Proxmox/Ansible/Terraform/firewall/monitoring/backup/CI) +that runs all of this on real hardware lives in `infra/` — see +`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`. Built and locally validated now; deployment is +pending purchase of the target Hetzner dedicated server. + --- ## Repository Layout diff --git a/docs/BACKUP_DR.md b/docs/BACKUP_DR.md new file mode 100644 index 0000000..192f88e --- /dev/null +++ b/docs/BACKUP_DR.md @@ -0,0 +1,65 @@ +# Backup & Disaster Recovery + +## What's backed up (Tier 1) vs. not + +| | Backed up | Why | +|---|---|---| +| Postgres data | Yes, daily | The only irreplaceable data — student records, cohort/session state, scenario attempts, assessments, achievements, audit events | +| `{{ rootnode_data_root }}` config/secrets (Vault files, env templates) | Yes, via the infra repo + Ansible Vault itself | Small, static, versioned | +| Infra code (`infra/`, Ansible, Terraform state) | Yes — it's in git; Terraform state should be backed up alongside Postgres if not using a remote backend | Needed to rebuild either VM from scratch | +| Disposable student Labs | **No, intentionally** | Ephemeral by design — a Lab is meant to be destroyed and recreated; backing it up would contradict the whole isolation/disposability model | +| Prometheus/Loki metrics & log history | No | Operational telemetry, not source-of-truth data; losing 30 days of dashboards on a rebuild is an inconvenience, not data loss | + +## Backup mechanism + +`infra/backup/backup-postgres.sh` (deployed to `/opt/rootnode/backup/` by the `backup` Ansible +role, run daily at 03:15 via cron): + +1. `pg_dump --format=plain --no-owner --no-privileges | gzip -9` +2. Sanity check: reject if the compressed dump is under 1KB (almost certainly means Postgres + was unreachable or empty, not a real backup). +3. `gpg --symmetric --cipher-algo AES256` — encrypt, then delete the plaintext dump immediately. +4. Optional off-host copy via `rsync -az` over SSH — works for Hetzner Storage Box or any SSH + target; that's the extension point for S3-compatible alternatives (swap the rsync line for + `aws s3 cp`/`rclone` if a different target is chosen later). +5. Retention pruning (`find ... -mtime +{{ backup_retention_days }} -delete`, default 30 days). +6. Writes Prometheus textfile-collector metrics + (`rootnode_backup_last_run_success`, `..._last_success_timestamp_seconds`, `..._size_bytes`) + atomically (tmp file + rename) so a half-written metric file is never scraped. + +## Restore + +`infra/backup/restore-postgres.sh` — requires an explicit target DB URL (never infers or +defaults to production, to prevent an accidental overwrite), shows the target's current tables, +requires typing `RESTORE` to confirm, then decrypts and pipes into `psql`. +`infra/backup/verify-restore.sh` — confirms all 8 expected tables exist plus the +`schema_migrations` row count, so a "successful" restore that silently missed a table is caught. + +**Genuinely tested**, not just written: `infra/backup/test-backup-restore.sh` runs the full +cycle against two disposable local Postgres containers, with the actual production migrations +applied (all 17 — 3 Provisioner + 14 Platform API, in the correct cross-service order), a +sample row inserted, backed up, restored into the second container, and verified byte-for-byte. +Confirmed passing, exit 0. + +## Disaster recovery scenarios + +| Scenario | RPO | RTO (estimate) | What happens | +|---|---|---|---| +| Platform VM lost, disks intact | 0 | ~30 min | `docs/RUNBOOK_VM_REBUILD.md` — Ansible re-run against the same disks | +| Platform VM + its disks lost | Up to 24h (since last backup) | ~1-2h | Rebuild VM (Terraform), re-run Ansible, restore Postgres from the latest off-host backup (`docs/RUNBOOK_POSTGRES_RESTORE.md`) | +| Lab Host VM lost, any reason | 0 (intentional) | ~30 min | Rebuild VM, re-run Ansible — no data to restore, all Labs were disposable and get recreated on next student login | +| Single disk failure in a mirror | 0 | 0 (no outage) | `docs/RUNBOOK_DISK_FAILURE.md` — pool stays online DEGRADED, replace and resilver | +| Whole storage pool lost (both mirror sides) | Up to 24h | ~1-2h | Same as "Platform VM + disks lost" above — this is precisely the scenario the off-host copy exists for; a local-only backup on the same failed pool is useless here | +| Encryption passphrase lost | N/A | Unrecoverable | The single most important secret to protect and back up separately (e.g., in the same secrets manager as other Vault secrets, not only in `production.vault.yml`) — losing it makes every existing backup file permanently unreadable | + +RPO is bounded by backup cadence (daily) — a mid-day incident can lose up to a day of student +activity. If that's ever unacceptable, the fix is a shorter cron interval or WAL archiving for +point-in-time recovery, not decided or built here since daily has not yet been proven +insufficient against real usage. + +## What backup does NOT cover + +Application code and container images — those live in Gitea and the image registry, both +already durable and versioned independently of this backup mechanism. TLS certificates — those +are regenerable (`docs/RUNBOOK_DOCKER_TLS_ROTATION.md`, `docs/RUNBOOK_TLS_RENEWAL.md`), not +backed up. diff --git a/docs/CAPACITY_PLANNING.md b/docs/CAPACITY_PLANNING.md new file mode 100644 index 0000000..6c8b658 --- /dev/null +++ b/docs/CAPACITY_PLANNING.md @@ -0,0 +1,59 @@ +# Capacity Planning + +How many concurrent student Labs `rn-lab-host-01` can actually run — determined from measured +data after deployment, not guessed or marketed now. + +## The model + +`infra/scripts/capacity-model.js`: + +``` +usable = hostTotal * (1 - overheadPct/100) - platformReserved +withMargin = usable * (1 - marginPct/100) +perLabLimit = floor(withMargin / perLabUsage) +recommended = min(cpuBoundLimit, memoryBoundLimit) # whichever runs out first +warning = floor(recommended * 0.8) +hardAdmission = recommended +``` + +Computed once for CPU, once for memory; the tighter (smaller) of the two constrains the actual +recommendation — a host can be memory-starved long before its CPU threads are exhausted, or +vice versa, and the model doesn't pretend otherwise. + +**Inputs** (`infra/environments/production/capacity.yml.example` — explicitly placeholder data +until real numbers exist): +- `host_cpu_threads`, `host_memory_mb` — from the real Lab Host VM, once sized +- `overhead_pct` — Proxmox/hypervisor + OS overhead reserved off the top +- `platform_reserved_cpu`/`platform_reserved_memory_mb` — headroom for the egress proxy, + node-exporter, cadvisor, promtail themselves (they run on the Lab Host too) +- `per_lab_cpu`/`per_lab_memory_mb` — **measured**, from `infra/scripts/loadtest/`, not assumed +- `safety_margin_pct` — extra headroom below the raw math, for burst/variance + +**Outputs**: recommended concurrent Labs, an 80%-of-recommended warning threshold (for +alerting — `LabHostHighMemory` in `docs/OBSERVABILITY.md`), and a hard admission-control +threshold the Provisioner's resource policy should enforce so it refuses to boot a Lab past that +point rather than degrading every running Lab. + +## Load-test tooling that produces the measured inputs + +`infra/scripts/loadtest/`: +- `boot-labs.js` — reuses `platform/test/session-helper.js` (built in Phase 3A) to drive the + real admin Prepare-Labs endpoint, booting N Labs and recording boot time/failure rate per Lab + into a JSON report. +- `sample-resources.sh` — samples real `docker stats` for every `rootnode-lab-*` container every + 5 seconds to CSV (timestamp, name, CPU%, mem usage, mem%, net I/O, block I/O, PIDs). + +Together these produce the `per_lab_cpu`/`per_lab_memory_mb` this model needs, plus data on how +boot time and failure rate change as concurrency rises — the dimensions this phase was told to +cover: single Lab boot, multiple simultaneous boots, steady-state active Labs, terminal +WebSocket load, scenario validation load, Gateway throughput, API request volume, Postgres +connection activity. + +## What this document deliberately does not claim + +No concurrency number in this repo (`capacity.yml.example`'s numbers included) is a real, +validated figure — every one is a placeholder until run against the real Lab Host VM with real +per-Lab measurements. Treating a placeholder as a marketing/sales number would be actively +misleading; the whole point of the model is that the real number gets filled in once, from +measurement, after deployment (`docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md` step 4 is the +trigger to re-measure and update it if the per-Lab footprint later changes). diff --git a/docs/ENGINEERING_SOP.md b/docs/ENGINEERING_SOP.md index c8f4b3c..0554392 100644 --- a/docs/ENGINEERING_SOP.md +++ b/docs/ENGINEERING_SOP.md @@ -280,3 +280,28 @@ Full design: `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`. - **Known limitations:** see the Phase 3B doc §20 — no automatic abandoned-attempt detection, no per-cohort needs-attention threshold configuration, raw-JSON pool_spec authoring in the admin UI (correct, not yet a guided form). + +## 13. Production Infrastructure (Phase 4) + +Full design: `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`, plus detail docs +`PRODUCTION_DEPLOYMENT.md`, `PRODUCTION_NETWORK_SECURITY.md`, `BACKUP_DR.md`, +`OBSERVABILITY.md`, `CAPACITY_PLANNING.md`, `PRODUCTION_VALIDATION_CHECKLIST.md`, and 12 +`RUNBOOK_*.md` files, all under `docs/`. Everything lives in `infra/` (Ansible, Terraform, +Compose, firewall, monitoring, backup, CI, load-test tooling). + +- **What it is:** the infrastructure-as-code needed to run rn-platform-01/rn-lab-host-01 on a + real Hetzner/Proxmox host — buildable and locally testable now, even though the dedicated + server hasn't been purchased. Status is explicitly **PHASE 4 BUILD COMPLETE / PRODUCTION + DEPLOYMENT PENDING HARDWARE**, not "deployed." +- **Where the boundary held:** the Platform VM / Lab Host VM security split from + `docs/LAB_ARCHITECTURE.md` is codified in firewall rulesets and mutual-TLS Docker config, not + weakened anywhere. +- **Known gap surfaced, not silently fixed:** the Gateway's Lab-reachability mechanism doesn't + survive the move to two real separate VMs — see `docs/PRODUCTION_NETWORK_SECURITY.md`'s + "Known gap" section for what's built (firewall policy) vs. what's an explicit, reviewed, + not-yet-built application follow-up (Provisioner/Gateway address handling). +- **Validated without the hardware:** real `tofu validate`, `ansible-playbook --syntax-check`, + `docker compose config`, `caddy validate`, `promtool`/`amtool`, and a genuine end-to-end + backup→encrypt→restore→verify cycle against disposable local Postgres with the real + production migrations applied. None of this proves the real Hetzner network boundary — that + needs `docs/PRODUCTION_VALIDATION_CHECKLIST.md` walked against the real server. diff --git a/docs/LAB_ARCHITECTURE.md b/docs/LAB_ARCHITECTURE.md index dc0a924..db65aa3 100644 --- a/docs/LAB_ARCHITECTURE.md +++ b/docs/LAB_ARCHITECTURE.md @@ -486,3 +486,13 @@ All four open questions this doc originally ended with are resolved — see the at the top and §2–§5 above. Next: `docs/PHASE1_PLATFORM_FOUNDATION.md` — the concrete build plan for the platform foundation (Postgres environment registry, provisioner skeleton, networking, egress, resource policy, reaper) against these decisions. + +## Phase 4: this boundary in production + +Phase 4 (`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`) codifies the Platform/Lab-Host split this +document designed as two real, separately-firewalled Proxmox VMs (`rn-platform-01`, +`rn-lab-host-01`) connected only by mutual-TLS Docker and a tightly scoped firewall allowance — +see `docs/PRODUCTION_NETWORK_SECURITY.md`. That document also records a gap Phase 4 surfaced but +did not resolve: the Gateway's `env.internal_ip` mechanism for reaching a Lab container assumes +Gateway and Lab share a Docker daemon, which breaks once they're genuinely on separate VMs. Read +that document's "Known gap" section before treating two-VM production as fully wired. diff --git a/docs/OBSERVABILITY.md b/docs/OBSERVABILITY.md new file mode 100644 index 0000000..cde9f10 --- /dev/null +++ b/docs/OBSERVABILITY.md @@ -0,0 +1,64 @@ +# Observability + +Self-hosted Prometheus + Grafana + Alertmanager + Loki/Promtail, deployed via +`infra/docker/platform/docker-compose.production.yml` on rn-platform-01, with lightweight agents +(node-exporter, cadvisor, promtail) also on rn-lab-host-01. No paid SaaS. + +This is operational observability — host/container/database health, uptime, resource pressure. +It is deliberately separate from the application's own instructor learning analytics +(Phase 3B's cohort/scenario/student dashboards) — different audience (ops vs. instructors), +different data (infrastructure vs. learning outcomes), and mixing them would make both worse. + +## What's real today + +- **Host metrics** (both VMs): node-exporter — CPU, memory, disk, network. +- **Container metrics** (Lab Host): cadvisor — per-container CPU/memory/network, including + student Lab containers (`rootnode-lab-*`). +- **Postgres metrics**: postgres-exporter — connections, transaction rate, table sizes. +- **Uptime/liveness**: blackbox-exporter probing `/healthz` on the Gateway + (`platform/gateway/src/server.js`) and the Platform API (`platform/api/src/server.js`), both + added this phase specifically for this purpose, plus the Provisioner's pre-existing `/health`. +- **Logs**: Loki + Promtail, scraping Docker container logs and journald from both VMs, with a + redaction pipeline stage that strips `password|token|secret=`-pattern values before they're + stored (`infra/monitoring/promtail/`). +- **Dashboards**: Grafana, reachable only via `https:///grafana/` and only from + `management_cidr` (see `docs/PRODUCTION_NETWORK_SECURITY.md`). One starter dashboard, + `infra/monitoring/grafana/provisioning/dashboards/rootnode-overview.json` — 8 panels (Platform + + Lab Host CPU/memory/disk, Postgres up/connections, running-Lab-count via cadvisor label + match [best-effort, caveated], uptime probes). + +## What's NOT built yet (honest accounting, not silently assumed) + +Application-level metrics — active student count, Lab provisioning success/failure rate, +Gateway request latency/error rate, per-endpoint API metrics — would need real instrumentation +added to the Platform API/Gateway/Provisioner (e.g. a `/metrics` endpoint via `prom-client`). +`infra/monitoring/prometheus/prometheus.yml.j2` has a commented-out scrape job as the landing +spot for this, explicitly marked as a follow-up, not wired to anything real yet. + +## Alerting + +`infra/monitoring/prometheus/rules/rootnode.yml` — two severities: + +**Critical** (fires fast, pages): PlatformAPIDown, GatewayDown, PostgresDown, LabHostDown, +BackupFailed, DiskCriticallyFull. + +**Warning** (informational, slower repeat): LabHostHighMemory, HighDiskUsage, BackupAging. +(RepeatedProvisioningFailures and ExcessiveFailedLogins are present as inert placeholders — +they need the not-yet-built application metrics above before they can fire on anything real.) + +Never alerts on an individual student's scenario failure or a single Lab crash — those are +expected, routine outcomes of normal usage, not operational incidents. + +`infra/monitoring/alertmanager/alertmanager.yml.j2` — critical alerts route separately with a +30-minute repeat interval, everything else at 4 hours, one webhook receiver +(`vault_alertmanager_webhook_url` — points at whatever channel gets configured at deploy time, +e.g. a Slack/Discord/Matrix webhook), and one inhibit rule so a LabHostDown critical suppresses +its own dependent warnings instead of paging twice for the same root cause. + +## Validation performed (local, real tooling — not hardware-dependent) + +`infra/monitoring/validate-monitoring.sh` wraps: `promtool check config` (Prometheus config), +`promtool check rules` (alert rules — 11 rules found, valid), `amtool check-config` +(Alertmanager — 1 inhibit rule, 2 receivers, valid). These prove syntactic/semantic correctness +of the configs, not that the real network paths between rn-platform-01 and rn-lab-host-01 work — +that requires the real hardware and is covered by `docs/PRODUCTION_VALIDATION_CHECKLIST.md`. diff --git a/docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md b/docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md new file mode 100644 index 0000000..a265b72 --- /dev/null +++ b/docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md @@ -0,0 +1,184 @@ +# Phase 4 — Production Infrastructure Package + +**Status: PHASE 4 BUILD COMPLETE. PRODUCTION DEPLOYMENT PENDING HARDWARE.** + +The dedicated Hetzner server this is designed for hasn't been purchased yet. Everything in this +document and the `infra/` tree it describes is buildable, testable (against local +Docker/Postgres/disposable certs), and reviewable **now** — none of it requires the real +hardware to exist. What genuinely can't happen until the hardware exists is called out +explicitly throughout, never silently assumed to already work. + +This document is the map. Detail docs: `docs/PRODUCTION_DEPLOYMENT.md` (step-by-step), +`docs/PRODUCTION_NETWORK_SECURITY.md` (network/firewall/TLS design), `docs/BACKUP_DR.md` +(backup/restore/disaster recovery), `docs/OBSERVABILITY.md` (metrics/logs/alerts), +`docs/CAPACITY_PLANNING.md` (sizing model), and the `docs/RUNBOOK_*.md` files (14 topics, +command-oriented). + +## 1. What was built + +``` +infra/ +├── proxmox/ manual bare-metal install docs + cloud-init template definition +├── terraform/ VM creation (rn-platform-01, rn-lab-host-01) — real `tofu validate` pass +├── ansible/ all host configuration — real `ansible-playbook --syntax-check` pass +│ ├── roles/ common, firewall, docker-tls, platform, lab-host, monitoring-agent, backup +│ ├── group_vars/ production.yml, staging.yml, vault template +│ └── site.yml top-level playbook, correct role ordering +├── docker/ production Compose files, Caddyfile, Postgres tuning — real `docker +│ compose config` + `caddy validate` pass +├── firewall/ iptables-restore rulesets for all 3 hosts — structural lint pass +├── monitoring/ Prometheus/Alertmanager/Loki/Promtail/Grafana config — real `promtool` +│ + `amtool` validation pass +├── backup/ Postgres backup/restore/verify scripts — REAL end-to-end test against +│ disposable local Postgres, including the actual production migrations +├── scripts/ deploy/rollback/smoke-test/CI/loadtest/capacity-model tooling +├── ci/ Gitea Actions workflow (build/test/stage automatic, prod manual) +└── environments/ dev/staging/production config templates, no secrets committed +``` + +18 new docs (this file + 5 detail docs + 12 runbooks), 4 updated (`LAB_ARCHITECTURE.md`, +`ENGINEERING_SOP.md`, `README.md`, `platform/README.md`). + +## 2. Architecture (unchanged from the design, now codified) + +``` +Hetzner Dedicated Server +└── Proxmox VE + ├── rn-platform-01 (frontend, Platform API, Gateway, Provisioner, Postgres, + │ Prometheus/Alertmanager/Grafana/Loki, backup cron, Caddy) + └── rn-lab-host-01 (Docker Engine w/ TLS, egress proxy, disposable student Labs, + node-exporter/cadvisor/promtail) +``` + +The Platform VM / Lab Host VM security boundary from `docs/LAB_ARCHITECTURE.md` is the one +thing this phase was explicitly told never to weaken — it's codified, not just described, in +`infra/firewall/` and `infra/ansible/roles/docker-tls/`. + +## 3. VM specifications + +| | rn-platform-01 | rn-lab-host-01 | +|---|---|---| +| Baseline (default in `infra/terraform/variables.tf`) | 4 cores, 16 GB RAM, 100 GB disk | 24 cores, 96 GB RAM, 500 GB disk | +| Recommended (once real host is ~16c/32t/128GB) | same — Postgres/API/Gateway/Provisioner don't need much | majority of remaining host resources | +| Max-growth | scale RAM with Postgres `shared_buffers` if instructor/cohort count grows a lot | scale with measured concurrent-Lab capacity (`docs/CAPACITY_PLANNING.md`) | +| Configurable via | `terraform.tfvars`' `platform_vm`/`lab_host_vm` objects | same | + +NUMA/CPU pinning: not configured — not useful at this scale (a handful of VMs on one physical +host, not a dozen contending for cache locality). Revisit only if real load testing shows a +reason to. + +## 4. Storage decision framework + +Not a final answer — `infra/environments/production/provisioner-config/README.md` and this +section exist so the final ZFS layout is a deployment-time decision made against real disks, +not guessed now. Candidate layouts (`infra/terraform/variables.tf`'s `vm_disk_datastore` names +whichever one gets chosen): + +- **4× NVMe**: mirrored vdevs (RAID10-equivalent) — two mirror pairs, one for + Postgres/platform data (I/O-sensitive, small), one for Lab Host disposable data + (throughput-sensitive, large, never needs redundancy since nothing on it is precious). +- **2× SSD/NVMe**: one mirrored vdev, logical dataset separation (ZFS datasets, not separate + pools) between platform (`{{ rootnode_data_root }}/postgres`, `/backups`) and Lab Host data. +- **Mixed SSD/HDD**: only if the purchased server's actual disks require it — HDD for + disposable Lab scratch space is acceptable (it's disposable), never for Postgres. + +**Never single-disk for Postgres data**, in any layout — a mirror is the floor, not a nice-to-have. + +Pre-deployment disk inspection checklist (model, size, SMART, power-on hours, wear/endurance, +sector errors, temperature, firmware): `docs/RUNBOOK_DISK_FAILURE.md`'s inspection section — +run once, against the real drives, before committing to a layout. + +## 5-8. Network, firewall, Docker TLS, public ingress + +Full detail: `docs/PRODUCTION_NETWORK_SECURITY.md`. Summary: two private VM networks, Docker TLS +(mutual, already application-supported since Phase 1 — this phase only needed the certificate +generation/deployment automation, not new app code), a single public origin (Caddy, automatic +HTTPS) proxying only to the Gateway, and one genuinely new architectural item this phase +surfaced and addressed at the network-policy layer: **Gateway-to-Lab reachability across two +real VMs** — see that doc's dedicated section for what's built vs. what still needs a small, +reviewed Provisioner/Gateway code change before real go-live. + +## 9-10. Production auth/cookie config, secrets + +`infra/environments/production/*.env.template` set `COOKIE_SECURE=1`/`TRUST_PROXY=1` explicitly +(dev's unsafe defaults are never inherited). Secrets: Ansible Vault (already bundled with +Ansible — no new tool to install/trust for this project's secret surface), template at +`infra/ansible/group_vars/production.vault.yml.template`. Nothing is committed — +`infra/.gitignore` covers `*.env`, TLS private key material, vault plaintext, Terraform state. + +## 11-13. Postgres, backup, DR + +Full detail: `docs/BACKUP_DR.md`. Summary: conservative tuning baseline (not hand-tuned before +real RAM is known), daily encrypted backups with an off-host adapter point (Storage Box/S3/any +SSH target), and a **real, tested** backup→encrypt→restore→verify cycle +(`infra/backup/test-backup-restore.sh`) — run against disposable local Postgres with the actual +production migrations applied, not a mock. + +## 14-17. Observability, logging, alerting, uptime + +Full detail: `docs/OBSERVABILITY.md`. Prometheus + Grafana + Loki/Promtail (the user's stated +Prometheus/Grafana preference, honored) + Alertmanager, all self-hosted, no paid dependency. +Deliberately does not duplicate the application's own instructor learning analytics +(Phase 3B) — operational observability and learning analytics stay separate concerns with +separate audiences, per the explicit instruction. Application-level metrics (active +students/Labs, provisioning failures, Gateway error rate) are a documented, **not-yet-built** +follow-up — see that doc's honest accounting of what's real today (host/container/Postgres +metrics, HTTP healthz probes) vs. what needs app instrumentation later. + +## 18-19. CI/CD, image strategy + +`infra/ci/gitea-workflows/build-and-test.yml` — automatic test/build/tag/staging-deploy, +**manual** production promotion (`infra/scripts/deploy-production.sh`), matching the stated +preference. SHA-tagged images always, semantic tags on deliberate releases, never `:latest` in +staging/production. Full detail: `infra/docker/IMAGE_STRATEGY.md`. + +## 20-21. Ansible, Terraform + +`infra/ansible/site.yml` — idempotent roles, doesn't assume the VMs already exist (Terraform +creates them first). Real `ansible-playbook --syntax-check` pass. `infra/terraform/` — real +`tofu validate` pass against the actual `bpg/proxmox` provider schema. Neither installs Proxmox +itself or builds the base VM template — both stay deliberate manual, documented, one-time steps +(`infra/proxmox/README.md`), per the explicit "don't over-automate the fragile one-time step" +instruction. + +## 22-23. Environment config, staging + +`infra/environments/{dev,staging,production}/` — explicit separation, production never +inherits dev defaults. Staging is designed to run in the homelab or a resource-constrained +slice of the real server later — no second dedicated server required. + +## 24-25. Load testing, capacity model + +`infra/scripts/loadtest/` (real Prepare-Labs-driven boot test + `docker stats` sampling) and +`infra/scripts/capacity-model.js` (a real, working worksheet — tested against example inputs, +produces CPU-vs-memory-bound recommended/warning/hard-admission-control thresholds). No +concurrency number is hard-coded as a marketing figure; the example config in +`infra/environments/production/capacity.yml.example` is explicitly flagged as placeholder data. + +## 26-27. Validation checklist, runbooks + +`docs/PRODUCTION_VALIDATION_CHECKLIST.md` — one master checklist spanning hardware through +academy-workflow verification. 12 runbooks (`docs/RUNBOOK_*.md`) — first deployment, normal +deploy, rollback, Postgres restore, VM rebuild (both hosts), Docker TLS rotation, TLS renewal, +owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host +resource pressure. + +## What this phase deliberately did NOT touch + +Learning system design, curriculum content, RBAC model, session architecture — all Phase 3A/3B, +unchanged. No parent portal, billing, held scenarios, or production OIDC provider integration. +RootNode production infrastructure is dedicated to RootNode Academy — never folded into +existing Synthos production servers, and canonical source stays Gitea, never GitHub `kubekosh`. + +## Known gap surfaced during this phase (not silently papered over) + +Building the real production Compose/network config surfaced that the Gateway's existing +mechanism for reaching a Lab container (`env.internal_ip`, a Docker-bridge-local address) only +works when Gateway and Lab share a Docker daemon — true in every dev/single-host setup so far, +**not** true once Labs run on a genuinely separate `rn-lab-host-01`. The network-policy half is +built (a firewall allowance for Gateway→Lab-Host on a published port range, +`infra/firewall/rn-lab-host-01.rules`); the application half (the Provisioner recording a +reachable external address for each Lab, and the Gateway using it) is an explicit, reviewed, +NOT-YET-BUILT follow-up — see `docs/PRODUCTION_NETWORK_SECURITY.md` for the two candidate +designs and why neither was silently chosen here. This must land before real two-VM go-live; +flagging it clearly was judged better than guessing at an architecture change mid-infra-phase. diff --git a/docs/PRODUCTION_DEPLOYMENT.md b/docs/PRODUCTION_DEPLOYMENT.md new file mode 100644 index 0000000..9815f1e --- /dev/null +++ b/docs/PRODUCTION_DEPLOYMENT.md @@ -0,0 +1,84 @@ +# Production Deployment + +Step-by-step, for when the real Hetzner host exists. See +`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` for the architecture this executes. + +## 0. One-time, manual, not automated (deliberately) + +1. Install Proxmox VE on the bare-metal host — `infra/proxmox/README.md`. +2. Inspect the real disks, pick a ZFS layout — `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4, + `docs/RUNBOOK_DISK_FAILURE.md`'s inspection checklist. +3. Build the cloud-init VM template Terraform will clone — + `infra/proxmox/cloud-init/user-data.yaml` + `infra/proxmox/README.md`. +4. Fill in `infra/terraform/terraform.tfvars` (copy from `.tfvars.example`) with the real + Proxmox API endpoint, node name, storage/datastore names, and chosen VM sizes. +5. Fill in `infra/ansible/inventory/production.ini` with the real VM IPs, and + `infra/ansible/group_vars/production.yml` with `platform_ip`, `lab_host_ip`, + `management_cidr`, `proxmox_mgmt_cidr`. +6. Copy `infra/ansible/group_vars/production.vault.yml.template` → + `production.vault.yml`, fill in every `vault_*` secret with real generated values, then + `ansible-vault encrypt group_vars/production.vault.yml`. Never commit the plaintext. + +## 1. Create the VMs + +```bash +cd infra/terraform +tofu init +tofu plan # review before applying against real hardware +tofu apply +``` + +Creates `rn-platform-01` and `rn-lab-host-01` from the cloud-init template. Confirm both boot +and are SSH-reachable from the management network before continuing. + +## 2. Generate Docker TLS certificates (offline, once) + +```bash +export ROOTNODE_TLS_BUNDLE_DIR=/secure/offline/path # never inside the repo +./infra/scripts/docker-tls/generate-docker-tls.sh +./infra/scripts/docker-tls/verify-docker-tls.sh +``` + +The CA private key stays in `ROOTNODE_TLS_BUNDLE_DIR` and never touches either VM — only the CA +cert (public) and the relevant leaf cert+key go to each host. + +## 3. Configure both hosts + +```bash +cd infra/ansible +ansible-playbook -i inventory/production.ini site.yml --ask-vault-pass +``` + +Role order matters and is fixed in `site.yml`: `common` → `firewall` → `docker-tls` → +`platform`/`lab-host` → `monitoring-agent` → `backup`. Firewall and TLS land before anything +that depends on them. + +## 4. First application deploy + +```bash +./infra/scripts/deploy-production.sh +``` + +Pulls the tagged images, runs the compose stack, applies Postgres migrations (automatic, on +Platform API/Provisioner startup — see `docs/RUNBOOK_FAILED_MIGRATION.md` if one fails). + +## 5. Verify + +```bash +./infra/scripts/production-smoke-test.sh +``` + +Curls `/healthz` on the Gateway and Platform API, confirms Postgres is reachable, confirms +Prometheus/Grafana/Alertmanager are up. Then walk +`docs/PRODUCTION_VALIDATION_CHECKLIST.md` in full before calling the deployment live. + +## Normal (non-first) deploys + +`docs/RUNBOOK_NORMAL_DEPLOY.md`. Rollback: `docs/RUNBOOK_ROLLBACK.md`. + +## What this document does NOT cover + +DNS/Cloudflare setup for the public origin (a one-time manual step outside this repo's scope — +point the A/AAAA record at the real host and confirm Caddy issues its certificate). Which +storage layout to pick (decision framework only, §4 of the master doc — the real answer depends +on the real disks). diff --git a/docs/PRODUCTION_NETWORK_SECURITY.md b/docs/PRODUCTION_NETWORK_SECURITY.md new file mode 100644 index 0000000..b5a1a49 --- /dev/null +++ b/docs/PRODUCTION_NETWORK_SECURITY.md @@ -0,0 +1,93 @@ +# Production Network & Security + +The Platform VM / Lab Host VM boundary is the core production control. This document is where +every network decision that touches it gets written down. Nothing here weakens it. + +## Topology + +``` +Internet ──HTTPS(443)──▶ Caddy (rn-platform-01) ──▶ Gateway (internal bridge) + │ +management_cidr ──SSH──▶ rn-platform-01 │ +management_cidr ──SSH──▶ rn-lab-host-01 │ +proxmox_mgmt_cidr ──22/8006──▶ Proxmox host │ + │ +rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd) +rn-lab-host-01 (promtail) ──3100──▶ rn-platform-01 (Loki) +rn-lab-host-01 (node-exporter/cadvisor) ◀──scrape── rn-platform-01 (Prometheus) +``` + +Firewall source of truth: `infra/firewall/{proxmox-host,rn-platform-01,rn-lab-host-01}.rules` +(iptables-restore format, deployed by the `firewall` Ansible role). Validate with +`infra/firewall/validate-firewall.sh`. + +## What is explicitly denied + +- Student Lab → another student Lab (no student-to-student traffic, ever). +- Student Lab → Proxmox management (both the host firewall and the `lab_subnet_range` DENY + rule in `rn-lab-host-01.rules`). +- Student Lab → Platform Postgres / Platform API / Gateway internals (logged DENY in + `rn-platform-01.rules` for anything from `lab_host_ip` to ports 5432/4100). +- rn-lab-host-01 → Postgres (same rule — the Lab Host itself has no legitimate reason to reach + Postgres; only the Provisioner on rn-platform-01 does, and it talks to Postgres over the + internal Docker bridge, never across the VM boundary). +- Any student egress except through the approved egress proxy (enforced by the Provisioner's + existing per-Lab `DOCKER-USER` chain manipulation from Phase 1 — unchanged by this phase). +- Direct public exposure of the Provisioner, Postgres, Docker TLS port, Proxmox UI, or Grafana + (Grafana is reachable only via Caddy's `/grafana/*` path, itself restricted to + `management_cidr` by a `caddy` matcher — see `infra/docker/platform/Caddyfile.j2`). + +## Docker TLS (Provisioner ↔ Lab Host) + +Mutual TLS on port 2376, restricted at the firewall to `platform_ip` only. Application support +already existed since Phase 1 (`platform/provisioner/src/docker.js`'s `buildClient()`); this +phase added the operational half: `infra/scripts/docker-tls/generate-docker-tls.sh` (CA + +server + client cert generation), the `docker-tls` Ansible role (deploys certs, configures +`dockerd`'s `daemon.json`), and `docs/RUNBOOK_DOCKER_TLS_ROTATION.md` for renewal. The CA +private key is generated offline and never committed or deployed to either VM — only the CA +cert (to verify the peer) and each host's own leaf cert+key. + +## Public ingress + +One origin, Caddy, automatic HTTPS. `reverse_proxy gateway:4200` — Caddy talks only to the +Gateway; nothing else is proxied. WebSocket upgrade works automatically (Caddy's default +`reverse_proxy` behavior, no special config needed for the terminal WebSocket path). Whether to +front this with Cloudflare or expose the origin directly is left as a deployment-time decision +(both work with Caddy's ACME flow; Cloudflare adds DDoS absorption at the cost of Cloudflare +being able to see plaintext unless "Full (strict)" mode is used) — not decided here since it +doesn't affect anything else in this document. + +## Known gap: Gateway-to-Lab reachability across two real VMs + +**This is the most important item in this document.** + +The Gateway currently reaches a running Lab container via `env.internal_ip` — a Docker +bridge-local address recorded by the Provisioner when it creates the container +(`platform/gateway/src/server.js`, `platform/provisioner`, built in Phase 2A). This works when +Gateway and the Lab share a Docker daemon, which has been true in every setup so far, including +dev and any single-host deployment. **It stops being true once Labs run on a genuinely separate +`rn-lab-host-01`** — a Docker bridge-internal IP on one host is not routable from a different +host by default. + +What's built now (network-policy half only): a firewall allowance in +`infra/firewall/rn-lab-host-01.rules` for `platform_ip` → `lab_gateway_port_range` +(`30000:32767` by default, configurable), so that once Lab containers publish ports on the Lab +Host's own interface, the Gateway VM is permitted to reach them. + +What's **not yet built** (application half — explicitly flagged, not silently decided here): +the Provisioner needs to either (a) publish each Lab container's relevant port(s) on the Lab +Host's interface and record a reachable `external_ip`/`external_port` alongside (or instead of) +`internal_ip`/`internal_port`, or (b) route Gateway→Lab traffic through a proxy/tunnel that +lives on the Lab Host and forwards to the container's internal address. Option (a) is simpler +and consistent with how the egress proxy and cadvisor are already handled in this phase's +compose files; option (b) avoids publishing a wide port range but adds a moving part. This is a +small, reviewable, application-level change — not a network-policy change — and should land as +its own reviewed commit before real two-VM production traffic depends on it. Until it lands, +Gateway↔Lab terminal/scenario traffic will not work correctly across two separate VMs, even +though every other production networking, firewall, and TLS boundary is in place and correct. + +## Cookie / auth production settings + +`infra/environments/production/*.env.template`: `COOKIE_SECURE=1`, `TRUST_PROXY=1`. Dev's +insecure defaults (`COOKIE_SECURE` unset, no proxy trust) are never inherited — production env +files are separate files, not a dev file with overrides layered on. diff --git a/docs/PRODUCTION_VALIDATION_CHECKLIST.md b/docs/PRODUCTION_VALIDATION_CHECKLIST.md new file mode 100644 index 0000000..11ba11d --- /dev/null +++ b/docs/PRODUCTION_VALIDATION_CHECKLIST.md @@ -0,0 +1,50 @@ +# Production Validation Checklist + +Walk this once, top to bottom, before calling a production deployment live. Every item here +needs the real hardware — this is the "DEPLOY LATER" half of Phase 4, not something this repo +can prove in advance. + +## Hardware / Proxmox +- [ ] Disk inspection done, layout chosen and matches what's actually in the server (`docs/RUNBOOK_DISK_FAILURE.md` inspection checklist, `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4) +- [ ] Proxmox VE installed, reachable only from `management_cidr`/`proxmox_mgmt_cidr` +- [ ] VM template built from real cloud-init (`infra/proxmox/README.md`) +- [ ] `tofu apply` created both VMs; both boot and are SSH-reachable + +## Platform VM +- [ ] `ansible-playbook site.yml` completed clean, all roles +- [ ] `docker compose ps` shows every service healthy (postgres, provisioner, api, gateway, reverse-proxy, prometheus, alertmanager, grafana, loki, promtail, exporters) +- [ ] `curl https:///healthz` (via Gateway) returns 200 +- [ ] `curl http://localhost:4300/healthz` (Platform API, from the VM) returns `{"ok":true,"db":"ok"}` +- [ ] TLS certificate issued and valid for the real domain + +## Lab Host VM +- [ ] Docker daemon TLS verified: `docker --tlsverify --tlscacert=... -H tcp://:2376 info` succeeds from the Platform VM, fails from anywhere else +- [ ] Provisioner successfully creates a real Lab against the real Lab Host +- [ ] Gateway can actually reach that Lab's terminal — **this depends on the not-yet-built application change documented in `docs/PRODUCTION_NETWORK_SECURITY.md`'s "Known gap" section; do not mark this item done until that change has landed and been verified for real** +- [ ] Egress proxy correctly restricts a Lab's outbound traffic (Provisioner's per-Lab `DOCKER-USER` chain) + +## Security +- [ ] `infra/firewall/validate-firewall.sh` run against the real hosts with real `iptables-restore --test`, not just the structural lint +- [ ] Confirm from an unauthorized IP: Postgres, Provisioner, Docker TLS port, Proxmox UI are all unreachable +- [ ] Confirm Student Lab → another Lab is blocked; confirm Student Lab → Proxmox is blocked +- [ ] `COOKIE_SECURE`/`TRUST_PROXY` active (check a real Set-Cookie header over HTTPS) + +## Academy (application-level smoke test) +- [ ] Owner bootstrap login works +- [ ] Instructor can create a cohort, roster a student +- [ ] Student can log in, boot a Lab, complete a scenario, see progress +- [ ] An assessment can be assigned, taken, and graded end-to-end +- [ ] Instructor analytics dashboards populate with real data + +## Backup +- [ ] Manual `backup-postgres.sh` run succeeds against real Postgres, real off-host target reachable +- [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness +- [ ] Cron job confirmed scheduled (`crontab -l`) + +## Observability +- [ ] Grafana dashboard shows real data for both VMs +- [ ] Trigger one alert deliberately (e.g. stop a service) and confirm it fires and reaches the webhook +- [ ] Loki has logs flowing from both VMs + +Only once every box above is checked against the real server does the deployment graduate from +"infrastructure package built" to "actually in production." diff --git a/docs/RUNBOOK_BACKUP_FAILURE.md b/docs/RUNBOOK_BACKUP_FAILURE.md new file mode 100644 index 0000000..ed02ceb --- /dev/null +++ b/docs/RUNBOOK_BACKUP_FAILURE.md @@ -0,0 +1,31 @@ +# Runbook: Backup Failure + +Triggered by the `BackupFailed`/`BackupAging` alerts (`infra/monitoring/prometheus/rules/rootnode.yml`). + +```bash +# 1. See what actually happened — the cron wrapper logs to syslog +journalctl -u rootnode-backup --since "24 hours ago" -o cat +# or, if the cron job never even ran: +grep rootnode-postgres-backup /var/log/syslog + +# 2. Run it manually to see the failure live +source /etc/rootnode/backup.env +/opt/rootnode/backup/backup-postgres.sh +``` + +**Common causes:** +- Disk full on `{{ rootnode_data_root }}/backups` — check `df -h`, prune more aggressively + (`BACKUP_RETENTION_DAYS`) or the ZFS dataset needs more space (see `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4). +- Off-host target unreachable (`BACKUP_OFFSITE_TARGET`) — check the SSH key/credentials + haven't expired, the target host/bucket is reachable: `ssh echo ok` or + the equivalent for whatever adapter is configured (`docs/BACKUP_DR.md`). +- `BACKUP_ENCRYPTION_PASSPHRASE` missing/wrong in `/etc/rootnode/backup.env` — re-check against + the value in Ansible Vault (`vault_backup_encryption_passphrase`); a mismatch here means even + a "successful" old backup becomes unrestorable, so treat this specific failure mode as urgent. +- The dump itself is suspiciously small (script's own sanity check tripped) — Postgres may be + unhealthy; check `docker exec rootnode-platform-postgres-1 pg_isready` first. + +**After fixing:** run the backup manually once to confirm success (metric updates immediately, +alert clears within one Prometheus scrape interval), then confirm the cron job is still +correctly scheduled (`crontab -l` as root, or re-run the `backup` Ansible role if the cron entry +itself is suspect). diff --git a/docs/RUNBOOK_DISK_FAILURE.md b/docs/RUNBOOK_DISK_FAILURE.md new file mode 100644 index 0000000..76e8f1f --- /dev/null +++ b/docs/RUNBOOK_DISK_FAILURE.md @@ -0,0 +1,44 @@ +# Runbook: Disk Failure + +Applies once real hardware/ZFS exists — see `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4 for +the storage decision framework this assumes (mirrored vdevs, never single-disk for Postgres). + +```bash +# 1. Confirm the failure +zpool status -v +smartctl -a /dev/sdX # the suspect drive + +# 2. If it's a mirrored vdev (the required layout for anything holding Postgres data — see §4), +# the pool stays online in DEGRADED state. No immediate data-loss risk, but no redundancy +# either until replaced — treat DEGRADED as urgent, not routine. +zpool status # confirm DEGRADED, not FAULTED/UNAVAIL (those mean act immediately, possibly + # restore from backup if the redundant mirror side is also compromised) + +# 3. Replace the physical drive, then: +zpool replace +zpool status # watch the resilver progress + +# 4. If a whole pool was lost (both mirror sides failed simultaneously — the scenario mirroring +# exists to make vanishingly unlikely, but plan for it anyway): +# - rn-lab-host-01's disposable data: nothing to recover, just rebuild the VM +# (docs/RUNBOOK_VM_REBUILD.md) +# - rn-platform-01's Postgres data: restore from the latest off-host backup +# (docs/RUNBOOK_POSTGRES_RESTORE.md) — this is exactly the scenario off-host backup +# copies exist for; a local-only backup on the SAME failed pool would be useless here +``` + +**Pre-deployment disk inspection checklist** (run once when the real drives are known, before +they go into service — `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4): + +```bash +for d in /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1; do # adjust to real device names + echo "== $d ==" + smartctl -i "$d" | grep -E 'Model|Serial|Capacity|Firmware' + smartctl -A "$d" | grep -iE 'power_on_hours|reallocated|pending|uncorrectable|wear|percentage.used|temperature' + smartctl -H "$d" | grep -i overall +done +``` +Record model, size, power-on hours, wear/endurance percentage, reallocated/pending sector +counts, temperature, and firmware version for every drive before committing to a ZFS layout — +a drive already showing wear or reallocated sectors shouldn't be trusted with Postgres data +even in a mirror. diff --git a/docs/RUNBOOK_DOCKER_TLS_ROTATION.md b/docs/RUNBOOK_DOCKER_TLS_ROTATION.md new file mode 100644 index 0000000..a595451 --- /dev/null +++ b/docs/RUNBOOK_DOCKER_TLS_ROTATION.md @@ -0,0 +1,31 @@ +# Runbook: Docker TLS Rotation + +Rotate on a schedule (recommend yearly, well before the ~2.25-year cert lifetime +`generate-docker-tls.sh` defaults to) or immediately if a client cert may have leaked. + +```bash +# 1. Generate a fresh bundle (new CA too — simplest, avoids any question about whether the old +# CA's private key was ever exposed) +cd infra/scripts/docker-tls +./generate-docker-tls.sh --out-dir ./out-$(date +%Y%m%d) --lab-host-ip --lab-host-dns rn-lab-host-01.internal +./verify-docker-tls.sh ./out-$(date +%Y%m%d) + +# 2. Redeploy — this restarts dockerd on rn-lab-host-01 (brief interruption to any in-flight +# Docker API call from the Provisioner, NOT to already-running student Labs — their +# containers keep running through a dockerd restart) +cd ../../ansible +ROOTNODE_TLS_BUNDLE_DIR=../scripts/docker-tls/out-$(date +%Y%m%d) \ + ansible-playbook site.yml -i inventory/production.ini --tags docker-tls --ask-vault-pass + +# 3. Verify the Provisioner can still reach rn-lab-host-01 +curl -s http://localhost:4100/health # from rn-platform-01 +``` + +**Troubleshooting a failed rotation:** +- `x509: certificate signed by unknown authority` — the new ca.pem wasn't deployed to both + sides, or an old one is cached; re-run the docker-tls role on BOTH hosts. +- Provisioner logs `ECONNREFUSED` on port 2376 — dockerd didn't actually restart with the new + daemon.json; check `systemctl status docker` on rn-lab-host-01 and the ExecStart override + (`infra/ansible/roles/docker-tls/templates/docker-override.conf.j2`) actually applied. +- Firewall — confirm `infra/firewall/rn-lab-host-01.rules`' 2376 rule still matches + rn-platform-01's actual IP (didn't change as part of this rotation, but worth a sanity check). diff --git a/docs/RUNBOOK_FAILED_MIGRATION.md b/docs/RUNBOOK_FAILED_MIGRATION.md new file mode 100644 index 0000000..7104975 --- /dev/null +++ b/docs/RUNBOOK_FAILED_MIGRATION.md @@ -0,0 +1,35 @@ +# Runbook: Failed Migration + +Both Platform API and Provisioner apply their own migrations automatically on startup +(`platform/api/src/db.js`, `platform/provisioner/src/db.js`), file-tracked in +`schema_migrations`, wrapped in a transaction per file — a migration either fully applies or +fully rolls back, it can't leave a table half-altered. + +**If a deploy fails because a migration errored:** + +```bash +# 1. Check what actually happened +docker logs rootnode-platform-api-1 --tail 50 # the error is right there — migration file + message +docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \ + "SELECT * FROM schema_migrations ORDER BY applied_at DESC LIMIT 10;" + +# 2. Since the failing migration's transaction rolled back, the schema is still in its +# PREVIOUS valid state — the app just won't start until the migration succeeds. Fix the +# root cause (usually a real data issue the migration didn't anticipate, or the +# cross-service ordering dependency — Provisioner's migrations must run before Platform +# API's, since scenario_attempts has a real FK to environments; see +# docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md). + +# 3. Redeploy once fixed — the migration runner is idempotent and will just pick up where it +# left off; already-applied migrations are skipped. +./infra/scripts/deploy-production.sh +``` + +**If you need to roll back the CODE but the migration already succeeded** (schema changed, +code that depended on it is being reverted): rolling back application images +(`docs/RUNBOOK_ROLLBACK.md`) does NOT undo a Postgres migration — there is no automatic +down-migration mechanism (deliberate, matches the existing check-before-alter, additive-only +migration convention documented in `docs/ENGINEERING_SOP.md` §6). If the old code genuinely +can't run against the new schema, restore from the pre-migration backup instead +(`docs/RUNBOOK_POSTGRES_RESTORE.md`) rather than trying to hand-write a down-migration under +pressure. diff --git a/docs/RUNBOOK_FIRST_DEPLOYMENT.md b/docs/RUNBOOK_FIRST_DEPLOYMENT.md new file mode 100644 index 0000000..af34127 --- /dev/null +++ b/docs/RUNBOOK_FIRST_DEPLOYMENT.md @@ -0,0 +1,34 @@ +# Runbook: First Production Deployment + +Prerequisites: Proxmox installed (`infra/proxmox/README.md`), VMs created +(`infra/terraform/`), DNS pointed at rn-platform-01's public IP. + +```bash +# 1. Generate Docker TLS material (offline, on your own machine — never on either VM) +cd infra/scripts/docker-tls +./generate-docker-tls.sh --out-dir ./out --lab-host-ip --lab-host-dns rn-lab-host-01.internal +./verify-docker-tls.sh ./out + +# 2. Fill in real values +cd ../../ansible +cp group_vars/production.vault.yml.template group_vars/production.vault.yml +$EDITOR group_vars/production.vault.yml # real passwords/secrets — openssl rand -hex 32 etc. +ansible-vault encrypt group_vars/production.vault.yml +$EDITOR group_vars/production.yml # real IPs, domain, CIDRs +$EDITOR inventory/production.ini # real IPs + +# 3. Deploy +ROOTNODE_TLS_BUNDLE_DIR=../scripts/docker-tls/out \ + ansible-playbook site.yml -i inventory/production.ini --ask-vault-pass + +# 4. Bootstrap the owner account (once, by a human) +ssh rootnode-deploy@rn-platform-01 +docker exec -it rootnode-platform-api-1 node bootstrap-owner.js you@example.com "Your Name" +# (password comes from BOOTSTRAP_OWNER_PASSWORD in the environment, or the CLI prompt — see +# platform/api/README.md) + +# 5. Verify +../../infra/scripts/production-smoke-test.sh +``` + +Then work through `docs/PRODUCTION_VALIDATION_CHECKLIST.md` in full before calling this done. diff --git a/docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md b/docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md new file mode 100644 index 0000000..d85e5e3 --- /dev/null +++ b/docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md @@ -0,0 +1,29 @@ +# Runbook: High Lab Host Resource Pressure + +Triggered by the `LabHostHighMemory`/`HighDiskUsage` warning alerts. + +```bash +# 1. How many Labs are actually running, and against what admission-control threshold? +curl -s http://localhost:4100/environments | jq '[.[] | select(.status=="running")] | length' +node infra/scripts/capacity-model.js # recommended/warning/hard thresholds + +# 2. If genuinely over capacity: this is expected pressure from real usage, not a bug. +# - Destroy idle/abandoned Labs early via /admin (per-cohort "Destroy Labs" after a class +# session ends, rather than waiting for the idle reaper's default window) +# - If a specific cohort's class just ended, use /admin -> Cohorts -> [cohort] -> Destroy +# Cohort Labs immediately rather than waiting + +# 3. If pressure is high with FEWER Labs running than capacity-model.js's threshold: something +# is consuming more per-Lab resources than the model assumes (a scenario doing something +# unusually heavy, a memory leak in a long-running Lab, zombie processes from a crashed +# reset). Check per-container usage: +docker stats --no-stream | grep rootnode-lab- + +# 4. Update infra/environments/production/capacity.yml with fresh measured numbers +# (infra/scripts/loadtest/) if the per-Lab footprint has genuinely changed, so the +# admission-control threshold reflects reality going forward. +``` + +Do not silently raise resource limits in `provisioner-config/resource-policy.yaml` to make an +alert go away without understanding why usage is higher than modeled — that just moves the +same pressure onto host RAM/CPU directly instead of onto individual Lab limits. diff --git a/docs/RUNBOOK_NORMAL_DEPLOY.md b/docs/RUNBOOK_NORMAL_DEPLOY.md new file mode 100644 index 0000000..b9bf4c0 --- /dev/null +++ b/docs/RUNBOOK_NORMAL_DEPLOY.md @@ -0,0 +1,17 @@ +# Runbook: Normal Deploy + +For a routine code change already merged to `main` on the canonical Gitea repo. + +1. CI (`infra/ci/gitea-workflows/build-and-test.yml`) runs automatically: tests, image build/tag/push, staging deploy, staging smoke test. +2. Review the staging smoke test result and, ideally, poke around `https://staging.` yourself. +3. Promote to production (manual, deliberate): + +```bash +cd infra/scripts +./deploy-production.sh sha-<12-char-sha> # or a release tag, e.g. v1.3.0 +./production-smoke-test.sh +``` + +4. Watch Grafana (`/grafana/`) for 10-15 minutes — error rates, Postgres connections, Lab Host + memory — before considering the deploy "settled." +5. If anything looks wrong: `docs/RUNBOOK_ROLLBACK.md`. diff --git a/docs/RUNBOOK_OWNER_RECOVERY.md b/docs/RUNBOOK_OWNER_RECOVERY.md new file mode 100644 index 0000000..701fd92 --- /dev/null +++ b/docs/RUNBOOK_OWNER_RECOVERY.md @@ -0,0 +1,27 @@ +# Runbook: Owner Account Recovery + +**Locked out with zero working owner accounts** (all deactivated, forgot the only owner's +password, etc.): + +```bash +ssh rootnode-deploy@rn-platform-01 +docker exec -it rootnode-platform-api-1 sh +node bootstrap-owner.js you@example.com "Your Name" +# bootstrap-owner.js refuses to run if an active owner already exists UNLESS the target email +# belongs to an EXISTING instructor row — in that case it promotes them to owner. Point it at +# any known instructor email to recover through them, or a brand-new email to create a fresh +# owner outright if the instructors table itself is in an unknown state. +``` + +**Forgot password, account still active and you have SSH access:** + +```bash +docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \ + "SELECT instructor_id, email, role, status FROM instructors WHERE role='owner';" +# Then use another owner's account to reset via /admin -> Instructors -> [account] -> Reset +# Password, OR re-run bootstrap-owner.js targeting that email (also resets the password). +``` + +**Never** hand-edit `password_hash` in the database — it's `scrypt(salt):hash` in a specific +format (`platform/api/src/auth/password.js`); a malformed value just breaks login rather than +setting a usable password. Always go through `bootstrap-owner.js` or the admin UI's reset flow. diff --git a/docs/RUNBOOK_POSTGRES_RESTORE.md b/docs/RUNBOOK_POSTGRES_RESTORE.md new file mode 100644 index 0000000..48dfe5c --- /dev/null +++ b/docs/RUNBOOK_POSTGRES_RESTORE.md @@ -0,0 +1,45 @@ +# Runbook: Postgres Restore + +**When to use:** Postgres corruption/data loss on rn-platform-01, or verifying a backup is +actually good (do this periodically even without an incident — an untested backup is a +hypothesis, not a backup). + +```bash +# 1. Find the backup to restore +ls -la /srv/rootnode-data/backups/ # or wherever backup_offsite_target points, if local copies are gone + +# 2. Create a FRESH target database — never restore over the live one directly +docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \ + "CREATE DATABASE rootnode_platform_restore_test;" + +# 3. Restore into it +source /etc/rootnode/backup.env # for BACKUP_ENCRYPTION_PASSPHRASE +/opt/rootnode/backup/restore-postgres.sh \ + /srv/rootnode-data/backups/rootnode-platform-.sql.gz.gpg \ + "postgres://rootnode_platform:$PGPASS@localhost:5432/rootnode_platform_restore_test" +# type RESTORE when prompted + +# 4. Verify +/opt/rootnode/backup/verify-restore.sh \ + "postgres://rootnode_platform:$PGPASS@localhost:5432/rootnode_platform_restore_test" + +# 5a. If this was a drill: drop the test database +docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \ + "DROP DATABASE rootnode_platform_restore_test;" + +# 5b. If this was a REAL incident: stop the app stack, rename the broken DB, rename the +# restored one into its place, restart. +systemctl stop rootnode-platform +docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \ + "ALTER DATABASE rootnode_platform RENAME TO rootnode_platform_broken_$(date +%s);" +docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \ + "ALTER DATABASE rootnode_platform_restore_test RENAME TO rootnode_platform;" +systemctl start rootnode-platform +``` + +Local, non-production dry run of this exact flow: `infra/backup/test-backup-restore.sh` (uses +disposable containers, safe to run anytime, no real data involved). + +**Recovery point objective:** up to 24 hours of academic data (daily backup cadence) — see +`docs/BACKUP_DR.md` for the full RPO/RTO discussion and why that's the honest number, not an +aspirational one. diff --git a/docs/RUNBOOK_ROLLBACK.md b/docs/RUNBOOK_ROLLBACK.md new file mode 100644 index 0000000..2365630 --- /dev/null +++ b/docs/RUNBOOK_ROLLBACK.md @@ -0,0 +1,18 @@ +# Runbook: Rollback + +```bash +cd infra/scripts +./rollback-production.sh # rolls back to the previous deployed tag automatically +# or, to go back further than one step: +./rollback-production.sh sha- + +./production-smoke-test.sh +``` + +This rolls back **application images only**. If the bad deploy included a database migration +that needs undoing too, schema changes aren't automatically reversible — see +`docs/RUNBOOK_FAILED_MIGRATION.md` instead (or in addition). + +After rolling back: file an issue / note in the deploy log (`/opt/rootnode/deploy-history.log` +already has the rollback recorded) about what went wrong before re-attempting the forward +deploy. diff --git a/docs/RUNBOOK_STUCK_LAB_CLEANUP.md b/docs/RUNBOOK_STUCK_LAB_CLEANUP.md new file mode 100644 index 0000000..a39cc12 --- /dev/null +++ b/docs/RUNBOOK_STUCK_LAB_CLEANUP.md @@ -0,0 +1,40 @@ +# Runbook: Stuck Lab Cleanup + +A Lab that's stuck `provisioning`/`resetting` past its expected boot window, or a container +that outlived its Postgres `environments` row (crash during teardown, etc.). + +**Via the admin UI (preferred, no host access needed):** `/admin` -> Students -> [student] -> +Environment -> Destroy, then Provision again if they need a fresh one immediately. + +**Via the API directly** (same effect, useful for scripting a batch cleanup): +```bash +curl -s -b -X DELETE https://academy.example.com/api/admin/environments/ +``` + +**A container the Provisioner itself has lost track of** (rare — e.g. a crash during +provisioning left an orphaned container/network/firewall chain with no matching `environments` +row, or vice versa): + +```bash +# On rn-lab-host-01 — list what's actually running +docker ps -a --format '{{.Names}}\t{{.Status}}' | grep '^rootnode-lab-' + +# Cross-reference against the Provisioner's own view +curl -s http://localhost:4100/environments | jq '.[] | {environment_id, status, container_name}' + +# Provisioner already does this automatically on its own restart (reconcileOnStartup, +# platform/provisioner/src/lifecycle.js) — a simple `systemctl restart rootnode-platform` +# (which restarts the provisioner container as part of the stack) is often the fastest fix for +# a batch of orphans rather than hand-cleaning each one. + +# Manual last resort — remove the container/network/firewall chain directly: +docker rm -f rootnode-lab- +docker network rm rootnode-lab-net- +iptables -D DOCKER-USER -s -j ROOTNODE-ENV- # remove the jump rule FIRST +iptables -F ROOTNODE-ENV- && iptables -X ROOTNODE-ENV- +``` + +Never manually delete an `environments` row from Postgres directly unless you've confirmed the +container/network/firewall chain are already gone — the Provisioner treats that table as +authoritative and will behave unpredictably against real Docker/iptables state it doesn't know +matches. diff --git a/docs/RUNBOOK_TLS_RENEWAL.md b/docs/RUNBOOK_TLS_RENEWAL.md new file mode 100644 index 0000000..625677d --- /dev/null +++ b/docs/RUNBOOK_TLS_RENEWAL.md @@ -0,0 +1,19 @@ +# Runbook: Public TLS Certificate Renewal + +If using Caddy's built-in Let's Encrypt (the default in `infra/docker/platform/Caddyfile.j2`): +**nothing to do** — Caddy renews automatically well before expiry, as long as port 80 stays +reachable for the ACME HTTP-01 challenge (`infra/firewall/rn-platform-01.rules` already allows +it). `infra/scripts/production-smoke-test.sh` checks the live expiry date on every run as an +early warning if auto-renewal ever silently breaks. + +If using Cloudflare origin certs instead (see `docs/PRODUCTION_NETWORK_SECURITY.md` §8 for +that decision): Cloudflare-issued origin certs are typically 15-year validity — this is a +once-per-network-topology-change concern, not a recurring one. Re-issue from the Cloudflare +dashboard (SSL/TLS -> Origin Server) and redeploy the cert/key via the `platform` Ansible role +if it ever needs replacing (domain change, suspected compromise). + +**Manual check:** +```bash +echo | openssl s_client -connect academy.example.com:443 -servername academy.example.com 2>/dev/null \ + | openssl x509 -noout -enddate +``` diff --git a/docs/RUNBOOK_VM_REBUILD.md b/docs/RUNBOOK_VM_REBUILD.md new file mode 100644 index 0000000..cf3eb21 --- /dev/null +++ b/docs/RUNBOOK_VM_REBUILD.md @@ -0,0 +1,31 @@ +# Runbook: Platform VM / Lab Host VM Rebuild + +Both VMs are designed to be disposable and rebuildable from code — nothing irreplaceable +should ever live only on either VM's disk outside of what's backed up (`docs/BACKUP_DR.md`). + +## rn-platform-01 lost + +Academic data survives (it's in Postgres backups, off-host). Instructor/student accounts and +progress are recoverable; anything since the last backup is not (`docs/BACKUP_DR.md`'s RPO). + +```bash +cd infra/terraform && terraform apply # recreates the VM from the template +cd ../ansible && ansible-playbook site.yml -i inventory/production.ini --limit platform --ask-vault-pass +# Then restore Postgres — docs/RUNBOOK_POSTGRES_RESTORE.md, step 5b variant (fresh DB this time, +# no need to rename since it's a fresh instance) +./infra/scripts/production-smoke-test.sh +``` + +## rn-lab-host-01 lost + +**No academic data is lost** — this host only ever holds disposable Lab containers by design +(`docs/BACKUP_DR.md`'s Tier distinction). Every enrolled student's Lab needs re-provisioning +after this, which is a normal "Prepare Labs" click, not a recovery operation. + +```bash +cd infra/terraform && terraform apply +cd ../ansible && ansible-playbook site.yml -i inventory/production.ini --limit lab_host --ask-vault-pass +# Re-generate Docker TLS material for the new host if its identity (IP/DNS) changed — +# docs/RUNBOOK_DOCKER_TLS_ROTATION.md +# From /admin: Prepare Labs again for any cohort that needs it. +``` diff --git a/platform/README.md b/platform/README.md index a271882..e9448f7 100644 --- a/platform/README.md +++ b/platform/README.md @@ -78,3 +78,12 @@ See `provisioner/README.md` (Phase 1), `docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md 2A), `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md` (Phase 3A identity limitations), and `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` §20 (Phase 3B learning/assessment limitations) for the full, honest lists. + +## Running this in production + +`../infra/` has the full production infrastructure package (Ansible, Terraform, firewall, +monitoring, backup, CI/CD) for deploying the Platform API/Gateway/Provisioner/Postgres onto a +dedicated `rn-platform-01` VM, with student Labs on a separate `rn-lab-host-01` VM. See +`../docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` for the overview and +`../docs/PRODUCTION_NETWORK_SECURITY.md` for a known gap in Gateway-to-Lab reachability across +two real VMs that must be closed before production go-live.