docs(phase4): production infrastructure documentation + runbooks
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs (PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR, OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST, and 12 runbooks (first deployment, normal deploy, rollback, Postgres restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure). PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine architectural gap this phase surfaced: the Gateway's existing env.internal_ip mechanism for reaching a Lab container assumes Gateway and Lab share a Docker daemon, which breaks once Labs run on a genuinely separate rn-lab-host-01 VM. The network-policy half is built (a firewall allowance for Gateway->Lab-Host on a published port range); the application half (Provisioner/Gateway address handling) is an explicit, reviewed, NOT-YET-BUILT follow-up flagged there rather than silently decided. Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and platform/README.md with Phase 4 pointers, consistent with how every prior phase updated these. Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
This commit is contained in:
@@ -194,6 +194,11 @@ Production topology and what's built vs. not yet: `docs/LAB_ARCHITECTURE.md`,
|
||||
`docs/PHASE1_PLATFORM_FOUNDATION.md`, `docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md`,
|
||||
`docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md`, `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`.
|
||||
|
||||
The production infrastructure package (Proxmox/Ansible/Terraform/firewall/monitoring/backup/CI)
|
||||
that runs all of this on real hardware lives in `infra/` — see
|
||||
`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`. Built and locally validated now; deployment is
|
||||
pending purchase of the target Hetzner dedicated server.
|
||||
|
||||
---
|
||||
|
||||
## Repository Layout
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
# Backup & Disaster Recovery
|
||||
|
||||
## What's backed up (Tier 1) vs. not
|
||||
|
||||
| | Backed up | Why |
|
||||
|---|---|---|
|
||||
| Postgres data | Yes, daily | The only irreplaceable data — student records, cohort/session state, scenario attempts, assessments, achievements, audit events |
|
||||
| `{{ rootnode_data_root }}` config/secrets (Vault files, env templates) | Yes, via the infra repo + Ansible Vault itself | Small, static, versioned |
|
||||
| Infra code (`infra/`, Ansible, Terraform state) | Yes — it's in git; Terraform state should be backed up alongside Postgres if not using a remote backend | Needed to rebuild either VM from scratch |
|
||||
| Disposable student Labs | **No, intentionally** | Ephemeral by design — a Lab is meant to be destroyed and recreated; backing it up would contradict the whole isolation/disposability model |
|
||||
| Prometheus/Loki metrics & log history | No | Operational telemetry, not source-of-truth data; losing 30 days of dashboards on a rebuild is an inconvenience, not data loss |
|
||||
|
||||
## Backup mechanism
|
||||
|
||||
`infra/backup/backup-postgres.sh` (deployed to `/opt/rootnode/backup/` by the `backup` Ansible
|
||||
role, run daily at 03:15 via cron):
|
||||
|
||||
1. `pg_dump --format=plain --no-owner --no-privileges | gzip -9`
|
||||
2. Sanity check: reject if the compressed dump is under 1KB (almost certainly means Postgres
|
||||
was unreachable or empty, not a real backup).
|
||||
3. `gpg --symmetric --cipher-algo AES256` — encrypt, then delete the plaintext dump immediately.
|
||||
4. Optional off-host copy via `rsync -az` over SSH — works for Hetzner Storage Box or any SSH
|
||||
target; that's the extension point for S3-compatible alternatives (swap the rsync line for
|
||||
`aws s3 cp`/`rclone` if a different target is chosen later).
|
||||
5. Retention pruning (`find ... -mtime +{{ backup_retention_days }} -delete`, default 30 days).
|
||||
6. Writes Prometheus textfile-collector metrics
|
||||
(`rootnode_backup_last_run_success`, `..._last_success_timestamp_seconds`, `..._size_bytes`)
|
||||
atomically (tmp file + rename) so a half-written metric file is never scraped.
|
||||
|
||||
## Restore
|
||||
|
||||
`infra/backup/restore-postgres.sh` — requires an explicit target DB URL (never infers or
|
||||
defaults to production, to prevent an accidental overwrite), shows the target's current tables,
|
||||
requires typing `RESTORE` to confirm, then decrypts and pipes into `psql`.
|
||||
`infra/backup/verify-restore.sh` — confirms all 8 expected tables exist plus the
|
||||
`schema_migrations` row count, so a "successful" restore that silently missed a table is caught.
|
||||
|
||||
**Genuinely tested**, not just written: `infra/backup/test-backup-restore.sh` runs the full
|
||||
cycle against two disposable local Postgres containers, with the actual production migrations
|
||||
applied (all 17 — 3 Provisioner + 14 Platform API, in the correct cross-service order), a
|
||||
sample row inserted, backed up, restored into the second container, and verified byte-for-byte.
|
||||
Confirmed passing, exit 0.
|
||||
|
||||
## Disaster recovery scenarios
|
||||
|
||||
| Scenario | RPO | RTO (estimate) | What happens |
|
||||
|---|---|---|---|
|
||||
| Platform VM lost, disks intact | 0 | ~30 min | `docs/RUNBOOK_VM_REBUILD.md` — Ansible re-run against the same disks |
|
||||
| Platform VM + its disks lost | Up to 24h (since last backup) | ~1-2h | Rebuild VM (Terraform), re-run Ansible, restore Postgres from the latest off-host backup (`docs/RUNBOOK_POSTGRES_RESTORE.md`) |
|
||||
| Lab Host VM lost, any reason | 0 (intentional) | ~30 min | Rebuild VM, re-run Ansible — no data to restore, all Labs were disposable and get recreated on next student login |
|
||||
| Single disk failure in a mirror | 0 | 0 (no outage) | `docs/RUNBOOK_DISK_FAILURE.md` — pool stays online DEGRADED, replace and resilver |
|
||||
| Whole storage pool lost (both mirror sides) | Up to 24h | ~1-2h | Same as "Platform VM + disks lost" above — this is precisely the scenario the off-host copy exists for; a local-only backup on the same failed pool is useless here |
|
||||
| Encryption passphrase lost | N/A | Unrecoverable | The single most important secret to protect and back up separately (e.g., in the same secrets manager as other Vault secrets, not only in `production.vault.yml`) — losing it makes every existing backup file permanently unreadable |
|
||||
|
||||
RPO is bounded by backup cadence (daily) — a mid-day incident can lose up to a day of student
|
||||
activity. If that's ever unacceptable, the fix is a shorter cron interval or WAL archiving for
|
||||
point-in-time recovery, not decided or built here since daily has not yet been proven
|
||||
insufficient against real usage.
|
||||
|
||||
## What backup does NOT cover
|
||||
|
||||
Application code and container images — those live in Gitea and the image registry, both
|
||||
already durable and versioned independently of this backup mechanism. TLS certificates — those
|
||||
are regenerable (`docs/RUNBOOK_DOCKER_TLS_ROTATION.md`, `docs/RUNBOOK_TLS_RENEWAL.md`), not
|
||||
backed up.
|
||||
@@ -0,0 +1,59 @@
|
||||
# Capacity Planning
|
||||
|
||||
How many concurrent student Labs `rn-lab-host-01` can actually run — determined from measured
|
||||
data after deployment, not guessed or marketed now.
|
||||
|
||||
## The model
|
||||
|
||||
`infra/scripts/capacity-model.js`:
|
||||
|
||||
```
|
||||
usable = hostTotal * (1 - overheadPct/100) - platformReserved
|
||||
withMargin = usable * (1 - marginPct/100)
|
||||
perLabLimit = floor(withMargin / perLabUsage)
|
||||
recommended = min(cpuBoundLimit, memoryBoundLimit) # whichever runs out first
|
||||
warning = floor(recommended * 0.8)
|
||||
hardAdmission = recommended
|
||||
```
|
||||
|
||||
Computed once for CPU, once for memory; the tighter (smaller) of the two constrains the actual
|
||||
recommendation — a host can be memory-starved long before its CPU threads are exhausted, or
|
||||
vice versa, and the model doesn't pretend otherwise.
|
||||
|
||||
**Inputs** (`infra/environments/production/capacity.yml.example` — explicitly placeholder data
|
||||
until real numbers exist):
|
||||
- `host_cpu_threads`, `host_memory_mb` — from the real Lab Host VM, once sized
|
||||
- `overhead_pct` — Proxmox/hypervisor + OS overhead reserved off the top
|
||||
- `platform_reserved_cpu`/`platform_reserved_memory_mb` — headroom for the egress proxy,
|
||||
node-exporter, cadvisor, promtail themselves (they run on the Lab Host too)
|
||||
- `per_lab_cpu`/`per_lab_memory_mb` — **measured**, from `infra/scripts/loadtest/`, not assumed
|
||||
- `safety_margin_pct` — extra headroom below the raw math, for burst/variance
|
||||
|
||||
**Outputs**: recommended concurrent Labs, an 80%-of-recommended warning threshold (for
|
||||
alerting — `LabHostHighMemory` in `docs/OBSERVABILITY.md`), and a hard admission-control
|
||||
threshold the Provisioner's resource policy should enforce so it refuses to boot a Lab past that
|
||||
point rather than degrading every running Lab.
|
||||
|
||||
## Load-test tooling that produces the measured inputs
|
||||
|
||||
`infra/scripts/loadtest/`:
|
||||
- `boot-labs.js` — reuses `platform/test/session-helper.js` (built in Phase 3A) to drive the
|
||||
real admin Prepare-Labs endpoint, booting N Labs and recording boot time/failure rate per Lab
|
||||
into a JSON report.
|
||||
- `sample-resources.sh` — samples real `docker stats` for every `rootnode-lab-*` container every
|
||||
5 seconds to CSV (timestamp, name, CPU%, mem usage, mem%, net I/O, block I/O, PIDs).
|
||||
|
||||
Together these produce the `per_lab_cpu`/`per_lab_memory_mb` this model needs, plus data on how
|
||||
boot time and failure rate change as concurrency rises — the dimensions this phase was told to
|
||||
cover: single Lab boot, multiple simultaneous boots, steady-state active Labs, terminal
|
||||
WebSocket load, scenario validation load, Gateway throughput, API request volume, Postgres
|
||||
connection activity.
|
||||
|
||||
## What this document deliberately does not claim
|
||||
|
||||
No concurrency number in this repo (`capacity.yml.example`'s numbers included) is a real,
|
||||
validated figure — every one is a placeholder until run against the real Lab Host VM with real
|
||||
per-Lab measurements. Treating a placeholder as a marketing/sales number would be actively
|
||||
misleading; the whole point of the model is that the real number gets filled in once, from
|
||||
measurement, after deployment (`docs/RUNBOOK_LAB_HOST_RESOURCE_PRESSURE.md` step 4 is the
|
||||
trigger to re-measure and update it if the per-Lab footprint later changes).
|
||||
@@ -280,3 +280,28 @@ Full design: `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`.
|
||||
- **Known limitations:** see the Phase 3B doc §20 — no automatic abandoned-attempt detection, no
|
||||
per-cohort needs-attention threshold configuration, raw-JSON pool_spec authoring in the admin
|
||||
UI (correct, not yet a guided form).
|
||||
|
||||
## 13. Production Infrastructure (Phase 4)
|
||||
|
||||
Full design: `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`, plus detail docs
|
||||
`PRODUCTION_DEPLOYMENT.md`, `PRODUCTION_NETWORK_SECURITY.md`, `BACKUP_DR.md`,
|
||||
`OBSERVABILITY.md`, `CAPACITY_PLANNING.md`, `PRODUCTION_VALIDATION_CHECKLIST.md`, and 12
|
||||
`RUNBOOK_*.md` files, all under `docs/`. Everything lives in `infra/` (Ansible, Terraform,
|
||||
Compose, firewall, monitoring, backup, CI, load-test tooling).
|
||||
|
||||
- **What it is:** the infrastructure-as-code needed to run rn-platform-01/rn-lab-host-01 on a
|
||||
real Hetzner/Proxmox host — buildable and locally testable now, even though the dedicated
|
||||
server hasn't been purchased. Status is explicitly **PHASE 4 BUILD COMPLETE / PRODUCTION
|
||||
DEPLOYMENT PENDING HARDWARE**, not "deployed."
|
||||
- **Where the boundary held:** the Platform VM / Lab Host VM security split from
|
||||
`docs/LAB_ARCHITECTURE.md` is codified in firewall rulesets and mutual-TLS Docker config, not
|
||||
weakened anywhere.
|
||||
- **Known gap surfaced, not silently fixed:** the Gateway's Lab-reachability mechanism doesn't
|
||||
survive the move to two real separate VMs — see `docs/PRODUCTION_NETWORK_SECURITY.md`'s
|
||||
"Known gap" section for what's built (firewall policy) vs. what's an explicit, reviewed,
|
||||
not-yet-built application follow-up (Provisioner/Gateway address handling).
|
||||
- **Validated without the hardware:** real `tofu validate`, `ansible-playbook --syntax-check`,
|
||||
`docker compose config`, `caddy validate`, `promtool`/`amtool`, and a genuine end-to-end
|
||||
backup→encrypt→restore→verify cycle against disposable local Postgres with the real
|
||||
production migrations applied. None of this proves the real Hetzner network boundary — that
|
||||
needs `docs/PRODUCTION_VALIDATION_CHECKLIST.md` walked against the real server.
|
||||
|
||||
@@ -486,3 +486,13 @@ All four open questions this doc originally ended with are resolved — see the
|
||||
at the top and §2–§5 above. Next: `docs/PHASE1_PLATFORM_FOUNDATION.md` — the concrete build
|
||||
plan for the platform foundation (Postgres environment registry, provisioner skeleton,
|
||||
networking, egress, resource policy, reaper) against these decisions.
|
||||
|
||||
## Phase 4: this boundary in production
|
||||
|
||||
Phase 4 (`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`) codifies the Platform/Lab-Host split this
|
||||
document designed as two real, separately-firewalled Proxmox VMs (`rn-platform-01`,
|
||||
`rn-lab-host-01`) connected only by mutual-TLS Docker and a tightly scoped firewall allowance —
|
||||
see `docs/PRODUCTION_NETWORK_SECURITY.md`. That document also records a gap Phase 4 surfaced but
|
||||
did not resolve: the Gateway's `env.internal_ip` mechanism for reaching a Lab container assumes
|
||||
Gateway and Lab share a Docker daemon, which breaks once they're genuinely on separate VMs. Read
|
||||
that document's "Known gap" section before treating two-VM production as fully wired.
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# Observability
|
||||
|
||||
Self-hosted Prometheus + Grafana + Alertmanager + Loki/Promtail, deployed via
|
||||
`infra/docker/platform/docker-compose.production.yml` on rn-platform-01, with lightweight agents
|
||||
(node-exporter, cadvisor, promtail) also on rn-lab-host-01. No paid SaaS.
|
||||
|
||||
This is operational observability — host/container/database health, uptime, resource pressure.
|
||||
It is deliberately separate from the application's own instructor learning analytics
|
||||
(Phase 3B's cohort/scenario/student dashboards) — different audience (ops vs. instructors),
|
||||
different data (infrastructure vs. learning outcomes), and mixing them would make both worse.
|
||||
|
||||
## What's real today
|
||||
|
||||
- **Host metrics** (both VMs): node-exporter — CPU, memory, disk, network.
|
||||
- **Container metrics** (Lab Host): cadvisor — per-container CPU/memory/network, including
|
||||
student Lab containers (`rootnode-lab-*`).
|
||||
- **Postgres metrics**: postgres-exporter — connections, transaction rate, table sizes.
|
||||
- **Uptime/liveness**: blackbox-exporter probing `/healthz` on the Gateway
|
||||
(`platform/gateway/src/server.js`) and the Platform API (`platform/api/src/server.js`), both
|
||||
added this phase specifically for this purpose, plus the Provisioner's pre-existing `/health`.
|
||||
- **Logs**: Loki + Promtail, scraping Docker container logs and journald from both VMs, with a
|
||||
redaction pipeline stage that strips `password|token|secret=`-pattern values before they're
|
||||
stored (`infra/monitoring/promtail/`).
|
||||
- **Dashboards**: Grafana, reachable only via `https://<origin>/grafana/` and only from
|
||||
`management_cidr` (see `docs/PRODUCTION_NETWORK_SECURITY.md`). One starter dashboard,
|
||||
`infra/monitoring/grafana/provisioning/dashboards/rootnode-overview.json` — 8 panels (Platform
|
||||
+ Lab Host CPU/memory/disk, Postgres up/connections, running-Lab-count via cadvisor label
|
||||
match [best-effort, caveated], uptime probes).
|
||||
|
||||
## What's NOT built yet (honest accounting, not silently assumed)
|
||||
|
||||
Application-level metrics — active student count, Lab provisioning success/failure rate,
|
||||
Gateway request latency/error rate, per-endpoint API metrics — would need real instrumentation
|
||||
added to the Platform API/Gateway/Provisioner (e.g. a `/metrics` endpoint via `prom-client`).
|
||||
`infra/monitoring/prometheus/prometheus.yml.j2` has a commented-out scrape job as the landing
|
||||
spot for this, explicitly marked as a follow-up, not wired to anything real yet.
|
||||
|
||||
## Alerting
|
||||
|
||||
`infra/monitoring/prometheus/rules/rootnode.yml` — two severities:
|
||||
|
||||
**Critical** (fires fast, pages): PlatformAPIDown, GatewayDown, PostgresDown, LabHostDown,
|
||||
BackupFailed, DiskCriticallyFull.
|
||||
|
||||
**Warning** (informational, slower repeat): LabHostHighMemory, HighDiskUsage, BackupAging.
|
||||
(RepeatedProvisioningFailures and ExcessiveFailedLogins are present as inert placeholders —
|
||||
they need the not-yet-built application metrics above before they can fire on anything real.)
|
||||
|
||||
Never alerts on an individual student's scenario failure or a single Lab crash — those are
|
||||
expected, routine outcomes of normal usage, not operational incidents.
|
||||
|
||||
`infra/monitoring/alertmanager/alertmanager.yml.j2` — critical alerts route separately with a
|
||||
30-minute repeat interval, everything else at 4 hours, one webhook receiver
|
||||
(`vault_alertmanager_webhook_url` — points at whatever channel gets configured at deploy time,
|
||||
e.g. a Slack/Discord/Matrix webhook), and one inhibit rule so a LabHostDown critical suppresses
|
||||
its own dependent warnings instead of paging twice for the same root cause.
|
||||
|
||||
## Validation performed (local, real tooling — not hardware-dependent)
|
||||
|
||||
`infra/monitoring/validate-monitoring.sh` wraps: `promtool check config` (Prometheus config),
|
||||
`promtool check rules` (alert rules — 11 rules found, valid), `amtool check-config`
|
||||
(Alertmanager — 1 inhibit rule, 2 receivers, valid). These prove syntactic/semantic correctness
|
||||
of the configs, not that the real network paths between rn-platform-01 and rn-lab-host-01 work —
|
||||
that requires the real hardware and is covered by `docs/PRODUCTION_VALIDATION_CHECKLIST.md`.
|
||||
@@ -0,0 +1,184 @@
|
||||
# Phase 4 — Production Infrastructure Package
|
||||
|
||||
**Status: PHASE 4 BUILD COMPLETE. PRODUCTION DEPLOYMENT PENDING HARDWARE.**
|
||||
|
||||
The dedicated Hetzner server this is designed for hasn't been purchased yet. Everything in this
|
||||
document and the `infra/` tree it describes is buildable, testable (against local
|
||||
Docker/Postgres/disposable certs), and reviewable **now** — none of it requires the real
|
||||
hardware to exist. What genuinely can't happen until the hardware exists is called out
|
||||
explicitly throughout, never silently assumed to already work.
|
||||
|
||||
This document is the map. Detail docs: `docs/PRODUCTION_DEPLOYMENT.md` (step-by-step),
|
||||
`docs/PRODUCTION_NETWORK_SECURITY.md` (network/firewall/TLS design), `docs/BACKUP_DR.md`
|
||||
(backup/restore/disaster recovery), `docs/OBSERVABILITY.md` (metrics/logs/alerts),
|
||||
`docs/CAPACITY_PLANNING.md` (sizing model), and the `docs/RUNBOOK_*.md` files (14 topics,
|
||||
command-oriented).
|
||||
|
||||
## 1. What was built
|
||||
|
||||
```
|
||||
infra/
|
||||
├── proxmox/ manual bare-metal install docs + cloud-init template definition
|
||||
├── terraform/ VM creation (rn-platform-01, rn-lab-host-01) — real `tofu validate` pass
|
||||
├── ansible/ all host configuration — real `ansible-playbook --syntax-check` pass
|
||||
│ ├── roles/ common, firewall, docker-tls, platform, lab-host, monitoring-agent, backup
|
||||
│ ├── group_vars/ production.yml, staging.yml, vault template
|
||||
│ └── site.yml top-level playbook, correct role ordering
|
||||
├── docker/ production Compose files, Caddyfile, Postgres tuning — real `docker
|
||||
│ compose config` + `caddy validate` pass
|
||||
├── firewall/ iptables-restore rulesets for all 3 hosts — structural lint pass
|
||||
├── monitoring/ Prometheus/Alertmanager/Loki/Promtail/Grafana config — real `promtool`
|
||||
│ + `amtool` validation pass
|
||||
├── backup/ Postgres backup/restore/verify scripts — REAL end-to-end test against
|
||||
│ disposable local Postgres, including the actual production migrations
|
||||
├── scripts/ deploy/rollback/smoke-test/CI/loadtest/capacity-model tooling
|
||||
├── ci/ Gitea Actions workflow (build/test/stage automatic, prod manual)
|
||||
└── environments/ dev/staging/production config templates, no secrets committed
|
||||
```
|
||||
|
||||
18 new docs (this file + 5 detail docs + 12 runbooks), 4 updated (`LAB_ARCHITECTURE.md`,
|
||||
`ENGINEERING_SOP.md`, `README.md`, `platform/README.md`).
|
||||
|
||||
## 2. Architecture (unchanged from the design, now codified)
|
||||
|
||||
```
|
||||
Hetzner Dedicated Server
|
||||
└── Proxmox VE
|
||||
├── rn-platform-01 (frontend, Platform API, Gateway, Provisioner, Postgres,
|
||||
│ Prometheus/Alertmanager/Grafana/Loki, backup cron, Caddy)
|
||||
└── rn-lab-host-01 (Docker Engine w/ TLS, egress proxy, disposable student Labs,
|
||||
node-exporter/cadvisor/promtail)
|
||||
```
|
||||
|
||||
The Platform VM / Lab Host VM security boundary from `docs/LAB_ARCHITECTURE.md` is the one
|
||||
thing this phase was explicitly told never to weaken — it's codified, not just described, in
|
||||
`infra/firewall/` and `infra/ansible/roles/docker-tls/`.
|
||||
|
||||
## 3. VM specifications
|
||||
|
||||
| | rn-platform-01 | rn-lab-host-01 |
|
||||
|---|---|---|
|
||||
| Baseline (default in `infra/terraform/variables.tf`) | 4 cores, 16 GB RAM, 100 GB disk | 24 cores, 96 GB RAM, 500 GB disk |
|
||||
| Recommended (once real host is ~16c/32t/128GB) | same — Postgres/API/Gateway/Provisioner don't need much | majority of remaining host resources |
|
||||
| Max-growth | scale RAM with Postgres `shared_buffers` if instructor/cohort count grows a lot | scale with measured concurrent-Lab capacity (`docs/CAPACITY_PLANNING.md`) |
|
||||
| Configurable via | `terraform.tfvars`' `platform_vm`/`lab_host_vm` objects | same |
|
||||
|
||||
NUMA/CPU pinning: not configured — not useful at this scale (a handful of VMs on one physical
|
||||
host, not a dozen contending for cache locality). Revisit only if real load testing shows a
|
||||
reason to.
|
||||
|
||||
## 4. Storage decision framework
|
||||
|
||||
Not a final answer — `infra/environments/production/provisioner-config/README.md` and this
|
||||
section exist so the final ZFS layout is a deployment-time decision made against real disks,
|
||||
not guessed now. Candidate layouts (`infra/terraform/variables.tf`'s `vm_disk_datastore` names
|
||||
whichever one gets chosen):
|
||||
|
||||
- **4× NVMe**: mirrored vdevs (RAID10-equivalent) — two mirror pairs, one for
|
||||
Postgres/platform data (I/O-sensitive, small), one for Lab Host disposable data
|
||||
(throughput-sensitive, large, never needs redundancy since nothing on it is precious).
|
||||
- **2× SSD/NVMe**: one mirrored vdev, logical dataset separation (ZFS datasets, not separate
|
||||
pools) between platform (`{{ rootnode_data_root }}/postgres`, `/backups`) and Lab Host data.
|
||||
- **Mixed SSD/HDD**: only if the purchased server's actual disks require it — HDD for
|
||||
disposable Lab scratch space is acceptable (it's disposable), never for Postgres.
|
||||
|
||||
**Never single-disk for Postgres data**, in any layout — a mirror is the floor, not a nice-to-have.
|
||||
|
||||
Pre-deployment disk inspection checklist (model, size, SMART, power-on hours, wear/endurance,
|
||||
sector errors, temperature, firmware): `docs/RUNBOOK_DISK_FAILURE.md`'s inspection section —
|
||||
run once, against the real drives, before committing to a layout.
|
||||
|
||||
## 5-8. Network, firewall, Docker TLS, public ingress
|
||||
|
||||
Full detail: `docs/PRODUCTION_NETWORK_SECURITY.md`. Summary: two private VM networks, Docker TLS
|
||||
(mutual, already application-supported since Phase 1 — this phase only needed the certificate
|
||||
generation/deployment automation, not new app code), a single public origin (Caddy, automatic
|
||||
HTTPS) proxying only to the Gateway, and one genuinely new architectural item this phase
|
||||
surfaced and addressed at the network-policy layer: **Gateway-to-Lab reachability across two
|
||||
real VMs** — see that doc's dedicated section for what's built vs. what still needs a small,
|
||||
reviewed Provisioner/Gateway code change before real go-live.
|
||||
|
||||
## 9-10. Production auth/cookie config, secrets
|
||||
|
||||
`infra/environments/production/*.env.template` set `COOKIE_SECURE=1`/`TRUST_PROXY=1` explicitly
|
||||
(dev's unsafe defaults are never inherited). Secrets: Ansible Vault (already bundled with
|
||||
Ansible — no new tool to install/trust for this project's secret surface), template at
|
||||
`infra/ansible/group_vars/production.vault.yml.template`. Nothing is committed —
|
||||
`infra/.gitignore` covers `*.env`, TLS private key material, vault plaintext, Terraform state.
|
||||
|
||||
## 11-13. Postgres, backup, DR
|
||||
|
||||
Full detail: `docs/BACKUP_DR.md`. Summary: conservative tuning baseline (not hand-tuned before
|
||||
real RAM is known), daily encrypted backups with an off-host adapter point (Storage Box/S3/any
|
||||
SSH target), and a **real, tested** backup→encrypt→restore→verify cycle
|
||||
(`infra/backup/test-backup-restore.sh`) — run against disposable local Postgres with the actual
|
||||
production migrations applied, not a mock.
|
||||
|
||||
## 14-17. Observability, logging, alerting, uptime
|
||||
|
||||
Full detail: `docs/OBSERVABILITY.md`. Prometheus + Grafana + Loki/Promtail (the user's stated
|
||||
Prometheus/Grafana preference, honored) + Alertmanager, all self-hosted, no paid dependency.
|
||||
Deliberately does not duplicate the application's own instructor learning analytics
|
||||
(Phase 3B) — operational observability and learning analytics stay separate concerns with
|
||||
separate audiences, per the explicit instruction. Application-level metrics (active
|
||||
students/Labs, provisioning failures, Gateway error rate) are a documented, **not-yet-built**
|
||||
follow-up — see that doc's honest accounting of what's real today (host/container/Postgres
|
||||
metrics, HTTP healthz probes) vs. what needs app instrumentation later.
|
||||
|
||||
## 18-19. CI/CD, image strategy
|
||||
|
||||
`infra/ci/gitea-workflows/build-and-test.yml` — automatic test/build/tag/staging-deploy,
|
||||
**manual** production promotion (`infra/scripts/deploy-production.sh`), matching the stated
|
||||
preference. SHA-tagged images always, semantic tags on deliberate releases, never `:latest` in
|
||||
staging/production. Full detail: `infra/docker/IMAGE_STRATEGY.md`.
|
||||
|
||||
## 20-21. Ansible, Terraform
|
||||
|
||||
`infra/ansible/site.yml` — idempotent roles, doesn't assume the VMs already exist (Terraform
|
||||
creates them first). Real `ansible-playbook --syntax-check` pass. `infra/terraform/` — real
|
||||
`tofu validate` pass against the actual `bpg/proxmox` provider schema. Neither installs Proxmox
|
||||
itself or builds the base VM template — both stay deliberate manual, documented, one-time steps
|
||||
(`infra/proxmox/README.md`), per the explicit "don't over-automate the fragile one-time step"
|
||||
instruction.
|
||||
|
||||
## 22-23. Environment config, staging
|
||||
|
||||
`infra/environments/{dev,staging,production}/` — explicit separation, production never
|
||||
inherits dev defaults. Staging is designed to run in the homelab or a resource-constrained
|
||||
slice of the real server later — no second dedicated server required.
|
||||
|
||||
## 24-25. Load testing, capacity model
|
||||
|
||||
`infra/scripts/loadtest/` (real Prepare-Labs-driven boot test + `docker stats` sampling) and
|
||||
`infra/scripts/capacity-model.js` (a real, working worksheet — tested against example inputs,
|
||||
produces CPU-vs-memory-bound recommended/warning/hard-admission-control thresholds). No
|
||||
concurrency number is hard-coded as a marketing figure; the example config in
|
||||
`infra/environments/production/capacity.yml.example` is explicitly flagged as placeholder data.
|
||||
|
||||
## 26-27. Validation checklist, runbooks
|
||||
|
||||
`docs/PRODUCTION_VALIDATION_CHECKLIST.md` — one master checklist spanning hardware through
|
||||
academy-workflow verification. 12 runbooks (`docs/RUNBOOK_*.md`) — first deployment, normal
|
||||
deploy, rollback, Postgres restore, VM rebuild (both hosts), Docker TLS rotation, TLS renewal,
|
||||
owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host
|
||||
resource pressure.
|
||||
|
||||
## What this phase deliberately did NOT touch
|
||||
|
||||
Learning system design, curriculum content, RBAC model, session architecture — all Phase 3A/3B,
|
||||
unchanged. No parent portal, billing, held scenarios, or production OIDC provider integration.
|
||||
RootNode production infrastructure is dedicated to RootNode Academy — never folded into
|
||||
existing Synthos production servers, and canonical source stays Gitea, never GitHub `kubekosh`.
|
||||
|
||||
## Known gap surfaced during this phase (not silently papered over)
|
||||
|
||||
Building the real production Compose/network config surfaced that the Gateway's existing
|
||||
mechanism for reaching a Lab container (`env.internal_ip`, a Docker-bridge-local address) only
|
||||
works when Gateway and Lab share a Docker daemon — true in every dev/single-host setup so far,
|
||||
**not** true once Labs run on a genuinely separate `rn-lab-host-01`. The network-policy half is
|
||||
built (a firewall allowance for Gateway→Lab-Host on a published port range,
|
||||
`infra/firewall/rn-lab-host-01.rules`); the application half (the Provisioner recording a
|
||||
reachable external address for each Lab, and the Gateway using it) is an explicit, reviewed,
|
||||
NOT-YET-BUILT follow-up — see `docs/PRODUCTION_NETWORK_SECURITY.md` for the two candidate
|
||||
designs and why neither was silently chosen here. This must land before real two-VM go-live;
|
||||
flagging it clearly was judged better than guessing at an architecture change mid-infra-phase.
|
||||
@@ -0,0 +1,84 @@
|
||||
# Production Deployment
|
||||
|
||||
Step-by-step, for when the real Hetzner host exists. See
|
||||
`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` for the architecture this executes.
|
||||
|
||||
## 0. One-time, manual, not automated (deliberately)
|
||||
|
||||
1. Install Proxmox VE on the bare-metal host — `infra/proxmox/README.md`.
|
||||
2. Inspect the real disks, pick a ZFS layout — `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4,
|
||||
`docs/RUNBOOK_DISK_FAILURE.md`'s inspection checklist.
|
||||
3. Build the cloud-init VM template Terraform will clone —
|
||||
`infra/proxmox/cloud-init/user-data.yaml` + `infra/proxmox/README.md`.
|
||||
4. Fill in `infra/terraform/terraform.tfvars` (copy from `.tfvars.example`) with the real
|
||||
Proxmox API endpoint, node name, storage/datastore names, and chosen VM sizes.
|
||||
5. Fill in `infra/ansible/inventory/production.ini` with the real VM IPs, and
|
||||
`infra/ansible/group_vars/production.yml` with `platform_ip`, `lab_host_ip`,
|
||||
`management_cidr`, `proxmox_mgmt_cidr`.
|
||||
6. Copy `infra/ansible/group_vars/production.vault.yml.template` →
|
||||
`production.vault.yml`, fill in every `vault_*` secret with real generated values, then
|
||||
`ansible-vault encrypt group_vars/production.vault.yml`. Never commit the plaintext.
|
||||
|
||||
## 1. Create the VMs
|
||||
|
||||
```bash
|
||||
cd infra/terraform
|
||||
tofu init
|
||||
tofu plan # review before applying against real hardware
|
||||
tofu apply
|
||||
```
|
||||
|
||||
Creates `rn-platform-01` and `rn-lab-host-01` from the cloud-init template. Confirm both boot
|
||||
and are SSH-reachable from the management network before continuing.
|
||||
|
||||
## 2. Generate Docker TLS certificates (offline, once)
|
||||
|
||||
```bash
|
||||
export ROOTNODE_TLS_BUNDLE_DIR=/secure/offline/path # never inside the repo
|
||||
./infra/scripts/docker-tls/generate-docker-tls.sh
|
||||
./infra/scripts/docker-tls/verify-docker-tls.sh
|
||||
```
|
||||
|
||||
The CA private key stays in `ROOTNODE_TLS_BUNDLE_DIR` and never touches either VM — only the CA
|
||||
cert (public) and the relevant leaf cert+key go to each host.
|
||||
|
||||
## 3. Configure both hosts
|
||||
|
||||
```bash
|
||||
cd infra/ansible
|
||||
ansible-playbook -i inventory/production.ini site.yml --ask-vault-pass
|
||||
```
|
||||
|
||||
Role order matters and is fixed in `site.yml`: `common` → `firewall` → `docker-tls` →
|
||||
`platform`/`lab-host` → `monitoring-agent` → `backup`. Firewall and TLS land before anything
|
||||
that depends on them.
|
||||
|
||||
## 4. First application deploy
|
||||
|
||||
```bash
|
||||
./infra/scripts/deploy-production.sh <image-tag>
|
||||
```
|
||||
|
||||
Pulls the tagged images, runs the compose stack, applies Postgres migrations (automatic, on
|
||||
Platform API/Provisioner startup — see `docs/RUNBOOK_FAILED_MIGRATION.md` if one fails).
|
||||
|
||||
## 5. Verify
|
||||
|
||||
```bash
|
||||
./infra/scripts/production-smoke-test.sh
|
||||
```
|
||||
|
||||
Curls `/healthz` on the Gateway and Platform API, confirms Postgres is reachable, confirms
|
||||
Prometheus/Grafana/Alertmanager are up. Then walk
|
||||
`docs/PRODUCTION_VALIDATION_CHECKLIST.md` in full before calling the deployment live.
|
||||
|
||||
## Normal (non-first) deploys
|
||||
|
||||
`docs/RUNBOOK_NORMAL_DEPLOY.md`. Rollback: `docs/RUNBOOK_ROLLBACK.md`.
|
||||
|
||||
## What this document does NOT cover
|
||||
|
||||
DNS/Cloudflare setup for the public origin (a one-time manual step outside this repo's scope —
|
||||
point the A/AAAA record at the real host and confirm Caddy issues its certificate). Which
|
||||
storage layout to pick (decision framework only, §4 of the master doc — the real answer depends
|
||||
on the real disks).
|
||||
@@ -0,0 +1,93 @@
|
||||
# Production Network & Security
|
||||
|
||||
The Platform VM / Lab Host VM boundary is the core production control. This document is where
|
||||
every network decision that touches it gets written down. Nothing here weakens it.
|
||||
|
||||
## Topology
|
||||
|
||||
```
|
||||
Internet ──HTTPS(443)──▶ Caddy (rn-platform-01) ──▶ Gateway (internal bridge)
|
||||
│
|
||||
management_cidr ──SSH──▶ rn-platform-01 │
|
||||
management_cidr ──SSH──▶ rn-lab-host-01 │
|
||||
proxmox_mgmt_cidr ──22/8006──▶ Proxmox host │
|
||||
│
|
||||
rn-platform-01 (Provisioner) ──TLS:2376──▶ rn-lab-host-01 (dockerd)
|
||||
rn-lab-host-01 (promtail) ──3100──▶ rn-platform-01 (Loki)
|
||||
rn-lab-host-01 (node-exporter/cadvisor) ◀──scrape── rn-platform-01 (Prometheus)
|
||||
```
|
||||
|
||||
Firewall source of truth: `infra/firewall/{proxmox-host,rn-platform-01,rn-lab-host-01}.rules`
|
||||
(iptables-restore format, deployed by the `firewall` Ansible role). Validate with
|
||||
`infra/firewall/validate-firewall.sh`.
|
||||
|
||||
## What is explicitly denied
|
||||
|
||||
- Student Lab → another student Lab (no student-to-student traffic, ever).
|
||||
- Student Lab → Proxmox management (both the host firewall and the `lab_subnet_range` DENY
|
||||
rule in `rn-lab-host-01.rules`).
|
||||
- Student Lab → Platform Postgres / Platform API / Gateway internals (logged DENY in
|
||||
`rn-platform-01.rules` for anything from `lab_host_ip` to ports 5432/4100).
|
||||
- rn-lab-host-01 → Postgres (same rule — the Lab Host itself has no legitimate reason to reach
|
||||
Postgres; only the Provisioner on rn-platform-01 does, and it talks to Postgres over the
|
||||
internal Docker bridge, never across the VM boundary).
|
||||
- Any student egress except through the approved egress proxy (enforced by the Provisioner's
|
||||
existing per-Lab `DOCKER-USER` chain manipulation from Phase 1 — unchanged by this phase).
|
||||
- Direct public exposure of the Provisioner, Postgres, Docker TLS port, Proxmox UI, or Grafana
|
||||
(Grafana is reachable only via Caddy's `/grafana/*` path, itself restricted to
|
||||
`management_cidr` by a `caddy` matcher — see `infra/docker/platform/Caddyfile.j2`).
|
||||
|
||||
## Docker TLS (Provisioner ↔ Lab Host)
|
||||
|
||||
Mutual TLS on port 2376, restricted at the firewall to `platform_ip` only. Application support
|
||||
already existed since Phase 1 (`platform/provisioner/src/docker.js`'s `buildClient()`); this
|
||||
phase added the operational half: `infra/scripts/docker-tls/generate-docker-tls.sh` (CA +
|
||||
server + client cert generation), the `docker-tls` Ansible role (deploys certs, configures
|
||||
`dockerd`'s `daemon.json`), and `docs/RUNBOOK_DOCKER_TLS_ROTATION.md` for renewal. The CA
|
||||
private key is generated offline and never committed or deployed to either VM — only the CA
|
||||
cert (to verify the peer) and each host's own leaf cert+key.
|
||||
|
||||
## Public ingress
|
||||
|
||||
One origin, Caddy, automatic HTTPS. `reverse_proxy gateway:4200` — Caddy talks only to the
|
||||
Gateway; nothing else is proxied. WebSocket upgrade works automatically (Caddy's default
|
||||
`reverse_proxy` behavior, no special config needed for the terminal WebSocket path). Whether to
|
||||
front this with Cloudflare or expose the origin directly is left as a deployment-time decision
|
||||
(both work with Caddy's ACME flow; Cloudflare adds DDoS absorption at the cost of Cloudflare
|
||||
being able to see plaintext unless "Full (strict)" mode is used) — not decided here since it
|
||||
doesn't affect anything else in this document.
|
||||
|
||||
## Known gap: Gateway-to-Lab reachability across two real VMs
|
||||
|
||||
**This is the most important item in this document.**
|
||||
|
||||
The Gateway currently reaches a running Lab container via `env.internal_ip` — a Docker
|
||||
bridge-local address recorded by the Provisioner when it creates the container
|
||||
(`platform/gateway/src/server.js`, `platform/provisioner`, built in Phase 2A). This works when
|
||||
Gateway and the Lab share a Docker daemon, which has been true in every setup so far, including
|
||||
dev and any single-host deployment. **It stops being true once Labs run on a genuinely separate
|
||||
`rn-lab-host-01`** — a Docker bridge-internal IP on one host is not routable from a different
|
||||
host by default.
|
||||
|
||||
What's built now (network-policy half only): a firewall allowance in
|
||||
`infra/firewall/rn-lab-host-01.rules` for `platform_ip` → `lab_gateway_port_range`
|
||||
(`30000:32767` by default, configurable), so that once Lab containers publish ports on the Lab
|
||||
Host's own interface, the Gateway VM is permitted to reach them.
|
||||
|
||||
What's **not yet built** (application half — explicitly flagged, not silently decided here):
|
||||
the Provisioner needs to either (a) publish each Lab container's relevant port(s) on the Lab
|
||||
Host's interface and record a reachable `external_ip`/`external_port` alongside (or instead of)
|
||||
`internal_ip`/`internal_port`, or (b) route Gateway→Lab traffic through a proxy/tunnel that
|
||||
lives on the Lab Host and forwards to the container's internal address. Option (a) is simpler
|
||||
and consistent with how the egress proxy and cadvisor are already handled in this phase's
|
||||
compose files; option (b) avoids publishing a wide port range but adds a moving part. This is a
|
||||
small, reviewable, application-level change — not a network-policy change — and should land as
|
||||
its own reviewed commit before real two-VM production traffic depends on it. Until it lands,
|
||||
Gateway↔Lab terminal/scenario traffic will not work correctly across two separate VMs, even
|
||||
though every other production networking, firewall, and TLS boundary is in place and correct.
|
||||
|
||||
## Cookie / auth production settings
|
||||
|
||||
`infra/environments/production/*.env.template`: `COOKIE_SECURE=1`, `TRUST_PROXY=1`. Dev's
|
||||
insecure defaults (`COOKIE_SECURE` unset, no proxy trust) are never inherited — production env
|
||||
files are separate files, not a dev file with overrides layered on.
|
||||
@@ -0,0 +1,50 @@
|
||||
# Production Validation Checklist
|
||||
|
||||
Walk this once, top to bottom, before calling a production deployment live. Every item here
|
||||
needs the real hardware — this is the "DEPLOY LATER" half of Phase 4, not something this repo
|
||||
can prove in advance.
|
||||
|
||||
## Hardware / Proxmox
|
||||
- [ ] Disk inspection done, layout chosen and matches what's actually in the server (`docs/RUNBOOK_DISK_FAILURE.md` inspection checklist, `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4)
|
||||
- [ ] Proxmox VE installed, reachable only from `management_cidr`/`proxmox_mgmt_cidr`
|
||||
- [ ] VM template built from real cloud-init (`infra/proxmox/README.md`)
|
||||
- [ ] `tofu apply` created both VMs; both boot and are SSH-reachable
|
||||
|
||||
## Platform VM
|
||||
- [ ] `ansible-playbook site.yml` completed clean, all roles
|
||||
- [ ] `docker compose ps` shows every service healthy (postgres, provisioner, api, gateway, reverse-proxy, prometheus, alertmanager, grafana, loki, promtail, exporters)
|
||||
- [ ] `curl https://<origin>/healthz` (via Gateway) returns 200
|
||||
- [ ] `curl http://localhost:4300/healthz` (Platform API, from the VM) returns `{"ok":true,"db":"ok"}`
|
||||
- [ ] TLS certificate issued and valid for the real domain
|
||||
|
||||
## Lab Host VM
|
||||
- [ ] Docker daemon TLS verified: `docker --tlsverify --tlscacert=... -H tcp://<lab_host_ip>:2376 info` succeeds from the Platform VM, fails from anywhere else
|
||||
- [ ] Provisioner successfully creates a real Lab against the real Lab Host
|
||||
- [ ] Gateway can actually reach that Lab's terminal — **this depends on the not-yet-built application change documented in `docs/PRODUCTION_NETWORK_SECURITY.md`'s "Known gap" section; do not mark this item done until that change has landed and been verified for real**
|
||||
- [ ] Egress proxy correctly restricts a Lab's outbound traffic (Provisioner's per-Lab `DOCKER-USER` chain)
|
||||
|
||||
## Security
|
||||
- [ ] `infra/firewall/validate-firewall.sh` run against the real hosts with real `iptables-restore --test`, not just the structural lint
|
||||
- [ ] Confirm from an unauthorized IP: Postgres, Provisioner, Docker TLS port, Proxmox UI are all unreachable
|
||||
- [ ] Confirm Student Lab → another Lab is blocked; confirm Student Lab → Proxmox is blocked
|
||||
- [ ] `COOKIE_SECURE`/`TRUST_PROXY` active (check a real Set-Cookie header over HTTPS)
|
||||
|
||||
## Academy (application-level smoke test)
|
||||
- [ ] Owner bootstrap login works
|
||||
- [ ] Instructor can create a cohort, roster a student
|
||||
- [ ] Student can log in, boot a Lab, complete a scenario, see progress
|
||||
- [ ] An assessment can be assigned, taken, and graded end-to-end
|
||||
- [ ] Instructor analytics dashboards populate with real data
|
||||
|
||||
## Backup
|
||||
- [ ] Manual `backup-postgres.sh` run succeeds against real Postgres, real off-host target reachable
|
||||
- [ ] A real restore (`restore-postgres.sh` + `verify-restore.sh`) has been rehearsed at least once against a throwaway target, not just the local test harness
|
||||
- [ ] Cron job confirmed scheduled (`crontab -l`)
|
||||
|
||||
## Observability
|
||||
- [ ] Grafana dashboard shows real data for both VMs
|
||||
- [ ] Trigger one alert deliberately (e.g. stop a service) and confirm it fires and reaches the webhook
|
||||
- [ ] Loki has logs flowing from both VMs
|
||||
|
||||
Only once every box above is checked against the real server does the deployment graduate from
|
||||
"infrastructure package built" to "actually in production."
|
||||
@@ -0,0 +1,31 @@
|
||||
# Runbook: Backup Failure
|
||||
|
||||
Triggered by the `BackupFailed`/`BackupAging` alerts (`infra/monitoring/prometheus/rules/rootnode.yml`).
|
||||
|
||||
```bash
|
||||
# 1. See what actually happened — the cron wrapper logs to syslog
|
||||
journalctl -u rootnode-backup --since "24 hours ago" -o cat
|
||||
# or, if the cron job never even ran:
|
||||
grep rootnode-postgres-backup /var/log/syslog
|
||||
|
||||
# 2. Run it manually to see the failure live
|
||||
source /etc/rootnode/backup.env
|
||||
/opt/rootnode/backup/backup-postgres.sh
|
||||
```
|
||||
|
||||
**Common causes:**
|
||||
- Disk full on `{{ rootnode_data_root }}/backups` — check `df -h`, prune more aggressively
|
||||
(`BACKUP_RETENTION_DAYS`) or the ZFS dataset needs more space (see `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4).
|
||||
- Off-host target unreachable (`BACKUP_OFFSITE_TARGET`) — check the SSH key/credentials
|
||||
haven't expired, the target host/bucket is reachable: `ssh <offsite-target-host> echo ok` or
|
||||
the equivalent for whatever adapter is configured (`docs/BACKUP_DR.md`).
|
||||
- `BACKUP_ENCRYPTION_PASSPHRASE` missing/wrong in `/etc/rootnode/backup.env` — re-check against
|
||||
the value in Ansible Vault (`vault_backup_encryption_passphrase`); a mismatch here means even
|
||||
a "successful" old backup becomes unrestorable, so treat this specific failure mode as urgent.
|
||||
- The dump itself is suspiciously small (script's own sanity check tripped) — Postgres may be
|
||||
unhealthy; check `docker exec rootnode-platform-postgres-1 pg_isready` first.
|
||||
|
||||
**After fixing:** run the backup manually once to confirm success (metric updates immediately,
|
||||
alert clears within one Prometheus scrape interval), then confirm the cron job is still
|
||||
correctly scheduled (`crontab -l` as root, or re-run the `backup` Ansible role if the cron entry
|
||||
itself is suspect).
|
||||
@@ -0,0 +1,44 @@
|
||||
# Runbook: Disk Failure
|
||||
|
||||
Applies once real hardware/ZFS exists — see `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4 for
|
||||
the storage decision framework this assumes (mirrored vdevs, never single-disk for Postgres).
|
||||
|
||||
```bash
|
||||
# 1. Confirm the failure
|
||||
zpool status -v
|
||||
smartctl -a /dev/sdX # the suspect drive
|
||||
|
||||
# 2. If it's a mirrored vdev (the required layout for anything holding Postgres data — see §4),
|
||||
# the pool stays online in DEGRADED state. No immediate data-loss risk, but no redundancy
|
||||
# either until replaced — treat DEGRADED as urgent, not routine.
|
||||
zpool status # confirm DEGRADED, not FAULTED/UNAVAIL (those mean act immediately, possibly
|
||||
# restore from backup if the redundant mirror side is also compromised)
|
||||
|
||||
# 3. Replace the physical drive, then:
|
||||
zpool replace <pool> <old-device> <new-device>
|
||||
zpool status # watch the resilver progress
|
||||
|
||||
# 4. If a whole pool was lost (both mirror sides failed simultaneously — the scenario mirroring
|
||||
# exists to make vanishingly unlikely, but plan for it anyway):
|
||||
# - rn-lab-host-01's disposable data: nothing to recover, just rebuild the VM
|
||||
# (docs/RUNBOOK_VM_REBUILD.md)
|
||||
# - rn-platform-01's Postgres data: restore from the latest off-host backup
|
||||
# (docs/RUNBOOK_POSTGRES_RESTORE.md) — this is exactly the scenario off-host backup
|
||||
# copies exist for; a local-only backup on the SAME failed pool would be useless here
|
||||
```
|
||||
|
||||
**Pre-deployment disk inspection checklist** (run once when the real drives are known, before
|
||||
they go into service — `docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` §4):
|
||||
|
||||
```bash
|
||||
for d in /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1; do # adjust to real device names
|
||||
echo "== $d =="
|
||||
smartctl -i "$d" | grep -E 'Model|Serial|Capacity|Firmware'
|
||||
smartctl -A "$d" | grep -iE 'power_on_hours|reallocated|pending|uncorrectable|wear|percentage.used|temperature'
|
||||
smartctl -H "$d" | grep -i overall
|
||||
done
|
||||
```
|
||||
Record model, size, power-on hours, wear/endurance percentage, reallocated/pending sector
|
||||
counts, temperature, and firmware version for every drive before committing to a ZFS layout —
|
||||
a drive already showing wear or reallocated sectors shouldn't be trusted with Postgres data
|
||||
even in a mirror.
|
||||
@@ -0,0 +1,31 @@
|
||||
# Runbook: Docker TLS Rotation
|
||||
|
||||
Rotate on a schedule (recommend yearly, well before the ~2.25-year cert lifetime
|
||||
`generate-docker-tls.sh` defaults to) or immediately if a client cert may have leaked.
|
||||
|
||||
```bash
|
||||
# 1. Generate a fresh bundle (new CA too — simplest, avoids any question about whether the old
|
||||
# CA's private key was ever exposed)
|
||||
cd infra/scripts/docker-tls
|
||||
./generate-docker-tls.sh --out-dir ./out-$(date +%Y%m%d) --lab-host-ip <real-ip> --lab-host-dns rn-lab-host-01.internal
|
||||
./verify-docker-tls.sh ./out-$(date +%Y%m%d)
|
||||
|
||||
# 2. Redeploy — this restarts dockerd on rn-lab-host-01 (brief interruption to any in-flight
|
||||
# Docker API call from the Provisioner, NOT to already-running student Labs — their
|
||||
# containers keep running through a dockerd restart)
|
||||
cd ../../ansible
|
||||
ROOTNODE_TLS_BUNDLE_DIR=../scripts/docker-tls/out-$(date +%Y%m%d) \
|
||||
ansible-playbook site.yml -i inventory/production.ini --tags docker-tls --ask-vault-pass
|
||||
|
||||
# 3. Verify the Provisioner can still reach rn-lab-host-01
|
||||
curl -s http://localhost:4100/health # from rn-platform-01
|
||||
```
|
||||
|
||||
**Troubleshooting a failed rotation:**
|
||||
- `x509: certificate signed by unknown authority` — the new ca.pem wasn't deployed to both
|
||||
sides, or an old one is cached; re-run the docker-tls role on BOTH hosts.
|
||||
- Provisioner logs `ECONNREFUSED` on port 2376 — dockerd didn't actually restart with the new
|
||||
daemon.json; check `systemctl status docker` on rn-lab-host-01 and the ExecStart override
|
||||
(`infra/ansible/roles/docker-tls/templates/docker-override.conf.j2`) actually applied.
|
||||
- Firewall — confirm `infra/firewall/rn-lab-host-01.rules`' 2376 rule still matches
|
||||
rn-platform-01's actual IP (didn't change as part of this rotation, but worth a sanity check).
|
||||
@@ -0,0 +1,35 @@
|
||||
# Runbook: Failed Migration
|
||||
|
||||
Both Platform API and Provisioner apply their own migrations automatically on startup
|
||||
(`platform/api/src/db.js`, `platform/provisioner/src/db.js`), file-tracked in
|
||||
`schema_migrations`, wrapped in a transaction per file — a migration either fully applies or
|
||||
fully rolls back, it can't leave a table half-altered.
|
||||
|
||||
**If a deploy fails because a migration errored:**
|
||||
|
||||
```bash
|
||||
# 1. Check what actually happened
|
||||
docker logs rootnode-platform-api-1 --tail 50 # the error is right there — migration file + message
|
||||
docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
|
||||
"SELECT * FROM schema_migrations ORDER BY applied_at DESC LIMIT 10;"
|
||||
|
||||
# 2. Since the failing migration's transaction rolled back, the schema is still in its
|
||||
# PREVIOUS valid state — the app just won't start until the migration succeeds. Fix the
|
||||
# root cause (usually a real data issue the migration didn't anticipate, or the
|
||||
# cross-service ordering dependency — Provisioner's migrations must run before Platform
|
||||
# API's, since scenario_attempts has a real FK to environments; see
|
||||
# docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md).
|
||||
|
||||
# 3. Redeploy once fixed — the migration runner is idempotent and will just pick up where it
|
||||
# left off; already-applied migrations are skipped.
|
||||
./infra/scripts/deploy-production.sh <same-or-fixed-tag>
|
||||
```
|
||||
|
||||
**If you need to roll back the CODE but the migration already succeeded** (schema changed,
|
||||
code that depended on it is being reverted): rolling back application images
|
||||
(`docs/RUNBOOK_ROLLBACK.md`) does NOT undo a Postgres migration — there is no automatic
|
||||
down-migration mechanism (deliberate, matches the existing check-before-alter, additive-only
|
||||
migration convention documented in `docs/ENGINEERING_SOP.md` §6). If the old code genuinely
|
||||
can't run against the new schema, restore from the pre-migration backup instead
|
||||
(`docs/RUNBOOK_POSTGRES_RESTORE.md`) rather than trying to hand-write a down-migration under
|
||||
pressure.
|
||||
@@ -0,0 +1,34 @@
|
||||
# Runbook: First Production Deployment
|
||||
|
||||
Prerequisites: Proxmox installed (`infra/proxmox/README.md`), VMs created
|
||||
(`infra/terraform/`), DNS pointed at rn-platform-01's public IP.
|
||||
|
||||
```bash
|
||||
# 1. Generate Docker TLS material (offline, on your own machine — never on either VM)
|
||||
cd infra/scripts/docker-tls
|
||||
./generate-docker-tls.sh --out-dir ./out --lab-host-ip <real-lab-host-ip> --lab-host-dns rn-lab-host-01.internal
|
||||
./verify-docker-tls.sh ./out
|
||||
|
||||
# 2. Fill in real values
|
||||
cd ../../ansible
|
||||
cp group_vars/production.vault.yml.template group_vars/production.vault.yml
|
||||
$EDITOR group_vars/production.vault.yml # real passwords/secrets — openssl rand -hex 32 etc.
|
||||
ansible-vault encrypt group_vars/production.vault.yml
|
||||
$EDITOR group_vars/production.yml # real IPs, domain, CIDRs
|
||||
$EDITOR inventory/production.ini # real IPs
|
||||
|
||||
# 3. Deploy
|
||||
ROOTNODE_TLS_BUNDLE_DIR=../scripts/docker-tls/out \
|
||||
ansible-playbook site.yml -i inventory/production.ini --ask-vault-pass
|
||||
|
||||
# 4. Bootstrap the owner account (once, by a human)
|
||||
ssh rootnode-deploy@rn-platform-01
|
||||
docker exec -it rootnode-platform-api-1 node bootstrap-owner.js [email protected] "Your Name"
|
||||
# (password comes from BOOTSTRAP_OWNER_PASSWORD in the environment, or the CLI prompt — see
|
||||
# platform/api/README.md)
|
||||
|
||||
# 5. Verify
|
||||
../../infra/scripts/production-smoke-test.sh
|
||||
```
|
||||
|
||||
Then work through `docs/PRODUCTION_VALIDATION_CHECKLIST.md` in full before calling this done.
|
||||
@@ -0,0 +1,29 @@
|
||||
# Runbook: High Lab Host Resource Pressure
|
||||
|
||||
Triggered by the `LabHostHighMemory`/`HighDiskUsage` warning alerts.
|
||||
|
||||
```bash
|
||||
# 1. How many Labs are actually running, and against what admission-control threshold?
|
||||
curl -s http://localhost:4100/environments | jq '[.[] | select(.status=="running")] | length'
|
||||
node infra/scripts/capacity-model.js # recommended/warning/hard thresholds
|
||||
|
||||
# 2. If genuinely over capacity: this is expected pressure from real usage, not a bug.
|
||||
# - Destroy idle/abandoned Labs early via /admin (per-cohort "Destroy Labs" after a class
|
||||
# session ends, rather than waiting for the idle reaper's default window)
|
||||
# - If a specific cohort's class just ended, use /admin -> Cohorts -> [cohort] -> Destroy
|
||||
# Cohort Labs immediately rather than waiting
|
||||
|
||||
# 3. If pressure is high with FEWER Labs running than capacity-model.js's threshold: something
|
||||
# is consuming more per-Lab resources than the model assumes (a scenario doing something
|
||||
# unusually heavy, a memory leak in a long-running Lab, zombie processes from a crashed
|
||||
# reset). Check per-container usage:
|
||||
docker stats --no-stream | grep rootnode-lab-
|
||||
|
||||
# 4. Update infra/environments/production/capacity.yml with fresh measured numbers
|
||||
# (infra/scripts/loadtest/) if the per-Lab footprint has genuinely changed, so the
|
||||
# admission-control threshold reflects reality going forward.
|
||||
```
|
||||
|
||||
Do not silently raise resource limits in `provisioner-config/resource-policy.yaml` to make an
|
||||
alert go away without understanding why usage is higher than modeled — that just moves the
|
||||
same pressure onto host RAM/CPU directly instead of onto individual Lab limits.
|
||||
@@ -0,0 +1,17 @@
|
||||
# Runbook: Normal Deploy
|
||||
|
||||
For a routine code change already merged to `main` on the canonical Gitea repo.
|
||||
|
||||
1. CI (`infra/ci/gitea-workflows/build-and-test.yml`) runs automatically: tests, image build/tag/push, staging deploy, staging smoke test.
|
||||
2. Review the staging smoke test result and, ideally, poke around `https://staging.<domain>` yourself.
|
||||
3. Promote to production (manual, deliberate):
|
||||
|
||||
```bash
|
||||
cd infra/scripts
|
||||
./deploy-production.sh sha-<12-char-sha> # or a release tag, e.g. v1.3.0
|
||||
./production-smoke-test.sh
|
||||
```
|
||||
|
||||
4. Watch Grafana (`/grafana/`) for 10-15 minutes — error rates, Postgres connections, Lab Host
|
||||
memory — before considering the deploy "settled."
|
||||
5. If anything looks wrong: `docs/RUNBOOK_ROLLBACK.md`.
|
||||
@@ -0,0 +1,27 @@
|
||||
# Runbook: Owner Account Recovery
|
||||
|
||||
**Locked out with zero working owner accounts** (all deactivated, forgot the only owner's
|
||||
password, etc.):
|
||||
|
||||
```bash
|
||||
ssh rootnode-deploy@rn-platform-01
|
||||
docker exec -it rootnode-platform-api-1 sh
|
||||
node bootstrap-owner.js [email protected] "Your Name"
|
||||
# bootstrap-owner.js refuses to run if an active owner already exists UNLESS the target email
|
||||
# belongs to an EXISTING instructor row — in that case it promotes them to owner. Point it at
|
||||
# any known instructor email to recover through them, or a brand-new email to create a fresh
|
||||
# owner outright if the instructors table itself is in an unknown state.
|
||||
```
|
||||
|
||||
**Forgot password, account still active and you have SSH access:**
|
||||
|
||||
```bash
|
||||
docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
|
||||
"SELECT instructor_id, email, role, status FROM instructors WHERE role='owner';"
|
||||
# Then use another owner's account to reset via /admin -> Instructors -> [account] -> Reset
|
||||
# Password, OR re-run bootstrap-owner.js targeting that email (also resets the password).
|
||||
```
|
||||
|
||||
**Never** hand-edit `password_hash` in the database — it's `scrypt(salt):hash` in a specific
|
||||
format (`platform/api/src/auth/password.js`); a malformed value just breaks login rather than
|
||||
setting a usable password. Always go through `bootstrap-owner.js` or the admin UI's reset flow.
|
||||
@@ -0,0 +1,45 @@
|
||||
# Runbook: Postgres Restore
|
||||
|
||||
**When to use:** Postgres corruption/data loss on rn-platform-01, or verifying a backup is
|
||||
actually good (do this periodically even without an incident — an untested backup is a
|
||||
hypothesis, not a backup).
|
||||
|
||||
```bash
|
||||
# 1. Find the backup to restore
|
||||
ls -la /srv/rootnode-data/backups/ # or wherever backup_offsite_target points, if local copies are gone
|
||||
|
||||
# 2. Create a FRESH target database — never restore over the live one directly
|
||||
docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
|
||||
"CREATE DATABASE rootnode_platform_restore_test;"
|
||||
|
||||
# 3. Restore into it
|
||||
source /etc/rootnode/backup.env # for BACKUP_ENCRYPTION_PASSPHRASE
|
||||
/opt/rootnode/backup/restore-postgres.sh \
|
||||
/srv/rootnode-data/backups/rootnode-platform-<timestamp>.sql.gz.gpg \
|
||||
"postgres://rootnode_platform:$PGPASS@localhost:5432/rootnode_platform_restore_test"
|
||||
# type RESTORE when prompted
|
||||
|
||||
# 4. Verify
|
||||
/opt/rootnode/backup/verify-restore.sh \
|
||||
"postgres://rootnode_platform:$PGPASS@localhost:5432/rootnode_platform_restore_test"
|
||||
|
||||
# 5a. If this was a drill: drop the test database
|
||||
docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
|
||||
"DROP DATABASE rootnode_platform_restore_test;"
|
||||
|
||||
# 5b. If this was a REAL incident: stop the app stack, rename the broken DB, rename the
|
||||
# restored one into its place, restart.
|
||||
systemctl stop rootnode-platform
|
||||
docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
|
||||
"ALTER DATABASE rootnode_platform RENAME TO rootnode_platform_broken_$(date +%s);"
|
||||
docker exec -it rootnode-platform-postgres-1 psql -U rootnode_platform -c \
|
||||
"ALTER DATABASE rootnode_platform_restore_test RENAME TO rootnode_platform;"
|
||||
systemctl start rootnode-platform
|
||||
```
|
||||
|
||||
Local, non-production dry run of this exact flow: `infra/backup/test-backup-restore.sh` (uses
|
||||
disposable containers, safe to run anytime, no real data involved).
|
||||
|
||||
**Recovery point objective:** up to 24 hours of academic data (daily backup cadence) — see
|
||||
`docs/BACKUP_DR.md` for the full RPO/RTO discussion and why that's the honest number, not an
|
||||
aspirational one.
|
||||
@@ -0,0 +1,18 @@
|
||||
# Runbook: Rollback
|
||||
|
||||
```bash
|
||||
cd infra/scripts
|
||||
./rollback-production.sh # rolls back to the previous deployed tag automatically
|
||||
# or, to go back further than one step:
|
||||
./rollback-production.sh sha-<older-sha>
|
||||
|
||||
./production-smoke-test.sh
|
||||
```
|
||||
|
||||
This rolls back **application images only**. If the bad deploy included a database migration
|
||||
that needs undoing too, schema changes aren't automatically reversible — see
|
||||
`docs/RUNBOOK_FAILED_MIGRATION.md` instead (or in addition).
|
||||
|
||||
After rolling back: file an issue / note in the deploy log (`/opt/rootnode/deploy-history.log`
|
||||
already has the rollback recorded) about what went wrong before re-attempting the forward
|
||||
deploy.
|
||||
@@ -0,0 +1,40 @@
|
||||
# Runbook: Stuck Lab Cleanup
|
||||
|
||||
A Lab that's stuck `provisioning`/`resetting` past its expected boot window, or a container
|
||||
that outlived its Postgres `environments` row (crash during teardown, etc.).
|
||||
|
||||
**Via the admin UI (preferred, no host access needed):** `/admin` -> Students -> [student] ->
|
||||
Environment -> Destroy, then Provision again if they need a fresh one immediately.
|
||||
|
||||
**Via the API directly** (same effect, useful for scripting a batch cleanup):
|
||||
```bash
|
||||
curl -s -b <owner-session-cookie> -X DELETE https://academy.example.com/api/admin/environments/<environment_id>
|
||||
```
|
||||
|
||||
**A container the Provisioner itself has lost track of** (rare — e.g. a crash during
|
||||
provisioning left an orphaned container/network/firewall chain with no matching `environments`
|
||||
row, or vice versa):
|
||||
|
||||
```bash
|
||||
# On rn-lab-host-01 — list what's actually running
|
||||
docker ps -a --format '{{.Names}}\t{{.Status}}' | grep '^rootnode-lab-'
|
||||
|
||||
# Cross-reference against the Provisioner's own view
|
||||
curl -s http://localhost:4100/environments | jq '.[] | {environment_id, status, container_name}'
|
||||
|
||||
# Provisioner already does this automatically on its own restart (reconcileOnStartup,
|
||||
# platform/provisioner/src/lifecycle.js) — a simple `systemctl restart rootnode-platform`
|
||||
# (which restarts the provisioner container as part of the stack) is often the fastest fix for
|
||||
# a batch of orphans rather than hand-cleaning each one.
|
||||
|
||||
# Manual last resort — remove the container/network/firewall chain directly:
|
||||
docker rm -f rootnode-lab-<id>
|
||||
docker network rm rootnode-lab-net-<id>
|
||||
iptables -D DOCKER-USER -s <subnet> -j ROOTNODE-ENV-<id> # remove the jump rule FIRST
|
||||
iptables -F ROOTNODE-ENV-<id> && iptables -X ROOTNODE-ENV-<id>
|
||||
```
|
||||
|
||||
Never manually delete an `environments` row from Postgres directly unless you've confirmed the
|
||||
container/network/firewall chain are already gone — the Provisioner treats that table as
|
||||
authoritative and will behave unpredictably against real Docker/iptables state it doesn't know
|
||||
matches.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Runbook: Public TLS Certificate Renewal
|
||||
|
||||
If using Caddy's built-in Let's Encrypt (the default in `infra/docker/platform/Caddyfile.j2`):
|
||||
**nothing to do** — Caddy renews automatically well before expiry, as long as port 80 stays
|
||||
reachable for the ACME HTTP-01 challenge (`infra/firewall/rn-platform-01.rules` already allows
|
||||
it). `infra/scripts/production-smoke-test.sh` checks the live expiry date on every run as an
|
||||
early warning if auto-renewal ever silently breaks.
|
||||
|
||||
If using Cloudflare origin certs instead (see `docs/PRODUCTION_NETWORK_SECURITY.md` §8 for
|
||||
that decision): Cloudflare-issued origin certs are typically 15-year validity — this is a
|
||||
once-per-network-topology-change concern, not a recurring one. Re-issue from the Cloudflare
|
||||
dashboard (SSL/TLS -> Origin Server) and redeploy the cert/key via the `platform` Ansible role
|
||||
if it ever needs replacing (domain change, suspected compromise).
|
||||
|
||||
**Manual check:**
|
||||
```bash
|
||||
echo | openssl s_client -connect academy.example.com:443 -servername academy.example.com 2>/dev/null \
|
||||
| openssl x509 -noout -enddate
|
||||
```
|
||||
@@ -0,0 +1,31 @@
|
||||
# Runbook: Platform VM / Lab Host VM Rebuild
|
||||
|
||||
Both VMs are designed to be disposable and rebuildable from code — nothing irreplaceable
|
||||
should ever live only on either VM's disk outside of what's backed up (`docs/BACKUP_DR.md`).
|
||||
|
||||
## rn-platform-01 lost
|
||||
|
||||
Academic data survives (it's in Postgres backups, off-host). Instructor/student accounts and
|
||||
progress are recoverable; anything since the last backup is not (`docs/BACKUP_DR.md`'s RPO).
|
||||
|
||||
```bash
|
||||
cd infra/terraform && terraform apply # recreates the VM from the template
|
||||
cd ../ansible && ansible-playbook site.yml -i inventory/production.ini --limit platform --ask-vault-pass
|
||||
# Then restore Postgres — docs/RUNBOOK_POSTGRES_RESTORE.md, step 5b variant (fresh DB this time,
|
||||
# no need to rename since it's a fresh instance)
|
||||
./infra/scripts/production-smoke-test.sh
|
||||
```
|
||||
|
||||
## rn-lab-host-01 lost
|
||||
|
||||
**No academic data is lost** — this host only ever holds disposable Lab containers by design
|
||||
(`docs/BACKUP_DR.md`'s Tier distinction). Every enrolled student's Lab needs re-provisioning
|
||||
after this, which is a normal "Prepare Labs" click, not a recovery operation.
|
||||
|
||||
```bash
|
||||
cd infra/terraform && terraform apply
|
||||
cd ../ansible && ansible-playbook site.yml -i inventory/production.ini --limit lab_host --ask-vault-pass
|
||||
# Re-generate Docker TLS material for the new host if its identity (IP/DNS) changed —
|
||||
# docs/RUNBOOK_DOCKER_TLS_ROTATION.md
|
||||
# From /admin: Prepare Labs again for any cohort that needs it.
|
||||
```
|
||||
@@ -78,3 +78,12 @@ See `provisioner/README.md` (Phase 1), `docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md
|
||||
2A), `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md` (Phase 3A identity limitations), and
|
||||
`docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` §20 (Phase 3B learning/assessment limitations) for the
|
||||
full, honest lists.
|
||||
|
||||
## Running this in production
|
||||
|
||||
`../infra/` has the full production infrastructure package (Ansible, Terraform, firewall,
|
||||
monitoring, backup, CI/CD) for deploying the Platform API/Gateway/Provisioner/Postgres onto a
|
||||
dedicated `rn-platform-01` VM, with student Labs on a separate `rn-lab-host-01` VM. See
|
||||
`../docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md` for the overview and
|
||||
`../docs/PRODUCTION_NETWORK_SECURITY.md` for a known gap in Gateway-to-Lab reachability across
|
||||
two real VMs that must be closed before production go-live.
|
||||
|
||||
Reference in New Issue
Block a user