ced4568
bf1e1b5c76
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs (PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR, OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST, and 12 runbooks (first deployment, normal deploy, rollback, Postgres restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure). PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine architectural gap this phase surfaced: the Gateway's existing env.internal_ip mechanism for reaching a Lab container assumes Gateway and Lab share a Docker daemon, which breaks once Labs run on a genuinely separate rn-lab-host-01 VM. The network-policy half is built (a firewall allowance for Gateway->Lab-Host on a published port range); the application half (Provisioner/Gateway address handling) is an explicit, reviewed, NOT-YET-BUILT follow-up flagged there rather than silently decided. Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and platform/README.md with Phase 4 pointers, consistent with how every prior phase updated these. Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
65 lines
3.9 KiB
Markdown
65 lines
3.9 KiB
Markdown
# Observability
|
|
|
|
Self-hosted Prometheus + Grafana + Alertmanager + Loki/Promtail, deployed via
|
|
`infra/docker/platform/docker-compose.production.yml` on rn-platform-01, with lightweight agents
|
|
(node-exporter, cadvisor, promtail) also on rn-lab-host-01. No paid SaaS.
|
|
|
|
This is operational observability — host/container/database health, uptime, resource pressure.
|
|
It is deliberately separate from the application's own instructor learning analytics
|
|
(Phase 3B's cohort/scenario/student dashboards) — different audience (ops vs. instructors),
|
|
different data (infrastructure vs. learning outcomes), and mixing them would make both worse.
|
|
|
|
## What's real today
|
|
|
|
- **Host metrics** (both VMs): node-exporter — CPU, memory, disk, network.
|
|
- **Container metrics** (Lab Host): cadvisor — per-container CPU/memory/network, including
|
|
student Lab containers (`rootnode-lab-*`).
|
|
- **Postgres metrics**: postgres-exporter — connections, transaction rate, table sizes.
|
|
- **Uptime/liveness**: blackbox-exporter probing `/healthz` on the Gateway
|
|
(`platform/gateway/src/server.js`) and the Platform API (`platform/api/src/server.js`), both
|
|
added this phase specifically for this purpose, plus the Provisioner's pre-existing `/health`.
|
|
- **Logs**: Loki + Promtail, scraping Docker container logs and journald from both VMs, with a
|
|
redaction pipeline stage that strips `password|token|secret=`-pattern values before they're
|
|
stored (`infra/monitoring/promtail/`).
|
|
- **Dashboards**: Grafana, reachable only via `https://<origin>/grafana/` and only from
|
|
`management_cidr` (see `docs/PRODUCTION_NETWORK_SECURITY.md`). One starter dashboard,
|
|
`infra/monitoring/grafana/provisioning/dashboards/rootnode-overview.json` — 8 panels (Platform
|
|
+ Lab Host CPU/memory/disk, Postgres up/connections, running-Lab-count via cadvisor label
|
|
match [best-effort, caveated], uptime probes).
|
|
|
|
## What's NOT built yet (honest accounting, not silently assumed)
|
|
|
|
Application-level metrics — active student count, Lab provisioning success/failure rate,
|
|
Gateway request latency/error rate, per-endpoint API metrics — would need real instrumentation
|
|
added to the Platform API/Gateway/Provisioner (e.g. a `/metrics` endpoint via `prom-client`).
|
|
`infra/monitoring/prometheus/prometheus.yml.j2` has a commented-out scrape job as the landing
|
|
spot for this, explicitly marked as a follow-up, not wired to anything real yet.
|
|
|
|
## Alerting
|
|
|
|
`infra/monitoring/prometheus/rules/rootnode.yml` — two severities:
|
|
|
|
**Critical** (fires fast, pages): PlatformAPIDown, GatewayDown, PostgresDown, LabHostDown,
|
|
BackupFailed, DiskCriticallyFull.
|
|
|
|
**Warning** (informational, slower repeat): LabHostHighMemory, HighDiskUsage, BackupAging.
|
|
(RepeatedProvisioningFailures and ExcessiveFailedLogins are present as inert placeholders —
|
|
they need the not-yet-built application metrics above before they can fire on anything real.)
|
|
|
|
Never alerts on an individual student's scenario failure or a single Lab crash — those are
|
|
expected, routine outcomes of normal usage, not operational incidents.
|
|
|
|
`infra/monitoring/alertmanager/alertmanager.yml.j2` — critical alerts route separately with a
|
|
30-minute repeat interval, everything else at 4 hours, one webhook receiver
|
|
(`vault_alertmanager_webhook_url` — points at whatever channel gets configured at deploy time,
|
|
e.g. a Slack/Discord/Matrix webhook), and one inhibit rule so a LabHostDown critical suppresses
|
|
its own dependent warnings instead of paging twice for the same root cause.
|
|
|
|
## Validation performed (local, real tooling — not hardware-dependent)
|
|
|
|
`infra/monitoring/validate-monitoring.sh` wraps: `promtool check config` (Prometheus config),
|
|
`promtool check rules` (alert rules — 11 rules found, valid), `amtool check-config`
|
|
(Alertmanager — 1 inhibit rule, 2 receivers, valid). These prove syntactic/semantic correctness
|
|
of the configs, not that the real network paths between rn-platform-01 and rn-lab-host-01 work —
|
|
that requires the real hardware and is covered by `docs/PRODUCTION_VALIDATION_CHECKLIST.md`.
|