# Observability

Self-hosted Prometheus + Grafana + Alertmanager + Loki/Promtail, deployed via
`infra/docker/platform/docker-compose.production.yml` on rn-platform-01, with lightweight agents
(node-exporter, cadvisor, promtail) also on rn-lab-host-01. No paid SaaS.

This is operational observability — host/container/database health, uptime, resource pressure.
It is deliberately separate from the application's own instructor learning analytics
(Phase 3B's cohort/scenario/student dashboards) — different audience (ops vs. instructors),
different data (infrastructure vs. learning outcomes), and mixing them would make both worse.

## What's real today

- **Host metrics** (both VMs): node-exporter — CPU, memory, disk, network.
- **Container metrics** (Lab Host): cadvisor — per-container CPU/memory/network, including
  student Lab containers (`rootnode-lab-*`).
- **Postgres metrics**: postgres-exporter — connections, transaction rate, table sizes.
- **Uptime/liveness**: blackbox-exporter probing `/healthz` on the Gateway
  (`platform/gateway/src/server.js`) and the Platform API (`platform/api/src/server.js`), both
  added this phase specifically for this purpose, plus the Provisioner's pre-existing `/health`.
- **Logs**: Loki + Promtail, scraping Docker container logs and journald from both VMs, with a
  redaction pipeline stage that strips `password|token|secret=`-pattern values before they're
  stored (`infra/monitoring/promtail/`).
- **Dashboards**: Grafana, reachable only via `https://<origin>/grafana/` and only from
  `management_cidr` (see `docs/PRODUCTION_NETWORK_SECURITY.md`). One starter dashboard,
  `infra/monitoring/grafana/provisioning/dashboards/rootnode-overview.json` — 8 panels (Platform
  + Lab Host CPU/memory/disk, Postgres up/connections, running-Lab-count via cadvisor label
  match [best-effort, caveated], uptime probes).

## What's NOT built yet (honest accounting, not silently assumed)

Application-level metrics — active student count, Lab provisioning success/failure rate,
Gateway request latency/error rate, per-endpoint API metrics — would need real instrumentation
added to the Platform API/Gateway/Provisioner (e.g. a `/metrics` endpoint via `prom-client`).
`infra/monitoring/prometheus/prometheus.yml.j2` has a commented-out scrape job as the landing
spot for this, explicitly marked as a follow-up, not wired to anything real yet.

## Alerting

`infra/monitoring/prometheus/rules/rootnode.yml` — two severities:

**Critical** (fires fast, pages): PlatformAPIDown, GatewayDown, PostgresDown, LabHostDown,
BackupFailed, DiskCriticallyFull.

**Warning** (informational, slower repeat): LabHostHighMemory, HighDiskUsage, BackupAging.
(RepeatedProvisioningFailures and ExcessiveFailedLogins are present as inert placeholders —
they need the not-yet-built application metrics above before they can fire on anything real.)

Never alerts on an individual student's scenario failure or a single Lab crash — those are
expected, routine outcomes of normal usage, not operational incidents.

`infra/monitoring/alertmanager/alertmanager.yml.j2` — critical alerts route separately with a
30-minute repeat interval, everything else at 4 hours, one webhook receiver
(`vault_alertmanager_webhook_url` — points at whatever channel gets configured at deploy time,
e.g. a Slack/Discord/Matrix webhook), and one inhibit rule so a LabHostDown critical suppresses
its own dependent warnings instead of paging twice for the same root cause.

## Validation performed (local, real tooling — not hardware-dependent)

`infra/monitoring/validate-monitoring.sh` wraps: `promtool check config` (Prometheus config),
`promtool check rules` (alert rules — 11 rules found, valid), `amtool check-config`
(Alertmanager — 1 inhibit rule, 2 receivers, valid). These prove syntactic/semantic correctness
of the configs, not that the real network paths between rn-platform-01 and rn-lab-host-01 work —
that requires the real hardware and is covered by `docs/PRODUCTION_VALIDATION_CHECKLIST.md`.