Add Phase 3 live observability deployment docs

2026-08-12 22:34:03 +00:00 · 2026-04-29 23:23:29 -05:00
parent 37037cafe6
commit c4d1fefe6c
3 changed files with 240 additions and 37 deletions
@@ -0,0 +1,131 @@
+# Blackbox Exporter Setup
+
+## Purpose
+
+Blackbox Exporter is used to monitor endpoint availability from Prometheus.
+
+---
+
+## Config Location
+
+/opt/blackbox/blackbox.yml
+
+---
+
+## Setup
+
+```bash
+sudo mkdir -p /opt/blackbox
+sudo nano /opt/blackbox/blackbox.yml
+
+## Config File
+
+Create or edit:
+
+/opt/blackbox/blackbox.yml
+
+Paste:
+
+    modules:
+      http_2xx:
+        prober: http
+        timeout: 10s
+        http:
+          method: GET
+          preferred_ip_protocol: ip4
+
+      http_2xx_insecure:
+        prober: http
+        timeout: 10s
+        http:
+          method: GET
+          preferred_ip_protocol: ip4
+          tls_config:
+            insecure_skip_verify: true
+
+      tcp_connect:
+        prober: tcp
+        timeout: 5s
+
+      icmp:
+        prober: icmp
+        timeout: 5s
+
+---
+
+## Remove Old Containers
+
+Run:
+
+    docker rm -f blackbox
+    docker rm -f blackbox-exporter
+
+---
+
+## Start Exporter
+
+Run:
+
+    docker run -d \
+      --name blackbox-exporter \
+      --restart unless-stopped \
+      -p 9115:9115 \
+      -v /opt/blackbox/blackbox.yml:/config/blackbox.yml:ro \
+      prom/blackbox-exporter:latest \
+      --config.file=/config/blackbox.yml
+
+---
+
+## Test
+
+Run:
+
+    curl "http://10.10.30.140:9115/probe?target=http://10.10.30.68:3000&module=http_2xx_insecure"
+
+Expected output should include:
+
+    probe_success 1
+
+---
+
+## Restart Prometheus
+
+Run:
+
+    systemctl restart prometheus
+
+---
+
+## Verify in Prometheus
+
+Open:
+
+    http://10.10.30.140:9090/targets
+
+You should now see:
+
+    blackbox-http-internal    UP
+
+---
+
+## Troubleshooting
+
+### Unknown module "http_2xx_insecure"
+
+Cause:
+Blackbox Exporter is running with an old or incorrect config.
+
+Fix:
+Restart the container after verifying the config file is mounted correctly.
+
+---
+
+### Port already allocated
+
+Cause:
+Another container is using port 9115.
+
+Fix:
+
+    docker ps --format "table {{.Names}}\t{{.Ports}}"
+    docker rm -f <container-name>
@@ -0,0 +1,60 @@
+# Phase 3 — Live Deployment Notes
+
+## Overview
+
+Phase 3 focused on moving Ced’s Observability Stack from documentation into a working live monitoring environment.
+
+The goal was to verify that Prometheus could collect real metrics from Ced’s HomeLab infrastructure and that Grafana could visualize those metrics through live dashboards.
+
+---
+
+## Environment
+
+| System | Purpose |
+|---|---|
+| Prometheus CT | Metrics collection |
+| Grafana CT | Dashboard visualization |
+| Blackbox Exporter | Service availability checks |
+| Node Exporter | Host-level metrics |
+| Unpoller / UniFi Exporter | UniFi network metrics |
+| TrueNAS Graphite Exporter | TrueNAS metrics |
+| Proxmox HA Cluster | Virtualization platform |
+| 12-node K3s Cluster | Kubernetes environment |
+
+---
+
+## Completed Work
+
+- Verified Prometheus target health
+- Repaired Blackbox Exporter configuration
+- Added custom Blackbox modules
+- Fixed HTTP probing for internal services
+- Verified Proxmox node exporter targets
+- Verified K3s node exporter targets
+- Installed UniFi exporter using Unpoller
+- Verified UniFi metrics on port 9130
+- Confirmed all Prometheus targets are UP
+- Verified Grafana dashboards are receiving live data
+
+---
+
+## Services Confirmed Working
+
+| Component | Status |
+|---|---|
+| Prometheus | UP |
+| Blackbox Exporter | UP |
+| Proxmox Node Exporters | UP |
+| K3s Node Exporters | UP |
+| Windows Exporter | UP |
+| TrueNAS Graphite Exporter | UP |
+| UniFi Exporter / Unpoller | UP |
+| Grafana Dashboards | Partially complete and receiving live data |
+
+---
+
+## Key Outcome
+
+At the end of this phase, Ced’s Observability Stack successfully collected metrics from multiple platforms across the homelab environment.
+
+This moved the project from a documentation-only repo into a real operating observability system.