Add Phase 3 live observability deployment docs

README.md

@@ -46,30 +46,34 @@ It simulates real-world **SRE / Platform Engineering environments**, delivering:
 
 ### Prerequisites
 
-- Linux server or VM
+* Linux server or VM
-- Python 3 installed
+* Python 3 installed
-- Prometheus installed
+* Prometheus installed
-- Grafana installed
+* Grafana installed
-- Network access to homelab systems
+* Network access to homelab systems
 
 ---
 
 ### Run Service Health Check
 
-bash python3 scripts/service-health-check.py 
+```bash
+python3 scripts/service-health-check.py
+```
 
 ---
 
 ### Run Prometheus
 
-bash prometheus --config.file=prometheus/prometheus.yml 
+```bash
+prometheus --config.file=prometheus/prometheus.yml
+```
 
 ---
 
 ### Access Services
 
-- Prometheus: http://localhost:9090
+* Prometheus: http://localhost:9090
-- Grafana: http://localhost:3000
+* Grafana: http://localhost:3000
 
 ---
 
@@ -110,15 +114,19 @@ flowchart TD
 ## 📸 Dashboards
 
 ### Infrastructure Overview
+
 Infrastructure Dashboard
 
 ### K3s Cluster Dashboard
+
 K3s Dashboard
 
 ### Proxmox HA Dashboard
+
 Proxmox Dashboard
 
 ### Service Uptime Dashboard
+
 Services Dashboard
 
 ---
@@ -270,9 +278,9 @@ http://<your-server-ip>:3000
 
 This observability stack is part of a larger ecosystem:
 
-- Ced’s HomeLab → Infrastructure layer
+* Ced’s HomeLab → Infrastructure layer
-- Ced’s Observability Stack → Metrics + monitoring layer
+* Ced’s Observability Stack → Metrics + monitoring layer
-- Ced’s NOC → Visualization and operations layer
+* Ced’s NOC → Visualization and operations layer
 
 Data flows from monitored systems into Prometheus, is visualized in Grafana, and feeds Ced’s NOC dashboard for real-time system visibility.
 
@@ -284,36 +292,40 @@ To verify the system is working correctly:
 
 ### Prometheus Targets
 
-- Navigate to: http://localhost:9090/targets
+* Navigate to: http://localhost:9090/targets
-- Confirm all targets show UP
+* Confirm all targets show UP
 
 ---
 
 ### Node Exporter
 
-bash curl http://<node-ip>:9100/metrics 
+```bash
+curl http://<node-ip>:9100/metrics
+```
 
 ---
 
 ### Service Health Check
 
-bash python3 scripts/service-health-check.py 
+```bash
+python3 scripts/service-health-check.py
+```
 
 ---
 
 ### Grafana
 
-- Confirm dashboards display real-time metrics
+* Confirm dashboards display real-time metrics
-- Verify data source connection to Prometheus
+* Verify data source connection to Prometheus
-- Check for active alerts
+* Check for active alerts
 
 ---
 
 ### Alert Testing
 
-- Stop a service or node temporarily
+* Stop a service or node temporarily
-- Confirm alert triggers in Prometheus
+* Confirm alert triggers in Prometheus
-- Confirm alert appears in Grafana
+* Confirm alert appears in Grafana
 
 ---
 
@@ -333,26 +345,26 @@ It is designed to demonstrate how distributed systems are monitored, analyzed, a
 
 Key capabilities include:
 
-- Monitoring a hybrid infrastructure environment (Proxmox HA + K3s cluster)
+* Monitoring a hybrid infrastructure environment (Proxmox HA + K3s cluster)
-- Collecting and visualizing system and service metrics
+* Collecting and visualizing system and service metrics
-- Tracking service availability and uptime
+* Tracking service availability and uptime
-- Detecting infrastructure and application-level failures
+* Detecting infrastructure and application-level failures
-- Supporting alert-driven operations
+* Supporting alert-driven operations
-- Integrating with a centralized NOC dashboard
+* Integrating with a centralized NOC dashboard
 
-This project represents a shift from running infrastructure to actively operating and maintaining it using observability principles aligned with SRE and platform engineering practice
+This project represents a shift from running infrastructure to actively operating and maintaining it using observability principles aligned with SRE and platform engineering practices.
 
 ---
 
 ## 🧠 Future Improvements
 
-- Loki log aggregation
+* Loki log aggregation
-- Tempo tracing
+* Tempo tracing
-- Cloudflare Access log ingestion
+* Cloudflare Access log ingestion
-- Automated remediation (self-healing infrastructure)
+* Automated remediation (self-healing infrastructure)
-- Grafana public demo dashboard
+* Grafana public demo dashboard
-- GitOps-based deployment (Argo CD / Flux)
+* GitOps-based deployment (Argo CD / Flux)
-- Multi-cluster Kubernetes monitoring
+* Multi-cluster Kubernetes monitoring
 
 ---
 

docs/blackbox-exporter-setup.md

@@ -0,0 +1,131 @@
+# Blackbox Exporter Setup
+
+## Purpose
+
+Blackbox Exporter is used to monitor endpoint availability from Prometheus.
+
+---
+
+## Config Location
+
+/opt/blackbox/blackbox.yml
+
+---
+
+## Setup
+
+```bash
+sudo mkdir -p /opt/blackbox
+sudo nano /opt/blackbox/blackbox.yml
+
+## Config File
+
+Create or edit:
+
+/opt/blackbox/blackbox.yml
+
+Paste:
+
+    modules:
+      http_2xx:
+        prober: http
+        timeout: 10s
+        http:
+          method: GET
+          preferred_ip_protocol: ip4
+
+      http_2xx_insecure:
+        prober: http
+        timeout: 10s
+        http:
+          method: GET
+          preferred_ip_protocol: ip4
+          tls_config:
+            insecure_skip_verify: true
+
+      tcp_connect:
+        prober: tcp
+        timeout: 5s
+
+      icmp:
+        prober: icmp
+        timeout: 5s
+
+---
+
+## Remove Old Containers
+
+Run:
+
+    docker rm -f blackbox
+    docker rm -f blackbox-exporter
+
+---
+
+## Start Exporter
+
+Run:
+
+    docker run -d \
+      --name blackbox-exporter \
+      --restart unless-stopped \
+      -p 9115:9115 \
+      -v /opt/blackbox/blackbox.yml:/config/blackbox.yml:ro \
+      prom/blackbox-exporter:latest \
+      --config.file=/config/blackbox.yml
+
+---
+
+## Test
+
+Run:
+
+    curl "http://10.10.30.140:9115/probe?target=http://10.10.30.68:3000&module=http_2xx_insecure"
+
+Expected output should include:
+
+    probe_success 1
+
+---
+
+## Restart Prometheus
+
+Run:
+
+    systemctl restart prometheus
+
+---
+
+## Verify in Prometheus
+
+Open:
+
+    http://10.10.30.140:9090/targets
+
+You should now see:
+
+    blackbox-http-internal    UP
+
+---
+
+## Troubleshooting
+
+### Unknown module "http_2xx_insecure"
+
+Cause:
+Blackbox Exporter is running with an old or incorrect config.
+
+Fix:
+Restart the container after verifying the config file is mounted correctly.
+
+---
+
+### Port already allocated
+
+Cause:
+Another container is using port 9115.
+
+Fix:
+
+    docker ps --format "table {{.Names}}\t{{.Ports}}"
+    docker rm -f <container-name>

docs/phase-3-live-deployment.md

@@ -0,0 +1,60 @@
+# Phase 3 — Live Deployment Notes
+
+## Overview
+
+Phase 3 focused on moving Ced’s Observability Stack from documentation into a working live monitoring environment.
+
+The goal was to verify that Prometheus could collect real metrics from Ced’s HomeLab infrastructure and that Grafana could visualize those metrics through live dashboards.
+
+---
+
+## Environment
+
+| System | Purpose |
+|---|---|
+| Prometheus CT | Metrics collection |
+| Grafana CT | Dashboard visualization |
+| Blackbox Exporter | Service availability checks |
+| Node Exporter | Host-level metrics |
+| Unpoller / UniFi Exporter | UniFi network metrics |
+| TrueNAS Graphite Exporter | TrueNAS metrics |
+| Proxmox HA Cluster | Virtualization platform |
+| 12-node K3s Cluster | Kubernetes environment |
+
+---
+
+## Completed Work
+
+- Verified Prometheus target health
+- Repaired Blackbox Exporter configuration
+- Added custom Blackbox modules
+- Fixed HTTP probing for internal services
+- Verified Proxmox node exporter targets
+- Verified K3s node exporter targets
+- Installed UniFi exporter using Unpoller
+- Verified UniFi metrics on port 9130
+- Confirmed all Prometheus targets are UP
+- Verified Grafana dashboards are receiving live data
+
+---
+
+## Services Confirmed Working
+
+| Component | Status |
+|---|---|
+| Prometheus | UP |
+| Blackbox Exporter | UP |
+| Proxmox Node Exporters | UP |
+| K3s Node Exporters | UP |
+| Windows Exporter | UP |
+| TrueNAS Graphite Exporter | UP |
+| UniFi Exporter / Unpoller | UP |
+| Grafana Dashboards | Partially complete and receiving live data |
+
+---
+
+## Key Outcome
+
+At the end of this phase, Ced’s Observability Stack successfully collected metrics from multiple platforms across the homelab environment.
+
+This moved the project from a documentation-only repo into a real operating observability system.