diff --git a/README.md b/README.md index f8142f1..67b532b 100644 --- a/README.md +++ b/README.md @@ -46,30 +46,34 @@ It simulates real-world **SRE / Platform Engineering environments**, delivering: ### Prerequisites -- Linux server or VM -- Python 3 installed -- Prometheus installed -- Grafana installed -- Network access to homelab systems +* Linux server or VM +* Python 3 installed +* Prometheus installed +* Grafana installed +* Network access to homelab systems --- ### Run Service Health Check -bash python3 scripts/service-health-check.py +```bash +python3 scripts/service-health-check.py +``` --- ### Run Prometheus -bash prometheus --config.file=prometheus/prometheus.yml +```bash +prometheus --config.file=prometheus/prometheus.yml +``` --- ### Access Services -- Prometheus: http://localhost:9090 -- Grafana: http://localhost:3000 +* Prometheus: http://localhost:9090 +* Grafana: http://localhost:3000 --- @@ -110,15 +114,19 @@ flowchart TD ## 📸 Dashboards ### Infrastructure Overview + Infrastructure Dashboard ### K3s Cluster Dashboard + K3s Dashboard ### Proxmox HA Dashboard + Proxmox Dashboard ### Service Uptime Dashboard + Services Dashboard --- @@ -270,9 +278,9 @@ http://:3000 This observability stack is part of a larger ecosystem: -- Ced’s HomeLab → Infrastructure layer -- Ced’s Observability Stack → Metrics + monitoring layer -- Ced’s NOC → Visualization and operations layer +* Ced’s HomeLab → Infrastructure layer +* Ced’s Observability Stack → Metrics + monitoring layer +* Ced’s NOC → Visualization and operations layer Data flows from monitored systems into Prometheus, is visualized in Grafana, and feeds Ced’s NOC dashboard for real-time system visibility. @@ -284,36 +292,40 @@ To verify the system is working correctly: ### Prometheus Targets -- Navigate to: http://localhost:9090/targets -- Confirm all targets show UP +* Navigate to: http://localhost:9090/targets +* Confirm all targets show UP --- ### Node Exporter -bash curl http://:9100/metrics +```bash +curl http://:9100/metrics +``` --- ### Service Health Check -bash python3 scripts/service-health-check.py +```bash +python3 scripts/service-health-check.py +``` --- ### Grafana -- Confirm dashboards display real-time metrics -- Verify data source connection to Prometheus -- Check for active alerts +* Confirm dashboards display real-time metrics +* Verify data source connection to Prometheus +* Check for active alerts --- ### Alert Testing -- Stop a service or node temporarily -- Confirm alert triggers in Prometheus -- Confirm alert appears in Grafana +* Stop a service or node temporarily +* Confirm alert triggers in Prometheus +* Confirm alert appears in Grafana --- @@ -333,26 +345,26 @@ It is designed to demonstrate how distributed systems are monitored, analyzed, a Key capabilities include: -- Monitoring a hybrid infrastructure environment (Proxmox HA + K3s cluster) -- Collecting and visualizing system and service metrics -- Tracking service availability and uptime -- Detecting infrastructure and application-level failures -- Supporting alert-driven operations -- Integrating with a centralized NOC dashboard +* Monitoring a hybrid infrastructure environment (Proxmox HA + K3s cluster) +* Collecting and visualizing system and service metrics +* Tracking service availability and uptime +* Detecting infrastructure and application-level failures +* Supporting alert-driven operations +* Integrating with a centralized NOC dashboard -This project represents a shift from running infrastructure to actively operating and maintaining it using observability principles aligned with SRE and platform engineering practice +This project represents a shift from running infrastructure to actively operating and maintaining it using observability principles aligned with SRE and platform engineering practices. --- -## 🧠 Future Improvements +## 🧠 Future Improvements -- Loki log aggregation -- Tempo tracing -- Cloudflare Access log ingestion -- Automated remediation (self-healing infrastructure) -- Grafana public demo dashboard -- GitOps-based deployment (Argo CD / Flux) -- Multi-cluster Kubernetes monitoring +* Loki log aggregation +* Tempo tracing +* Cloudflare Access log ingestion +* Automated remediation (self-healing infrastructure) +* Grafana public demo dashboard +* GitOps-based deployment (Argo CD / Flux) +* Multi-cluster Kubernetes monitoring --- diff --git a/docs/blackbox-exporter-setup.md b/docs/blackbox-exporter-setup.md new file mode 100644 index 0000000..19bdedd --- /dev/null +++ b/docs/blackbox-exporter-setup.md @@ -0,0 +1,131 @@ +# Blackbox Exporter Setup + +## Purpose + +Blackbox Exporter is used to monitor endpoint availability from Prometheus. + +--- + +## Config Location + +/opt/blackbox/blackbox.yml + +--- + +## Setup + +```bash +sudo mkdir -p /opt/blackbox +sudo nano /opt/blackbox/blackbox.yml + +## Config File + +Create or edit: + +/opt/blackbox/blackbox.yml + +Paste: + + modules: + http_2xx: + prober: http + timeout: 10s + http: + method: GET + preferred_ip_protocol: ip4 + + http_2xx_insecure: + prober: http + timeout: 10s + http: + method: GET + preferred_ip_protocol: ip4 + tls_config: + insecure_skip_verify: true + + tcp_connect: + prober: tcp + timeout: 5s + + icmp: + prober: icmp + timeout: 5s + +--- + +## Remove Old Containers + +Run: + + docker rm -f blackbox + docker rm -f blackbox-exporter + +--- + +## Start Exporter + +Run: + + docker run -d \ + --name blackbox-exporter \ + --restart unless-stopped \ + -p 9115:9115 \ + -v /opt/blackbox/blackbox.yml:/config/blackbox.yml:ro \ + prom/blackbox-exporter:latest \ + --config.file=/config/blackbox.yml + +--- + +## Test + +Run: + + curl "http://10.10.30.140:9115/probe?target=http://10.10.30.68:3000&module=http_2xx_insecure" + +Expected output should include: + + probe_success 1 + +--- + +## Restart Prometheus + +Run: + + systemctl restart prometheus + +--- + +## Verify in Prometheus + +Open: + + http://10.10.30.140:9090/targets + +You should now see: + + blackbox-http-internal UP + +--- + +## Troubleshooting + +### Unknown module "http_2xx_insecure" + +Cause: +Blackbox Exporter is running with an old or incorrect config. + +Fix: +Restart the container after verifying the config file is mounted correctly. + +--- + +### Port already allocated + +Cause: +Another container is using port 9115. + +Fix: + + docker ps --format "table {{.Names}}\t{{.Ports}}" + docker rm -f \ No newline at end of file diff --git a/docs/phase-3-live-deployment.md b/docs/phase-3-live-deployment.md new file mode 100644 index 0000000..07967c1 --- /dev/null +++ b/docs/phase-3-live-deployment.md @@ -0,0 +1,60 @@ +# Phase 3 — Live Deployment Notes + +## Overview + +Phase 3 focused on moving Ced’s Observability Stack from documentation into a working live monitoring environment. + +The goal was to verify that Prometheus could collect real metrics from Ced’s HomeLab infrastructure and that Grafana could visualize those metrics through live dashboards. + +--- + +## Environment + +| System | Purpose | +|---|---| +| Prometheus CT | Metrics collection | +| Grafana CT | Dashboard visualization | +| Blackbox Exporter | Service availability checks | +| Node Exporter | Host-level metrics | +| Unpoller / UniFi Exporter | UniFi network metrics | +| TrueNAS Graphite Exporter | TrueNAS metrics | +| Proxmox HA Cluster | Virtualization platform | +| 12-node K3s Cluster | Kubernetes environment | + +--- + +## Completed Work + +- Verified Prometheus target health +- Repaired Blackbox Exporter configuration +- Added custom Blackbox modules +- Fixed HTTP probing for internal services +- Verified Proxmox node exporter targets +- Verified K3s node exporter targets +- Installed UniFi exporter using Unpoller +- Verified UniFi metrics on port 9130 +- Confirmed all Prometheus targets are UP +- Verified Grafana dashboards are receiving live data + +--- + +## Services Confirmed Working + +| Component | Status | +|---|---| +| Prometheus | UP | +| Blackbox Exporter | UP | +| Proxmox Node Exporters | UP | +| K3s Node Exporters | UP | +| Windows Exporter | UP | +| TrueNAS Graphite Exporter | UP | +| UniFi Exporter / Unpoller | UP | +| Grafana Dashboards | Partially complete and receiving live data | + +--- + +## Key Outcome + +At the end of this phase, Ced’s Observability Stack successfully collected metrics from multiple platforms across the homelab environment. + +This moved the project from a documentation-only repo into a real operating observability system. \ No newline at end of file