From b291126015553b768bbc670a607a687c7e2cc4af Mon Sep 17 00:00:00 2001 From: Chase Dumphord Date: Wed, 6 May 2026 02:28:34 -0500 Subject: [PATCH] docs: complete README rewrite with live dashboard documentation --- README.md | 609 ++++++++++++++++++++---------------------------------- 1 file changed, 224 insertions(+), 385 deletions(-) diff --git a/README.md b/README.md index ad20752..a2822d3 100644 --- a/README.md +++ b/README.md @@ -1,412 +1,251 @@ -# πŸš€ Ced’s Observability Stack +# Ced's Observability Stack β€” Production Monitoring Platform -![Status](https://img.shields.io/badge/Status-Active%20Development-blue) -![Platform](https://img.shields.io/badge/Platform-Proxmox%20%7C%20K3s-orange) -![Monitoring](https://img.shields.io/badge/Stack-Prometheus%20%7C%20Grafana-red) -![Alerting](https://img.shields.io/badge/Alerting-Alertmanager-yellow) -![License](https://img.shields.io/badge/License-MIT-green) +> A production-style observability platform delivering full-stack visibility across a hybrid homelab infrastructure β€” Proxmox HA cluster, 12-node K3s cluster, network gear, storage, and edge systems. Built on Prometheus, Grafana, and Alertmanager with live dashboards running 24/7. + +[![NOC Health](https://img.shields.io/badge/NOC%20Health-100%25-1D9E75?style=flat-square)](#production-dashboards) +[![Grafana](https://img.shields.io/badge/Grafana-Live-F46800?style=flat-square)](https://grafana.cedshomelab.com) +[![Stack](https://img.shields.io/badge/Stack-Prometheus%20%7C%20Grafana%20%7C%20Alertmanager-326CE5?style=flat-square)](#stack) +[![Live NOC](https://img.shields.io/badge/Live%20NOC-noc.chasedumphord.com-085041?style=flat-square)](https://noc.chasedumphord.com) +[![Portfolio](https://img.shields.io/badge/Portfolio-chasedumphord.com-0F6E56?style=flat-square)](https://chasedumphord.com) --- -## 🧠 Executive Summary +## What This Is -**Ced’s Observability Stack** is a production-style monitoring, metrics, and alerting platform built to provide full visibility into a distributed hybrid infrastructure. +This is the observability layer for Ced's HomeLab β€” the system that answers the question: *"Is everything actually working?"* -It simulates real-world **SRE / Platform Engineering environments**, delivering: +It collects metrics from every layer of the infrastructure stack, visualizes them in purpose-built Grafana dashboards, and routes alerts through Alertmanager when something breaks. Three production dashboards serve different operational needs β€” an executive NOC view for quick status checks, a deep drill-down dashboard for troubleshooting, and a dedicated K3s cluster dashboard for Kubernetes visibility. -* πŸ“Š Real-time infrastructure monitoring -* βš™οΈ Kubernetes observability (12-node K3s cluster) -* πŸ–₯️ Proxmox HA cluster visibility -* 🌐 Service uptime + network health tracking -* 🚨 Alerting pipelines (Alertmanager) -* πŸ“ˆ Operational dashboards (Grafana) - -> 🎯 **Goal:** Demonstrate enterprise-level observability practices across virtualization, Kubernetes, and self-hosted infrastructure. +This isn't a demo. Prometheus is actively scraping targets. Grafana is displaying live data. The dashboards have been through real incidents and refined based on what actually matters during an outage. --- -## πŸ—οΈ Environment Overview - -### Core Infrastructure - -| System | Purpose | -| ------------------------ | ---------------------------------- | -| πŸ–₯️ Proxmox HA Cluster | Virtualization & high availability | -| ☸️ K3s Cluster (12-node) | Container orchestration | -| πŸ’Ύ TrueNAS | Storage services | -| 🌐 Nginx Proxy Manager | Reverse proxy & routing | -| ☁️ Cloudflare | DNS, tunnels, external protection | -| πŸ“Š Grafana | Visualization dashboards | -| πŸ“‘ Prometheus | Metrics collection | -| 🚨 Alertmanager | Alert routing | - ---- - -## πŸš€ Quick Start - -### Prerequisites - -* Linux server or VM -* Python 3 installed -* Prometheus installed -* Grafana installed -* Network access to homelab systems - ---- - -### Run Service Health Check - -```bash -python3 scripts/service-health-check.py -``` - ---- - -### Run Prometheus - -```bash -prometheus --config.file=prometheus/prometheus.yml -``` - ---- - -### Access Services - -* Prometheus: http://localhost:9090 -* Grafana: http://localhost:3000 - ---- - -## πŸ“‘ Monitored Systems - -| Target | Example Metrics | -| ----------------------- | ------------------------------------ | -| πŸ–₯️ Proxmox Nodes | CPU, memory, storage, VM + HA status | -| ☸️ K3s Nodes | Node readiness, resource usage | -| πŸ“¦ Kubernetes Workloads | Pods, deployments, restarts | -| 🌐 Network Services | Uptime, latency, TCP checks | -| πŸ’Ύ TrueNAS | Storage + service availability | -| πŸ”€ Nginx Proxy Manager | Reverse proxy health | -| πŸ“Š Dashy / NOC | Dashboard availability | -| 🎬 Jellyfin | Media service uptime | - ---- - -## 🧩 Architecture +## Architecture ```mermaid flowchart TD - A[Proxmox HA Cluster] --> P[Prometheus] - B[12-Node K3s Cluster] --> P - C[Node Exporters] --> P - D[Service Health Checks] --> P - E[Proxmox Exporter] --> P + subgraph Sources["Metric Sources"] + PVE[Proxmox HA Cluster
6 Nodes] + K3S[K3s Cluster
12 Nodes] + NODE[Node Exporter
All Hosts] + KSM[kube-state-metrics] + WIN[Windows Exporter
PrimeStation] + BB[Blackbox Exporter
HTTP/TCP Probes] + TN[TrueNAS
Graphite Exporter] + UNI[UniFi / Unpoller
Network Metrics] + end - P --> G[Grafana Dashboards] - P --> AM[Alertmanager] + subgraph Core["Observability Core"] + PROM[Prometheus] + AM[Alertmanager] + end - AM --> N[Email / Discord / Slack Alerts] - G --> NOC[Ced's NOC Dashboard] + subgraph Dashboards["Grafana Dashboards"] + CMD[Production Command Center v3
Executive NOC View] + DEEP[Deep Observability v3
Full Drill-Down] + K3SD[K3s Elite Observability v1
Kubernetes Focus] + end + + subgraph Output["Operations"] + NOC[Ced's NOC
noc.chasedumphord.com] + ALERT[Alert Notifications] + end + + PVE & K3S & NODE & KSM & WIN & BB & TN & UNI --> PROM + PROM --> AM + PROM --> CMD & DEEP & K3SD + CMD --> NOC + AM --> ALERT ``` --- -## πŸ“Έ Dashboards - -### Infrastructure Overview - -Infrastructure Dashboard - -### K3s Cluster Dashboard - -K3s Dashboard - -### Proxmox HA Dashboard - -Proxmox Dashboard - -### Service Uptime Dashboard - -Services Dashboard - ---- - -## βš™οΈ Core Components - -### πŸ“‘ Prometheus - -Collects metrics from: - -* Kubernetes endpoints -* Node exporters -* Proxmox exporter -* Custom health scripts -* Static service targets - ---- - -### πŸ“Š Grafana - -Provides dashboards for: - -* Cluster health -* Resource utilization -* Storage trends -* Service uptime -* Alert visibility - ---- - -### 🚨 Alertmanager - -Handles alerting for: - -* Node failures -* High CPU / memory -* Service outages -* Pod crash loops -* Proxmox HA issues - ---- - -## πŸ“ Repo Structure - -``` -ceds-observability-stack/ -β”œβ”€β”€ architecture/ -β”œβ”€β”€ prometheus/ -β”œβ”€β”€ grafana/ -β”œβ”€β”€ exporters/ -β”œβ”€β”€ alerting/ -β”œβ”€β”€ scripts/ -└── docs/ -``` - ---- - -## πŸ“Έ Dashboard Preview - -* πŸ”Ή Infrastructure Overview -* πŸ”Ή K3s Cluster Health -* πŸ”Ή Proxmox Cluster Status -* πŸ”Ή Service Uptime Dashboard - ---- - -## πŸš€ Deployment (High-Level) - -```bash -# Clone repo -git clone https://github.com/ced4568/ceds-observability-stack.git - -# Navigate to project -cd ceds-observability-stack - -# Deploy Prometheus + exporters -# (Add your actual deployment steps here) - -# Access Grafana -http://:3000 -``` - ---- - -## 🎯 Project Roadmap - -### Phase 1 β€” Foundation - -* [x] Architecture design -* [x] Repo structure -* [ ] Prometheus base config -* [ ] Grafana datasource - -### Phase 2 β€” Metrics Collection - -* [ ] Node exporter -* [ ] K3s metrics -* [ ] Proxmox exporter -* [ ] Uptime checks - -### Phase 3 β€” Dashboards - -* [ ] Infrastructure dashboard -* [ ] K3s dashboard -* [ ] Proxmox dashboard -* [ ] Service uptime dashboard - -### Phase 4 β€” Alerting - -* [ ] Alertmanager setup -* [ ] Alert rules -* [ ] Notification testing - -### Phase 5 β€” Portfolio Polish - -* [ ] Screenshots -* [ ] Architecture diagrams -* [ ] Setup guide -* [ ] Troubleshooting docs - ---- - -## 🧠 Skills Demonstrated - -* πŸ“Š Infrastructure Monitoring -* ☸️ Kubernetes Operations -* πŸ“‘ Prometheus Configuration -* πŸ“ˆ Grafana Dashboarding -* 🚨 Alert Engineering -* 🐧 Linux Administration -* πŸ–₯️ Proxmox Virtualization -* βš™οΈ SRE Principles -* πŸ—οΈ Platform Engineering - ---- - -## πŸ”— Related Projects - -| Project | Purpose | -| ----------------- | --------------------------------- | -| Ced’s HomeLab | Full infrastructure ecosystem | -| Ced’s NOC | Visualization + status dashboards | -| Ced’s K3s HomeLab | Kubernetes architecture | -| Ced’s APRS iGate | Networking + RF integration | - ---- - -## πŸ”— Integration - -This observability stack is part of a larger ecosystem: - -* Ced’s HomeLab β†’ Infrastructure layer -* Ced’s Observability Stack β†’ Metrics + monitoring layer -* Ced’s NOC β†’ Visualization and operations layer - -Data flows from monitored systems into Prometheus, is visualized in Grafana, and feeds Ced’s NOC dashboard for real-time system visibility. - ---- - -## πŸ§ͺ Verification - -To verify the system is working correctly: - -### Prometheus Targets - -* Navigate to: http://localhost:9090/targets -* Confirm all targets show UP - ---- - -### Node Exporter - -```bash -curl http://:9100/metrics -``` - ---- - -### Service Health Check - -```bash -python3 scripts/service-health-check.py -``` - ---- - -### Grafana - -* Confirm dashboards display real-time metrics -* Verify data source connection to Prometheus -* Check for active alerts - ---- - -### Alert Testing - -* Stop a service or node temporarily -* Confirm alert triggers in Prometheus -* Confirm alert appears in Grafana - ---- - -## πŸ“Œ Status - -🟒 **Active Development** - -This project is continuously evolving as part of Ced’s HomeLab ecosystem and professional portfolio. - ---- - -## πŸ’‘ Why This Project Matters - -This project simulates a production-style observability system used in modern infrastructure environments. - -It is designed to demonstrate how distributed systems are monitored, analyzed, and maintained in real-world engineering teams. - -Key capabilities include: - -* Monitoring a hybrid infrastructure environment (Proxmox HA + K3s cluster) -* Collecting and visualizing system and service metrics -* Tracking service availability and uptime -* Detecting infrastructure and application-level failures -* Supporting alert-driven operations -* Integrating with a centralized NOC dashboard - -This project represents a shift from running infrastructure to actively operating and maintaining it using observability principles aligned with SRE and platform engineering practices. - ---- - -## 🧠 Future Improvements - -* Loki log aggregation -* Tempo tracing -* Cloudflare Access log ingestion -* Automated remediation (self-healing infrastructure) -* Grafana public demo dashboard -* GitOps-based deployment (Argo CD / Flux) -* Multi-cluster Kubernetes monitoring - ---- - -## Current Deployment Status - -Ced’s Observability Stack is now actively collecting live metrics from Ced’s HomeLab. - -Current working components: - -- Prometheus metrics collection -- Grafana dashboard visualization -- Proxmox node exporter monitoring -- K3s node exporter monitoring -- kube-state-metrics for Kubernetes object state -- Windows exporter for PrimeStation -- Blackbox HTTP/TCP endpoint probing -- TrueNAS Graphite exporter -- UniFi exporter using Unpoller - -Some Grafana panels are being refined as dashboard queries are aligned with available Prometheus metrics. - ---- - ## Production Dashboards -| Dashboard | Purpose | -|---|---| -| Ced's NOC - Production Command Center v3 | Stable executive NOC view for service availability, K3s, Proxmox, PrimeStation, and latency | -| Ced's NOC - Deep Observability v3 | Full drill-down dashboard for Proxmox, K3s, services, UniFi, PrimeStation, and alerts | -| Ced's K3s Elite Observability v1 | Dedicated K3s dashboard using node-exporter and kube-state-metrics | +Three purpose-built dashboards serving different operational needs. All live at `grafana.cedshomelab.com`. + +### Production Command Center v3 + +Executive NOC view for quick infrastructure status checks. Designed for portfolio demonstrations, interviews, and daily operational awareness. + +![Production Command Center](./screenshots/production-command-center-v3.png) + +**What it shows:** +- NOC Health Score β€” single number representing overall infrastructure health +- Core Services UP / DOWN counters +- Average service latency with trend graph +- PrimeStation online status +- Core Service Availability (UP/DOWN) +- Service Latency Trend over time +- K3s Nodes Online / Proxmox Nodes Online +- Per-service status tiles: TrueNAS, NPM, Jellyfin, Dashy, Home Assistant, Grafana, Prometheus, Proxmox UI + +**Built for:** Portfolio presentations, interview demos, daily ops check --- -## Phase 3 Live Metrics Milestone +### Deep Observability v3 -- [x] Prometheus running -- [x] Grafana connected to Prometheus +Full drill-down dashboard for active troubleshooting and infrastructure analysis. Covers every layer of the stack in one view. + +![Deep Observability](./screenshots/deep-observability-v3.png) + +**What it shows:** +- Prometheus target health summary +- Proxmox HA cluster metrics β€” nodes, storage, VM status +- K3s cluster health β€” nodes, pods, deployments +- Windows/Network/Backup section β€” PrimeStation, UniFi, TrueNAS +- HTTP/TCP probe results via Blackbox Exporter +- HTTPS response time trends + +**Built for:** Active incident response, performance analysis, infrastructure troubleshooting + +--- + +### K3s Elite Observability v1 + +Focused Kubernetes dashboard using node-exporter and kube-state-metrics for deep cluster visibility. + +![K3s Elite Observability](./screenshots/k3s-elite-observability-v1.png) + +**What it shows:** +- Cluster Summary: Nodes Online, Nodes Ready, Pods Running, Pods Pending, Pods Failed, Restart count +- Node CPU usage per node +- Node Memory usage with historical trend +- Pod Phase by Namespace +- Container Restarts by Pod +- Pods Not Running +- Nodes Not Ready +- Node Load average +- Node Disk Usage + +**Built for:** Kubernetes operations, cluster health monitoring, capacity planning + +--- + +## Full Exporter Stack + +Every metric source in the infrastructure is actively scraped by Prometheus. + +| Exporter | Target | Metrics | +|----------|--------|---------| +| Node Exporter | All Proxmox + K3s nodes | CPU, RAM, disk, network per host | +| kube-state-metrics | K3s cluster | Pod state, deployment health, replica counts | +| Proxmox Exporter | Proxmox HA cluster | Node status, VM health, HA state | +| Windows Exporter | PrimeStation | CPU, RAM, disk, network for main workstation | +| Blackbox Exporter | HTTP/TCP endpoints | Service uptime, response time, probe results | +| TrueNAS Graphite Exporter | TrueNAS | Storage pool health, dataset usage | +| Unpoller (UniFi Exporter) | UniFi Dream Router | Network device metrics, client counts, throughput | +| metrics-server | K3s | Real-time resource usage for kubectl top | + +--- + +## Infrastructure Coverage + +| System | Monitoring Status | +|--------|------------------| +| Proxmox HA Cluster (6 nodes) | βœ… Live β€” node exporter + Proxmox exporter | +| K3s Cluster (12 nodes) | βœ… Live β€” node exporter + kube-state-metrics | +| TrueNAS | βœ… Live β€” Graphite exporter | +| Nginx Proxy Manager | βœ… Live β€” Blackbox HTTP probe | +| Home Assistant | βœ… Live β€” Blackbox HTTP probe | +| Dashy | βœ… Live β€” Blackbox HTTP probe | +| Jellyfin | βœ… Live β€” Blackbox HTTP probe | +| UniFi Dream Router | βœ… Live β€” Unpoller exporter | +| PrimeStation (Windows) | βœ… Live β€” Windows exporter | +| Grafana | βœ… Live β€” self-monitored | +| Prometheus | βœ… Live β€” self-monitored | +| Public endpoints | βœ… Live β€” Blackbox external probes | + +--- + +## Repository Structure + +``` +ceds-observability-stack/ +β”œβ”€β”€ architecture/ # Architecture diagrams +β”œβ”€β”€ prometheus/ +β”‚ └── prometheus.yml # Scrape configs and target definitions +β”œβ”€β”€ grafana/ +β”‚ └── dashboards/ # Dashboard JSON exports +β”œβ”€β”€ exporters/ # Exporter configs (node, blackbox, unpoller) +β”œβ”€β”€ alerting/ # Alertmanager config and alert rules +β”œβ”€β”€ scripts/ +β”‚ └── service-health-check.py +└── docs/ # Setup guides and notes +``` + +--- + +## Quick Start + +**Prerequisites:** Linux server or VM, Prometheus, Grafana, network access to homelab systems. + +```bash +# Run Prometheus with config +prometheus --config.file=prometheus/prometheus.yml + +# Verify targets are up +# Navigate to: http://localhost:9090/targets + +# Run service health check script +python3 scripts/service-health-check.py +``` + +**Access:** +- Prometheus: `http://:9090` +- Grafana: `http://:3000` +- Live (external): `https://grafana.cedshomelab.com` + +--- + +## Roadmap + +### Completed +- [x] Prometheus running and scraping all targets +- [x] Grafana connected to Prometheus datasource - [x] Proxmox node exporters reporting -- [x] K3s node exporters reporting +- [x] K3s node exporters reporting across all 12 nodes - [x] kube-state-metrics installed and reporting -- [x] Windows exporter reporting -- [x] Blackbox Exporter repaired -- [x] Internal HTTP probes working -- [x] UniFi exporter installed -- [x] Grafana dashboards receiving live data +- [x] Windows Exporter on PrimeStation +- [x] Blackbox Exporter β€” internal HTTP/TCP probing +- [x] UniFi Exporter via Unpoller +- [x] TrueNAS Graphite Exporter +- [x] Production Command Center v3 β€” live +- [x] Deep Observability v3 β€” live +- [x] K3s Elite Observability v1 β€” live + +### In Progress +- [ ] Alertmanager alert rules library +- [ ] Alert notification channels (email / Discord) +- [ ] Loki log aggregation +- [ ] Grafana public demo dashboard +- [ ] Tempo distributed tracing +- [ ] GitOps deployment via ArgoCD +- [ ] Automated remediation (self-healing infrastructure) +- [ ] Cloudflare Access log ingestion +- [ ] Multi-cluster Kubernetes monitoring + +--- + +## Related Projects + +| Project | Role in Stack | +|---------|--------------| +| [ceds-homelab](https://github.com/ced4568/ceds-homelab) | Infrastructure layer β€” Proxmox, TrueNAS, networking | +| [ced-k3s-homelab](https://github.com/ced4568/ced-k3s-homelab) | Orchestration layer β€” 12-node K3s cluster | +| [ceds-aprs-igate](https://github.com/ced4568/ceds-aprs-igate) | Edge layer β€” RF ingestion nodes | +| [ced-portfolio](https://github.com/ced4568/ced-portfolio) | Portfolio β€” chasedumphord.com | + +--- + +## Author + +**Chase Dumphord (Ced)** +Digital Systems Engineer Β· GE Aerospace Β· Oxford, MS + +[![Portfolio](https://img.shields.io/badge/Portfolio-chasedumphord.com-0F6E56?style=flat-square)](https://chasedumphord.com) +[![LinkedIn](https://img.shields.io/badge/LinkedIn-chase--dumphord-0A66C2?style=flat-square)](https://www.linkedin.com/in/chase-dumphord/) +[![GitHub](https://img.shields.io/badge/GitHub-ced4568-181717?style=flat-square)](https://github.com/ced4568) +[![Grafana](https://img.shields.io/badge/Grafana-Live%20%28Login%20Required%29-F46800?style=flat-square)](https://grafana.cedshomelab.com) +[![Live NOC](https://img.shields.io/badge/NOC-noc.chasedumphord.com-1D9E75?style=flat-square)](https://noc.chasedumphord.com) \ No newline at end of file