mirror of
https://github.com/ced4568/ceds-observability-stack.git
synced 2026-08-12 22:34:03 +00:00
docs: complete README rewrite with live dashboard documentation
This commit is contained in:
@@ -1,412 +1,251 @@
|
|||||||
# 🚀 Ced’s Observability Stack
|
# Ced's Observability Stack — Production Monitoring Platform
|
||||||
|
|
||||||
|
> A production-style observability platform delivering full-stack visibility across a hybrid homelab infrastructure — Proxmox HA cluster, 12-node K3s cluster, network gear, storage, and edge systems. Built on Prometheus, Grafana, and Alertmanager with live dashboards running 24/7.
|
||||||
|
|
||||||
|
[](#production-dashboards)
|
||||||
|
[](https://grafana.cedshomelab.com)
|
||||||
|
[](#stack)
|
||||||
|
[](https://noc.chasedumphord.com)
|
||||||
|
[](https://chasedumphord.com)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🧠 Executive Summary
|
## What This Is
|
||||||
|
|
||||||
**Ced’s Observability Stack** is a production-style monitoring, metrics, and alerting platform built to provide full visibility into a distributed hybrid infrastructure.
|
This is the observability layer for Ced's HomeLab — the system that answers the question: *"Is everything actually working?"*
|
||||||
|
|
||||||
It simulates real-world **SRE / Platform Engineering environments**, delivering:
|
It collects metrics from every layer of the infrastructure stack, visualizes them in purpose-built Grafana dashboards, and routes alerts through Alertmanager when something breaks. Three production dashboards serve different operational needs — an executive NOC view for quick status checks, a deep drill-down dashboard for troubleshooting, and a dedicated K3s cluster dashboard for Kubernetes visibility.
|
||||||
|
|
||||||
* 📊 Real-time infrastructure monitoring
|
This isn't a demo. Prometheus is actively scraping targets. Grafana is displaying live data. The dashboards have been through real incidents and refined based on what actually matters during an outage.
|
||||||
* ⚙️ Kubernetes observability (12-node K3s cluster)
|
|
||||||
* 🖥️ Proxmox HA cluster visibility
|
|
||||||
* 🌐 Service uptime + network health tracking
|
|
||||||
* 🚨 Alerting pipelines (Alertmanager)
|
|
||||||
* 📈 Operational dashboards (Grafana)
|
|
||||||
|
|
||||||
> 🎯 **Goal:** Demonstrate enterprise-level observability practices across virtualization, Kubernetes, and self-hosted infrastructure.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🏗️ Environment Overview
|
## Architecture
|
||||||
|
|
||||||
### Core Infrastructure
|
|
||||||
|
|
||||||
| System | Purpose |
|
|
||||||
| ------------------------ | ---------------------------------- |
|
|
||||||
| 🖥️ Proxmox HA Cluster | Virtualization & high availability |
|
|
||||||
| ☸️ K3s Cluster (12-node) | Container orchestration |
|
|
||||||
| 💾 TrueNAS | Storage services |
|
|
||||||
| 🌐 Nginx Proxy Manager | Reverse proxy & routing |
|
|
||||||
| ☁️ Cloudflare | DNS, tunnels, external protection |
|
|
||||||
| 📊 Grafana | Visualization dashboards |
|
|
||||||
| 📡 Prometheus | Metrics collection |
|
|
||||||
| 🚨 Alertmanager | Alert routing |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🚀 Quick Start
|
|
||||||
|
|
||||||
### Prerequisites
|
|
||||||
|
|
||||||
* Linux server or VM
|
|
||||||
* Python 3 installed
|
|
||||||
* Prometheus installed
|
|
||||||
* Grafana installed
|
|
||||||
* Network access to homelab systems
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Run Service Health Check
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python3 scripts/service-health-check.py
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Run Prometheus
|
|
||||||
|
|
||||||
```bash
|
|
||||||
prometheus --config.file=prometheus/prometheus.yml
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Access Services
|
|
||||||
|
|
||||||
* Prometheus: http://localhost:9090
|
|
||||||
* Grafana: http://localhost:3000
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📡 Monitored Systems
|
|
||||||
|
|
||||||
| Target | Example Metrics |
|
|
||||||
| ----------------------- | ------------------------------------ |
|
|
||||||
| 🖥️ Proxmox Nodes | CPU, memory, storage, VM + HA status |
|
|
||||||
| ☸️ K3s Nodes | Node readiness, resource usage |
|
|
||||||
| 📦 Kubernetes Workloads | Pods, deployments, restarts |
|
|
||||||
| 🌐 Network Services | Uptime, latency, TCP checks |
|
|
||||||
| 💾 TrueNAS | Storage + service availability |
|
|
||||||
| 🔀 Nginx Proxy Manager | Reverse proxy health |
|
|
||||||
| 📊 Dashy / NOC | Dashboard availability |
|
|
||||||
| 🎬 Jellyfin | Media service uptime |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🧩 Architecture
|
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
flowchart TD
|
flowchart TD
|
||||||
A[Proxmox HA Cluster] --> P[Prometheus]
|
subgraph Sources["Metric Sources"]
|
||||||
B[12-Node K3s Cluster] --> P
|
PVE[Proxmox HA Cluster<br/>6 Nodes]
|
||||||
C[Node Exporters] --> P
|
K3S[K3s Cluster<br/>12 Nodes]
|
||||||
D[Service Health Checks] --> P
|
NODE[Node Exporter<br/>All Hosts]
|
||||||
E[Proxmox Exporter] --> P
|
KSM[kube-state-metrics]
|
||||||
|
WIN[Windows Exporter<br/>PrimeStation]
|
||||||
|
BB[Blackbox Exporter<br/>HTTP/TCP Probes]
|
||||||
|
TN[TrueNAS<br/>Graphite Exporter]
|
||||||
|
UNI[UniFi / Unpoller<br/>Network Metrics]
|
||||||
|
end
|
||||||
|
|
||||||
P --> G[Grafana Dashboards]
|
subgraph Core["Observability Core"]
|
||||||
P --> AM[Alertmanager]
|
PROM[Prometheus]
|
||||||
|
AM[Alertmanager]
|
||||||
|
end
|
||||||
|
|
||||||
AM --> N[Email / Discord / Slack Alerts]
|
subgraph Dashboards["Grafana Dashboards"]
|
||||||
G --> NOC[Ced's NOC Dashboard]
|
CMD[Production Command Center v3<br/>Executive NOC View]
|
||||||
|
DEEP[Deep Observability v3<br/>Full Drill-Down]
|
||||||
|
K3SD[K3s Elite Observability v1<br/>Kubernetes Focus]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph Output["Operations"]
|
||||||
|
NOC[Ced's NOC<br/>noc.chasedumphord.com]
|
||||||
|
ALERT[Alert Notifications]
|
||||||
|
end
|
||||||
|
|
||||||
|
PVE & K3S & NODE & KSM & WIN & BB & TN & UNI --> PROM
|
||||||
|
PROM --> AM
|
||||||
|
PROM --> CMD & DEEP & K3SD
|
||||||
|
CMD --> NOC
|
||||||
|
AM --> ALERT
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 📸 Dashboards
|
|
||||||
|
|
||||||
### Infrastructure Overview
|
|
||||||
|
|
||||||
Infrastructure Dashboard
|
|
||||||
|
|
||||||
### K3s Cluster Dashboard
|
|
||||||
|
|
||||||
K3s Dashboard
|
|
||||||
|
|
||||||
### Proxmox HA Dashboard
|
|
||||||
|
|
||||||
Proxmox Dashboard
|
|
||||||
|
|
||||||
### Service Uptime Dashboard
|
|
||||||
|
|
||||||
Services Dashboard
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## ⚙️ Core Components
|
|
||||||
|
|
||||||
### 📡 Prometheus
|
|
||||||
|
|
||||||
Collects metrics from:
|
|
||||||
|
|
||||||
* Kubernetes endpoints
|
|
||||||
* Node exporters
|
|
||||||
* Proxmox exporter
|
|
||||||
* Custom health scripts
|
|
||||||
* Static service targets
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 📊 Grafana
|
|
||||||
|
|
||||||
Provides dashboards for:
|
|
||||||
|
|
||||||
* Cluster health
|
|
||||||
* Resource utilization
|
|
||||||
* Storage trends
|
|
||||||
* Service uptime
|
|
||||||
* Alert visibility
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 🚨 Alertmanager
|
|
||||||
|
|
||||||
Handles alerting for:
|
|
||||||
|
|
||||||
* Node failures
|
|
||||||
* High CPU / memory
|
|
||||||
* Service outages
|
|
||||||
* Pod crash loops
|
|
||||||
* Proxmox HA issues
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📁 Repo Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
ceds-observability-stack/
|
|
||||||
├── architecture/
|
|
||||||
├── prometheus/
|
|
||||||
├── grafana/
|
|
||||||
├── exporters/
|
|
||||||
├── alerting/
|
|
||||||
├── scripts/
|
|
||||||
└── docs/
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📸 Dashboard Preview
|
|
||||||
|
|
||||||
* 🔹 Infrastructure Overview
|
|
||||||
* 🔹 K3s Cluster Health
|
|
||||||
* 🔹 Proxmox Cluster Status
|
|
||||||
* 🔹 Service Uptime Dashboard
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🚀 Deployment (High-Level)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Clone repo
|
|
||||||
git clone https://github.com/ced4568/ceds-observability-stack.git
|
|
||||||
|
|
||||||
# Navigate to project
|
|
||||||
cd ceds-observability-stack
|
|
||||||
|
|
||||||
# Deploy Prometheus + exporters
|
|
||||||
# (Add your actual deployment steps here)
|
|
||||||
|
|
||||||
# Access Grafana
|
|
||||||
http://<your-server-ip>:3000
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🎯 Project Roadmap
|
|
||||||
|
|
||||||
### Phase 1 — Foundation
|
|
||||||
|
|
||||||
* [x] Architecture design
|
|
||||||
* [x] Repo structure
|
|
||||||
* [ ] Prometheus base config
|
|
||||||
* [ ] Grafana datasource
|
|
||||||
|
|
||||||
### Phase 2 — Metrics Collection
|
|
||||||
|
|
||||||
* [ ] Node exporter
|
|
||||||
* [ ] K3s metrics
|
|
||||||
* [ ] Proxmox exporter
|
|
||||||
* [ ] Uptime checks
|
|
||||||
|
|
||||||
### Phase 3 — Dashboards
|
|
||||||
|
|
||||||
* [ ] Infrastructure dashboard
|
|
||||||
* [ ] K3s dashboard
|
|
||||||
* [ ] Proxmox dashboard
|
|
||||||
* [ ] Service uptime dashboard
|
|
||||||
|
|
||||||
### Phase 4 — Alerting
|
|
||||||
|
|
||||||
* [ ] Alertmanager setup
|
|
||||||
* [ ] Alert rules
|
|
||||||
* [ ] Notification testing
|
|
||||||
|
|
||||||
### Phase 5 — Portfolio Polish
|
|
||||||
|
|
||||||
* [ ] Screenshots
|
|
||||||
* [ ] Architecture diagrams
|
|
||||||
* [ ] Setup guide
|
|
||||||
* [ ] Troubleshooting docs
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🧠 Skills Demonstrated
|
|
||||||
|
|
||||||
* 📊 Infrastructure Monitoring
|
|
||||||
* ☸️ Kubernetes Operations
|
|
||||||
* 📡 Prometheus Configuration
|
|
||||||
* 📈 Grafana Dashboarding
|
|
||||||
* 🚨 Alert Engineering
|
|
||||||
* 🐧 Linux Administration
|
|
||||||
* 🖥️ Proxmox Virtualization
|
|
||||||
* ⚙️ SRE Principles
|
|
||||||
* 🏗️ Platform Engineering
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🔗 Related Projects
|
|
||||||
|
|
||||||
| Project | Purpose |
|
|
||||||
| ----------------- | --------------------------------- |
|
|
||||||
| Ced’s HomeLab | Full infrastructure ecosystem |
|
|
||||||
| Ced’s NOC | Visualization + status dashboards |
|
|
||||||
| Ced’s K3s HomeLab | Kubernetes architecture |
|
|
||||||
| Ced’s APRS iGate | Networking + RF integration |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🔗 Integration
|
|
||||||
|
|
||||||
This observability stack is part of a larger ecosystem:
|
|
||||||
|
|
||||||
* Ced’s HomeLab → Infrastructure layer
|
|
||||||
* Ced’s Observability Stack → Metrics + monitoring layer
|
|
||||||
* Ced’s NOC → Visualization and operations layer
|
|
||||||
|
|
||||||
Data flows from monitored systems into Prometheus, is visualized in Grafana, and feeds Ced’s NOC dashboard for real-time system visibility.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🧪 Verification
|
|
||||||
|
|
||||||
To verify the system is working correctly:
|
|
||||||
|
|
||||||
### Prometheus Targets
|
|
||||||
|
|
||||||
* Navigate to: http://localhost:9090/targets
|
|
||||||
* Confirm all targets show UP
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Node Exporter
|
|
||||||
|
|
||||||
```bash
|
|
||||||
curl http://<node-ip>:9100/metrics
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Service Health Check
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python3 scripts/service-health-check.py
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Grafana
|
|
||||||
|
|
||||||
* Confirm dashboards display real-time metrics
|
|
||||||
* Verify data source connection to Prometheus
|
|
||||||
* Check for active alerts
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Alert Testing
|
|
||||||
|
|
||||||
* Stop a service or node temporarily
|
|
||||||
* Confirm alert triggers in Prometheus
|
|
||||||
* Confirm alert appears in Grafana
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📌 Status
|
|
||||||
|
|
||||||
🟢 **Active Development**
|
|
||||||
|
|
||||||
This project is continuously evolving as part of Ced’s HomeLab ecosystem and professional portfolio.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 💡 Why This Project Matters
|
|
||||||
|
|
||||||
This project simulates a production-style observability system used in modern infrastructure environments.
|
|
||||||
|
|
||||||
It is designed to demonstrate how distributed systems are monitored, analyzed, and maintained in real-world engineering teams.
|
|
||||||
|
|
||||||
Key capabilities include:
|
|
||||||
|
|
||||||
* Monitoring a hybrid infrastructure environment (Proxmox HA + K3s cluster)
|
|
||||||
* Collecting and visualizing system and service metrics
|
|
||||||
* Tracking service availability and uptime
|
|
||||||
* Detecting infrastructure and application-level failures
|
|
||||||
* Supporting alert-driven operations
|
|
||||||
* Integrating with a centralized NOC dashboard
|
|
||||||
|
|
||||||
This project represents a shift from running infrastructure to actively operating and maintaining it using observability principles aligned with SRE and platform engineering practices.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🧠 Future Improvements
|
|
||||||
|
|
||||||
* Loki log aggregation
|
|
||||||
* Tempo tracing
|
|
||||||
* Cloudflare Access log ingestion
|
|
||||||
* Automated remediation (self-healing infrastructure)
|
|
||||||
* Grafana public demo dashboard
|
|
||||||
* GitOps-based deployment (Argo CD / Flux)
|
|
||||||
* Multi-cluster Kubernetes monitoring
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Current Deployment Status
|
|
||||||
|
|
||||||
Ced’s Observability Stack is now actively collecting live metrics from Ced’s HomeLab.
|
|
||||||
|
|
||||||
Current working components:
|
|
||||||
|
|
||||||
- Prometheus metrics collection
|
|
||||||
- Grafana dashboard visualization
|
|
||||||
- Proxmox node exporter monitoring
|
|
||||||
- K3s node exporter monitoring
|
|
||||||
- kube-state-metrics for Kubernetes object state
|
|
||||||
- Windows exporter for PrimeStation
|
|
||||||
- Blackbox HTTP/TCP endpoint probing
|
|
||||||
- TrueNAS Graphite exporter
|
|
||||||
- UniFi exporter using Unpoller
|
|
||||||
|
|
||||||
Some Grafana panels are being refined as dashboard queries are aligned with available Prometheus metrics.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Production Dashboards
|
## Production Dashboards
|
||||||
|
|
||||||
| Dashboard | Purpose |
|
Three purpose-built dashboards serving different operational needs. All live at `grafana.cedshomelab.com`.
|
||||||
|---|---|
|
|
||||||
| Ced's NOC - Production Command Center v3 | Stable executive NOC view for service availability, K3s, Proxmox, PrimeStation, and latency |
|
### Production Command Center v3
|
||||||
| Ced's NOC - Deep Observability v3 | Full drill-down dashboard for Proxmox, K3s, services, UniFi, PrimeStation, and alerts |
|
|
||||||
| Ced's K3s Elite Observability v1 | Dedicated K3s dashboard using node-exporter and kube-state-metrics |
|
Executive NOC view for quick infrastructure status checks. Designed for portfolio demonstrations, interviews, and daily operational awareness.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
**What it shows:**
|
||||||
|
- NOC Health Score — single number representing overall infrastructure health
|
||||||
|
- Core Services UP / DOWN counters
|
||||||
|
- Average service latency with trend graph
|
||||||
|
- PrimeStation online status
|
||||||
|
- Core Service Availability (UP/DOWN)
|
||||||
|
- Service Latency Trend over time
|
||||||
|
- K3s Nodes Online / Proxmox Nodes Online
|
||||||
|
- Per-service status tiles: TrueNAS, NPM, Jellyfin, Dashy, Home Assistant, Grafana, Prometheus, Proxmox UI
|
||||||
|
|
||||||
|
**Built for:** Portfolio presentations, interview demos, daily ops check
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Phase 3 Live Metrics Milestone
|
### Deep Observability v3
|
||||||
|
|
||||||
- [x] Prometheus running
|
Full drill-down dashboard for active troubleshooting and infrastructure analysis. Covers every layer of the stack in one view.
|
||||||
- [x] Grafana connected to Prometheus
|
|
||||||
|
|
||||||
|
|
||||||
|
**What it shows:**
|
||||||
|
- Prometheus target health summary
|
||||||
|
- Proxmox HA cluster metrics — nodes, storage, VM status
|
||||||
|
- K3s cluster health — nodes, pods, deployments
|
||||||
|
- Windows/Network/Backup section — PrimeStation, UniFi, TrueNAS
|
||||||
|
- HTTP/TCP probe results via Blackbox Exporter
|
||||||
|
- HTTPS response time trends
|
||||||
|
|
||||||
|
**Built for:** Active incident response, performance analysis, infrastructure troubleshooting
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### K3s Elite Observability v1
|
||||||
|
|
||||||
|
Focused Kubernetes dashboard using node-exporter and kube-state-metrics for deep cluster visibility.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
**What it shows:**
|
||||||
|
- Cluster Summary: Nodes Online, Nodes Ready, Pods Running, Pods Pending, Pods Failed, Restart count
|
||||||
|
- Node CPU usage per node
|
||||||
|
- Node Memory usage with historical trend
|
||||||
|
- Pod Phase by Namespace
|
||||||
|
- Container Restarts by Pod
|
||||||
|
- Pods Not Running
|
||||||
|
- Nodes Not Ready
|
||||||
|
- Node Load average
|
||||||
|
- Node Disk Usage
|
||||||
|
|
||||||
|
**Built for:** Kubernetes operations, cluster health monitoring, capacity planning
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Full Exporter Stack
|
||||||
|
|
||||||
|
Every metric source in the infrastructure is actively scraped by Prometheus.
|
||||||
|
|
||||||
|
| Exporter | Target | Metrics |
|
||||||
|
|----------|--------|---------|
|
||||||
|
| Node Exporter | All Proxmox + K3s nodes | CPU, RAM, disk, network per host |
|
||||||
|
| kube-state-metrics | K3s cluster | Pod state, deployment health, replica counts |
|
||||||
|
| Proxmox Exporter | Proxmox HA cluster | Node status, VM health, HA state |
|
||||||
|
| Windows Exporter | PrimeStation | CPU, RAM, disk, network for main workstation |
|
||||||
|
| Blackbox Exporter | HTTP/TCP endpoints | Service uptime, response time, probe results |
|
||||||
|
| TrueNAS Graphite Exporter | TrueNAS | Storage pool health, dataset usage |
|
||||||
|
| Unpoller (UniFi Exporter) | UniFi Dream Router | Network device metrics, client counts, throughput |
|
||||||
|
| metrics-server | K3s | Real-time resource usage for kubectl top |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Infrastructure Coverage
|
||||||
|
|
||||||
|
| System | Monitoring Status |
|
||||||
|
|--------|------------------|
|
||||||
|
| Proxmox HA Cluster (6 nodes) | ✅ Live — node exporter + Proxmox exporter |
|
||||||
|
| K3s Cluster (12 nodes) | ✅ Live — node exporter + kube-state-metrics |
|
||||||
|
| TrueNAS | ✅ Live — Graphite exporter |
|
||||||
|
| Nginx Proxy Manager | ✅ Live — Blackbox HTTP probe |
|
||||||
|
| Home Assistant | ✅ Live — Blackbox HTTP probe |
|
||||||
|
| Dashy | ✅ Live — Blackbox HTTP probe |
|
||||||
|
| Jellyfin | ✅ Live — Blackbox HTTP probe |
|
||||||
|
| UniFi Dream Router | ✅ Live — Unpoller exporter |
|
||||||
|
| PrimeStation (Windows) | ✅ Live — Windows exporter |
|
||||||
|
| Grafana | ✅ Live — self-monitored |
|
||||||
|
| Prometheus | ✅ Live — self-monitored |
|
||||||
|
| Public endpoints | ✅ Live — Blackbox external probes |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Repository Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
ceds-observability-stack/
|
||||||
|
├── architecture/ # Architecture diagrams
|
||||||
|
├── prometheus/
|
||||||
|
│ └── prometheus.yml # Scrape configs and target definitions
|
||||||
|
├── grafana/
|
||||||
|
│ └── dashboards/ # Dashboard JSON exports
|
||||||
|
├── exporters/ # Exporter configs (node, blackbox, unpoller)
|
||||||
|
├── alerting/ # Alertmanager config and alert rules
|
||||||
|
├── scripts/
|
||||||
|
│ └── service-health-check.py
|
||||||
|
└── docs/ # Setup guides and notes
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
**Prerequisites:** Linux server or VM, Prometheus, Grafana, network access to homelab systems.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run Prometheus with config
|
||||||
|
prometheus --config.file=prometheus/prometheus.yml
|
||||||
|
|
||||||
|
# Verify targets are up
|
||||||
|
# Navigate to: http://localhost:9090/targets
|
||||||
|
|
||||||
|
# Run service health check script
|
||||||
|
python3 scripts/service-health-check.py
|
||||||
|
```
|
||||||
|
|
||||||
|
**Access:**
|
||||||
|
- Prometheus: `http://<server-ip>:9090`
|
||||||
|
- Grafana: `http://<server-ip>:3000`
|
||||||
|
- Live (external): `https://grafana.cedshomelab.com`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Roadmap
|
||||||
|
|
||||||
|
### Completed
|
||||||
|
- [x] Prometheus running and scraping all targets
|
||||||
|
- [x] Grafana connected to Prometheus datasource
|
||||||
- [x] Proxmox node exporters reporting
|
- [x] Proxmox node exporters reporting
|
||||||
- [x] K3s node exporters reporting
|
- [x] K3s node exporters reporting across all 12 nodes
|
||||||
- [x] kube-state-metrics installed and reporting
|
- [x] kube-state-metrics installed and reporting
|
||||||
- [x] Windows exporter reporting
|
- [x] Windows Exporter on PrimeStation
|
||||||
- [x] Blackbox Exporter repaired
|
- [x] Blackbox Exporter — internal HTTP/TCP probing
|
||||||
- [x] Internal HTTP probes working
|
- [x] UniFi Exporter via Unpoller
|
||||||
- [x] UniFi exporter installed
|
- [x] TrueNAS Graphite Exporter
|
||||||
- [x] Grafana dashboards receiving live data
|
- [x] Production Command Center v3 — live
|
||||||
|
- [x] Deep Observability v3 — live
|
||||||
|
- [x] K3s Elite Observability v1 — live
|
||||||
|
|
||||||
|
### In Progress
|
||||||
|
- [ ] Alertmanager alert rules library
|
||||||
|
- [ ] Alert notification channels (email / Discord)
|
||||||
|
- [ ] Loki log aggregation
|
||||||
|
- [ ] Grafana public demo dashboard
|
||||||
|
- [ ] Tempo distributed tracing
|
||||||
|
- [ ] GitOps deployment via ArgoCD
|
||||||
|
- [ ] Automated remediation (self-healing infrastructure)
|
||||||
|
- [ ] Cloudflare Access log ingestion
|
||||||
|
- [ ] Multi-cluster Kubernetes monitoring
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Related Projects
|
||||||
|
|
||||||
|
| Project | Role in Stack |
|
||||||
|
|---------|--------------|
|
||||||
|
| [ceds-homelab](https://github.com/ced4568/ceds-homelab) | Infrastructure layer — Proxmox, TrueNAS, networking |
|
||||||
|
| [ced-k3s-homelab](https://github.com/ced4568/ced-k3s-homelab) | Orchestration layer — 12-node K3s cluster |
|
||||||
|
| [ceds-aprs-igate](https://github.com/ced4568/ceds-aprs-igate) | Edge layer — RF ingestion nodes |
|
||||||
|
| [ced-portfolio](https://github.com/ced4568/ced-portfolio) | Portfolio — chasedumphord.com |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Author
|
||||||
|
|
||||||
|
**Chase Dumphord (Ced)**
|
||||||
|
Digital Systems Engineer · GE Aerospace · Oxford, MS
|
||||||
|
|
||||||
|
[](https://chasedumphord.com)
|
||||||
|
[](https://www.linkedin.com/in/chase-dumphord/)
|
||||||
|
[](https://github.com/ced4568)
|
||||||
|
[](https://grafana.cedshomelab.com)
|
||||||
|
[](https://noc.chasedumphord.com)
|
||||||
Reference in New Issue
Block a user