From af73789fe489dcb4b89fee03b2cab91d3d4d50b5 Mon Sep 17 00:00:00 2001 From: Chase Dumphord Date: Tue, 28 Apr 2026 00:29:25 -0500 Subject: [PATCH] Added Readme V2 & Folder layout --- README.md | 241 +++++++++++++++++++++ alerting/alerting-strategy.md | 0 alerting/alertmanager.yml | 0 architecture/observability-architecture.md | 0 docs/roadmap.md | 0 docs/setup-guide.md | 0 docs/troubleshooting.md | 0 exporters/k3s-metrics.md | 0 exporters/node-exporter.md | 0 exporters/proxmox-exporter.md | 0 grafana/dashboards/dashboard-index.md | 0 grafana/datasource-prometheus.yml | 0 prometheus/alert-rules.yml | 0 prometheus/prometheus.yml | 0 scripts/service-health-check.py | 0 15 files changed, 241 insertions(+) create mode 100644 README.md create mode 100644 alerting/alerting-strategy.md create mode 100644 alerting/alertmanager.yml create mode 100644 architecture/observability-architecture.md create mode 100644 docs/roadmap.md create mode 100644 docs/setup-guide.md create mode 100644 docs/troubleshooting.md create mode 100644 exporters/k3s-metrics.md create mode 100644 exporters/node-exporter.md create mode 100644 exporters/proxmox-exporter.md create mode 100644 grafana/dashboards/dashboard-index.md create mode 100644 grafana/datasource-prometheus.yml create mode 100644 prometheus/alert-rules.yml create mode 100644 prometheus/prometheus.yml create mode 100644 scripts/service-health-check.py diff --git a/README.md b/README.md new file mode 100644 index 0000000..24b4b0b --- /dev/null +++ b/README.md @@ -0,0 +1,241 @@ +# πŸš€ Ced’s Observability Stack + +![Status](https://img.shields.io/badge/Status-Active%20Development-blue) +![Platform](https://img.shields.io/badge/Platform-Proxmox%20%7C%20K3s-orange) +![Monitoring](https://img.shields.io/badge/Stack-Prometheus%20%7C%20Grafana-red) +![Alerting](https://img.shields.io/badge/Alerting-Alertmanager-yellow) +![License](https://img.shields.io/badge/License-MIT-green) + +--- + +## 🧠 Executive Summary + +**Ced’s Observability Stack** is a production-style monitoring, metrics, and alerting platform built to provide full visibility into a distributed hybrid infrastructure. + +It simulates real-world **SRE / Platform Engineering environments**, delivering: + +* πŸ“Š Real-time infrastructure monitoring +* βš™οΈ Kubernetes observability (12-node K3s cluster) +* πŸ–₯️ Proxmox HA cluster visibility +* 🌐 Service uptime + network health tracking +* 🚨 Alerting pipelines (Alertmanager) +* πŸ“ˆ Operational dashboards (Grafana) + +> 🎯 **Goal:** Demonstrate enterprise-level observability practices across virtualization, Kubernetes, and self-hosted infrastructure. + +--- + +## πŸ—οΈ Environment Overview + +### Core Infrastructure + +| System | Purpose | +| ------------------------ | ---------------------------------- | +| πŸ–₯️ Proxmox HA Cluster | Virtualization & high availability | +| ☸️ K3s Cluster (12-node) | Container orchestration | +| πŸ’Ύ TrueNAS | Storage services | +| 🌐 Nginx Proxy Manager | Reverse proxy & routing | +| ☁️ Cloudflare | DNS, tunnels, external protection | +| πŸ“Š Grafana | Visualization dashboards | +| πŸ“‘ Prometheus | Metrics collection | +| 🚨 Alertmanager | Alert routing | + +--- + +## πŸ“‘ Monitored Systems + +| Target | Example Metrics | +| ----------------------- | ------------------------------------ | +| πŸ–₯️ Proxmox Nodes | CPU, memory, storage, VM + HA status | +| ☸️ K3s Nodes | Node readiness, resource usage | +| πŸ“¦ Kubernetes Workloads | Pods, deployments, restarts | +| 🌐 Network Services | Uptime, latency, TCP checks | +| πŸ’Ύ TrueNAS | Storage + service availability | +| πŸ”€ Nginx Proxy Manager | Reverse proxy health | +| πŸ“Š Dashy / NOC | Dashboard availability | +| 🎬 Jellyfin | Media service uptime | + +--- + +## 🧩 Architecture + +```mermaid +flowchart TD + A[Proxmox HA Cluster] --> P[Prometheus] + B[12-Node K3s Cluster] --> P + C[Node Exporters] --> P + D[Service Health Checks] --> P + E[Proxmox Exporter] --> P + + P --> G[Grafana Dashboards] + P --> AM[Alertmanager] + + AM --> N[Email / Discord / Slack Alerts] + G --> NOC[Ced's NOC Dashboard] +``` + +--- + +## βš™οΈ Core Components + +### πŸ“‘ Prometheus + +Collects metrics from: + +* Kubernetes endpoints +* Node exporters +* Proxmox exporter +* Custom health scripts +* Static service targets + +--- + +### πŸ“Š Grafana + +Provides dashboards for: + +* Cluster health +* Resource utilization +* Storage trends +* Service uptime +* Alert visibility + +--- + +### 🚨 Alertmanager + +Handles alerting for: + +* Node failures +* High CPU / memory +* Service outages +* Pod crash loops +* Proxmox HA issues + +--- + +## πŸ“ Repo Structure + +``` +ceds-observability-stack/ +β”œβ”€β”€ architecture/ +β”œβ”€β”€ prometheus/ +β”œβ”€β”€ grafana/ +β”œβ”€β”€ exporters/ +β”œβ”€β”€ alerting/ +β”œβ”€β”€ scripts/ +└── docs/ +``` + +--- + +## πŸ“Έ Dashboard Preview (Add Your Screenshots) + +> πŸ“Œ Replace with real screenshots from your Grafana dashboards + +* πŸ”Ή Infrastructure Overview +* πŸ”Ή K3s Cluster Health +* πŸ”Ή Proxmox Cluster Status +* πŸ”Ή Service Uptime Dashboard + +--- + +## πŸš€ Deployment (High-Level) + +```bash +# Clone repo +git clone https://github.com/ced4568/ceds-observability-stack.git + +# Navigate to project +cd ceds-observability-stack + +# Deploy Prometheus + exporters +# (Add your actual deployment steps here) + +# Access Grafana +http://:3000 +``` + +--- + +## 🎯 Project Roadmap + +### Phase 1 β€” Foundation + +* [x] Architecture design +* [x] Repo structure +* [ ] Prometheus base config +* [ ] Grafana datasource + +### Phase 2 β€” Metrics Collection + +* [ ] Node exporter +* [ ] K3s metrics +* [ ] Proxmox exporter +* [ ] Uptime checks + +### Phase 3 β€” Dashboards + +* [ ] Infrastructure dashboard +* [ ] K3s dashboard +* [ ] Proxmox dashboard +* [ ] Service uptime dashboard + +### Phase 4 β€” Alerting + +* [ ] Alertmanager setup +* [ ] Alert rules +* [ ] Notification testing + +### Phase 5 β€” Portfolio Polish + +* [ ] Screenshots +* [ ] Architecture diagrams +* [ ] Setup guide +* [ ] Troubleshooting docs + +--- + +## 🧠 Skills Demonstrated + +* πŸ“Š Infrastructure Monitoring +* ☸️ Kubernetes Operations +* πŸ“‘ Prometheus Configuration +* πŸ“ˆ Grafana Dashboarding +* 🚨 Alert Engineering +* 🐧 Linux Administration +* πŸ–₯️ Proxmox Virtualization +* βš™οΈ SRE Principles +* πŸ—οΈ Platform Engineering + +--- + +## πŸ”— Related Projects + +| Project | Purpose | +| ----------------- | --------------------------------- | +| Ced’s HomeLab | Full infrastructure ecosystem | +| Ced’s NOC | Visualization + status dashboards | +| Ced’s K3s HomeLab | Kubernetes architecture | +| Ced’s APRS iGate | Networking + RF integration | + +--- + +## πŸ“Œ Status + +🟒 **Active Development** + +This project is continuously evolving as part of Ced’s HomeLab ecosystem and professional portfolio. + +--- + +## πŸ’Ό Why This Project Matters + +This repository demonstrates the ability to: + +* Design and operate **distributed systems** +* Implement **observability at scale** +* Build **production-style monitoring stacks** +* Apply **real-world SRE practices** + +> πŸš€ Designed as a **portfolio-grade project** for career growth, promotion, and technical leadership visibility. diff --git a/alerting/alerting-strategy.md b/alerting/alerting-strategy.md new file mode 100644 index 0000000..e69de29 diff --git a/alerting/alertmanager.yml b/alerting/alertmanager.yml new file mode 100644 index 0000000..e69de29 diff --git a/architecture/observability-architecture.md b/architecture/observability-architecture.md new file mode 100644 index 0000000..e69de29 diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 0000000..e69de29 diff --git a/docs/setup-guide.md b/docs/setup-guide.md new file mode 100644 index 0000000..e69de29 diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 0000000..e69de29 diff --git a/exporters/k3s-metrics.md b/exporters/k3s-metrics.md new file mode 100644 index 0000000..e69de29 diff --git a/exporters/node-exporter.md b/exporters/node-exporter.md new file mode 100644 index 0000000..e69de29 diff --git a/exporters/proxmox-exporter.md b/exporters/proxmox-exporter.md new file mode 100644 index 0000000..e69de29 diff --git a/grafana/dashboards/dashboard-index.md b/grafana/dashboards/dashboard-index.md new file mode 100644 index 0000000..e69de29 diff --git a/grafana/datasource-prometheus.yml b/grafana/datasource-prometheus.yml new file mode 100644 index 0000000..e69de29 diff --git a/prometheus/alert-rules.yml b/prometheus/alert-rules.yml new file mode 100644 index 0000000..e69de29 diff --git a/prometheus/prometheus.yml b/prometheus/prometheus.yml new file mode 100644 index 0000000..e69de29 diff --git a/scripts/service-health-check.py b/scripts/service-health-check.py new file mode 100644 index 0000000..e69de29