diff --git a/README-dashboard.png b/README-dashboard.png deleted file mode 100644 index c43bb94..0000000 Binary files a/README-dashboard.png and /dev/null differ diff --git a/README.md b/README.md index aadc036..b5d7215 100644 --- a/README.md +++ b/README.md @@ -1,166 +1,187 @@ -# ๐Ÿš€ CedOps โ€” AI Infrastructure Operations NOC +# ๐Ÿค– CedOps โ€” AI Infrastructure Operations NOC -![Grafana](https://img.shields.io/badge/Grafana-Dashboard-orange) -![Prometheus](https://img.shields.io/badge/Prometheus-Monitoring-red) -![Docker](https://img.shields.io/badge/Docker-Containers-blue) -![Tailscale](https://img.shields.io/badge/Tailscale-ZeroTrust-purple) -![Proxmox](https://img.shields.io/badge/Proxmox-Virtualization-orange) -![Platform Engineering](https://img.shields.io/badge/Focus-Platform%20Engineering-green) +> A live AI operations platform running on Hetzner VPS โ€” 8 autonomous agents orchestrated by Paperclip, monitored in real-time by a Prometheus + Grafana NOC dashboard. Built by a Digital Systems Engineer at GE Aerospace. -![CedOps AI Infrastructure NOC Dashboard](README-dashboard.png) +[![Live NOC](https://img.shields.io/badge/Live%20NOC-noc.chasedumphord.com-1D9E75?style=flat-square)](https://noc.chasedumphord.com) +[![Paperclip](https://img.shields.io/badge/Paperclip-AI%20Agent%20Platform-6B46C1?style=flat-square)](https://paperclip.synthossystems.com) +[![Synthos Systems](https://img.shields.io/badge/Synthos%20Systems-synthossystems.com-0EA5E9?style=flat-square)](https://synthossystems.com) +[![Portfolio](https://img.shields.io/badge/Portfolio-chasedumphord.com-0F6E56?style=flat-square)](https://chasedumphord.com) +[![Agents](https://img.shields.io/badge/AI%20Agents-8%20Active-orange?style=flat-square)](#-ai-agent-roster) +[![Platform](https://img.shields.io/badge/Platform-Hetzner%20VPS-D4001A?style=flat-square)](#๏ธ-infrastructure-layer) +[![Monitoring](https://img.shields.io/badge/Monitoring-Prometheus%20%2B%20Grafana-F46800?style=flat-square)](#-observability--noc) -## Overview +--- -CedOps is an **AI-Assisted Infrastructure Operations Platform** designed to provide centralized monitoring, observability, uptime management, and operational visibility across a hybrid environment of: +## ๐Ÿ“ธ NOC Dashboard -- ๐Ÿ  Homelab Infrastructure -- โ˜๏ธ VPS Cloud Infrastructure -- ๐Ÿค– AI Operations Systems -- ๐ŸŒ Public-Facing Services -- ๐Ÿ“ก Network & Platform Health +[![CedOps NOC Dashboard](docs/screenshots/dashboard_v1.png)](docs/screenshots/dashboard_v1.png) + +> **Live:** All 5 public services reporting UP ยท NOC Health Score 100% ยท VPS Disk 27% ยท Prometheus scraping every 30s + +--- + +## ๐Ÿงญ Overview + +CedOps is a **production-style AI operations platform** built to simulate a real-world Platform Engineering and AI Infrastructure environment. The entire stack runs on a Hetzner VPS and is monitored continuously by a live Grafana NOC dashboard. The platform combines: -- **Prometheus** for metrics collection -- **Grafana** for visualization -- **Blackbox Exporter** for service monitoring -- **Node Exporter** for infrastructure metrics -- **Tailscale** for secure private observability -- **Proxmox VE** for virtualization -- **Docker** for service deployment +- **Paperclip** โ€” AI Agent Orchestration Platform managing 8 autonomous agents +- **SugeBot** โ€” Hermes AI agent deployed as Telegram Operations Assistant (CTO/Ops role) +- **n8n** โ€” Automation platform for workflow execution and inter-service routing +- **OpenRouter** โ€” AI model gateway serving as the LLM backbone for all agents +- **Prometheus + Grafana** โ€” Full observability NOC monitoring the entire stack in real-time +- **Nginx Proxy Manager** โ€” Reverse proxy + SSL management across all services +- **Docker + Portainer** โ€” Container infrastructure and management layer --- -# ๐ŸŽฏ Project Goals +## ๐Ÿ—๏ธ Architecture Overview -CedOps was built to simulate a **real-world Platform Engineering / Infrastructure Operations environment**. - -Goals: - -โœ… Single-pane-of-glass infrastructure visibility -โœ… Secure observability without exposing services publicly -โœ… Hybrid monitoring (on-prem + cloud) -โœ… Public service uptime tracking -โœ… AI platform monitoring -โœ… Portfolio-grade infrastructure engineering project - ---- - -# ๐Ÿ—๏ธ Architecture Overview - -```text - Internet - โ”‚ - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ”‚ Public Services โ”‚ - โ”‚ synthossystems.com โ”‚ - โ”‚ noc.chasedumphord.com โ”‚ - โ”‚ cedshomelab.com โ”‚ - โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ - Blackbox Exporter - โ”‚ - โ–ผ - Prometheus Monitoring - (10.10.30.140) - โ”‚ - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ”‚ โ”‚ โ”‚ - โ–ผ โ–ผ โ–ผ - Proxmox Nodes K3s Cluster Hetzner VPS - Node Exporter Monitoring synthos-node-1 - โ”‚ - Node Exporter - โ”‚ - Tailscale +``` + Internet + โ”‚ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Cloudflare Edge โ”‚ + โ”‚ DNS ยท Tunnels ยท Proxy โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Hetzner VPS โ”‚ + โ”‚ synthos-node-1 โ”‚ + โ”‚ โ”‚ + โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ + โ”‚ โ”‚ Nginx Proxy Manager โ”‚ โ”‚ + โ”‚ โ”‚ Reverse Proxy / SSL โ”‚ โ”‚ + โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ + โ”‚ โ”‚ โ”‚ + โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ + โ”‚ โ”‚ Portainer โ”‚ โ”‚ + โ”‚ โ”‚ Docker Management โ”‚ โ”‚ + โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ + โ”‚ โ”‚ โ”‚ + โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ + โ”‚ โ”‚ CedOps Platform โ”‚ โ”‚ + โ”‚ โ”‚ โ”‚ โ”‚ + โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚ + โ”‚ โ”‚ โ”‚ Paperclip โ”‚ โ”‚ SugeBot โ”‚ โ”‚ โ”‚ + โ”‚ โ”‚ โ”‚ AI Orch. โ”‚ โ”‚ Telegram Ops โ”‚ โ”‚ โ”‚ + โ”‚ โ”‚ โ”‚ 8 Agents โ”‚ โ”‚ (Hermes Agent) โ”‚ โ”‚ โ”‚ + โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ + โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ + โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚ + โ”‚ โ”‚ โ”‚ OpenRouter โ”‚ โ”‚ n8n โ”‚ โ”‚ โ”‚ + โ”‚ โ”‚ โ”‚ AI Gateway โ”‚ โ”‚ Automation โ”‚ โ”‚ โ”‚ + โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ + โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ + โ”‚ โ”‚ + โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ + โ”‚ โ”‚ Observability Stack โ”‚ โ”‚ + โ”‚ โ”‚ Prometheus โ†’ Grafana NOC Dashboard โ”‚ โ”‚ + โ”‚ โ”‚ Blackbox Exporter โ†’ Service Uptime โ”‚ โ”‚ + โ”‚ โ”‚ Node Exporter โ†’ VPS Resource Health โ”‚ โ”‚ + โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ ``` --- -# ๐Ÿ–ฅ๏ธ Dashboard Features +## ๐Ÿค– AI Agent Roster -## Executive Health +All agents run inside **Paperclip** โ€” the AI Agent Orchestration Platform at `paperclip.synthossystems.com`. -- VPS Status -- Public Services Status -- NOC Health Score -- Uptime Metrics +| Agent | Role | Status | +|---|---|---| +| **Suge** | CTO / Operations (Hermes Agent) โ€” also deployed as SugeBot on Telegram | โœ… Live | +| **CEO** | Executive decision-making and strategic oversight | โœ… Live | +| **ClaudeCoder** | Software engineering and code generation | โœ… Live | +| **Engineer** | Infrastructure and systems engineering | โœ… Live | +| **Marketing** | Marketing strategy and content | โœ… Live | +| **Offer Builder Agent** | Sales offer generation and pipeline | โœ… Live | +| **Research** | Market research and intelligence gathering | โœ… Live | +| **Sales Agent** | Lead qualification and outreach | โœ… Live | -## Infrastructure Monitoring - -- CPU Usage -- Memory Utilization -- Disk Usage -- Network Traffic - -## Public Service Monitoring - -Tracked via Blackbox Exporter: - -- CedOps -- n8n -- Paperclip -- Synthos Website -- Ced's Home Lab -- Grafana -- Prometheus - -## Reliability Monitoring - -- HTTP Status Codes -- Response Times -- Endpoint Availability -- Service Reachability +> **Suge** serves dual duty โ€” the primary Hermes-style ops agent inside Paperclip, and the external-facing **SugeBot** on Telegram for real-time operational commands. --- -# ๐Ÿ“ธ Dashboard Screenshots +## ๐Ÿ–ฅ๏ธ Infrastructure Layer -## CedOps Infrastructure NOC (V1) +All services run on a **Hetzner VPS** (`synthos-node-1`) behind Cloudflare Tunnels with zero exposed ports. -![Dashboard V1](docs/screenshots/dashboard_v1.png) +| Service | Role | URL | +|---|---|---| +| **VPS / Portainer** | Docker container management | Internal | +| **Nginx Proxy Manager** | Reverse proxy + SSL termination | Internal | +| **Homepage** | CedOps command center dashboard | `ops.synthossystems.com` | +| **Synthos Website** | Business-facing website | `synthossystems.com` | +| **Ced's Home Lab** | Homelab / Portfolio platform | `cedshomelab.com` | + +**Traffic Flow:** +``` +Internet โ†’ Cloudflare Edge โ†’ Tunnel โ†’ Nginx Proxy Manager โ†’ Docker Services +``` --- -# ๐Ÿ”’ Security Architecture +## ๐Ÿ“Š Observability & NOC -CedOps prioritizes **secure observability**. +The CedOps NOC dashboard monitors the entire VPS stack in real-time. -### No Public Prometheus Exposure +### Executive Health Panel +- **Synthos VPS Status** โ€” UP/DOWN state with instant alerting +- **Public Services Up** โ€” count of healthy endpoints (target: 5/5) +- **NOC Health Score** โ€” composite platform health percentage +- **VPS Uptime** โ€” continuous uptime tracking in days -Infrastructure metrics are collected using: +### VPS Resource Health +- CPU Usage % โ€” `synthos-node-1` CPU time series +- Memory Utilization % โ€” real-time memory pressure +- Disk Used % โ€” current at 27% +- Network Traffic โ€” RX/TX bandwidth graphs (30s refresh) -**Tailscale private networking** +### Public Service Availability -Benefits: +Tracked via **Blackbox Exporter** HTTP probes: -- Encrypted communication -- Zero Trust access -- No public metrics exposure -- Private infrastructure scraping +| Service | Endpoint | Status | +|---|---|---| +| cedshomelab | cedshomelab.com | โœ… UP | +| n8n | n8n.synthossystems.com | โœ… UP | +| cedops | ops.synthossystems.com | โœ… UP | +| paperclip | paperclip.synthossystems.com | โœ… UP | +| synthos-website | synthossystems.com | โœ… UP | -### Monitoring Stack - -| Tool | Purpose | -|------|---------| -| Prometheus | Metrics collection | -| Grafana | Visualization | -| Blackbox Exporter | Uptime checks | -| Node Exporter | Linux system metrics | -| Tailscale | Secure networking | -| Docker | Service deployment | -| Proxmox | Virtualization | +### Reliability Metrics +- HTTP Status Codes per endpoint (all returning `200`) +- Public Endpoint Response Time (ms) โ€” time series graph +- SSL Expiry Days โ€” tracked per certificate with warning thresholds --- -# ๐Ÿ“‚ Repository Structure +## ๐Ÿ”’ Security Architecture -```text +CedOps is designed with **zero public infrastructure exposure**. + +| Control | Implementation | +|---|---| +| No open ports | Cloudflare Tunnel handles all ingress | +| No public Prometheus | Metrics stay private; only Grafana is exposed | +| SSL everywhere | NPM handles termination for all services | +| Container isolation | Docker network segmentation via Portainer | +| Secret management | No secrets in this repo โ€” handled via environment variables | + +--- + +## ๐Ÿ“‚ Repository Structure + +``` cedops-ai-infrastructure-noc/ โ”‚ โ”œโ”€โ”€ docs/ โ”‚ โ”œโ”€โ”€ architecture/ โ”‚ โ”œโ”€โ”€ screenshots/ +โ”‚ โ”‚ โ””โ”€โ”€ dashboard_v1.png โ”‚ โ””โ”€โ”€ sops/ โ”‚ โ”œโ”€โ”€ grafana/ @@ -176,78 +197,315 @@ cedops-ai-infrastructure-noc/ โ””โ”€โ”€ README.md ``` ---- - -# ๐Ÿ› ๏ธ Skills Demonstrated - -### Platform Engineering -- Infrastructure Monitoring -- Observability Design -- Service Reliability -- Secure Networking - -### DevOps -- Linux Administration -- Docker -- YAML Configuration -- Troubleshooting - -### Cloud & Infrastructure -- Hetzner VPS -- Proxmox Virtualization -- Hybrid Infrastructure -- Private Networking - -### Monitoring & Observability -- Prometheus -- Grafana -- Blackbox Exporter -- Alerting -- Metrics Design +> **Note:** Prometheus and Blackbox configs are sanitized for public sharing. Production configs are managed privately via environment variables and are never committed to this repo. --- -# ๐Ÿ’ผ Resume Relevance +## ๐Ÿ› ๏ธ Stack -This project demonstrates real-world experience with: - -**Platform Engineering** -**Infrastructure Monitoring** -**Observability Engineering** -**DevOps Workflows** -**Cloud + On-Prem Hybrid Systems** +| Layer | Technology | +|---|---| +| **Compute** | Hetzner VPS (synthos-node-1) | +| **Containers** | Docker + Portainer | +| **Reverse Proxy** | Nginx Proxy Manager | +| **DNS & Tunnels** | Cloudflare | +| **AI Orchestration** | Paperclip | +| **AI Gateway** | OpenRouter | +| **Automation** | n8n | +| **Ops Agent** | SugeBot (Hermes / Telegram) | +| **Metrics** | Prometheus + Node Exporter + Blackbox Exporter | +| **Visualization** | Grafana | --- -# ๐Ÿ”ฎ Roadmap +## ๐Ÿ”ฎ Roadmap -## V2 Dashboard +### V2 Dashboard +- [ ] Agent activity metrics (Paperclip runs, success rate, token spend) +- [ ] SugeBot command log panel +- [ ] Per-agent health monitoring +- [ ] n8n workflow execution tracking +- [ ] Telegram alert integration via SugeBot -- SSL Expiration Monitoring -- Alert Feed -- SLA Tracking -- Reliability Scoring -- Capacity Metrics - -## Future Enhancements - -- Telegram Alerts -- Discord Notifications -- AI Incident Detection -- Automated Remediation -- Multi-Environment Monitoring +### Platform Enhancements +- [ ] AI Incident Detection โ€” anomaly-triggered agent escalation +- [ ] Automated Remediation โ€” n8n workflows triggered by Prometheus alerts +- [ ] Multi-environment monitoring (homelab + VPS unified NOC) +- [ ] Discord alert channel --- -# ๐Ÿ‘จโ€๐Ÿ’ป Author +## ๐Ÿ‘ค Author -### Chase Dumphord +**Chase Dumphord** +Digital Systems Engineer | Platform Engineering | AI Infrastructure | Observability -Digital Systems Engineer | Infrastructure & Platform Engineering | Observability | AI Systems +[![LinkedIn](https://img.shields.io/badge/LinkedIn-chase--dumphord-0A66C2?style=flat-square)](https://www.linkedin.com/in/toochase-dumphord/) +[![GitHub](https://img.shields.io/badge/GitHub-ced4568-181717?style=flat-square)](https://github.com/ced4568) +[![Live NOC](https://img.shields.io/badge/Live%20NOC-noc.chasedumphord.com-1D9E75?style=flat-square)](https://noc.chasedumphord.com) +[![Portfolio](https://img.shields.io/badge/Portfolio-chasedumphord.com-0F6E56?style=flat-square)](https://chasedumphord.com) -GitHub: https://github.com/ced4568 +--- -Portfolio: -- https://chasedumphord.com -- https://noc.chasedumphord.com -- https://cedshomelab.com \ No newline at end of file +## ๐Ÿ”— Related Repos + +| Repo | Description | +|---|---| +| [ceds-homelab](https://github.com/ced4568/ceds-homelab) | 6-node Proxmox cluster + 12-node K3s + full homelab infrastructure | +| [ced-k3s-homelab](https://github.com/ced4568/ced-k3s-homelab) | 12-node Raspberry Pi K3s cluster detail | +# ๐Ÿค– CedOps โ€” AI Infrastructure Operations NOC + +> A live AI operations platform running on Hetzner VPS โ€” 8 autonomous agents orchestrated by Paperclip, monitored in real-time by a Prometheus + Grafana NOC dashboard. Built by a Digital Systems Engineer at GE Aerospace. + +[![Live NOC](https://img.shields.io/badge/Live%20NOC-noc.chasedumphord.com-1D9E75?style=flat-square)](https://noc.chasedumphord.com) +[![Paperclip](https://img.shields.io/badge/Paperclip-AI%20Agent%20Platform-6B46C1?style=flat-square)](https://paperclip.synthossystems.com) +[![Synthos Systems](https://img.shields.io/badge/Synthos%20Systems-synthossystems.com-0EA5E9?style=flat-square)](https://synthossystems.com) +[![Portfolio](https://img.shields.io/badge/Portfolio-chasedumphord.com-0F6E56?style=flat-square)](https://chasedumphord.com) +[![Agents](https://img.shields.io/badge/AI%20Agents-8%20Active-orange?style=flat-square)](#-ai-agent-roster) +[![Platform](https://img.shields.io/badge/Platform-Hetzner%20VPS-D4001A?style=flat-square)](#๏ธ-infrastructure-layer) +[![Monitoring](https://img.shields.io/badge/Monitoring-Prometheus%20%2B%20Grafana-F46800?style=flat-square)](#-observability--noc) + +--- + +## ๐Ÿ“ธ NOC Dashboard + +[![CedOps NOC Dashboard](docs/screenshots/dashboard_v1.png)](docs/screenshots/dashboard_v1.png) + +> **Live:** All 5 public services reporting UP ยท NOC Health Score 100% ยท VPS Disk 27% ยท Prometheus scraping every 30s + +--- + +## ๐Ÿงญ Overview + +CedOps is a **production-style AI operations platform** built to simulate a real-world Platform Engineering and AI Infrastructure environment. The entire stack runs on a Hetzner VPS and is monitored continuously by a live Grafana NOC dashboard. + +The platform combines: + +- **Paperclip** โ€” AI Agent Orchestration Platform managing 8 autonomous agents +- **SugeBot** โ€” Hermes AI agent deployed as Telegram Operations Assistant (CTO/Ops role) +- **n8n** โ€” Automation platform for workflow execution and inter-service routing +- **OpenRouter** โ€” AI model gateway serving as the LLM backbone for all agents +- **Prometheus + Grafana** โ€” Full observability NOC monitoring the entire stack in real-time +- **Nginx Proxy Manager** โ€” Reverse proxy + SSL management across all services +- **Docker + Portainer** โ€” Container infrastructure and management layer + +--- + +## ๐Ÿ—๏ธ Architecture Overview + +``` + Internet + โ”‚ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Cloudflare Edge โ”‚ + โ”‚ DNS ยท Tunnels ยท Proxy โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Hetzner VPS โ”‚ + โ”‚ synthos-node-1 โ”‚ + โ”‚ โ”‚ + โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ + โ”‚ โ”‚ Nginx Proxy Manager โ”‚ โ”‚ + โ”‚ โ”‚ Reverse Proxy / SSL โ”‚ โ”‚ + โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ + โ”‚ โ”‚ โ”‚ + โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ + โ”‚ โ”‚ Portainer โ”‚ โ”‚ + โ”‚ โ”‚ Docker Management โ”‚ โ”‚ + โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ + โ”‚ โ”‚ โ”‚ + โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ + โ”‚ โ”‚ CedOps Platform โ”‚ โ”‚ + โ”‚ โ”‚ โ”‚ โ”‚ + โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚ + โ”‚ โ”‚ โ”‚ Paperclip โ”‚ โ”‚ SugeBot โ”‚ โ”‚ โ”‚ + โ”‚ โ”‚ โ”‚ AI Orch. โ”‚ โ”‚ Telegram Ops โ”‚ โ”‚ โ”‚ + โ”‚ โ”‚ โ”‚ 8 Agents โ”‚ โ”‚ (Hermes Agent) โ”‚ โ”‚ โ”‚ + โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ + โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ + โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚ + โ”‚ โ”‚ โ”‚ OpenRouter โ”‚ โ”‚ n8n โ”‚ โ”‚ โ”‚ + โ”‚ โ”‚ โ”‚ AI Gateway โ”‚ โ”‚ Automation โ”‚ โ”‚ โ”‚ + โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ + โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ + โ”‚ โ”‚ + โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ + โ”‚ โ”‚ Observability Stack โ”‚ โ”‚ + โ”‚ โ”‚ Prometheus โ†’ Grafana NOC Dashboard โ”‚ โ”‚ + โ”‚ โ”‚ Blackbox Exporter โ†’ Service Uptime โ”‚ โ”‚ + โ”‚ โ”‚ Node Exporter โ†’ VPS Resource Health โ”‚ โ”‚ + โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +--- + +## ๐Ÿค– AI Agent Roster + +All agents run inside **Paperclip** โ€” the AI Agent Orchestration Platform at `paperclip.synthossystems.com`. + +| Agent | Role | Status | +|---|---|---| +| **Suge** | CTO / Operations (Hermes Agent) โ€” also deployed as SugeBot on Telegram | โœ… Live | +| **CEO** | Executive decision-making and strategic oversight | โœ… Live | +| **ClaudeCoder** | Software engineering and code generation | โœ… Live | +| **Engineer** | Infrastructure and systems engineering | โœ… Live | +| **Marketing** | Marketing strategy and content | โœ… Live | +| **Offer Builder Agent** | Sales offer generation and pipeline | โœ… Live | +| **Research** | Market research and intelligence gathering | โœ… Live | +| **Sales Agent** | Lead qualification and outreach | โœ… Live | + +> **Suge** serves dual duty โ€” the primary Hermes-style ops agent inside Paperclip, and the external-facing **SugeBot** on Telegram for real-time operational commands. + +--- + +## ๐Ÿ–ฅ๏ธ Infrastructure Layer + +All services run on a **Hetzner VPS** (`synthos-node-1`) behind Cloudflare Tunnels with zero exposed ports. + +| Service | Role | URL | +|---|---|---| +| **VPS / Portainer** | Docker container management | Internal | +| **Nginx Proxy Manager** | Reverse proxy + SSL termination | Internal | +| **Homepage** | CedOps command center dashboard | `ops.synthossystems.com` | +| **Synthos Website** | Business-facing website | `synthossystems.com` | +| **Ced's Home Lab** | Homelab / Portfolio platform | `cedshomelab.com` | + +**Traffic Flow:** +``` +Internet โ†’ Cloudflare Edge โ†’ Tunnel โ†’ Nginx Proxy Manager โ†’ Docker Services +``` + +--- + +## ๐Ÿ“Š Observability & NOC + +The CedOps NOC dashboard monitors the entire VPS stack in real-time. + +### Executive Health Panel +- **Synthos VPS Status** โ€” UP/DOWN state with instant alerting +- **Public Services Up** โ€” count of healthy endpoints (target: 5/5) +- **NOC Health Score** โ€” composite platform health percentage +- **VPS Uptime** โ€” continuous uptime tracking in days + +### VPS Resource Health +- CPU Usage % โ€” `synthos-node-1` CPU time series +- Memory Utilization % โ€” real-time memory pressure +- Disk Used % โ€” current at 27% +- Network Traffic โ€” RX/TX bandwidth graphs (30s refresh) + +### Public Service Availability + +Tracked via **Blackbox Exporter** HTTP probes: + +| Service | Endpoint | Status | +|---|---|---| +| cedshomelab | cedshomelab.com | โœ… UP | +| n8n | n8n.synthossystems.com | โœ… UP | +| cedops | ops.synthossystems.com | โœ… UP | +| paperclip | paperclip.synthossystems.com | โœ… UP | +| synthos-website | synthossystems.com | โœ… UP | + +### Reliability Metrics +- HTTP Status Codes per endpoint (all returning `200`) +- Public Endpoint Response Time (ms) โ€” time series graph +- SSL Expiry Days โ€” tracked per certificate with warning thresholds + +--- + +## ๐Ÿ”’ Security Architecture + +CedOps is designed with **zero public infrastructure exposure**. + +| Control | Implementation | +|---|---| +| No open ports | Cloudflare Tunnel handles all ingress | +| No public Prometheus | Metrics stay private; only Grafana is exposed | +| SSL everywhere | NPM handles termination for all services | +| Container isolation | Docker network segmentation via Portainer | +| Secret management | No secrets in this repo โ€” handled via environment variables | + +--- + +## ๐Ÿ“‚ Repository Structure + +``` +cedops-ai-infrastructure-noc/ +โ”‚ +โ”œโ”€โ”€ docs/ +โ”‚ โ”œโ”€โ”€ architecture/ +โ”‚ โ”œโ”€โ”€ screenshots/ +โ”‚ โ”‚ โ””โ”€โ”€ dashboard_v1.png +โ”‚ โ””โ”€โ”€ sops/ +โ”‚ +โ”œโ”€โ”€ grafana/ +โ”‚ โ””โ”€โ”€ cedops-ai-infrastructure-noc-v1.json +โ”‚ +โ”œโ”€โ”€ prometheus/ +โ”‚ โ”œโ”€โ”€ prometheus-example.yml +โ”‚ โ””โ”€โ”€ blackbox-targets-example.yml +โ”‚ +โ”œโ”€โ”€ diagrams/ +โ”‚ โ””โ”€โ”€ cedops-architecture.mmd +โ”‚ +โ””โ”€โ”€ README.md +``` + +> **Note:** Prometheus and Blackbox configs are sanitized for public sharing. Production configs are managed privately via environment variables and are never committed to this repo. + +--- + +## ๐Ÿ› ๏ธ Stack + +| Layer | Technology | +|---|---| +| **Compute** | Hetzner VPS (synthos-node-1) | +| **Containers** | Docker + Portainer | +| **Reverse Proxy** | Nginx Proxy Manager | +| **DNS & Tunnels** | Cloudflare | +| **AI Orchestration** | Paperclip | +| **AI Gateway** | OpenRouter | +| **Automation** | n8n | +| **Ops Agent** | SugeBot (Hermes / Telegram) | +| **Metrics** | Prometheus + Node Exporter + Blackbox Exporter | +| **Visualization** | Grafana | + +--- + +## ๐Ÿ”ฎ Roadmap + +### V2 Dashboard +- [ ] Agent activity metrics (Paperclip runs, success rate, token spend) +- [ ] SugeBot command log panel +- [ ] Per-agent health monitoring +- [ ] n8n workflow execution tracking +- [ ] Telegram alert integration via SugeBot + +### Platform Enhancements +- [ ] AI Incident Detection โ€” anomaly-triggered agent escalation +- [ ] Automated Remediation โ€” n8n workflows triggered by Prometheus alerts +- [ ] Multi-environment monitoring (homelab + VPS unified NOC) +- [ ] Discord alert channel + +--- + +## ๐Ÿ‘ค Author + +**Chase Dumphord** +Digital Systems Engineer | Platform Engineering | AI Infrastructure | Observability + +[![LinkedIn](https://img.shields.io/badge/LinkedIn-chase--dumphord-0A66C2?style=flat-square)](https://www.linkedin.com/in/toochase-dumphord/) +[![GitHub](https://img.shields.io/badge/GitHub-ced4568-181717?style=flat-square)](https://github.com/ced4568) +[![Live NOC](https://img.shields.io/badge/Live%20NOC-noc.chasedumphord.com-1D9E75?style=flat-square)](https://noc.chasedumphord.com) +[![Portfolio](https://img.shields.io/badge/Portfolio-chasedumphord.com-0F6E56?style=flat-square)](https://chasedumphord.com) + +--- + +## ๐Ÿ”— Related Repos + +| Repo | Description | +|---|---| +| [ceds-homelab](https://github.com/ced4568/ceds-homelab) | 6-node Proxmox cluster + 12-node K3s + full homelab infrastructure | +| [ced-noc](https://github.com/ced4568/ceds-noc) | Custom-built public NOC dashboard | +| [ced-k3s-homelab](https://github.com/ced4568/ced-k3s-homelab) | 12-nodeRaspberry Pi K3s cluster detail | \ No newline at end of file diff --git a/grafana/cedops-ai-infrastructure-noc-v2.json b/grafana/cedops-ai-infrastructure-noc-v2.json new file mode 100644 index 0000000..10d1068 --- /dev/null +++ b/grafana/cedops-ai-infrastructure-noc-v2.json @@ -0,0 +1,1555 @@ +{ + "annotations": { + "list": [] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 1, + "id": null, + "links": [], + "panels": [ + { + "id": 1, + "type": "row", + "title": "Executive Reliability Overview", + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 0 + }, + "collapsed": false, + "panels": [] + }, + { + "id": 2, + "type": "stat", + "title": "Synthos VPS", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 0, + "y": 1 + }, + "targets": [ + { + "expr": "up{instance=\"synthos-node-1\"}", + "legendFormat": "Synthos VPS", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "green", + "value": 1 + } + ] + }, + "mappings": [ + { + "type": "value", + "options": { + "0": { + "text": "DOWN", + "color": "red" + }, + "1": { + "text": "UP", + "color": "green" + } + } + } + ] + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ], + "fields": "" + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto" + } + }, + { + "id": 3, + "type": "stat", + "title": "NOC Health Score", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 4, + "w": 5, + "x": 4, + "y": 1 + }, + "targets": [ + { + "expr": "100 * avg(probe_success{job=\"blackbox-http-public\",site=~\"synthos-systems|ceds-home-lab\"})", + "legendFormat": "NOC Health Score", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "percent", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "yellow", + "value": 80 + }, + { + "color": "green", + "value": 95 + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ], + "fields": "" + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto" + } + }, + { + "id": 4, + "type": "stat", + "title": "24h Public SLA", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 4, + "w": 5, + "x": 9, + "y": 1 + }, + "targets": [ + { + "expr": "100 * avg(avg_over_time(probe_success{job=\"blackbox-http-public\",site=~\"synthos-systems|ceds-home-lab\"}[24h]))", + "legendFormat": "24h Public SLA", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "percent", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "yellow", + "value": 95 + }, + { + "color": "green", + "value": 99 + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ], + "fields": "" + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto" + } + }, + { + "id": 5, + "type": "stat", + "title": "Active Alerts", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 4, + "w": 5, + "x": 14, + "y": 1 + }, + "targets": [ + { + "expr": "sum(ALERTS{alertstate=\"firing\"}) OR vector(0)", + "legendFormat": "Active Alerts", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 1 + }, + { + "color": "red", + "value": 3 + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ], + "fields": "" + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto" + } + }, + { + "id": 6, + "type": "stat", + "title": "Min SSL Days", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 4, + "w": 5, + "x": 19, + "y": 1 + }, + "targets": [ + { + "expr": "min((probe_ssl_earliest_cert_expiry{job=\"blackbox-http-public\",site=~\"synthos-systems|ceds-home-lab\"} - time()) / 86400)", + "legendFormat": "Min SSL Days", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "d", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "yellow", + "value": 14 + }, + { + "color": "green", + "value": 30 + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ], + "fields": "" + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto" + } + }, + { + "id": 7, + "type": "row", + "title": "VPS Capacity & Host Health", + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 5 + }, + "collapsed": false, + "panels": [] + }, + { + "id": 8, + "type": "timeseries", + "title": "VPS CPU Usage %", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 0, + "y": 6 + }, + "targets": [ + { + "expr": "100 - (avg by(instance)(rate(node_cpu_seconds_total{instance=\"synthos-node-1\",mode=\"idle\"}[5m])) * 100)", + "legendFormat": "CPU", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "percent", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 70 + }, + { + "color": "red", + "value": 90 + } + ] + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 9, + "type": "timeseries", + "title": "VPS Memory Usage %", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 8, + "y": 6 + }, + "targets": [ + { + "expr": "(1 - (node_memory_MemAvailable_bytes{instance=\"synthos-node-1\"} / node_memory_MemTotal_bytes{instance=\"synthos-node-1\"})) * 100", + "legendFormat": "Memory", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "percent", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 75 + }, + { + "color": "red", + "value": 90 + } + ] + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 10, + "type": "timeseries", + "title": "VPS Disk Usage %", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 16, + "y": 6 + }, + "targets": [ + { + "expr": "100 - ((node_filesystem_avail_bytes{instance=\"synthos-node-1\",mountpoint=\"/\",fstype!=\"rootfs\"} * 100) / node_filesystem_size_bytes{instance=\"synthos-node-1\",mountpoint=\"/\",fstype!=\"rootfs\"})", + "legendFormat": "Disk /", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "percent", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 70 + }, + { + "color": "red", + "value": 85 + } + ] + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 11, + "type": "timeseries", + "title": "VPS Network Traffic", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 0, + "y": 13 + }, + "targets": [ + { + "expr": "sum by(instance)(rate(node_network_receive_bytes_total{instance=\"synthos-node-1\",device!~\"lo|veth.*|docker.*|br-.*\"}[5m]))", + "legendFormat": "RX", + "refId": "A" + }, + { + "expr": "sum by(instance)(rate(node_network_transmit_bytes_total{instance=\"synthos-node-1\",device!~\"lo|veth.*|docker.*|br-.*\"}[5m]))", + "legendFormat": "TX", + "refId": "B" + } + ], + "fieldConfig": { + "defaults": { + "unit": "Bps", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 12, + "type": "timeseries", + "title": "VPS Load Average", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 13 + }, + "targets": [ + { + "expr": "node_load1{instance=\"synthos-node-1\"}", + "legendFormat": "1m", + "refId": "A" + }, + { + "expr": "node_load5{instance=\"synthos-node-1\"}", + "legendFormat": "5m", + "refId": "B" + }, + { + "expr": "node_load15{instance=\"synthos-node-1\"}", + "legendFormat": "15m", + "refId": "C" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 13, + "type": "row", + "title": "Public Services: Availability, Latency, Status", + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 20 + }, + "collapsed": false, + "panels": [] + }, + { + "id": 14, + "type": "stat", + "title": "Public Services Up", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 0, + "y": 21 + }, + "targets": [ + { + "expr": "sum(probe_success{job=\"blackbox-http-public\",site=~\"synthos-systems|ceds-home-lab\"})", + "legendFormat": "Public Services Up", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "yellow", + "value": 3 + }, + { + "color": "green", + "value": 5 + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ], + "fields": "" + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto" + } + }, + { + "id": 15, + "type": "stat", + "title": "Public Services Total", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 6, + "y": 21 + }, + "targets": [ + { + "expr": "count(probe_success{job=\"blackbox-http-public\",site=~\"synthos-systems|ceds-home-lab\"})", + "legendFormat": "Public Services Total", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ], + "fields": "" + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto" + } + }, + { + "id": 16, + "type": "stat", + "title": "Avg Response Time", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 12, + "y": 21 + }, + "targets": [ + { + "expr": "avg(probe_duration_seconds{job=\"blackbox-http-public\",site=~\"synthos-systems|ceds-home-lab\"})", + "legendFormat": "Avg Response Time", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "s", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 1 + }, + { + "color": "red", + "value": 3 + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ], + "fields": "" + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto" + } + }, + { + "id": 17, + "type": "stat", + "title": "HTTP 2xx Services", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 18, + "y": 21 + }, + "targets": [ + { + "expr": "count(probe_http_status_code{job=\"blackbox-http-public\",site=~\"synthos-systems|ceds-home-lab\"} >= 200 <= 299)", + "legendFormat": "HTTP 2xx Services", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "yellow", + "value": 3 + }, + { + "color": "green", + "value": 5 + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ], + "fields": "" + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto" + } + }, + { + "id": 18, + "type": "timeseries", + "title": "Public Endpoint Response Time", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 25 + }, + "targets": [ + { + "expr": "probe_duration_seconds{job=\"blackbox-http-public\",site=~\"synthos-systems|ceds-home-lab\"}", + "legendFormat": "{{service}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "s", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 1 + }, + { + "color": "red", + "value": 3 + } + ] + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 19, + "type": "timeseries", + "title": "HTTP Status Codes", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 25 + }, + "targets": [ + { + "expr": "probe_http_status_code{job=\"blackbox-http-public\",site=~\"synthos-systems|ceds-home-lab\"}", + "legendFormat": "{{service}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 20, + "type": "row", + "title": "SSL / SLA / Reliability", + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 33 + }, + "collapsed": false, + "panels": [] + }, + { + "id": 21, + "type": "timeseries", + "title": "SSL Certificate Days Remaining", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 34 + }, + "targets": [ + { + "expr": "(probe_ssl_earliest_cert_expiry{job=\"blackbox-http-public\",site=~\"synthos-systems|ceds-home-lab\"} - time()) / 86400", + "legendFormat": "{{service}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "d", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "yellow", + "value": 14 + }, + { + "color": "green", + "value": 30 + } + ] + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 22, + "type": "timeseries", + "title": "24h Service SLA %", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 34 + }, + "targets": [ + { + "expr": "avg_over_time(probe_success{job=\"blackbox-http-public\",site=~\"synthos-systems|ceds-home-lab\"}[24h]) * 100", + "legendFormat": "{{service}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "percent", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "yellow", + "value": 95 + }, + { + "color": "green", + "value": 99 + } + ] + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 23, + "type": "timeseries", + "title": "Probe Failure Timeline", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 24, + "x": 0, + "y": 42 + }, + "targets": [ + { + "expr": "1 - probe_success{job=\"blackbox-http-public\",site=~\"synthos-systems|ceds-home-lab\"}", + "legendFormat": "{{service}} failure", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 24, + "type": "row", + "title": "Prometheus / Scrape Health", + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 49 + }, + "collapsed": false, + "panels": [] + }, + { + "id": 25, + "type": "stat", + "title": "Targets Down", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 0, + "y": 50 + }, + "targets": [ + { + "expr": "count(up == 0)", + "legendFormat": "Targets Down", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 1 + }, + { + "color": "red", + "value": 3 + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ], + "fields": "" + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto" + } + }, + { + "id": 26, + "type": "stat", + "title": "Node Exporters Up", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 6, + "y": 50 + }, + "targets": [ + { + "expr": "sum(up{job=~\".*node-exporter.*\"})", + "legendFormat": "Node Exporters Up", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "yellow", + "value": 3 + }, + { + "color": "green", + "value": 6 + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ], + "fields": "" + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto" + } + }, + { + "id": 27, + "type": "stat", + "title": "Blackbox Public Up", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 12, + "y": 50 + }, + "targets": [ + { + "expr": "sum(probe_success{job=\"blackbox-http-public\"})", + "legendFormat": "Blackbox Public Up", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "yellow", + "value": 3 + }, + { + "color": "green", + "value": 5 + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ], + "fields": "" + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto" + } + }, + { + "id": 28, + "type": "stat", + "title": "TSDB Head Series", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 18, + "y": 50 + }, + "targets": [ + { + "expr": "prometheus_tsdb_head_series", + "legendFormat": "TSDB Head Series", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 100000 + }, + { + "color": "red", + "value": 250000 + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ], + "fields": "" + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto" + } + }, + { + "id": 29, + "type": "timeseries", + "title": "Scrape Duration by Job", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 0, + "y": 54 + }, + "targets": [ + { + "expr": "scrape_duration_seconds", + "legendFormat": "{{job}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "s", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 30, + "type": "timeseries", + "title": "Samples Scraped by Job", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 54 + }, + "targets": [ + { + "expr": "scrape_samples_scraped", + "legendFormat": "{{job}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 31, + "type": "row", + "title": "Ops Notes / Cost Awareness", + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 61 + }, + "collapsed": false, + "panels": [] + }, + { + "id": 32, + "type": "text", + "title": "CedOps Operational Notes", + "gridPos": { + "h": 7, + "w": 12, + "x": 0, + "y": 62 + }, + "options": { + "mode": "markdown", + "content": "### Operating Model\n\n- **V1** = executive NOC overview\n- **V2** = platform engineering drill-down\n- VPS metrics are scraped over **Tailscale**, not the public internet\n- Public services are monitored through **Blackbox Exporter**\n- Designed for portfolio, interview, and operations use\n\n### Next Improvements\n\n- Telegram/Discord alert routing\n- Container-level metrics via cAdvisor\n- Synthetic checks for Paperclip/n8n workflows\n- Incident timeline and MTTR tracking" + } + }, + { + "id": 33, + "type": "text", + "title": "Cost + Capacity Awareness", + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 62 + }, + "options": { + "mode": "markdown", + "content": "### Cost-Aware Platform Notes\n\nTrack whether the VPS is right-sized:\n\n- CPU sustained over 70% = review workload sizing\n- Memory sustained over 75% = review containers/models\n- Disk over 70% = plan cleanup or expansion\n- Public latency over 1s = investigate routing/app performance\n\nThis section shows business-aware platform engineering: reliability, cost, and capacity together." + } + } + ], + "refresh": "30s", + "schemaVersion": 39, + "style": "dark", + "tags": [ + "cedops", + "synthos", + "prometheus", + "blackbox", + "platform-engineering", + "v2" + ], + "templating": { + "list": [ + { + "current": {}, + "hide": 0, + "includeAll": false, + "label": "Prometheus", + "multi": false, + "name": "DS_PROMETHEUS", + "options": [], + "query": "prometheus", + "refresh": 1, + "regex": "", + "type": "datasource" + } + ] + }, + "time": { + "from": "now-24h", + "to": "now" + }, + "timepicker": {}, + "timezone": "browser", + "title": "CedOps AI Infrastructure NOC V2", + "uid": "cedops-ai-infra-noc-v2", + "version": 1, + "weekStart": "" +} \ No newline at end of file diff --git a/image.png b/image.png deleted file mode 100644 index 77d0b9e..0000000 Binary files a/image.png and /dev/null differ