diff --git a/README.md b/README.md index e5b8168..b4c0e89 100644 --- a/README.md +++ b/README.md @@ -14,38 +14,58 @@ ## ๐Ÿ“ธ NOC Dashboard -### V2 โ€” Current +CedOps dashboards are intentionally versioned to demonstrate increasing operational maturity โ€” from foundational observability to SRE-style incident response and reliability engineering. + +### V3 โ€” Current (SRE / Incident Response) + +[![CedOps NOC Dashboard V3](docs/screenshots/dashboard_v3-overview.png)](docs/screenshots/dashboard_v3-overview.png) + +#### Incident Response & Dependencies + +[![CedOps Incident View](docs/screenshots/dashboard_v3-incidents.png)](docs/screenshots/dashboard_v3-incidents.png) + +#### Reliability / SLO Monitoring + +[![CedOps SLO View](docs/screenshots/dashboard_v3-slo.png)](docs/screenshots/dashboard_v3-slo.png) + +--- + +### V2 โ€” Operational Reliability [![CedOps NOC Dashboard V2](docs/screenshots/dashboard_v2.png)](docs/screenshots/dashboard_v2.png) -### V1 +--- -[![CedOps NOC Dashboard V1](docs/screenshots/dashboard_V1.png)](docs/screenshots/dashboard_V1.png) +### V1 โ€” Foundational Observability -> **Live:** All 5 public services reporting UP ยท NOC Health Score 100% ยท VPS Disk 27% ยท Prometheus scraping every 30s +[![CedOps NOC Dashboard V1](docs/screenshots/dashboard_v1.png)](docs/screenshots/dashboard_v1.png) + +> **Live:** Public services monitored every 30 seconds ยท Prometheus scraping via Tailscale ยท Grafana visualizing hybrid infrastructure health --- ## ๐Ÿงญ Overview -CedOps is a **production-style AI operations platform** built to simulate a real-world Platform Engineering and AI Infrastructure environment. The VPS stack runs on Hetzner and is monitored by Prometheus hosted on a Proxmox homelab node, communicating over a private Tailscale network โ€” with everything visualized in a live Grafana NOC dashboard. +CedOps is a **production-style AI operations platform** built to simulate a real-world Platform Engineering and AI Infrastructure environment. + +The VPS stack runs on Hetzner and is monitored by Prometheus hosted on a Proxmox homelab node, communicating over a private Tailscale network โ€” with everything visualized in a live Grafana NOC dashboard. The platform combines: -- **Paperclip** โ€” AI Agent Orchestration Platform managing 8 autonomous agents -- **SugeBot** โ€” Hermes AI agent deployed as Telegram Operations Assistant (CTO/Ops role) -- **n8n** โ€” Automation platform for workflow execution and inter-service routing -- **OpenRouter** โ€” AI model gateway serving as the LLM backbone for all agents -- **Prometheus** โ€” Hosted on Proxmox homelab, scraping VPS services over Tailscale -- **Grafana** โ€” Full observability NOC dashboard visualizing the entire stack in real-time -- **Nginx Proxy Manager** โ€” Reverse proxy + SSL management across all services -- **Docker + Portainer** โ€” Container infrastructure and management layer +* **Paperclip** โ€” AI Agent Orchestration Platform managing 8 autonomous agents +* **SugeBot** โ€” Hermes AI agent deployed as Telegram Operations Assistant (CTO/Ops role) +* **n8n** โ€” Automation platform for workflow execution and inter-service routing +* **OpenRouter** โ€” AI model gateway serving as the LLM backbone for all agents +* **Prometheus** โ€” Hosted on Proxmox homelab, scraping VPS services over Tailscale +* **Grafana** โ€” Full observability NOC dashboard visualizing the entire stack in real-time +* **Nginx Proxy Manager** โ€” Reverse proxy + SSL management across all services +* **Docker + Portainer** โ€” Container infrastructure and management layer --- ## ๐Ÿ—๏ธ Architecture Overview -``` +```text Internet โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” @@ -101,16 +121,16 @@ The platform combines: All agents run inside **Paperclip** โ€” the AI Agent Orchestration Platform at `paperclip.synthossystems.com`. -| Agent | Role | Status | -|---|---|---| -| **Suge** | CTO / Operations (Hermes Agent) โ€” also deployed as SugeBot on Telegram | โœ… Live | -| **CEO** | Executive decision-making and strategic oversight | โœ… Live | -| **ClaudeCoder** | Software engineering and code generation | โœ… Live | -| **Engineer** | Infrastructure and systems engineering | โœ… Live | -| **Marketing** | Marketing strategy and content | โœ… Live | -| **Offer Builder Agent** | Sales offer generation and pipeline | โœ… Live | -| **Research** | Market research and intelligence gathering | โœ… Live | -| **Sales Agent** | Lead qualification and outreach | โœ… Live | +| Agent | Role | Status | +| ----------------------- | ---------------------------------------------------------------------- | ------ | +| **Suge** | CTO / Operations (Hermes Agent) โ€” also deployed as SugeBot on Telegram | โœ… Live | +| **CEO** | Executive decision-making and strategic oversight | โœ… Live | +| **ClaudeCoder** | Software engineering and code generation | โœ… Live | +| **Engineer** | Infrastructure and systems engineering | โœ… Live | +| **Marketing** | Marketing strategy and content | โœ… Live | +| **Offer Builder Agent** | Sales offer generation and pipeline | โœ… Live | +| **Research** | Market research and intelligence gathering | โœ… Live | +| **Sales Agent** | Lead qualification and outreach | โœ… Live | > **Suge** serves dual duty โ€” the primary Hermes-style ops agent inside Paperclip, and the external-facing **SugeBot** on Telegram for real-time operational commands. @@ -120,16 +140,17 @@ All agents run inside **Paperclip** โ€” the AI Agent Orchestration Platform at ` All services run on a **Hetzner VPS** (`synthos-node-1`) behind Cloudflare Tunnels with zero exposed ports. -| Service | Role | URL | -|---|---|---| -| **VPS / Portainer** | Docker container management | Internal | -| **Nginx Proxy Manager** | Reverse proxy + SSL termination | Internal | -| **Homepage** | CedOps command center dashboard | `ops.synthossystems.com` | -| **Synthos Website** | Business-facing website | `synthossystems.com` | -| **Ced's Home Lab** | Homelab / Portfolio platform | `cedshomelab.com` | +| Service | Role | URL | +| ----------------------- | ------------------------------- | ------------------------ | +| **VPS / Portainer** | Docker container management | Internal | +| **Nginx Proxy Manager** | Reverse proxy + SSL termination | Internal | +| **Homepage** | CedOps command center dashboard | `ops.synthossystems.com` | +| **Synthos Website** | Business-facing website | `synthossystems.com` | +| **Ced's Home Lab** | Homelab / Portfolio platform | `cedshomelab.com` | **Traffic Flow:** -``` + +```text Internet โ†’ Cloudflare Edge โ†’ Tunnel โ†’ Nginx Proxy Manager โ†’ Docker Services ``` @@ -137,36 +158,44 @@ Internet โ†’ Cloudflare Edge โ†’ Tunnel โ†’ Nginx Proxy Manager โ†’ Docker Servi ## ๐Ÿ“Š Observability & NOC -Prometheus runs on the **Proxmox homelab** and scrapes the Hetzner VPS over a **Tailscale private tunnel** โ€” keeping metrics collection completely off the public internet. Grafana visualizes everything in the live NOC dashboard. +Prometheus runs on the **Proxmox homelab** and scrapes the Hetzner VPS over a **Tailscale private tunnel** โ€” keeping metrics collection completely off the public internet. + +Grafana visualizes everything in the live NOC dashboard. ### Executive Health Panel -- **Synthos VPS Status** โ€” UP/DOWN state with instant alerting -- **Public Services Up** โ€” count of healthy endpoints (target: 5/5) -- **NOC Health Score** โ€” composite platform health percentage -- **VPS Uptime** โ€” continuous uptime tracking in days + +* **Synthos VPS Status** โ€” UP/DOWN state with instant alerting +* **Public Services Up** โ€” count of healthy endpoints (target: 5/5) +* **NOC Health Score** โ€” composite platform health percentage +* **VPS Uptime** โ€” continuous uptime tracking in days ### VPS Resource Health -- CPU Usage % โ€” `synthos-node-1` CPU time series -- Memory Utilization % โ€” real-time memory pressure -- Disk Used % โ€” current at 27% -- Network Traffic โ€” RX/TX bandwidth graphs (30s refresh) + +* CPU Usage % โ€” `synthos-node-1` CPU time series +* Memory Utilization % โ€” real-time memory pressure +* Disk Used % โ€” infrastructure capacity monitoring +* Network Traffic โ€” RX/TX bandwidth graphs (30s refresh) ### Public Service Availability -Tracked via **Blackbox Exporter** HTTP probes: +Tracked via **Blackbox Exporter** HTTP probes. -| Service | Endpoint | Status | -|---|---|---| -| cedshomelab | cedshomelab.com | โœ… UP | -| n8n | n8n.synthossystems.com | โœ… UP | -| cedops | ops.synthossystems.com | โœ… UP | -| paperclip | paperclip.synthossystems.com | โœ… UP | -| synthos-website | synthossystems.com | โœ… UP | +| Service | Endpoint | Status | +| --------------- | ---------------------------- | ------ | +| cedshomelab | cedshomelab.com | โœ… UP | +| n8n | n8n.synthossystems.com | โœ… UP | +| cedops | ops.synthossystems.com | โœ… UP | +| paperclip | paperclip.synthossystems.com | โœ… UP | +| synthos-website | synthossystems.com | โœ… UP | ### Reliability Metrics -- HTTP Status Codes per endpoint (all returning `200`) -- Public Endpoint Response Time (ms) โ€” time series graph -- SSL Expiry Days โ€” tracked per certificate with warning thresholds + +* HTTP Status Codes per endpoint +* Public Endpoint Response Time +* SSL Expiry Days tracking +* Service availability monitoring +* SLO / SLA visibility +* Incident timeline monitoring --- @@ -174,31 +203,35 @@ Tracked via **Blackbox Exporter** HTTP probes: CedOps is designed with **zero public infrastructure exposure**. -| Control | Implementation | -|---|---| -| No open ports | Cloudflare Tunnel handles all ingress | -| Private metrics pipeline | Prometheus โ†’ Tailscale โ†’ VPS; never public | -| SSL everywhere | NPM handles termination for all services | -| Container isolation | Docker network segmentation via Portainer | -| Secret management | No secrets in this repo โ€” handled via environment variables | +| Control | Implementation | +| ------------------------ | ----------------------------------------------------------- | +| No open ports | Cloudflare Tunnel handles all ingress | +| Private metrics pipeline | Prometheus โ†’ Tailscale โ†’ VPS; never public | +| SSL everywhere | NPM handles termination for all services | +| Container isolation | Docker network segmentation via Portainer | +| Secret management | No secrets in this repo โ€” handled via environment variables | --- ## ๐Ÿ“‚ Repository Structure -``` +```text cedops-ai-infrastructure-noc/ โ”‚ โ”œโ”€โ”€ docs/ โ”‚ โ”œโ”€โ”€ architecture/ โ”‚ โ”œโ”€โ”€ screenshots/ โ”‚ โ”‚ โ”œโ”€โ”€ dashboard_v1.png -โ”‚ โ”‚ โ””โ”€โ”€ dashboard_v2.png +โ”‚ โ”‚ โ”œโ”€โ”€ dashboard_v2.png +โ”‚ โ”‚ โ”œโ”€โ”€ dashboard_v3-overview.png +โ”‚ โ”‚ โ”œโ”€โ”€ dashboard_v3-incidents.png +โ”‚ โ”‚ โ””โ”€โ”€ dashboard_v3-slo.png โ”‚ โ””โ”€โ”€ sops/ โ”‚ โ”œโ”€โ”€ grafana/ โ”‚ โ”œโ”€โ”€ cedops-ai-infrastructure-noc-v1.json -โ”‚ โ””โ”€โ”€ cedops-ai-infrastructure-noc-v2.json +โ”‚ โ”œโ”€โ”€ cedops-ai-infrastructure-noc-v2.json +โ”‚ โ””โ”€โ”€ cedops-sre-incident-response-v3.json โ”‚ โ”œโ”€โ”€ Prometheus/ โ”‚ โ”œโ”€โ”€ prometheus-example.yml @@ -210,55 +243,76 @@ cedops-ai-infrastructure-noc/ โ””โ”€โ”€ README.md ``` -> **Note:** Prometheus and Blackbox configs are sanitized for public sharing. Production configs are managed privately via environment variables and are never committed to this repo. +> **Note:** Prometheus and Blackbox configs are sanitized for public sharing. Production configs are managed privately and never committed to this repo. --- ## ๐Ÿ› ๏ธ Stack -| Layer | Technology | -|---|---| -| **Compute** | Hetzner VPS (synthos-node-1) | -| **Containers** | Docker + Portainer | -| **Reverse Proxy** | Nginx Proxy Manager | -| **DNS & Tunnels** | Cloudflare | -| **Private Network** | Tailscale (Proxmox โ†” VPS metrics tunnel) | -| **AI Orchestration** | Paperclip | -| **AI Gateway** | OpenRouter | -| **Automation** | n8n | -| **Ops Agent** | SugeBot (Hermes / Telegram) | -| **Metrics** | Prometheus (Proxmox) + Node Exporter + Blackbox Exporter | -| **Visualization** | Grafana | -| **Virtualization** | Proxmox VE (homelab, hosts Prometheus + Grafana) | +| Layer | Technology | +| -------------------- | ---------------------------------------------- | +| **Compute** | Hetzner VPS (synthos-node-1) | +| **Containers** | Docker + Portainer | +| **Reverse Proxy** | Nginx Proxy Manager | +| **DNS & Tunnels** | Cloudflare | +| **Private Network** | Tailscale (Proxmox โ†” VPS metrics tunnel) | +| **AI Orchestration** | Paperclip | +| **AI Gateway** | OpenRouter | +| **Automation** | n8n | +| **Ops Agent** | SugeBot (Hermes / Telegram) | +| **Metrics** | Prometheus + Node Exporter + Blackbox Exporter | +| **Visualization** | Grafana | +| **Virtualization** | Proxmox VE | --- ## โœ… What's Built -### V1 Dashboard -- [x] VPS UP/DOWN status panel -- [x] Public service availability (5 endpoints via Blackbox Exporter) -- [x] VPS resource health (CPU, Memory, Disk, Network) -- [x] HTTP status codes per endpoint -- [x] Public endpoint response times -- [x] SSL certificate expiry tracking -- [x] Cross-environment scraping via Tailscale (Proxmox โ†’ Hetzner VPS) +### V1 Dashboard โ€” Foundational Observability -### V2 Dashboard -- [x] Agent activity metrics (Paperclip runs, success rate, token spend) -- [x] SugeBot command log panel -- [x] Per-agent health monitoring -- [x] n8n workflow execution tracking -- [x] Telegram alert integration via SugeBot +* [x] VPS UP/DOWN status panel +* [x] Public service availability monitoring +* [x] VPS resource health (CPU, Memory, Disk, Network) +* [x] HTTP status code monitoring +* [x] Endpoint response time tracking +* [x] SSL certificate monitoring +* [x] Cross-environment scraping via Tailscale + +### V2 Dashboard โ€” Operational Reliability + +* [x] SSL expiration tracking +* [x] Service reliability scoring +* [x] Public service availability metrics +* [x] Latency monitoring +* [x] Capacity awareness +* [x] Prometheus scrape visibility +* [x] Infrastructure health scoring + +### V3 Dashboard โ€” SRE / Incident Response + +* [x] Incident command overview +* [x] Service dependency monitoring +* [x] Alert timeline visibility +* [x] SLO/SLA monitoring +* [x] Error budget awareness +* [x] Incident response workflow +* [x] Infrastructure pressure detection +* [x] Reliability engineering metrics --- ## ๐Ÿ”ฎ Roadmap -- [ ] AI Incident Detection โ€” anomaly-triggered agent escalation -- [ ] Automated Remediation โ€” n8n workflows triggered by Prometheus alerts -- [ ] Multi-environment monitoring (homelab + VPS unified NOC) -- [ ] Discord alert channel +### V4 โ€” AI-Assisted Operations + +* [ ] AI Incident Detection +* [ ] Automated Remediation (n8n workflows triggered by alerts) +* [ ] Telegram alert integration via SugeBot +* [ ] Discord notifications +* [ ] Paperclip agent telemetry dashboard +* [ ] Agent execution analytics +* [ ] Token spend / cost observability +* [ ] Multi-environment monitoring (Homelab + VPS unified NOC) --- @@ -276,8 +330,9 @@ Digital Systems Engineer | Platform Engineering | AI Infrastructure | Observabil ## ๐Ÿ”— Related Repos -| Repo | Description | -|---|---| -| [ceds-homelab](https://github.com/ced4568/ceds-homelab) | 6-node Proxmox cluster + 12-node K3s + full homelab infrastructure | -| [ced-k3s-homelab](https://github.com/ced4568/ced-k3s-homelab) | 12-node Raspberry Pi K3s cluster detail | -| [ceds-noc](https://github.com/ced4568/ceds-noc) | Custom-built public NOC status page | \ No newline at end of file +| Repo | Description | +| ------------------------------------------------------------- | ------------------------------------------------------------------ | +| [ceds-homelab](https://github.com/ced4568/ceds-homelab) | 6-node Proxmox cluster + 12-node K3s + full homelab infrastructure | +| [ced-k3s-homelab](https://github.com/ced4568/ced-k3s-homelab) | 12-node Raspberry Pi K3s cluster detail | +| [ceds-observability-stack](https://github.com/ced4568) | Observability stack configs and dashboards | +| [ceds-noc](https://github.com/ced4568/ceds-noc) | Custom-built public NOC status page | diff --git a/docs/screenshots/dashboard_v3-incidents.png b/docs/screenshots/dashboard_v3-incidents.png new file mode 100644 index 0000000..f962ce5 Binary files /dev/null and b/docs/screenshots/dashboard_v3-incidents.png differ diff --git a/docs/screenshots/dashboard_v3-overview.png b/docs/screenshots/dashboard_v3-overview.png new file mode 100644 index 0000000..bc60e06 Binary files /dev/null and b/docs/screenshots/dashboard_v3-overview.png differ diff --git a/docs/screenshots/dashboard_v3-slo.png b/docs/screenshots/dashboard_v3-slo.png new file mode 100644 index 0000000..512cbbb Binary files /dev/null and b/docs/screenshots/dashboard_v3-slo.png differ