mirror of
https://github.com/ced4568/cedops-ai-infrastructure-noc.git
synced 2026-08-12 21:34:02 +00:00
Add CedOps V3 SRE dashboard and README updates
This commit is contained in:
@@ -14,38 +14,58 @@
|
||||
|
||||
## 📸 NOC Dashboard
|
||||
|
||||
### V2 — Current
|
||||
CedOps dashboards are intentionally versioned to demonstrate increasing operational maturity — from foundational observability to SRE-style incident response and reliability engineering.
|
||||
|
||||
### V3 — Current (SRE / Incident Response)
|
||||
|
||||
[](docs/screenshots/dashboard_v3-overview.png)
|
||||
|
||||
#### Incident Response & Dependencies
|
||||
|
||||
[](docs/screenshots/dashboard_v3-incidents.png)
|
||||
|
||||
#### Reliability / SLO Monitoring
|
||||
|
||||
[](docs/screenshots/dashboard_v3-slo.png)
|
||||
|
||||
---
|
||||
|
||||
### V2 — Operational Reliability
|
||||
|
||||
[](docs/screenshots/dashboard_v2.png)
|
||||
|
||||
### V1
|
||||
---
|
||||
|
||||
[](docs/screenshots/dashboard_V1.png)
|
||||
### V1 — Foundational Observability
|
||||
|
||||
> **Live:** All 5 public services reporting UP · NOC Health Score 100% · VPS Disk 27% · Prometheus scraping every 30s
|
||||
[](docs/screenshots/dashboard_v1.png)
|
||||
|
||||
> **Live:** Public services monitored every 30 seconds · Prometheus scraping via Tailscale · Grafana visualizing hybrid infrastructure health
|
||||
|
||||
---
|
||||
|
||||
## 🧭 Overview
|
||||
|
||||
CedOps is a **production-style AI operations platform** built to simulate a real-world Platform Engineering and AI Infrastructure environment. The VPS stack runs on Hetzner and is monitored by Prometheus hosted on a Proxmox homelab node, communicating over a private Tailscale network — with everything visualized in a live Grafana NOC dashboard.
|
||||
CedOps is a **production-style AI operations platform** built to simulate a real-world Platform Engineering and AI Infrastructure environment.
|
||||
|
||||
The VPS stack runs on Hetzner and is monitored by Prometheus hosted on a Proxmox homelab node, communicating over a private Tailscale network — with everything visualized in a live Grafana NOC dashboard.
|
||||
|
||||
The platform combines:
|
||||
|
||||
- **Paperclip** — AI Agent Orchestration Platform managing 8 autonomous agents
|
||||
- **SugeBot** — Hermes AI agent deployed as Telegram Operations Assistant (CTO/Ops role)
|
||||
- **n8n** — Automation platform for workflow execution and inter-service routing
|
||||
- **OpenRouter** — AI model gateway serving as the LLM backbone for all agents
|
||||
- **Prometheus** — Hosted on Proxmox homelab, scraping VPS services over Tailscale
|
||||
- **Grafana** — Full observability NOC dashboard visualizing the entire stack in real-time
|
||||
- **Nginx Proxy Manager** — Reverse proxy + SSL management across all services
|
||||
- **Docker + Portainer** — Container infrastructure and management layer
|
||||
* **Paperclip** — AI Agent Orchestration Platform managing 8 autonomous agents
|
||||
* **SugeBot** — Hermes AI agent deployed as Telegram Operations Assistant (CTO/Ops role)
|
||||
* **n8n** — Automation platform for workflow execution and inter-service routing
|
||||
* **OpenRouter** — AI model gateway serving as the LLM backbone for all agents
|
||||
* **Prometheus** — Hosted on Proxmox homelab, scraping VPS services over Tailscale
|
||||
* **Grafana** — Full observability NOC dashboard visualizing the entire stack in real-time
|
||||
* **Nginx Proxy Manager** — Reverse proxy + SSL management across all services
|
||||
* **Docker + Portainer** — Container infrastructure and management layer
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Architecture Overview
|
||||
|
||||
```
|
||||
```text
|
||||
Internet
|
||||
│
|
||||
┌─────────────┴─────────────┐
|
||||
@@ -102,7 +122,7 @@ The platform combines:
|
||||
All agents run inside **Paperclip** — the AI Agent Orchestration Platform at `paperclip.synthossystems.com`.
|
||||
|
||||
| Agent | Role | Status |
|
||||
|---|---|---|
|
||||
| ----------------------- | ---------------------------------------------------------------------- | ------ |
|
||||
| **Suge** | CTO / Operations (Hermes Agent) — also deployed as SugeBot on Telegram | ✅ Live |
|
||||
| **CEO** | Executive decision-making and strategic oversight | ✅ Live |
|
||||
| **ClaudeCoder** | Software engineering and code generation | ✅ Live |
|
||||
@@ -121,7 +141,7 @@ All agents run inside **Paperclip** — the AI Agent Orchestration Platform at `
|
||||
All services run on a **Hetzner VPS** (`synthos-node-1`) behind Cloudflare Tunnels with zero exposed ports.
|
||||
|
||||
| Service | Role | URL |
|
||||
|---|---|---|
|
||||
| ----------------------- | ------------------------------- | ------------------------ |
|
||||
| **VPS / Portainer** | Docker container management | Internal |
|
||||
| **Nginx Proxy Manager** | Reverse proxy + SSL termination | Internal |
|
||||
| **Homepage** | CedOps command center dashboard | `ops.synthossystems.com` |
|
||||
@@ -129,7 +149,8 @@ All services run on a **Hetzner VPS** (`synthos-node-1`) behind Cloudflare Tunne
|
||||
| **Ced's Home Lab** | Homelab / Portfolio platform | `cedshomelab.com` |
|
||||
|
||||
**Traffic Flow:**
|
||||
```
|
||||
|
||||
```text
|
||||
Internet → Cloudflare Edge → Tunnel → Nginx Proxy Manager → Docker Services
|
||||
```
|
||||
|
||||
@@ -137,26 +158,30 @@ Internet → Cloudflare Edge → Tunnel → Nginx Proxy Manager → Docker Servi
|
||||
|
||||
## 📊 Observability & NOC
|
||||
|
||||
Prometheus runs on the **Proxmox homelab** and scrapes the Hetzner VPS over a **Tailscale private tunnel** — keeping metrics collection completely off the public internet. Grafana visualizes everything in the live NOC dashboard.
|
||||
Prometheus runs on the **Proxmox homelab** and scrapes the Hetzner VPS over a **Tailscale private tunnel** — keeping metrics collection completely off the public internet.
|
||||
|
||||
Grafana visualizes everything in the live NOC dashboard.
|
||||
|
||||
### Executive Health Panel
|
||||
- **Synthos VPS Status** — UP/DOWN state with instant alerting
|
||||
- **Public Services Up** — count of healthy endpoints (target: 5/5)
|
||||
- **NOC Health Score** — composite platform health percentage
|
||||
- **VPS Uptime** — continuous uptime tracking in days
|
||||
|
||||
* **Synthos VPS Status** — UP/DOWN state with instant alerting
|
||||
* **Public Services Up** — count of healthy endpoints (target: 5/5)
|
||||
* **NOC Health Score** — composite platform health percentage
|
||||
* **VPS Uptime** — continuous uptime tracking in days
|
||||
|
||||
### VPS Resource Health
|
||||
- CPU Usage % — `synthos-node-1` CPU time series
|
||||
- Memory Utilization % — real-time memory pressure
|
||||
- Disk Used % — current at 27%
|
||||
- Network Traffic — RX/TX bandwidth graphs (30s refresh)
|
||||
|
||||
* CPU Usage % — `synthos-node-1` CPU time series
|
||||
* Memory Utilization % — real-time memory pressure
|
||||
* Disk Used % — infrastructure capacity monitoring
|
||||
* Network Traffic — RX/TX bandwidth graphs (30s refresh)
|
||||
|
||||
### Public Service Availability
|
||||
|
||||
Tracked via **Blackbox Exporter** HTTP probes:
|
||||
Tracked via **Blackbox Exporter** HTTP probes.
|
||||
|
||||
| Service | Endpoint | Status |
|
||||
|---|---|---|
|
||||
| --------------- | ---------------------------- | ------ |
|
||||
| cedshomelab | cedshomelab.com | ✅ UP |
|
||||
| n8n | n8n.synthossystems.com | ✅ UP |
|
||||
| cedops | ops.synthossystems.com | ✅ UP |
|
||||
@@ -164,9 +189,13 @@ Tracked via **Blackbox Exporter** HTTP probes:
|
||||
| synthos-website | synthossystems.com | ✅ UP |
|
||||
|
||||
### Reliability Metrics
|
||||
- HTTP Status Codes per endpoint (all returning `200`)
|
||||
- Public Endpoint Response Time (ms) — time series graph
|
||||
- SSL Expiry Days — tracked per certificate with warning thresholds
|
||||
|
||||
* HTTP Status Codes per endpoint
|
||||
* Public Endpoint Response Time
|
||||
* SSL Expiry Days tracking
|
||||
* Service availability monitoring
|
||||
* SLO / SLA visibility
|
||||
* Incident timeline monitoring
|
||||
|
||||
---
|
||||
|
||||
@@ -175,7 +204,7 @@ Tracked via **Blackbox Exporter** HTTP probes:
|
||||
CedOps is designed with **zero public infrastructure exposure**.
|
||||
|
||||
| Control | Implementation |
|
||||
|---|---|
|
||||
| ------------------------ | ----------------------------------------------------------- |
|
||||
| No open ports | Cloudflare Tunnel handles all ingress |
|
||||
| Private metrics pipeline | Prometheus → Tailscale → VPS; never public |
|
||||
| SSL everywhere | NPM handles termination for all services |
|
||||
@@ -186,19 +215,23 @@ CedOps is designed with **zero public infrastructure exposure**.
|
||||
|
||||
## 📂 Repository Structure
|
||||
|
||||
```
|
||||
```text
|
||||
cedops-ai-infrastructure-noc/
|
||||
│
|
||||
├── docs/
|
||||
│ ├── architecture/
|
||||
│ ├── screenshots/
|
||||
│ │ ├── dashboard_v1.png
|
||||
│ │ └── dashboard_v2.png
|
||||
│ │ ├── dashboard_v2.png
|
||||
│ │ ├── dashboard_v3-overview.png
|
||||
│ │ ├── dashboard_v3-incidents.png
|
||||
│ │ └── dashboard_v3-slo.png
|
||||
│ └── sops/
|
||||
│
|
||||
├── grafana/
|
||||
│ ├── cedops-ai-infrastructure-noc-v1.json
|
||||
│ └── cedops-ai-infrastructure-noc-v2.json
|
||||
│ ├── cedops-ai-infrastructure-noc-v2.json
|
||||
│ └── cedops-sre-incident-response-v3.json
|
||||
│
|
||||
├── Prometheus/
|
||||
│ ├── prometheus-example.yml
|
||||
@@ -210,14 +243,14 @@ cedops-ai-infrastructure-noc/
|
||||
└── README.md
|
||||
```
|
||||
|
||||
> **Note:** Prometheus and Blackbox configs are sanitized for public sharing. Production configs are managed privately via environment variables and are never committed to this repo.
|
||||
> **Note:** Prometheus and Blackbox configs are sanitized for public sharing. Production configs are managed privately and never committed to this repo.
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Stack
|
||||
|
||||
| Layer | Technology |
|
||||
|---|---|
|
||||
| -------------------- | ---------------------------------------------- |
|
||||
| **Compute** | Hetzner VPS (synthos-node-1) |
|
||||
| **Containers** | Docker + Portainer |
|
||||
| **Reverse Proxy** | Nginx Proxy Manager |
|
||||
@@ -227,38 +260,59 @@ cedops-ai-infrastructure-noc/
|
||||
| **AI Gateway** | OpenRouter |
|
||||
| **Automation** | n8n |
|
||||
| **Ops Agent** | SugeBot (Hermes / Telegram) |
|
||||
| **Metrics** | Prometheus (Proxmox) + Node Exporter + Blackbox Exporter |
|
||||
| **Metrics** | Prometheus + Node Exporter + Blackbox Exporter |
|
||||
| **Visualization** | Grafana |
|
||||
| **Virtualization** | Proxmox VE (homelab, hosts Prometheus + Grafana) |
|
||||
| **Virtualization** | Proxmox VE |
|
||||
|
||||
---
|
||||
|
||||
## ✅ What's Built
|
||||
|
||||
### V1 Dashboard
|
||||
- [x] VPS UP/DOWN status panel
|
||||
- [x] Public service availability (5 endpoints via Blackbox Exporter)
|
||||
- [x] VPS resource health (CPU, Memory, Disk, Network)
|
||||
- [x] HTTP status codes per endpoint
|
||||
- [x] Public endpoint response times
|
||||
- [x] SSL certificate expiry tracking
|
||||
- [x] Cross-environment scraping via Tailscale (Proxmox → Hetzner VPS)
|
||||
### V1 Dashboard — Foundational Observability
|
||||
|
||||
### V2 Dashboard
|
||||
- [x] Agent activity metrics (Paperclip runs, success rate, token spend)
|
||||
- [x] SugeBot command log panel
|
||||
- [x] Per-agent health monitoring
|
||||
- [x] n8n workflow execution tracking
|
||||
- [x] Telegram alert integration via SugeBot
|
||||
* [x] VPS UP/DOWN status panel
|
||||
* [x] Public service availability monitoring
|
||||
* [x] VPS resource health (CPU, Memory, Disk, Network)
|
||||
* [x] HTTP status code monitoring
|
||||
* [x] Endpoint response time tracking
|
||||
* [x] SSL certificate monitoring
|
||||
* [x] Cross-environment scraping via Tailscale
|
||||
|
||||
### V2 Dashboard — Operational Reliability
|
||||
|
||||
* [x] SSL expiration tracking
|
||||
* [x] Service reliability scoring
|
||||
* [x] Public service availability metrics
|
||||
* [x] Latency monitoring
|
||||
* [x] Capacity awareness
|
||||
* [x] Prometheus scrape visibility
|
||||
* [x] Infrastructure health scoring
|
||||
|
||||
### V3 Dashboard — SRE / Incident Response
|
||||
|
||||
* [x] Incident command overview
|
||||
* [x] Service dependency monitoring
|
||||
* [x] Alert timeline visibility
|
||||
* [x] SLO/SLA monitoring
|
||||
* [x] Error budget awareness
|
||||
* [x] Incident response workflow
|
||||
* [x] Infrastructure pressure detection
|
||||
* [x] Reliability engineering metrics
|
||||
|
||||
---
|
||||
|
||||
## 🔮 Roadmap
|
||||
|
||||
- [ ] AI Incident Detection — anomaly-triggered agent escalation
|
||||
- [ ] Automated Remediation — n8n workflows triggered by Prometheus alerts
|
||||
- [ ] Multi-environment monitoring (homelab + VPS unified NOC)
|
||||
- [ ] Discord alert channel
|
||||
### V4 — AI-Assisted Operations
|
||||
|
||||
* [ ] AI Incident Detection
|
||||
* [ ] Automated Remediation (n8n workflows triggered by alerts)
|
||||
* [ ] Telegram alert integration via SugeBot
|
||||
* [ ] Discord notifications
|
||||
* [ ] Paperclip agent telemetry dashboard
|
||||
* [ ] Agent execution analytics
|
||||
* [ ] Token spend / cost observability
|
||||
* [ ] Multi-environment monitoring (Homelab + VPS unified NOC)
|
||||
|
||||
---
|
||||
|
||||
@@ -277,7 +331,8 @@ Digital Systems Engineer | Platform Engineering | AI Infrastructure | Observabil
|
||||
## 🔗 Related Repos
|
||||
|
||||
| Repo | Description |
|
||||
|---|---|
|
||||
| ------------------------------------------------------------- | ------------------------------------------------------------------ |
|
||||
| [ceds-homelab](https://github.com/ced4568/ceds-homelab) | 6-node Proxmox cluster + 12-node K3s + full homelab infrastructure |
|
||||
| [ced-k3s-homelab](https://github.com/ced4568/ced-k3s-homelab) | 12-node Raspberry Pi K3s cluster detail |
|
||||
| [ceds-observability-stack](https://github.com/ced4568) | Observability stack configs and dashboards |
|
||||
| [ceds-noc](https://github.com/ced4568/ceds-noc) | Custom-built public NOC status page |
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 214 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 313 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 231 KiB |
Reference in New Issue
Block a user