Ceds-HomeLab/cedops-ai-infrastructure-noc
1
1
Fork 0
mirror of https://github.com/ced4568/cedops-ai-infrastructure-noc.git synced 2026-08-13 05:44:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
fcfe879095cffa46f0ad3872b199b537ff1e6a30
cedops-ai-infrastructure-noc/README.md
T

339 lines
16 KiB
Markdown
Raw Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 🤖 CedOps — AI Infrastructure Operations NOC
> A live AI operations platform running on Hetzner VPS — 8 autonomous agents orchestrated by Paperclip, monitored in real-time by Prometheus (hosted on Proxmox homelab) scraping over Tailscale, visualized in a Grafana NOC dashboard.
[![Live NOC](https://img.shields.io/badge/Live%20NOC-noc.chasedumphord.com-1D9E75?style=flat-square)](https://noc.chasedumphord.com)
[![Paperclip](https://img.shields.io/badge/Paperclip-AI%20Agent%20Platform-6B46C1?style=flat-square)](https://paperclip.synthossystems.com)
[![Synthos Systems](https://img.shields.io/badge/Synthos%20Systems-synthossystems.com-0EA5E9?style=flat-square)](https://synthossystems.com)
[![Portfolio](https://img.shields.io/badge/Portfolio-chasedumphord.com-0F6E56?style=flat-square)](https://chasedumphord.com)
[![Agents](https://img.shields.io/badge/AI%20Agents-8%20Active-orange?style=flat-square)](#-ai-agent-roster)
[![Platform](https://img.shields.io/badge/Platform-Hetzner%20VPS-D4001A?style=flat-square)](#️-infrastructure-layer)
[![Monitoring](https://img.shields.io/badge/Monitoring-Prometheus%20%2B%20Grafana-F46800?style=flat-square)](#-observability--noc)
---
## 📸 NOC Dashboard
CedOps dashboards are intentionally versioned to demonstrate increasing operational maturity — from foundational observability to SRE-style incident response and reliability engineering.
### V3 — Current (SRE / Incident Response)
[![CedOps NOC Dashboard V3](docs/screenshots/dashboard_v3-overview.png)](docs/screenshots/dashboard_v3-overview.png)
#### Incident Response & Dependencies
[![CedOps Incident View](docs/screenshots/dashboard_v3-incidents.png)](docs/screenshots/dashboard_v3-incidents.png)
#### Reliability / SLO Monitoring
[![CedOps SLO View](docs/screenshots/dashboard_v3-slo.png)](docs/screenshots/dashboard_v3-slo.png)
---
### V2 — Operational Reliability
[![CedOps NOC Dashboard V2](docs/screenshots/dashboard_v2.png)](docs/screenshots/dashboard_v2.png)
---
### V1 — Foundational Observability
[![CedOps NOC Dashboard V1](docs/screenshots/dashboard_v1.png)](docs/screenshots/dashboard_v1.png)
> **Live:** Public services monitored every 30 seconds · Prometheus scraping via Tailscale · Grafana visualizing hybrid infrastructure health
---
## 🧭 Overview
CedOps is a **production-style AI operations platform** built to simulate a real-world Platform Engineering and AI Infrastructure environment.
The VPS stack runs on Hetzner and is monitored by Prometheus hosted on a Proxmox homelab node, communicating over a private Tailscale network — with everything visualized in a live Grafana NOC dashboard.
The platform combines:
* **Paperclip** — AI Agent Orchestration Platform managing 8 autonomous agents
* **SugeBot** — Hermes AI agent deployed as Telegram Operations Assistant (CTO/Ops role)
* **n8n** — Automation platform for workflow execution and inter-service routing
* **OpenRouter** — AI model gateway serving as the LLM backbone for all agents
* **Prometheus** — Hosted on Proxmox homelab, scraping VPS services over Tailscale
* **Grafana** — Full observability NOC dashboard visualizing the entire stack in real-time
* **Nginx Proxy Manager** — Reverse proxy + SSL management across all services
* **Docker + Portainer** — Container infrastructure and management layer
---
## 🏗️ Architecture Overview
```text
Internet
│
┌─────────────┴─────────────┐
│ Cloudflare Edge │
│ DNS · Tunnels · Proxy │
└─────────────┬─────────────┘
│
┌─────────────▼─────────────┐
│ Hetzner VPS │
│ synthos-node-1 │
│ │
│ ┌──────────────────────┐ │
│ │ Nginx Proxy Manager │ │
│ │ Reverse Proxy / SSL │ │
│ └──────────┬───────────┘ │
│ │ │
│ ┌──────────▼───────────┐ │
│ │ Portainer │ │
│ │ Docker Management │ │
│ └──────────┬───────────┘ │
│ │ │
│ ┌──────────▼──────────────────────────────────┐ │
│ │ CedOps Platform │ │
│ │ │ │
│ │ ┌─────────────┐ ┌──────────────────────┐ │ │
│ │ │ Paperclip │ │ SugeBot │ │ │
│ │ │ AI Orch. │ │ Telegram Ops │ │ │
│ │ │ 8 Agents │ │ (Hermes Agent) │ │ │
│ │ └──────┬──────┘ └──────────┬───────────┘ │ │
│ │ │ │ │ │
│ │ ┌──────▼──────┐ ┌──────────▼───────────┐ │ │
│ │ │ OpenRouter │ │ n8n │ │ │
│ │ │ AI Gateway │ │ Automation │ │ │
│ │ └─────────────┘ └──────────────────────┘ │ │
│ └───────────────────────────────────────────────┘ │
│ ▲ Node Exporter + Blackbox │
└──────────────┼──────────────────────────────────────┘
│ Tailscale (encrypted private tunnel)
┌──────────────┴──────────────────────┐
│ Proxmox Homelab │
│ │
│ ┌─────────────┐ ┌───────────────┐ │
│ │ Prometheus │ │ Grafana │ │
│ │ Metrics │→ │ NOC Dashboard │ │
│ │ Collection │ │ │ │
│ └─────────────┘ └───────────────┘ │
└─────────────────────────────────────┘
```
---
## 🤖 AI Agent Roster
All agents run inside **Paperclip** — the AI Agent Orchestration Platform at `paperclip.synthossystems.com`.
| Agent | Role | Status |
| ----------------------- | ---------------------------------------------------------------------- | ------ |
| **Suge** | CTO / Operations (Hermes Agent) — also deployed as SugeBot on Telegram | ✅ Live |
| **CEO** | Executive decision-making and strategic oversight | ✅ Live |
| **ClaudeCoder** | Software engineering and code generation | ✅ Live |
| **Engineer** | Infrastructure and systems engineering | ✅ Live |
| **Marketing** | Marketing strategy and content | ✅ Live |
| **Offer Builder Agent** | Sales offer generation and pipeline | ✅ Live |
| **Research** | Market research and intelligence gathering | ✅ Live |
| **Sales Agent** | Lead qualification and outreach | ✅ Live |
> **Suge** serves dual duty — the primary Hermes-style ops agent inside Paperclip, and the external-facing **SugeBot** on Telegram for real-time operational commands.
---
## 🖥️ Infrastructure Layer
All services run on a **Hetzner VPS** (`synthos-node-1`) behind Cloudflare Tunnels with zero exposed ports.
| Service | Role | URL |
| ----------------------- | ------------------------------- | ------------------------ |
| **VPS / Portainer** | Docker container management | Internal |
| **Nginx Proxy Manager** | Reverse proxy + SSL termination | Internal |
| **Homepage** | CedOps command center dashboard | `ops.synthossystems.com` |
| **Synthos Website** | Business-facing website | `synthossystems.com` |
| **Ced's Home Lab** | Homelab / Portfolio platform | `cedshomelab.com` |
**Traffic Flow:**
```text
Internet → Cloudflare Edge → Tunnel → Nginx Proxy Manager → Docker Services
```
---
## 📊 Observability & NOC
Prometheus runs on the **Proxmox homelab** and scrapes the Hetzner VPS over a **Tailscale private tunnel** — keeping metrics collection completely off the public internet.
Grafana visualizes everything in the live NOC dashboard.
### Executive Health Panel
* **Synthos VPS Status** — UP/DOWN state with instant alerting
* **Public Services Up** — count of healthy endpoints (target: 5/5)
* **NOC Health Score** — composite platform health percentage
* **VPS Uptime** — continuous uptime tracking in days
### VPS Resource Health
* CPU Usage % — `synthos-node-1` CPU time series
* Memory Utilization % — real-time memory pressure
* Disk Used % — infrastructure capacity monitoring
* Network Traffic — RX/TX bandwidth graphs (30s refresh)
### Public Service Availability
Tracked via **Blackbox Exporter** HTTP probes.
| Service | Endpoint | Status |
| --------------- | ---------------------------- | ------ |
| cedshomelab | cedshomelab.com | ✅ UP |
| n8n | n8n.synthossystems.com | ✅ UP |
| cedops | ops.synthossystems.com | ✅ UP |
| paperclip | paperclip.synthossystems.com | ✅ UP |
| synthos-website | synthossystems.com | ✅ UP |
### Reliability Metrics
* HTTP Status Codes per endpoint
* Public Endpoint Response Time
* SSL Expiry Days tracking
* Service availability monitoring
* SLO / SLA visibility
* Incident timeline monitoring
---
## 🔒 Security Architecture
CedOps is designed with **zero public infrastructure exposure**.
| Control | Implementation |
| ------------------------ | ----------------------------------------------------------- |
| No open ports | Cloudflare Tunnel handles all ingress |
| Private metrics pipeline | Prometheus → Tailscale → VPS; never public |
| SSL everywhere | NPM handles termination for all services |
| Container isolation | Docker network segmentation via Portainer |
| Secret management | No secrets in this repo — handled via environment variables |
---
## 📂 Repository Structure
```text
cedops-ai-infrastructure-noc/
│
├── docs/
│ ├── architecture/
│ ├── screenshots/
│ │ ├── dashboard_v1.png
│ │ ├── dashboard_v2.png
│ │ ├── dashboard_v3-overview.png
│ │ ├── dashboard_v3-incidents.png
│ │ └── dashboard_v3-slo.png
│ └── sops/
│
├── grafana/
│ ├── cedops-ai-infrastructure-noc-v1.json
│ ├── cedops-ai-infrastructure-noc-v2.json
│ └── cedops-sre-incident-response-v3.json
│
├── Prometheus/
│ ├── prometheus-example.yml
│ └── blackbox-targets-example.yml
│
├── diagrams/
│ └── cedops-architecture.mmd
│
└── README.md
```
> **Note:** Prometheus and Blackbox configs are sanitized for public sharing. Production configs are managed privately and never committed to this repo.
---
## 🛠️ Stack
| Layer | Technology |
| -------------------- | ---------------------------------------------- |
| **Compute** | Hetzner VPS (synthos-node-1) |
| **Containers** | Docker + Portainer |
| **Reverse Proxy** | Nginx Proxy Manager |
| **DNS & Tunnels** | Cloudflare |
| **Private Network** | Tailscale (Proxmox ↔ VPS metrics tunnel) |
| **AI Orchestration** | Paperclip |
| **AI Gateway** | OpenRouter |
| **Automation** | n8n |
| **Ops Agent** | SugeBot (Hermes / Telegram) |
| **Metrics** | Prometheus + Node Exporter + Blackbox Exporter |
| **Visualization** | Grafana |
| **Virtualization** | Proxmox VE |
---
## ✅ What's Built
### V1 Dashboard — Foundational Observability
* [x] VPS UP/DOWN status panel
* [x] Public service availability monitoring
* [x] VPS resource health (CPU, Memory, Disk, Network)
* [x] HTTP status code monitoring
* [x] Endpoint response time tracking
* [x] SSL certificate monitoring
* [x] Cross-environment scraping via Tailscale
### V2 Dashboard — Operational Reliability
* [x] SSL expiration tracking
* [x] Service reliability scoring
* [x] Public service availability metrics
* [x] Latency monitoring
* [x] Capacity awareness
* [x] Prometheus scrape visibility
* [x] Infrastructure health scoring
### V3 Dashboard — SRE / Incident Response
* [x] Incident command overview
* [x] Service dependency monitoring
* [x] Alert timeline visibility
* [x] SLO/SLA monitoring
* [x] Error budget awareness
* [x] Incident response workflow
* [x] Infrastructure pressure detection
* [x] Reliability engineering metrics
---
## 🔮 Roadmap
### V4 — AI-Assisted Operations
* [ ] AI Incident Detection
* [ ] Automated Remediation (n8n workflows triggered by alerts)
* [ ] Telegram alert integration via SugeBot
* [ ] Discord notifications
* [ ] Paperclip agent telemetry dashboard
* [ ] Agent execution analytics
* [ ] Token spend / cost observability
* [ ] Multi-environment monitoring (Homelab + VPS unified NOC)
---
## 👤 Author
**Chase Dumphord**
Digital Systems Engineer | Platform Engineering | AI Infrastructure | Observability
[![LinkedIn](https://img.shields.io/badge/LinkedIn-chase--dumphord-0A66C2?style=flat-square)](https://www.linkedin.com/in/toochase-dumphord/)
[![GitHub](https://img.shields.io/badge/GitHub-ced4568-181717?style=flat-square)](https://github.com/ced4568)
[![Live NOC](https://img.shields.io/badge/Live%20NOC-noc.chasedumphord.com-1D9E75?style=flat-square)](https://noc.chasedumphord.com)
[![Portfolio](https://img.shields.io/badge/Portfolio-chasedumphord.com-0F6E56?style=flat-square)](https://chasedumphord.com)
---
## 🔗 Related Repos
| Repo | Description |
| ------------------------------------------------------------- | ------------------------------------------------------------------ |
| [ceds-homelab](https://github.com/ced4568/ceds-homelab) | 6-node Proxmox cluster + 12-node K3s + full homelab infrastructure |
| [ced-k3s-homelab](https://github.com/ced4568/ced-k3s-homelab) | 12-node Raspberry Pi K3s cluster detail |
| [ceds-observability-stack](https://github.com/ced4568) | Observability stack configs and dashboards |
| [ceds-noc](https://github.com/ced4568/ceds-noc) | Custom-built public NOC status page |
Reference in New Issue View Git Blame Copy Permalink