Ceds-HomeLab/cedops-ai-infrastructure-noc
1
1
Fork 0
mirror of https://github.com/ced4568/cedops-ai-infrastructure-noc.git synced 2026-08-12 21:34:02 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
f62323e5f1cfeda4cce6d06ef36d24e30ca626d4
cedops-ai-infrastructure-noc/README.md
T

339 lines
16 KiB
Markdown
Raw Blame History

# 🤖 CedOps AI Infrastructure Operations NOC
> A live AI operations platform running on Hetzner VPS 8 autonomous agents orchestrated by Paperclip, monitored in real time by Prometheus (hosted on Proxmox homelab) scraping over Tailscale, visualized in a Grafana NOC dashboard.
[![Live NOC](https://img.shields.io/badge/Live%20NOC-noc.chasedumphord.com-1D9E75?style=flat-square)](https://noc.chasedumphord.com)
[![Paperclip](https://img.shields.io/badge/Paperclip-AI%20Agent%20Platform-6B46C1?style=flat-square)](https://paperclip.synthossystems.com)
[![Synthos Systems](https://img.shields.io/badge/Synthos%20Systems-synthossystems.com-0EA5E9?style=flat-square)](https://synthossystems.com)
[![Portfolio](https://img.shields.io/badge/Portfolio-chasedumphord.com-0F6E56?style=flat-square)](https://chasedumphord.com)
[![Agents](https://img.shields.io/badge/AI%20Agents-8%20Active-orange?style=flat-square)](#ai-agent-roster)
[![Platform](https://img.shields.io/badge/Platform-Hetzner%20VPS-D4001A?style=flat-square)](#infrastructure-layer)
[![Monitoring](https://img.shields.io/badge/Monitoring-Prometheus%20%2B%20Grafana-F46800?style=flat-square)](#observability--noc)
---
## NOC Dashboard
CedOps dashboards are intentionally versioned to demonstrate increasing operational maturity from foundational observability to SRE style incident response and reliability engineering.
### V3 Current (SRE / Incident Response)
[![CedOps NOC Dashboard V3](docs/screenshots/dashboard_v3-overview.png)](docs/screenshots/dashboard_v3-overview.png)
#### Incident Response & Dependencies
[![CedOps Incident View](docs/screenshots/dashboard_v3-incidents.png)](docs/screenshots/dashboard_v3-incidents.png)
#### Reliability / SLO Monitoring
[![CedOps SLO View](docs/screenshots/dashboard_v3-slo.png)](docs/screenshots/dashboard_v3-slo.png)
---
### V2 Operational Reliability
[![CedOps NOC Dashboard V2](docs/screenshots/dashboard_v2.png)](docs/screenshots/dashboard_v2.png)
---
### V1 Foundational Observability
[![CedOps NOC Dashboard V1](docs/screenshots/dashboard_v1.png)](docs/screenshots/dashboard_v1.png)
> **Live:** Public services monitored every 30 seconds · Prometheus scraping via Tailscale · Grafana visualizing hybrid infrastructure health
---
## Overview
CedOps is a **production-style AI operations platform** built to simulate a real world Platform Engineering and AI Infrastructure environment.
The VPS stack runs on Hetzner and is monitored by Prometheus hosted on a Proxmox homelab node, communicating over a private Tailscale network with everything visualized in a live Grafana NOC dashboard.
The platform combines:
* **Paperclip** — AI Agent Orchestration Platform managing 8 autonomous agents
* **SugeBot** — Hermes AI agent deployed as Telegram Operations Assistant (CTO/Ops role)
* **n8n** — Automation platform for workflow execution and inter-service routing
* **OpenRouter** — AI model gateway serving as the LLM backbone for all agents
* **Prometheus** — Hosted on Proxmox homelab, scraping VPS services over Tailscale
* **Grafana** — Full observability NOC dashboard visualizing the entire stack in real-time
* **Nginx Proxy Manager** — Reverse proxy + SSL management across all services
* **Docker + Portainer** — Container infrastructure and management layer
---
## Architecture Overview
```text
Internet
│
┌─────────────┴─────────────┐
│ Cloudflare Edge │
│ DNS · Tunnels · Proxy │
└─────────────┬─────────────┘
│
┌─────────────▼─────────────┐
│ Hetzner VPS │
│ synthos-node-1 │
│ │
│ ┌──────────────────────┐ │
│ │ Nginx Proxy Manager │ │
│ │ Reverse Proxy / SSL │ │
│ └──────────┬───────────┘ │
│ │ │
│ ┌──────────▼───────────┐ │
│ │ Portainer │ │
│ │ Docker Management │ │
│ └──────────┬───────────┘ │
│ │ │
│ ┌──────────▼──────────────────────────────────┐ │
│ │ CedOps Platform │ │
│ │ │ │
│ │ ┌─────────────┐ ┌──────────────────────┐ │ │
│ │ │ Paperclip │ │ SugeBot │ │ │
│ │ │ AI Orch. │ │ Telegram Ops │ │ │
│ │ │ 8 Agents │ │ (Hermes Agent) │ │ │
│ │ └──────┬──────┘ └──────────┬───────────┘ │ │
│ │ │ │ │ │
│ │ ┌──────▼──────┐ ┌──────────▼───────────┐ │ │
│ │ │ OpenRouter │ │ n8n │ │ │
│ │ │ AI Gateway │ │ Automation │ │ │
│ │ └─────────────┘ └──────────────────────┘ │ │
│ └───────────────────────────────────────────────┘ │
│ ▲ Node Exporter + Blackbox │
└──────────────┼──────────────────────────────────────┘
│ Tailscale (encrypted private tunnel)
┌──────────────┴──────────────────────┐
│ Proxmox Homelab │
│ │
│ ┌─────────────┐ ┌───────────────┐ │
│ │ Prometheus │ │ Grafana │ │
│ │ Metrics │→ │ NOC Dashboard │ │
│ │ Collection │ │ │ │
│ └─────────────┘ └───────────────┘ │
└─────────────────────────────────────┘
```
---
## AI Agent Roster
All agents run inside **Paperclip** the AI Agent Orchestration Platform at `paperclip.synthossystems.com`.
| Agent | Role | Status |
| ----------------------- | ---------------------------------------------------------------------- | ------ |
| **Suge** | CTO / Operations (Hermes Agent) — also deployed as SugeBot on Telegram | ✅ Live |
| **CEO** | Executive decision-making and strategic oversight | ✅ Live |
| **ClaudeCoder** | Software engineering and code generation | ✅ Live |
| **Engineer** | Infrastructure and systems engineering | ✅ Live |
| **Marketing** | Marketing strategy and content | ✅ Live |
| **Offer Builder Agent** | Sales offer generation and pipeline | ✅ Live |
| **Research** | Market research and intelligence gathering | ✅ Live |
| **Sales Agent** | Lead qualification and outreach | ✅ Live |
> **Suge** serves dual duty the primary Hermes style ops agent inside Paperclip, and the external-facing **SugeBot** on Telegram for real-time operational commands.
---
## Infrastructure Layer
All services run on a **Hetzner VPS** (`synthos-node-1`) behind Cloudflare Tunnels with zero exposed ports.
| Service | Role | URL |
| ----------------------- | ------------------------------- | ------------------------ |
| **VPS / Portainer** | Docker container management | Internal |
| **Nginx Proxy Manager** | Reverse proxy + SSL termination | Internal |
| **Homepage** | CedOps command center dashboard | `ops.synthossystems.com` |
| **Synthos Website** | Business-facing website | `synthossystems.com` |
| **Ced's Home Lab** | Homelab / Portfolio platform | `cedshomelab.com` |
**Traffic Flow:**
```text
Internet → Cloudflare Edge → Tunnel → Nginx Proxy Manager → Docker Services
```
---
## Observability & NOC
Prometheus runs on the **Proxmox homelab** and scrapes the Hetzner VPS over a **Tailscale private tunnel** keeping metrics collection completely off the public internet.
Grafana visualizes everything in the live NOC dashboard.
### Executive Health Panel
* **Synthos VPS Status** UP/DOWN state with instant alerting
* **Public Services Up** count of healthy endpoints (target: 5/5)
* **NOC Health Score** composite platform health percentage
* **VPS Uptime** continuous uptime tracking in days
### VPS Resource Health
* CPU Usage % `synthos-node-1` CPU time series
* Memory Utilization % real-time memory pressure
* Disk Used % infrastructure capacity monitoring
* Network Traffic RX/TX bandwidth graphs (30s refresh)
### Public Service Availability
Tracked via **Blackbox Exporter** HTTP probes.
| Service | Endpoint | Status |
| --------------- | ---------------------------- | ------ |
| cedshomelab | cedshomelab.com | ✅ UP |
| n8n | n8n.synthossystems.com | ✅ UP |
| cedops | ops.synthossystems.com | ✅ UP |
| paperclip | paperclip.synthossystems.com | ✅ UP |
| synthos-website | synthossystems.com | ✅ UP |
### Reliability Metrics
* HTTP Status Codes per endpoint
* Public Endpoint Response Time
* SSL Expiry Days tracking
* Service availability monitoring
* SLO / SLA visibility
* Incident timeline monitoring
---
## Security Architecture
CedOps is designed with **zero public infrastructure exposure**.
| Control | Implementation |
| ------------------------ | ----------------------------------------------------------- |
| No open ports | Cloudflare Tunnel handles all ingress |
| Private metrics pipeline | Prometheus → Tailscale → VPS; never public |
| SSL everywhere | NPM handles termination for all services |
| Container isolation | Docker network segmentation via Portainer |
| Secret management | No secrets in this repo handled via environment variables |
---
## Repository Structure
```text
cedops-ai-infrastructure-noc/
│
├── docs/
│ ├── architecture/
│ ├── screenshots/
│ │ ├── dashboard_v1.png
│ │ ├── dashboard_v2.png
│ │ ├── dashboard_v3-overview.png
│ │ ├── dashboard_v3-incidents.png
│ │ └── dashboard_v3-slo.png
│ └── sops/
│
├── grafana/
│ ├── cedops-ai-infrastructure-noc-v1.json
│ ├── cedops-ai-infrastructure-noc-v2.json
│ └── cedops-sre-incident-response-v3.json
│
├── Prometheus/
│ ├── prometheus-example.yml
│ └── blackbox-targets-example.yml
│
├── diagrams/
│ └── cedops-architecture.mmd
│
└── README.md
```
> **Note:** Prometheus and Blackbox configs are sanitized for public sharing. Production configs are managed privately and never committed to this repo.
---
## Stack
| Layer | Technology |
| -------------------- | ---------------------------------------------- |
| **Compute** | Hetzner VPS (synthos-node-1) |
| **Containers** | Docker + Portainer |
| **Reverse Proxy** | Nginx Proxy Manager |
| **DNS & Tunnels** | Cloudflare |
| **Private Network** | Tailscale (Proxmox ↔ VPS metrics tunnel) |
| **AI Orchestration** | Paperclip |
| **AI Gateway** | OpenRouter |
| **Automation** | n8n |
| **Ops Agent** | SugeBot (Hermes / Telegram) |
| **Metrics** | Prometheus + Node Exporter + Blackbox Exporter |
| **Visualization** | Grafana |
| **Virtualization** | Proxmox VE |
---
## What's Built
### V1 Dashboard — Foundational Observability
* [x] VPS UP/DOWN status panel
* [x] Public service availability monitoring
* [x] VPS resource health (CPU, Memory, Disk, Network)
* [x] HTTP status code monitoring
* [x] Endpoint response time tracking
* [x] SSL certificate monitoring
* [x] Cross-environment scraping via Tailscale
### V2 Dashboard — Operational Reliability
* [x] SSL expiration tracking
* [x] Service reliability scoring
* [x] Public service availability metrics
* [x] Latency monitoring
* [x] Capacity awareness
* [x] Prometheus scrape visibility
* [x] Infrastructure health scoring
### V3 Dashboard — SRE / Incident Response
* [x] Incident command overview
* [x] Service dependency monitoring
* [x] Alert timeline visibility
* [x] SLO/SLA monitoring
* [x] Error budget awareness
* [x] Incident response workflow
* [x] Infrastructure pressure detection
* [x] Reliability engineering metrics
---
## Roadmap
### V4 — AI-Assisted Operations
* [ ] AI Incident Detection
* [ ] Automated Remediation (n8n workflows triggered by alerts)
* [ ] Telegram alert integration via SugeBot
* [ ] Discord notifications
* [ ] Paperclip agent telemetry dashboard
* [ ] Agent execution analytics
* [ ] Token spend / cost observability
* [ ] Multi-environment monitoring (Homelab + VPS unified NOC)
---
## Author
**Chase Dumphord**
DevOps and Cloud Infrastructure Engineer | Platform Engineering | AI Infrastructure | Observability
[![LinkedIn](https://img.shields.io/badge/LinkedIn-chase--dumphord-0A66C2?style=flat-square)](https://www.linkedin.com/in/chase-dumphord/)
[![GitHub](https://img.shields.io/badge/GitHub-ced4568-181717?style=flat-square)](https://github.com/ced4568)
[![Live NOC](https://img.shields.io/badge/Live%20NOC-noc.chasedumphord.com-1D9E75?style=flat-square)](https://noc.chasedumphord.com)
[![Portfolio](https://img.shields.io/badge/Portfolio-chasedumphord.com-0F6E56?style=flat-square)](https://chasedumphord.com)
---
## Related Repos
| Repo | Description |
| ------------------------------------------------------------- | ------------------------------------------------------------------ |
| [ceds-homelab](https://github.com/ced4568/ceds-homelab) | 6 node Proxmox cluster + 12 node K3s + full homelab infrastructure |
| [ced-k3s-homelab](https://github.com/ced4568/ced-k3s-homelab) | 12 node Raspberry Pi K3s cluster detail |
| [ceds-observability-stack](https://github.com/ced4568/ceds-observability-stack) | Observability stack configs and dashboards |
| [ceds-noc](https://github.com/ced4568/ceds-noc) | Custom-built public NOC status page |
Reference in New Issue View Git Blame Copy Permalink