# 🤖 CedOps — AI Infrastructure Operations NOC

> A live AI operations platform running on Hetzner VPS — 8 autonomous agents orchestrated by Paperclip, monitored in real-time by Prometheus (hosted on Proxmox homelab) scraping over Tailscale, visualized in a Grafana NOC dashboard.

[![Live NOC](https://img.shields.io/badge/Live%20NOC-noc.chasedumphord.com-1D9E75?style=flat-square)](https://noc.chasedumphord.com)
[![Paperclip](https://img.shields.io/badge/Paperclip-AI%20Agent%20Platform-6B46C1?style=flat-square)](https://paperclip.synthossystems.com)
[![Synthos Systems](https://img.shields.io/badge/Synthos%20Systems-synthossystems.com-0EA5E9?style=flat-square)](https://synthossystems.com)
[![Portfolio](https://img.shields.io/badge/Portfolio-chasedumphord.com-0F6E56?style=flat-square)](https://chasedumphord.com)
[![Agents](https://img.shields.io/badge/AI%20Agents-8%20Active-orange?style=flat-square)](#-ai-agent-roster)
[![Platform](https://img.shields.io/badge/Platform-Hetzner%20VPS-D4001A?style=flat-square)](#️-infrastructure-layer)
[![Monitoring](https://img.shields.io/badge/Monitoring-Prometheus%20%2B%20Grafana-F46800?style=flat-square)](#-observability--noc)

---

## 📸 NOC Dashboard

### V2 — Current

[![CedOps NOC Dashboard V2](docs/screenshots/dashboard_V2.png)](docs/screenshots/dashboard_V2.png)

### V1

[![CedOps NOC Dashboard V1](docs/screenshots/dashboard_V1.png)](docs/screenshots/dashboard_V1.png)

> **Live:** All 5 public services reporting UP · NOC Health Score 100% · VPS Disk 27% · Prometheus scraping every 30s

---

## 🧭 Overview

CedOps is a **production-style AI operations platform** built to simulate a real-world Platform Engineering and AI Infrastructure environment. The VPS stack runs on Hetzner and is monitored by Prometheus hosted on a Proxmox homelab node, communicating over a private Tailscale network — with everything visualized in a live Grafana NOC dashboard.

The platform combines:

- **Paperclip** — AI Agent Orchestration Platform managing 8 autonomous agents
- **SugeBot** — Hermes AI agent deployed as Telegram Operations Assistant (CTO/Ops role)
- **n8n** — Automation platform for workflow execution and inter-service routing
- **OpenRouter** — AI model gateway serving as the LLM backbone for all agents
- **Prometheus** — Hosted on Proxmox homelab, scraping VPS services over Tailscale
- **Grafana** — Full observability NOC dashboard visualizing the entire stack in real-time
- **Nginx Proxy Manager** — Reverse proxy + SSL management across all services
- **Docker + Portainer** — Container infrastructure and management layer

---

## 🏗️ Architecture Overview

```
                        Internet
                            │
              ┌─────────────┴─────────────┐
              │      Cloudflare Edge       │
              │   DNS · Tunnels · Proxy    │
              └─────────────┬─────────────┘
                            │
              ┌─────────────▼─────────────┐
              │       Hetzner VPS          │
              │    synthos-node-1          │
              │                           │
              │  ┌──────────────────────┐ │
              │  │  Nginx Proxy Manager  │ │
              │  │  Reverse Proxy / SSL  │ │
              │  └──────────┬───────────┘ │
              │             │             │
              │  ┌──────────▼───────────┐ │
              │  │      Portainer        │ │
              │  │  Docker Management    │ │
              │  └──────────┬───────────┘ │
              │             │             │
              │  ┌──────────▼──────────────────────────────────┐ │
              │  │              CedOps Platform                  │ │
              │  │                                               │ │
              │  │  ┌─────────────┐   ┌──────────────────────┐ │ │
              │  │  │  Paperclip  │   │       SugeBot         │ │ │
              │  │  │ AI Orch.    │   │   Telegram Ops        │ │ │
              │  │  │  8 Agents   │   │   (Hermes Agent)      │ │ │
              │  │  └──────┬──────┘   └──────────┬───────────┘ │ │
              │  │         │                      │             │ │
              │  │  ┌──────▼──────┐   ┌──────────▼───────────┐ │ │
              │  │  │ OpenRouter  │   │         n8n           │ │ │
              │  │  │ AI Gateway  │   │     Automation        │ │ │
              │  │  └─────────────┘   └──────────────────────┘ │ │
              │  └───────────────────────────────────────────────┘ │
              │              ▲ Node Exporter + Blackbox             │
              └──────────────┼──────────────────────────────────────┘
                             │ Tailscale (encrypted private tunnel)
              ┌──────────────┴──────────────────────┐
              │         Proxmox Homelab              │
              │                                      │
              │  ┌─────────────┐  ┌───────────────┐ │
              │  │ Prometheus  │  │    Grafana     │ │
              │  │ Metrics     │→ │  NOC Dashboard │ │
              │  │ Collection  │  │                │ │
              │  └─────────────┘  └───────────────┘ │
              └─────────────────────────────────────┘
```

---

## 🤖 AI Agent Roster

All agents run inside **Paperclip** — the AI Agent Orchestration Platform at `paperclip.synthossystems.com`.

| Agent | Role | Status |
|---|---|---|
| **Suge** | CTO / Operations (Hermes Agent) — also deployed as SugeBot on Telegram | ✅ Live |
| **CEO** | Executive decision-making and strategic oversight | ✅ Live |
| **ClaudeCoder** | Software engineering and code generation | ✅ Live |
| **Engineer** | Infrastructure and systems engineering | ✅ Live |
| **Marketing** | Marketing strategy and content | ✅ Live |
| **Offer Builder Agent** | Sales offer generation and pipeline | ✅ Live |
| **Research** | Market research and intelligence gathering | ✅ Live |
| **Sales Agent** | Lead qualification and outreach | ✅ Live |

> **Suge** serves dual duty — the primary Hermes-style ops agent inside Paperclip, and the external-facing **SugeBot** on Telegram for real-time operational commands.

---

## 🖥️ Infrastructure Layer

All services run on a **Hetzner VPS** (`synthos-node-1`) behind Cloudflare Tunnels with zero exposed ports.

| Service | Role | URL |
|---|---|---|
| **VPS / Portainer** | Docker container management | Internal |
| **Nginx Proxy Manager** | Reverse proxy + SSL termination | Internal |
| **Homepage** | CedOps command center dashboard | `ops.synthossystems.com` |
| **Synthos Website** | Business-facing website | `synthossystems.com` |
| **Ced's Home Lab** | Homelab / Portfolio platform | `cedshomelab.com` |

**Traffic Flow:**
```
Internet → Cloudflare Edge → Tunnel → Nginx Proxy Manager → Docker Services
```

---

## 📊 Observability & NOC

Prometheus runs on the **Proxmox homelab** and scrapes the Hetzner VPS over a **Tailscale private tunnel** — keeping metrics collection completely off the public internet. Grafana visualizes everything in the live NOC dashboard.

### Executive Health Panel
- **Synthos VPS Status** — UP/DOWN state with instant alerting
- **Public Services Up** — count of healthy endpoints (target: 5/5)
- **NOC Health Score** — composite platform health percentage
- **VPS Uptime** — continuous uptime tracking in days

### VPS Resource Health
- CPU Usage % — `synthos-node-1` CPU time series
- Memory Utilization % — real-time memory pressure
- Disk Used % — current at 27%
- Network Traffic — RX/TX bandwidth graphs (30s refresh)

### Public Service Availability

Tracked via **Blackbox Exporter** HTTP probes:

| Service | Endpoint | Status |
|---|---|---|
| cedshomelab | cedshomelab.com | ✅ UP |
| n8n | n8n.synthossystems.com | ✅ UP |
| cedops | ops.synthossystems.com | ✅ UP |
| paperclip | paperclip.synthossystems.com | ✅ UP |
| synthos-website | synthossystems.com | ✅ UP |

### Reliability Metrics
- HTTP Status Codes per endpoint (all returning `200`)
- Public Endpoint Response Time (ms) — time series graph
- SSL Expiry Days — tracked per certificate with warning thresholds

---

## 🔒 Security Architecture

CedOps is designed with **zero public infrastructure exposure**.

| Control | Implementation |
|---|---|
| No open ports | Cloudflare Tunnel handles all ingress |
| Private metrics pipeline | Prometheus → Tailscale → VPS; never public |
| SSL everywhere | NPM handles termination for all services |
| Container isolation | Docker network segmentation via Portainer |
| Secret management | No secrets in this repo — handled via environment variables |

---

## 📂 Repository Structure

```
cedops-ai-infrastructure-noc/
│
├── docs/
│   ├── architecture/
│   ├── screenshots/
│   │   ├── dashboard_v1.png
│   │   └── dashboard_v2.png
│   └── sops/
│
├── grafana/
│   ├── cedops-ai-infrastructure-noc-v1.json
│   └── cedops-ai-infrastructure-noc-v2.json
│
├── Prometheus/
│   ├── prometheus-example.yml
│   └── blackbox-targets-example.yml
│
├── diagrams/
│   └── cedops-architecture.mmd
│
└── README.md
```

> **Note:** Prometheus and Blackbox configs are sanitized for public sharing. Production configs are managed privately via environment variables and are never committed to this repo.

---

## 🛠️ Stack

| Layer | Technology |
|---|---|
| **Compute** | Hetzner VPS (synthos-node-1) |
| **Containers** | Docker + Portainer |
| **Reverse Proxy** | Nginx Proxy Manager |
| **DNS & Tunnels** | Cloudflare |
| **Private Network** | Tailscale (Proxmox ↔ VPS metrics tunnel) |
| **AI Orchestration** | Paperclip |
| **AI Gateway** | OpenRouter |
| **Automation** | n8n |
| **Ops Agent** | SugeBot (Hermes / Telegram) |
| **Metrics** | Prometheus (Proxmox) + Node Exporter + Blackbox Exporter |
| **Visualization** | Grafana |
| **Virtualization** | Proxmox VE (homelab, hosts Prometheus + Grafana) |

---

## ✅ What's Built

### V1 Dashboard
- [x] VPS UP/DOWN status panel
- [x] Public service availability (5 endpoints via Blackbox Exporter)
- [x] VPS resource health (CPU, Memory, Disk, Network)
- [x] HTTP status codes per endpoint
- [x] Public endpoint response times
- [x] SSL certificate expiry tracking
- [x] Cross-environment scraping via Tailscale (Proxmox → Hetzner VPS)

### V2 Dashboard
- [x] Agent activity metrics (Paperclip runs, success rate, token spend)
- [x] SugeBot command log panel
- [x] Per-agent health monitoring
- [x] n8n workflow execution tracking
- [x] Telegram alert integration via SugeBot

---

## 🔮 Roadmap

- [ ] AI Incident Detection — anomaly-triggered agent escalation
- [ ] Automated Remediation — n8n workflows triggered by Prometheus alerts
- [ ] Multi-environment monitoring (homelab + VPS unified NOC)
- [ ] Discord alert channel

---

## 👤 Author

**Chase Dumphord**
Digital Systems Engineer | Platform Engineering | AI Infrastructure | Observability

[![LinkedIn](https://img.shields.io/badge/LinkedIn-chase--dumphord-0A66C2?style=flat-square)](https://www.linkedin.com/in/toochase-dumphord/)
[![GitHub](https://img.shields.io/badge/GitHub-ced4568-181717?style=flat-square)](https://github.com/ced4568)
[![Live NOC](https://img.shields.io/badge/Live%20NOC-noc.chasedumphord.com-1D9E75?style=flat-square)](https://noc.chasedumphord.com)
[![Portfolio](https://img.shields.io/badge/Portfolio-chasedumphord.com-0F6E56?style=flat-square)](https://chasedumphord.com)

---

## 🔗 Related Repos

| Repo | Description |
|---|---|
| [ceds-homelab](https://github.com/ced4568/ceds-homelab) | 6-node Proxmox cluster + 12-node K3s + full homelab infrastructure |
| [ced-k3s-homelab](https://github.com/ced4568/ced-k3s-homelab) | 12-node Raspberry Pi K3s cluster detail |
| [ceds-noc](https://github.com/ced4568/ceds-noc) | Custom-built public NOC status page |